From time to time, a developer will attempt to add the ASP.NET Membership/Forms Authentication tables to their Entity Framework model (or LINQ to SQL, NHibernate, etc.) model. Before doing this, they will often have created referential constraints against these tables. When the mapping doesn’t work out quite the way they planned, they will ask how to make the mapping work.

There’s only one correct answer to this question: Don’t do it at all!

There are a number of good reasons why you should not make your database and code depend upon the SQL Membership Provider database schema. In this post, I will focus on a few of the most important:

• Separation of concerns
• Membership and authentication providers are supposed to be interchangeable
• The SQL Membership Provider database schema is an implementation detail

Separation of Concerns

Your application’s data model is designed to fit your application domain. It will change based upon the needs of the end users. It should not have to change because Microsoft decides to update the SQL membership provider, as well. That would violate the single responsibility principle. It is often dangerous to combine data which is not closely related into a single data model. This danger is compounded when data from two separate domains, written by entirely different companies, and designed for orthogonal purposes is shoehorned into a single entity model.

Membership providers are supposed to be interchangeable

One of the most important design intentions of the ASP.NET authentication and membership provider model is to make it easy to interchange providers. If you decide to stop using the SQL membership provider in transition to Open ID, domain authentication, Facebook authentication, etc., this should be a matter of, at most, a couple of days work to migrate data from one provider to the other, rather than a complete rewrite of your application, starting with the database and moving out from there.

The SQL Membership Provider database schema is an implementation detail

The publicly-documented interfaces to membership and forms authentication are the Membership and FormsAuthentication types, respectively, as well as the relevant sections of the Web.config file. If you write your code around these types, you have a reasonable expectation that your code will continue to work when the .NET framework is next updated. On the other hand, if you query the database directly, there is no guarantee that the schema will not change when the next version of .NET ships. If Microsoft makes a security-related change to the SQL membership provider, then it is conceivable that the schema could even change with a service pack. The cost of relying on an implementation detail is that you never really know.

Share This | Email this page to a friend

Now when autoencode is set to true we encode the data coming from server and not only when we post it (secutity fix)

Prior to this, you were required to encode the data yourself.

Now personally, I think that should be the default. But it would have been a breaking change for the grid, since there are a few cases where you want to display unencoded data (I’ll discuss these exceptional cases in a second).

It’s really easy to make this the default for your application. Set the grid’s defaults before you create any grids:

    $.jgrid.defaults = $.extend($.jgrid.defaults, {
            autoencode: true,
            datatype: 'json',
            // etc....

You can override this in the rare cases when you don’t want encoded data by setting autoencode false when setting up your grid:

    $("#grid").jqGrid({
                    autoencode: false,
                    url: "/Some/Path",
                    // etc....,

So why wouldn’t you want to use this setting? I can think of two reasons:

1. The server is returning markup which you don’t want to encode. If you have a server method which is returning markup (e.g., "<img src=…"), then you won’t want the grid to encode it. It then becomes your responsibility to sanitize the rest of the grid data on the server.
2. There is a performance cost to doing the encoding in JavaScript. If you are absolutely certain that the server method you’re calling can return only sanitized data, then you can turn off the auto encoding and save this cost. Note that many JSON encoders, such as the "Json()" method in ASP.NET MVC will not encode HTML by default.

Note that you can only set autoencode at the grid level, not at the column level.

If you use custom formatters in your grid, then you should note that the strings they return will not be HTML encoded, even if you set autoencode true. This is probably a good thing, since it is common to use custom formatters to return HTML. However, it means that if you write a custom formatter, then you must ensure that any user data contained in the string and returns is encoded. You can use the grid’s encoder, which is little more than a string replace on angle brackets, in a custom formatter as follows:

    myFormatter: function(cellval, opts, action) {
        if (cellval) {
            return $.jgrid.htmlEncode(cellval.Name + "");
        };
        return "";
    }

Share This | Email this page to a friend

Obviously, if you use a source control tool which locks files on check out, then working concurrently on just about anything will be impossible. So I’m going to presume that your source control tool supports working concurrently, without locks, and has a decent merge tool to handle conflicts at check-in.

We tend to think of the EDMX files as having three sections:

• CSDL, which describes the client schema. This will be used when generating code for your entity classes.
• SSDL, which describes the storage schema, more commonly known as your database metadata.
• MSL, which describes the relationship between the first two.

In reality, however, the EDMX file has two main sections. The first, edmx:Runtime, contains the three sub-sections described above. The second, edmx:Designer, contains a couple of important properties, such as MetadataArtifactProcessing, plus a lot of information about the position of objects on the GUI designer, in a sub-node called edmx:Diagrams. The illustration below should make this clear:

Why is this important? My experience is that any decent merge tool doesn’t tend to have any problem merging what we typically think of as the "EDMX", namely the CSDL, SSDL, and MSL. It is well-formed XML with descriptive tags; it’s the sort of thing that text merge tools tend to do very well with. There are a couple of tips worth knowing, but before I describe them, I’d like to focus on the more common issue with EDMX merges.

In my experience, if two people change the layout of entities in the designer and then attempt to merge, the merge will probably fail. Moreover, you will find it difficult to impossible to fix this up manually, something which is generally fairly easy with merges of source code or other parts of the EDMX. When you look at what is in the edmx:Diagrams node, you will quickly understand why:

I strongly suspect that this "feature" is not specific to the Entity Framework designer, but rather is shared amongst all of Visual Studio’s various diagramming tools. At any rate, I think you will have a difficult time merging XML like this, so I have two recommendations regarding merging and EDMX files:

• If you are going to have two or more developers work on an EDMX file concurrently, then don’t change anything in the designer. If two or more people do this, it almost guarantees a conflict on check-in.
• The other side of the coin is that if you must change something in the designer, then you should give a heads-up to other developers, and exclusively lock the EDMX file while you work.

That said, we don’t generally use the GUI designer at all. It is too unwieldy to have more than a dozen or so types on a single design surface. On the other hand, the tree-style Model Browser is quite useful. I suggest using the Model Browser instead of the GUI designer to navigate your entity model.

Perhaps a future version of the Entity Framework designer will allow for multiple diagrams within a single entity model, somewhat like SQL Server diagrams. If each diagram had its own node in the XML, you might even be able to merge them.

If you steer clear of issues with the designer, then you will typically find that EDMX changes can be automatically merged by any decent merge tool (unless, of course, they actually conflict). In the unlikely event, however, that your merge tool reports a conflict, and you don’t think it’s an actual conflict between your changes and those made by the other developer, it does help to understand how the "Update Model from Database" wizard updates existing EDMX:

1. The wizard will generate new SSDL from scratch. It will then go through the newly generated SSDL, and replace each matching node in the existing SSDL with the newly generated version. There are a few SSDL features which are not supported by the designer/wizard, such as will, and these will be left alone, if they exist in your model.
2. Any new entities or new properties of existing entities will have corresponding nodes added to the CSDL and MSL. But existing entities and properties will not have their CSDL and MSL updated, because the wizard presumes that you want to keep any changes you may have made to them.

Therefore, when multiple developers are updating an EDMX file concurrently, they should use the same database. This will tend to make SSDL changes merge without conflict, and allows you to simply overwrite any false conflicts (merge tool failures) in the SSDL, because you can be confident that it was generated from the same object in the same database. But if you’ve manually customized your SSDL (i.e., edited the SSDL section of the EDMX as text), then you should keep an eye on your changes, to make sure that the wizard does not overwrite them. Most people never do this, though.

On the other hand, changes to the CSDL and MSL must be reviewed carefully in the unlikely event that there are conflicts in the merge, because you, or the other developer, may have made customizations to the mapping or entity types which you want to keep.

Finally, it is worth mentioning that the Entity Framework version 4 supports "code only" models which do not use EDMX files at all. If you prefer designing your entity model via source code, you can choose this option.

Share This | Email this page to a friend

Many people ask how to do a "left join" in LINQ to SQL, and unfortunately, the answer they nearly always get — "Use DefaultIfEmpty!" — is, in my opinion, terrible advice. Let’s implement the same query with and without the join keyword, and then compare the readability of the queries, the functionality, the knowledge required to write the query, and the maintainability of the code. I think you will find that using associations wins on every single criterion I examine.

I’m going to use the Northwind demo database for this example, since many people are familiar with its structure. I created a LINQ to SQL model for Northwind by simply dragging all of the tables in the database onto the LINQ to SQL designer. The only change I made to the model generated by the designer is to rename the "Employee1" property on the generated "Employee" type to the more descriptive name "Supervisor." I’m using LINQ to SQL for this demo, but everything I’m saying here applies equally to LINQ to Entities.

"Left" Joins

Let’s imagine that I am asked to produce a web page listing all employees in a company, along with their supervisor, if any. This requires a "left join," in SQL terms, because not all employees have a supervisor. I’ll project onto a presentation model, just like I do in LINQ to Entities. Using the association properties generated by LINQ to SQL, this is quite simple:

This is fairly readable. The one thing that you need to know is that both LINQ to SQL and LINQ to Entities coalesce nulls. This means that on a row where e.Supervisor is null, you will not get a NullReferenceException in the assignment to SupervisorName and SupervisorBirthDate, as you would with LINQ to Objects. Instead, null will be assigned. Therefore, it is important that EmployeeListItem.SupervisorBirthDate is of type DateTime? (a.k.a. Nullable<DateTime>) instead of the non-nullable DateTime.

Let’s compare that to the equivalent query using the join syntax, using the mysteriously popular DefaultIfEmpty trick:

Yuck! This is far less readable than the query above. Yet these two queries produce exactly the same results. If you don’t believe me, download the sample project attached to this post and try it yourself. Perhaps even worse than the general unreadability of the "join version" is the fact that this query requires knowledge of the structure of the database which is already present in the DBML (or EDMX, in the case of the Entity Framework) model. This is a problem for two reasons. First, it’s an opportunity for programmers to make a mistake, which the first query eliminates. Second, it’s a potential maintenance issue if the foreign key definition ever changes in the database.

"Inner" Joins

Now let’s compare an example of an "inner join" using both my recommended method of associations and the LINQ join keyword. Here’s the association version. It’s so readable that there is very little to say about it:

Here’s the join version:

Again, these two queries do exactly the same thing, as you can confirm for yourself by running the demo project. The join version here shares all of the faults of the "left join" version above.

API Consistency

Thus far, I’ve been discussing LINQ to SQL. But what if I have an Employee instance and that like to examine the employee’s supervisor? I might write code like this:

Now compare that code with the "association" and "join" query forms. You will see that using the associations makes the LINQ to SQL queries much more closely resemble how you work with the materialized entity objects in "regular" code. Again, I think this makes your code easier to read.

What About Performance?

Not surprisingly, the SQL generated by the equivalent association and join versions is close to identical. I would not expect to ever see a performance difference between the two syntaxes in terms of query execution time.

Is It Ever Correct to Use join?

The advantages of using associations are so strong that you may wonder why join exists in LINQ at all. Associations, however, are only helpful when they actually exist. There may be times when you need to "join" based on values which are not actually foreign keys. Or you may need to join between LINQ to SQL and LINQ to Objects.

Running the Demo Project

Here’s a the demo project. To build and run it, you’ll need to Visual Studio 2010 Beta, SQL Server (Express is fine), and the Northwind demo database I linked above. Open the project, find the Web.config file in Solution Explorer and open it. Change the connectionString to point to your SQL Server. Now you should be able to run the application.

Share This | Email this page to a friend

Share This | Email this page to a friend

Imagine that you’ve been asked to write a new web application to track employees for a customer, Chotchkies restaurant. The application must use ASP.NET MVC and the ADO.NET Entity Framework. The user interaction designers have mocked up the following interface for editing an employee:

Upon seeing this mockup, your first recommendation is to fire the user interface designers. But management declines to follow your recommendation. So how should you implement this? Since this is a new application, you have no database, no entity model, nothing. Where to begin?

Presentation Models

For a variety of reasons, I always use strongly-typed views in my ASP.NET MVC applications. Also, I use presentation models instead of using entity types directly as the model type for my views.This allows the user interface and data models to evolve independently.

Because I am likely to want to build user interfaces using ASP.NET MVC 2’s Dynamic Templated Views, I need to decorate the view model with presentation concerns, like noting that the "Flair count" field is read-only in this particular view.

Especially in view of the less-than-compelling user interface markup, it’s important to realize that the design of the user interface is likely to change wildly over the course of implementing the application. When actual users begin to test the application, they will request changes to the user interface, and you need to be able to adapt to this.

This has two important implications: You must get a prototype to testers and end-users as fast as possible, and your user interface must not be deeply coupled to the rest of the application, given the high likelihood of change.

So based upon the user interface prototype above, the following presentation model might be reasonable:

It’s now trivial to write an action which can be used to develop a view for editing the employee, and which matches the UI prototype above. Such an action might look like:

I’ll spare you the HTML. At this point, I have a running application which the user interaction designer can approve, and testers can try out. My total effort, thus far, is a few minutes of work. If I had started by designing a database and the data model, I would still have nothing to show for my efforts.

Entity Model

At some point however, you need to create a database and an entity model. That was, after all, part of the requirements you were given for the application. Knowing that there might be a need to reference people who are not employees, the entity model is going to have to look very different than the presentation model above. After some discussions with the business analyst, you learn that Chotchkies management is very serious about tracking employee flair, so a first attempt at an entity model might look like this:

Now you can generate a database. At this point, you have a presentation model and an entity model, and need only wire them together.

Projection

Taking a collection of instances of one type, and mapping their properties onto a collection of instances of another type is often called mapping or projection. With older ORMs, which have either no or very limited LINQ support, this can be quite tedious, requiring manual code, the use of tools like AutoMapper, and a good deal of thinking about eager loading, lazy loading, and optimizing situations like the fact that we don’t actually want to load all of the flair for an employee here; we just need the count.

The Entity Framework, on the other hand, makes this very easy, as we can just express the relationship between the entity model and the presentation model with a LINQ expression:

I’m glossing over some details here. In a real-world application, for example, we would use the repository pattern rather than grabbing the context directly in the controller. But the point of this post is shown in the code above:

By expressing the relationship between the entity model and the presentation model as a LINQ query, it is no longer necessary to worry about implementation details like eager loading versus lazy loading, optimizing the SQL for the Count and avoiding loading big properties not actually used here. All of this can be — and is — derived from the query above, automatically.

Share This | Email this page to a friend

There is no good reason not to use static analysis. If you are the sort of person who doesn’t allow hints and warnings in your code, and has configured your build to fail on any hint or warning (and I hope that you are), then you’ll love static analysis; it takes this kind of discipline to the next level. Sure, it’s useful to find existing errors in your code, but the most important benefit is that it prevents new errors from being introduced.

The only remotely valid argument I’ve heard against using static analysis is that when you first use a static analysis tool against a legacy code base, you will typically see hundreds of rules violations, many of them incorrect. Cleaning all this up will be a lot of work, which you usually can’t take on all at once.

Unlike compiler hints and warnings, static analysis rule violations often happen when there is no underlying bug. When you develop new code, you can analyze any newly introduced rule violation and exclude it if necessary. But that’s not practical to do with legacy code. My approach to this problem is very simple: Turn off rules in your static analysis tool until all the rule violations go away.

What’s the point of buying a static analysis tool and then turning off half the rules? Well, the other half are still turned on, and they are now a part of your automated build, making it impossible to introduce new code which violates them. Moreover, you can go back and turn on the other rules a reasonable pace, say, one or two a week, fixing real issues that you find or excluding the particular violation if it turns out to be a non-issue, until they are all turned back on.

Once you have even a few static analysis rules turned on in your automated build, you’re getting some degree of quality control for very little expense. Integration testing costs money, every time you do it. Unit testing costs money every time you write new tests. Static analysis just works and keeps working, and costs you nothing more than a software upgrade once every couple years or so.

So static analysis in general is a good thing. What about CodeHealer in particular? Well, we have been using it for around three years now. It has been stable, and I’ve seen new versions released regularly. Also, I have never seen anything better for Delphi.

Digression: Attributes

When Delphi 2010 was released with the enhanced RTTI and custom attributes feature, many Delphi developers wondered what this feature would be useful for. In general, your source code describes the behavior of your application, and attributes describe your mechanisms (thanks to Eric Lippert for this concise description). Static analysis is an excellent example. Imagine that I want to exclude a static analysis rule in a particular case. This has nothing to do with the desired functionality of the application; rather, it’s more like a parameter to the automated build process.

That may sound a bit abstract, so let’s look at a specific example. FxCop, a static analysis tool for .NET code, has rule that should not introduce private types which are never used. But FxCop cannot detect the instantiation of types inside of a LINQ query. So in this case the rule is making a mistake; it is failing to detect that a type is, in fact, instantiated in my code. One possible solution to this issue would be to try and write a better rule. But that’s a lot of work, and there are only about two cases of this false positive in our entire source tree. So it’s easier to just suppress this message in the case of a false positive, which you can do with an attribute:

[System.Diagnostics.CodeAnalysis.SuppressMessage(
    "Microsoft.Performance",
    "CA1812:AvoidUninstantiatedInternalClasses",
    Justification = "It is instantiated in a query, which FxCop can't see.")]
private class ConsumerRow //...

This is far easier (and, I think, a better design) than what CodeHealer currently offers. I hope that future versions of CodeHealer will use this method, now that Delphi has attribute support.

Share This | Email this page to a friend

Share This | Email this page to a friend

New Features

Some of the new features of preview 2 have been discussed elsewhere, so I won’t rehash them. But I’ve also noticed that there is a new attribute, [RequireHttps], which does what you would expect, when added to an action, and a new HTML helper, Html.HttpMethodOverride, which makes it easy to take a POST request and code as if the request were actually PUT or DELETE, by adding a hidden input containing a special value on the HTML form. This allows you to write your server in a more RESTful style, which will be suitable for user agents which know about the PUT and DELETE verbs, while maintaining compatibility with those (like browsers) which do not.

MvcHtmlString

After installing the new assembly into the GAC, I attempted to compile our existing projects. I got a compilation error on some of our custom HTML helpers, as the MVC extension methods like Html.RouteLink have been changed to return an instance of type MvcHtmlString instead of String. In many cases, I could just change the return type. However, the clear intention of the framework designers is that any HTML helpers which return HTML (with angle brackets) instead of "plain" text should return a MvcHtmlString instead of a String. So, in addition to changing the return types of methods where necessary to get those methods to compile, I also wanted to change the return types of any custom HTML helper which returned HTML containing angle brackets. It makes no difference today, but in ASP.NET 4, there will be a new syntax to guard against XSS attacks, and it makes sense to get ready for this by returning the correct type.

The trick here is that MvcHtmlString does not have a public constructor, so it’s not immediately obvious how to create one of these things. Perusing the MVC source code, I noticed that it does have a static Create method, and comments elsewhere in the source code indicate that this method should be used instead of the detected constructor. So you can now write a helper like this:

public static MvcHtmlString Fud(this HtmlHelper helper)
{
    return MvcHtmlString.Create("<acronym title=\"Fear, Uncertainty, and Doubt\">FUD</acronym>");
}

JsonRequestBehavior

ASP.NET MVC will now, by default, throw an exception when an action attempts to return JSON in response to a GET request. Unless you read the release notes carefully, you won’t discover this until runtime. This is in order to proactively defend against a particular cross-site attack. It is good to be safe by default, but you are only vulnerable to this attack under the following combination of circumstances:

• The data you’re returning is worth stealing.
• The root data object in the JSON response is an array.
• The requesting browser is not IE 8, or some other browser which doesn’t allow __defineSetter__

It turns out that in our application we almost never return JSON results with the root object as an array in a GET. So the best fix was generally just to tell the framework that we have examined the risk and determined it returning JSON in this case, by changing code like:

return Json(model);

…to:

return Json(model, JsonRequestBehavior.AllowGet);

Unfortunately, I had to make this change in a lot of places. Still, I agree with the framework designers: Better safe than sorry.

JavaScript Files

The MicrosoftAjax.js, MicrosoftAjax.debug.js, MicrosoftMvcAjax.debug.js, and MicrosoftMvcAjax.debug.js files have all changed for ASP.NET MVC 2. Existing ASP.NET MVC 1.0 projects will have old versions of these files. The upgrade instructions in the Release Notes don’t mention this, but (they do; I just missed it) you should replace these files with the newer versions when upgrading to MVC 2. To do this, create a new MVC 2 project, copy the JavaScript files from the Scripts folder in this new project, and paste them over the existing files in the project you are upgrading.

ModelBindingContext Changes

One of the new features of preview 2 is client-side validation and custom metadata providers for validations. In order to implement this feature, some of the implementation details of model binding have changed. This won’t affect most people, but the fix was a little tricky, and required a good bit of examination of the MVC source code, so I’ll list it anyway. Prior to preview 2, I had code in a unit test for a custom model binder like this:

        internal static T Bind<T>(string prefix, FormCollection collection, ModelStateDictionary modelState) where T:class
        {
            var mbc = new ModelBindingContext()
            {
                ModelName = prefix,
                ModelState = modelState,
                ModelType = typeof(T),
                ValueProvider = collection.ToValueProvider()
            };
            IModelBinder binder = new MyModelBinder();
            var cc = new ControllerContext();
            return binder.BindModel(cc, mbc) as T;
        }

With Preview 2, on the other hand, I have to do this:

        internal static T Bind<T>(string prefix, FormCollection collection, ModelStateDictionary modelState) where T:class
        {
            var mbc = new ModelBindingContext()
            {
                ModelMetadata = ModelMetadataProviders.Current.GetMetadataForType(null, typeof(T)),
                ModelName = prefix,
                ModelState = modelState,
                ValueProvider = collection.ToValueProvider()
            };
            IModelBinder binder = new MyModelBinder();
            var cc = new ControllerContext();
            return binder.BindModel(cc, mbc) as T;
        }

Not hard once you know the trick.

Share This | Email this page to a friend

Let me say at the outset that although all three systems work somewhat differently, I don’t see an overwhelming advantage to any one design. Generally, you can do what you need to do in all three environments. I’m writing this article not to claim that any one system is better than the others, but to point out some of the subtleties in the implementations.

Before I get started, I’d like to thank Barry Kelly for his useful feedback on my first draft of this article.

Compiling Instantiations

Every implementation of generic types works via a two-step process. First, you define a generic type or method with a "placeholder" for a specific type, which will be substituted later on. Later (exactly when depends upon the language), the type is "instantiated." Note that instantiating a generic type is very different from instantiating an object. The former happens within the compiler, whereas the latter happens at runtime.

Instantiation is triggered when some code uses a generic type or method with a specific type parameter, and means that based upon the generic definition and the types or values passed when the generic is used, a specific implementation is substituted in order to allow the generation of machine code. Instantiation is one of the most important differences between real generic types and using non-generic types with casts. In the end, different machine code is generated for instantiations for different type parameters.

In C# and Delphi, there is a language feature which is solely dedicated to implementing generic types and methods. In C++, on the other hand, the "templates" language feature can be used to implement generic types and methods, among many, many other things. It is even possible to do general-purpose programming using templates, which C++ programmers call "metaprogramming."

C++ templates require the template source code to be available when the code using the template is compiled. This is because the compiler does not actually compile the template as a separate entity, but rather instantiates it "in-place" and only compiles the instantiation. The C++ compiler is effectively doing code generation, substituting the type parameter (or value) for the placeholder for the type, and generating new code for the instantiation. Update: Moritz Beutel elaborates on this in his excellent comment on this post. You should read the full comment, but the short version is that the manner in which templates are compiled can result in errors in the code which uses the template appearing (from compiler error messages), incorrectly, to be errors in the template itself. Moreover, the implementation of most C++ compilers makes this problem even worse than what is necessary in order to implement the C++ standard.

In Delphi and C#, on the other hand, the generic type or method in the code which uses the generic type or method can be compiled separately. Therefore, you can compile a library which contains a generic type, and later on compile an executable which uses a instantiation of that type and has a reference to the binary library, rather than to the source code for the library.

Another way to think of this difference is that in C++, a template will not be compiled at all until it is used. In Delphi and C#, on the other hand, a generic type or method must be compiled before it can be used.

In Delphi, the compiler uses a feature closely related to the method inlining feature. This causes the compiler to store the relevant bits of the abstract syntax tree for the generic type parameter in the compiled DCU. When the code which uses the generic type is compiled, this bit of the abstract syntax tree is read and included in the abstract syntax tree for the code which uses the generic type, so that when machine code is produced, based on the new, “compound” abstract syntax tree, it looks, to the code emitter, like the type was defined with the type parameter "hard coded." Instead of linking to compiled code in the DCU, the code which uses the generic type emits new code for the instantiation into its own DCU.

Because generic instantiation is performed in the same area of the Delphi compiler which does method inlining, there are some limitations on what you can do in a generic method, or a method of a generic type. Like inlined methods, these methods cannot contain ASM. Also, calls to these methods cannot be inlined. These restrictions are limitations of the implementation, not of the language design, and could theoretically be removed in a future version of the compiler.

C# generics use the .NET Framework 2.0+, which has native support for generic types. The C# compiler emits IL which specifies that a generic type should be used, with certain type parameters. The .NET framework implements these types using one instantiation for any reference type, and custom instantiations for value types. (Don’t confuse “instantiation” with “instance” in the preceding sentence; they mean entirely different things in this context. There are usually many instances of one instantiation.) This is because a reference to a reference type is always the same size, whereas value types can be many different sizes. Later, the IL will be JITted into machine code, and, as with compiled C++ or Delphi code, types don’t really exist at the machine code level. In .NET, generic type instantiation and JITting are two distinct operations.

So one important difference in generics implementations is when the instantiation occurs. It occurs very early in C++ compilation, somewhat later for Delphi compilation, and as late as possible for .NET compilation.

Custom Specializations

Another very important difference is that C++ allows custom instantiations, called specializations, including specializations by value. With C# and Delphi, on the other hand, the only way to instantiate a generic type is to use that type with an explicit type parameter. The implementation will always be the same, with the exception of the type of the type parameter. Because C++ allows custom instantiations, it is easy for a programmer to write different implementations of a method, for example, for different integer values. Like operator overloading, this is a powerful feature which requires considerable self-restraint to avoid abuse.

Constraints

Delphi and C# both have a generic constraint feature, which allows/requires the developer of a generic or method type to limit which type parameter values can be passed. For example, a generic type which needs to iterate over some list of data could require that the type parameter support IEnumerable, in either language. This allows the developer of the generic type to make her intentions for the use of the type very clear. It also allows the IDE to provide code completion/IntelliSense on the type parameter, within the definition of the generic type. Also, it allows a user of the generic type to be confident that they are passing a legal value for the type parameter without having to compile their code to find out.

In C++, on the other hand, there is not presently any such feature. A more powerful/complex feature called "concepts" was considered for, but ultimately removed from, C++0x.

An implication of the lack of constraints is that C++ templates are duck typed. If a generic method calls some method, Foo on a type passed as the generic type parameter, then the template is going to compile just fine so long as the type parameter passed contains some method called Foo with the appropriate signature, no matter where or how it is defined.

Covariance and Contravariance

Let’s say I have a function which takes an argument of type IEnumerable<TParent>. Can I pass an argument of type IEnumerable<TChild>; to that function? What if the argument type were List<TParent>; instead of IEnumerable<TParent>? Or what if the generic type was the function result rather than the function argument? The formal names for these problems are covariance and contravariance. The precise details are too complicated to explain in this article, but the examples above summarize the most common times you run into the problem.

Delphi generics and C++ templates do not support covariance and contravariance. So the answers to the questions above are no, no, and no, although there are, of course, workarounds, like copying the data into a new list. In C# 4.0, function arguments and results can be declared covariant or contravariant, so the examples above can be made to work where appropriate. "Where appropriate" involves non-trivial subtleties hinted at above, and exemplified by the fact that arrays in .NET have (intentionally) broken covariance. However, the BCL routines in the .NET Framework 4.0 have been annotated to support covariance and contravariance when appropriate, so developers will benefit from the feature without having to fully understand it.

Share This | Email this page to a friend

• Most tools tend to work on either compiled assemblies or source code. NDepend uses both.
• Most tools implement their tests in compiled, executable code. NDepend, on the other hand, uses a proprietary, domain-specific language called CQL.
• NDepend can enhance its analysis by importing analysis data from related builds, like NCover, Team System coverage analysis, and can compare multiple builds of the same project.

As with other static analysis tools, however, NDepend is designed to be used both within Visual Studio and as part of an automated build/continuous integration process.

NDepend claims to support "any .NET language including C#, VB.NET, MC++ and C++\CLI." To put this claim to the test, I ran it against a Delphi Prism application. Indeed, it did produce useful results in the analysis, although there were some quirks. It couldn’t find the assembly from the solution, but after I located it manually analysis could run. I suspect NDepend doesn’t parse the source code of Prism applications as it does with C#.

What NDepend Does

The name "NDepend" presumably comes from the dependency analysis feature. NDepend will display dependencies between assemblies either as a cross tab or a directed graph. See the website for illustrations of this feature.

NDepend also does static analysis along the same lines as Microsoft FxCop and StyleCop, only in a much more interactive manner. By combining the features I’ve already mentioned, you can easily ask queries like, "What are the most complex methods I’ve changed recently which have no unit test coverage?"

All of NDepend’s features are available in its own GUI application, from the command line, within Visual Studio, and in a custom MS build task. So, as with most static analysis tools, you can integrate NDepend into a continuous build environment in order to set enforceable standards on code complexity and the like.

Installation

NDepend does not have an installer. You need to create a folder for the application, and unzip an archive file into that folder. To install the integration with Visual Studio, you click a button in the GUI application, once it’s running. I find point and click installers a bit more convenient, but it was certainly easy to get the application working.

Default Analysis of a Do-Nothing Application

NDepend is quite strict even in its default configuration. For example, I ran it on a do-nothing, "File->New" ASP.NET MVC 2 Preview 1 application, with no code of my own added. It reported quite a number of warnings.

Many of the warnings were incorrect, because NDepend did not automatically find the System.Web.Mvc assembly, which is not stored along with the rest of the .NET framework, even though it is in the GAC. In general, the current version of NDepend will only find assemblies which are in the solution folder or are in the .NET Framework folders automatically. You should add other folders for referenced assemblies manually, even if those assemblies are GACced. To fix this, I had to do the following steps:

1. Go to the Project Properties tab in the GUI.
2. Click the "View Folders" button.
3. Click the "Browse" button and manually locate the folder containing the required assembly.

With these steps completed, the quality of the analysis improved quite a lot.

NDepend shows the results in its GUI and also produces an HTML report. The HTML report for the do-nothing application was 14 pages long, and the report for our production web application ran over 50 pages. This can be kind of overwhelming for a new user. The issues flagged in the do-nothing application included (with my comments in parentheses):

• Methods with > 10 lines of code should be commented. (I disagree; methods should be commented when their intention cannot be made clear in code. This has little relationship to method length and is difficult to impossible to flag with static analysis.)
• Types with no state should probably be static. (Worth considering.)
• Types with no subtypes could be sealed. (NDepend makes it easy to turn this off for public types.)
• Classes with small instance sizes, etc., could be structs. (Maybe, if you want value semantics.)
• Avoid namespaces with few types. (This is expected on a File->New application; would work better in a "real" app. FxCop has the same rule.)
• You should’t have types outside a declared namespace. (I agree.)
• Avoid [un] boxing. (I agree; great test!)
• Types which "could be internal." (NDepend can’t pick up instantiation via reflection, making this less useful in the specific case of MVC.)
• Fields which are assigned only by the constructors of their types should be marked read-only. (I agree; this is a really useful test.)
• Type names should begin with an uppercase letter. The default MVC application includes a type called "_Default".

For the sake of comparison, I analyzed the same application using FxCop. This produced 11 warnings, about half correct, and about half incorrect. Incorrect warnings included not recognizing the TLA MVC, asserting that Application_Start should be static, and claiming that the application should target .NET Framework 3.5 SP 1 (as opposed to just 3.5), something which Visual Studio 2008 does not allow. Probably correct warnings included that the application should have the CLSCompliant attribute, and that the _Default type was oddly named.

Customizing the Analysis For a Real Application

NDepend’s use of its CQL language to implement its test makes individual test very easy to customize. For example, our production web application includes the Microsoft DynamicQuery library, which is distributed as a single CS file. Because we included this in one of our assemblies instead of creating a separate assembly for it, NDepend analyzes it along with the rest of our assembly. It contains many long methods, and other things which NDepend does not like, but because this is Microsoft’s code, and not ours, I would like exclude it from the analysis. So I can, for example, change the test for "method too long" from:

// <Name>Methods too big (NbLinesOfCode)</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE NbLinesOfCode > 30 ORDER BY NbLinesOfCode DESC
// METHODS WHERE NbLinesOfCode > 30 are extremely complex and
// should be split in smaller methods
// (except if they are automatically generated by a tool).
// See the definition of the NbLinesOfCode metric here http://www.ndepend.com/Metrics.aspx#NbLinesOfCode

to:

WARN IF Count > 0 IN SELECT TOP 10 METHODS
    WHERE NbLinesOfCode > 30
        AND !(FullNameLike "^System.Linq.Dynamic") 
    ORDER BY NbLinesOfCode DESC

What if you don’t know how to write a regular expression? Well, then you need to learn how to write a regular expression. Regular expressions are the only option for this kind of name matching in NDepend.

It might seem at first that using a proprietary language would be a barrier to using the tool. After all, other tools, like FxCop can be customized using C# and other .NET languages. Doesn’t a proprietary syntax like CQL discourage customization? Not really, in my experience. The real win of NDepend is that you can customize the rules interactively, within the GUI. In contrast to building a custom FxCop rule and restarting FxCop, this is a huge win, because it encourages the use of ad hoc queries and trial and error to build the ideal heuristic for your application. Coupled with auto complete in the CQL editor, it makes the relatively minor inconvenience of dealing with a proprietary language almost irrelevant. Yes, I would prefer to use LINQ, but the interactivity of CQL is a much bigger win.

However, you don’t get the same kind of customizability at a global level. You can include and exclude assemblies from analysis globally, but that’s about it. So while you can include and exclude code based on almost any imaginable criterion at the level of an individual test, I have not found any way to do an exclusion like the one I demonstrate above globally. (Note, however, that in a comment on this blog a week ago, a member of the NDepend team suggested that features along this line may be coming next year.)

I should add that most static analysis tools seem to work this way. They typically make it fairly easy to include and exclude entire tests or specific tests on specific cases, but don’t typically include a way to say, "Don’t run any test on any bit of code which meets the following criteria, ever." So NDepend is no worse than other tools I’ve used in this regard, and it’s features to customize individual tests are considerably better than most other tools, although the learning curve is a bit steeper.

Like every static analysis tool I’ve ever tried, NDepend does not produce useful results for generated code. Since this is not an NDepend-specific issue, I discussed it in an earlier post, along with some tips for excluding generated code from NDepend’s analysis.

Documentation

The documentation in the NDepend online help is a bit thin. But the authors have written a number of informative blog posts on how to use specific features and tips for getting the most out of the tool. Once you realize that these posts exist and know where to find them, it is quite a bit easier to read up on how to use the tool effectively.

Usability

I’m of two, contrasting opinions regarding the usability of NDepend. I found the initial learning curve to be quite steep, and elements of the user interface to be not very discoverable. On the other hand, once I figured out how to use the application, which took me a few weeks (maybe I’m just slow), I found the application to be very usable, and I could do what I needed to do very quickly. Watching the videos on the NDepend website helps quite a lot in getting over this initial hump. I’ve also had some e-mail discussions with the makers of the software on how they might improve some of the things this new user found initially confusing in their next release.

For those new to the application, here are some things which may help you get started:

• The CQL query editor has auto complete, so it’s fairly easy to customize a query even if you’re unfamiliar with the language. However, if you intend to do any serious work in CQL, it is worth seeking out and spending some time reviewing the CQL examples on the NDepend website.
• After you do an initial analysis, look at the error list before considering the analysis results. Adjust the configuration (by adding assemblies or search directories) to clean up any of the errors before moving on to looking at the analysis results. There are a few valid C# language constructs that NDepend’s source code parser won’t recognize. I got a warning from a class constraint on a generic type. I understand this will be fixed in a future version. You can ignore these warnings; they’re harmless.
• Whatever you want to do within the application, the place to start is by selecting the appropriate view configuration from the View menu. So if you want to alter a query to return more specific results, you would select View -> Reset Views to work with the CQL language.
• The CQL Query Result tab will display a query, and has been Edit button, you can never added a query in that window. To edit a query, go to the CQL Queries tab, select the query you’d like to edit, and click the Edit Query button. A new tab will appear which allows you to actually edit query.
• When you click on a query in the CQL Queries window, you will only see the results if the query returns any rows. If the query returns no rows, then the results window will not appear.
• Whenever you analyze the project in the NDepend GUI, NDepend creates a lengthy HTML report, and opens it in your web browser. But all of the information in the report is also available within the GUI application, as well. So you can see the same information in whichever place is most convenient for you to navigate.

Performance

NDepend is fast, sometimes surprisingly so, given that it is looking at both binaries and source code. It generally takes less time to run an analysis than it does to compile a solution. I never experienced any performance problems in the NDepend GUI, either.

Conclusion

Do you need NDepend? It depends. NDepend will be useful to developers and teams who have a functioning continuous integration process (and you should) and want to impose standards on this sort of code which can be checked in to the repository. But if you’re going to go down this road, you should probably start with FxCop, which is free and better known. NDepend, however, can take this kind of analysis much, much farther than would be possible even with custom FxCop rules, like comparing multiple versions of an application and mandating unit test coverage. I also find it much easier to customize.

If you do have an automated build procedure in place and you’re not already doing static analysis, it’s time to start! It can be a bit overwhelming to run static analysis for the first time on a legacy project and see hundreds of warnings or errors. But that’s easy to get over; simply disable all the warnings but one, and start cleaning things up a bit at a time. You can make the analysis stricter as time goes on. You don’t have to start by fixing everything. Even a few rules in place will prevent new such errors being introduced to the source base.

It will be also useful to developers confronting a large source base for the first time, particularly if there are more than a few assemblies involved.

While NDepend might be useful to a single developer reviewing their own code, and not part of an automated process, I’ve found that static analysis will really only be used regularly when it is automated. Few developers have the discipline to run static analysis and cleanup any problems uncovered without integrating it into build automation.

Disclaimer

This review is based on a review copy supplied by the makers of NDepend. If you think that has colored my impressions of the tool, well, I disagree, but at least you know.

Share This | Email this page to a friend

I think I know what the answer is. Very often, someone will suggest an approach to a problem which strikes me as technically wrong or otherwise misguided. Like a lot of geeks, I have an almost compulsive desire to do things correctly, especially in areas where (to me, anyway) there is a clear-cut difference between "the right way" and "the wrong way" of doing it. Over and over again, I have said "no" in a way which ranged somewhere between insufficiently respectful and inappropriate.

I think I can explain why I do this (and I’ll do so below, for anyone who actually cares), but the important part is this: I need to teach myself to say "No" in a much more constructive way.

When I hear (what I feel is) a bad idea, I think I have this little moment of panic about the bad idea getting into our code, and having to maintain it for 10 years, because goodness knows I have a lot of bad ideas to maintain in our VCS. It’s like I’m worried that simply mentioning the idea will doom it (or us) to implementation, so I went to put the barrier in front of it before it can go any further.

This may accomplish the technical goal, but saying "No" in an unconstructive way is poor leadership. I don’t want to create a climate where people hesitate to suggest solutions to a problem, because they might be incorrect. Indeed, I learned as least as much, if not more, from my mistakes is from my successes, so discussing a bad idea may contribute as much to a project is discussing a good idea. Rather than just saying, "That won’t work because…" I need to help the person who suggested the idea explore its implications and work out the ups and downs on their own.

Share This | Email this page to a friend

Therefore, I’ve updated the About page to include formal licensing terms. In particular, any source code I’ve written and posted here is available under the MIT license, which should allow you to do almost anything you want to with it, but read the About page for the particulars.

Thanks to Robert Claypool for pointing this out.

One other bit of news about this site: Readers who prefer to be informed of new posts via Twitter can now follow the blog there.

Share This | Email this page to a friend

A Useful, But Imperfect, Tool

Static analysis can be incredibly useful. It finds errors in much the same way that compilers report hints and warnings, but at a significantly deeper level. Compilers typically report a hint or a warning only in cases where the compiler is completely certain that there is a problem with the code, like a completely unused variable. Static analysis tools, on the other hand, can report errors in cases where, for example, the code itself might function, but appears to be too complex to be maintainable. Obviously, there is a gray area between maintainable and unmaintainable, and every static analysis tool I have ever tried has returned false positives. FxCop, for example, will complain that an internal type is never instantiated if the only time it is ever instantiated is inside of a LINQ to Entities query; it doesn’t seem to be able to "see into the Expression."

For Code Written By Humans Only

Static analysis is close to useless for generated code, because it is generated by a tool, tends to always function correctly, and is presumably never maintained by humans, at least directly. But generated code tends to really set off alarms on static analysis tools, due to high line counts, cyclomatic complexities, and numbers of parameters, no unit tests, unusual type and member names, etc. In other words, in generated code functionality is much more important than maintainability. But in non-generated code, maintainability is at least as important as functionality. The best solution is to convince your static analysis tool to ignore the generated code altogether, but this is often easier said than done.

When I run FxCop or NDepend against the source code for our commercial applications, for example, they report a large number of errors, with the majority coming from generated code. The reality is that no tool I have used can consistently distinguish between unmaintainable code written by code generators and unmaintainable code written by programmers, automatically. Therefore, in order to get the maximum benefit from static analysis, we have some additional work to do in order to help the tools look at only the code where their analysis will return useful information for developers.

Excluding Generated Code From Static Analysis

The .NET framework includes an attribute, GeneratedCode, which is intended to tell tools such as FxCop to ignore parts of your code. Also, FxCop has its own attribute, SuppressMessage which instructs the tool not to report individual messages in specific cases, and individual tests can be turned off. CodeHealer can be told to suppress a certain number of messages, and you can turn individual tests on and off.

Even with these tools, however, the line between generated and non-generated code is not always clear. FxCop, for example, looks only at compiled assemblies, not at source code. If you have a partial class containing both generated code and human-written code, it’s important to recall that FxCop cannot tell the difference between the parts of the class which came from the generated code file and the parts which came from the non-generated file. The distinction simply does not exist in the compiled assembly, because "partial" classes are combined at compile time by the C# compiler. So you’re left with the choice of putting the GeneratedCode attribute on the entire class, causing static analysis to miss code you wrote, or not putting it on the entire class, resulting in static analysis errors from the generated code. The best solution is probably to keep any code you write as part of a partial class as simple as possible, such as calls into another class which does the real work. Considering that partial classes are typically used for things like forms, or LINQ to SQL types, this is a good practice anyway.

Code generation can also work by producing IL or assembly directly, without a source file at all. Obviously, tools which look only at source code, like CodeHealer or Microsoft StyleCop will miss this sort of code (which is a good thing) by their very design. But tools which reflect into compiled code will see this sort of code as no different from any other code in the assembly unless they are specifically designed to exclude it.

For example, when you have an anonymous type with a large number of members in C#, the compiler will generate (in IL) a constructor with an equivalent number of arguments. NDepend flags these as excessive numbers of arguments, and, indeed, it would be correct if this were a real method which I had written.

Improving Heuristics to Avoid False Positives

NDepend offers an additional means of filtering data. The tool has a domain specific language for querying your source code, called CQL, and you can use this to filter the specific areas analyzed by NDepend. I can modify the NDepend rules by editing the CQL, so I can, for example, exclude the cases I’ve just described (generated constructors with large numbers of parameters) by changing the default rule for "too many arguments," which looks like this:

// <Name>Methods with too many parameters (NbParameters)</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE NbParameters > 5 ORDER BY NbParameters DESC
// METHODS WHERE NbParameters > 5 might be painful to call and might degrade performance.
// You should prefer using additional properties/fields to the declaring type to handle
// numerous states. Another alternative is to provide a class or structure dedicated to
// handle arguments passing (for example see the class System.Diagnostics.ProcessStartInfo
// and the method System.Diagnostics.Process.Start(ProcessStartInfo))
// See the definition of the NbParameters metric here http://www.ndepend.com/Metrics.aspx#NbParameters

to this:

WARN IF Count > 0 IN SELECT TOP 10 METHODS 
    WHERE NbParameters > 5 
        AND !IsGeneratedByCompiler
    ORDER BY NbParameters DESC

…and the false positives go away. Unfortunately, this only fixes one test. Similar CQL appears in other default NDepend metrics, like this one:

// <Name>Quick summary of methods to refactor</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS /*OUT OF "YourGeneratedCode" */ WHERE 

                                           // Metrics' definitions
     (  NbLinesOfCode > 30 OR              // http://www.ndepend.com/Metrics.aspx#NbLinesOfCode
        NbILInstructions > 200 OR          // http://www.ndepend.com/Metrics.aspx#NbILInstructions
        CyclomaticComplexity > 20 OR       // http://www.ndepend.com/Metrics.aspx#CC
        ILCyclomaticComplexity > 50 OR     // http://www.ndepend.com/Metrics.aspx#ILCC
        ILNestingDepth > 4 OR              // http://www.ndepend.com/Metrics.aspx#ILNestingDepth
        NbParameters > 5 OR                // http://www.ndepend.com/Metrics.aspx#NbParameters
        NbVariables > 8 OR                 // http://www.ndepend.com/Metrics.aspx#NbVariables
        NbOverloads > 6 )                  // http://www.ndepend.com/Metrics.aspx#NbOverloads
     AND 

     // Here are some ways to avoid taking account of generated methods.
     !( NameIs "InitializeComponent()" OR
        // NDepend.CQL.GeneratedAttribute is defined in the redistributable assembly $NDependInstallDir$\Lib\NDepend.CQL.dll
        // You can define your own attribute to mark "Generated".
        HasAttribute "OPTIONAL:NDepend.CQL.GeneratedAttribute")

These must all be fixed individually. On the other hand, at least they can be fixed by easily changing the heuristics of the rule, instead of by excluding specific cases, as with many other static analysis tools.

One Time Pain

The real win from static analysis tools comes from integrating them into an automated build process. Tools like FinalBuilder and MSBuild can easily be configured to run all of the tools mentioned so far. This allows you to fail the build if potentially unmaintainable code is checked in. The first time you use any static analysis tool, you will probably need to spend a substantial effort in order to get the project to the point where it will not fail the build with existing code. This may involve turning off individual rules (you can gradually turn them on later), fixing specific violations, and excluding code such as generated code. You will then need to integrate the tool into your automated build process. This is, in all honesty, a fair bit of work, but it’s really a one-time pain. After that, the analysis will pay you back for your effort, day in and day out.

Share This | Email this page to a friend

Share This | Email this page to a friend

These are very bad approaches, for two important reasons:

1. They don’t work. If your action result is cached by ASP.NET, then the action will not even run the next time it is requested. AuthorizeAttribute handles this correctly. Code you write in a Controller cannot handle this. Code you write in a custom action filter could work, if you cut and paste the implementation from AuthorizeAttribute. But AuthorizeAttribute is unsealed, so why cut and paste, when you can subtype?
2. The modularity is wrong. You should aim to develop MVC sites which can be used with any authentication (or role) provider, whether it is ASP.NET membership, domain authentication, OpenId, or a custom membership provider. Wiring authentication concerns into a Controller makes this extremely difficult.

I’ve already hinted at the correct solution for the problem: If you cannot find an out of the box membership provider which does what you need, then look harder. If you still can’t find one, write your own. Here is a video and an article on how to do it. Similarly, the role provider can also be customized. In other words, the correct extensibility point for membership in an MVC site is exactly the same as the extensibility point for membership in a non-MVC ASP.NET site.

AuthorizeAttribute does a very simple thing in a somewhat complicated way. The simple thing it does is to tie the MVC framework into the existing ASP.NET membership system, which is quite customizable. That’s why I am skeptical about claims that AuthorizeAttribute can never serve someone’s needs. The implementation is more complex than one might imagine is required on first consideration because it must both be thread safe and cooperate with caching, which makes attempting to re-invent it at a different level a doubly bad idea.

Although AuthorizeAttribute doesn’t do very much, you may decide that it’s specific behavior is not correct for your project. In that case, it is generally a better idea to subtype AuthorizeAttribute instead of re-inventing your own. This allows you to preserve desirable behavior (like cooperating with caching) while customizing the behavior you need. You will need to override the AuthorizeCore method, taking careful note of the comments (in the MVC source code for the inherited method) on thread-safety.

One final word about customizing security: Your general approach should be to use proven, off the shelf, actively maintained solutions and customize as little as possible. If the problem you are trying to solve is not strictly security-related (like adding a custom field to a user profile), then do not alter (and potentially compromise) your security in order to solve it. Instead, find a more appropriate solution, like storing the information elsewhere, such as in an ASP.NET profile or your own DB. Doing security correctly is hard, and the consequences of small mistakes can be high. Even experts in the field make mistakes which can take years to uncover. Even if you follow to the letter everything which is known about implementing good security, today, your implementation may be obsolete tomorrow.

Share This | Email this page to a friend

How to Think Like the ADO.NET Entity Framework

(and why you might want to bother learning to do so in the first place)

In order to use the ADO.NET Entity Framework effectively, you must understand its value-based data model, which is distinctly different from the SQL and ORM approaches to representing and querying data. Programmers who approach the Entity Framework as "just another ORM" often find themselves frustrated or disappointed, a common reaction to trying to force a square peg into a round hole. This presentation examines the Entity Data Model in-depth, explaining where it fits into the context of a larger application, and demonstrates different techniques for retrieving and altering data. Also covered are performance considerations and future directions.

Sound like something you’d like to learn about? The conference is free, but you must register to attend. I’ll be on at 5:00 PDT; you can convert that to your local time.

Embarcadero has me in the Delphi Prism track, but my examples will be in C#.

If .NET’s not your cup of tea, there are plenty of other sessions you can join.

Update: The conference is over, but you can download a recording of my session.

Share This | Email this page to a friend

If you can persuade someone to use your modified compiler, or if you can modify the compiler on their machine, then you can infect the executables they produce, even if they recompile the compiler itself from source code! If that seems unlikely or difficult, read the whole paper. It’s easier than you think. Thompson concludes:

The moral is obvious. You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untested code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.

Now it seems that somebody has actually gone and done this with old versions of Delphi. Frankly, I’m surprised it took so long. Perhaps this is just the first time we’ve noticed. But the problem described by Thompson is so insidious that in the quarter century since he gave his address (which really just gave additional prominence to an issue discovered decades before that) it’s almost remarkable that this technique is not more widespread. It’s also interesting that this specific case is relatively innocuous; the virus does nothing more than reproduce itself, and only affects a very old compiler. Is this a test shot? Someone whose curiosity got the better of them?

I don’t know. But the historical perspective is clear: This is not about Delphi, and it’s not about the malware of the week. It’s an old question of trust becoming new again.

Here is one proposal for countering these attacks. Interestingly, the technique can sometimes find unintended compiler bugs, as well. They can be hard to distinguish sometimes!

Share This | Email this page to a friend

Share This | Email this page to a friend

Change Manager 5.1.1 will add support for InterBase 2007 and 2009 databases to: compare, version, and synchronize schemas; compare, synchronize, move, and mask data; and compare, version, and monitor version configurations for compliance and tracking.

And for any Firebird SQL users out there, you can join the party too because Change Manager will support versions 1.5 and 2.0 of that database. Move your data to and from any platform we support and move and compare schemas between Firebird and InterBase.

Also, it’s worth noting you can get Change Manager for free with the purchase of another Embarcadero tool, including Delphi or InterBase.

Share This | Email this page to a friend

Warning: The following comic has been demonstrated to severely lower developer productivity upon first introduction.

Sydney Padua has done the impossible: Created a programming comic which I like as much as XKCD. You can start at the beginning or read the latest, thrilling episode. She also did a special for the BBC. Navigation is a bit awkward (look for the "next" and "back" links above each post), but entirely worth the effort.

Share This | Email this page to a friend

The Give Camp organizers found charities who needed web development donations and split the registered developers into teams based upon skill sets and organization needs. I was assigned to the team for the Shiloh Family Institute College Choir Concert Series, a small organization which sponsors a series of gospel choir performances once a year. They needed a web site to promote the organization, list upcoming events, allow their patrons to register for e-mail alerts of scheduled concerts, and think their donors. Critically, the site had to be easy to maintain for administrators without computer expertise. My team included Brahma Ghosh, Brian Jackett, Joel Broughton, Joseph Ours, and Peter Vasys. Prior to the start of Give Camp, Brahma, the project manager, met with representatives of the organization and produced a mockup of each page we would build in the site.

We decided to build the sites using (non-MVC) ASP.NET, because that was the "least common denominator" of the skill sets of our team members. The requirements for the site specified displaying 10-20 concerts and around 10 sponsors. In view of the tiny data set and the emphasis on ease-of-use for the folks who will be maintaining the site, we decided against using a database or off-the-shelf content management system. Instead, we built the site such that the content areas would simply be editable in a WYSIWYG editor if you are logged into the site. So the site maintainers only needed to be instructed as to how to log in, and the rest is obvious.

Here is a part of what a logged-in user sees on the site homepage; you can compare that with the production site in order to see the difference.

WYSIWYG editor on site home page

At the beginning of Friday night, we met our clients, discussed the requirements for the project, settled on which technology we would use to build the site, and got to work. One of the first things we needed to do was to register a domain name and secure hosting so that the site could go live on Sunday. Also, we set up a Subversion repository on assembla.com. Then we split up the work and started writing code. The project manager created a solution for the project and checked it into subversion. I started creating repositories for the data which would be editable on the site. Because each dataset would only hold 10 through 20 records maximum, we decided to use XML files instead of a database to hold the data. Performance is not a problem with such a small dataset, and using files instead of a database allows non-technical users to easily move the site from one hosting provider to another. Also, we used the repository pattern, so it’s possible to switch out the storage without changing a line of web client code in the event the site ever grows beyond its current scale. The first thing I did was check in the public interfaces for the repositories so that the people creating the web pages could start work before I was done with the implementation. Other team members created web pages, started the site design, and began implementing a Google Maps feature on the "upcoming concert" page.

We wrote unit tests for all of the repositories in the site. In view of the very short amount of time we had to develop the site, that might seem surprising. In fact, however, writing unit tests allowed us to test the repository functionality before the web pages which used the repositories were finished, so even though it was a little extra work, it improved our ability to work concurrently.

I went home to get some sleep around 2 a.m. Friday night, but some people worked through the whole night. The Give Camp organizers supplied us with food, caffeine, and entertainment. I spent most of Saturday implementing a feature to allow the site maintainers to add and remove sponsors, including a feature to upload sponsor logos and thumbnail them to the correct size for the site. By the end of Saturday, just after midnight, all of the major features of the site except for the Google Maps were working, and we were down to fixing bugs and typos. A pleasant surprise for me was that the food provided was consistently good and occasionally even healthy; the Give Camp organizers had recruited some local restaurants as sponsors, and a grateful charity brought in some large containers of fruit.

Incidentally, there are some photos of Give Camp on Flickr.

We spent the first half of Sunday cleaning up defects, and the second half testing and fixing typos in the content. Some of the other team members implemented the Google Maps feature in two different ways in order to be sure that we would have at least one which worked before we deployed the site to production.

At 4:00 p.m. on Sunday, all eight teams presented their work. There was a real diversity of projects. In our case, we built a simple site from scratch and finished it the same weekend. Other teams added new features to existing sites. One team was given three or four different sites using different technologies with the instructions to make them have a common look and feel. Every one of the clients appeared completely delighted with the end results.

Whenever I read "studies" which claim that the majority of software projects fail, I have to balance these claims against my own personal experience, which demonstrates that projects can be successful even with very little time and team members who have never spoken to each other before the instant the project begins. So why did this project work out so well?

• Our project manager did excellent work up front with the client, so we knew exactly what we needed to build.
• The client had a good sense of priorities for the project.
• The team was highly motivated. We were all participating by choice. The Give Camp organizers kept the emphasis on fun. The deadline was challenging, but realistic.
• The Give Camp organizers did a great job splitting up the teams. We had people with project management skills, design skills, and development skills.
• We used off-the-shelf functionality where appropriate. For example, the mailing list registration is handled via the MailChimp API. The map feature is done with Google Maps. Event photos are hosted on Flickr. We used their API to show a single image when there was only one image of an event available, and a slideshow when multiple images were available.
• We built only what was needed. Using an off-the-shelf CMS would have delivered more features, but at the cost of being difficult for the client to maintain, both in terms of the user interface and a database on the back end. The client made it clear that ease-of-use was paramount, and we stuck to that.
• There was no dead weight on the team. A single, unmotivated developer can slow down an entire team. Since everyone was there by choice, and everyone was qualified for the work, we didn’t have to deal with any problem team members.
• We had our choice of technologies, so we could choose a platform with which everyone on the team had at least some familiarity.

Give Camp was the most fun I’ve had coding in some time, even though (non-MVC) ASP.NET is not my preferred platform. The emphasis on fun, the opportunity to try things outside of my comfort zone, and the excellent team atmosphere made it a great experience.

Share This | Email this page to a friend

Given that calling Skip(0) introduces a performance penalty, but there is a reason for introducing that penalty, should we avoid making this call? The short answer is yes, the performance improvement is worth it, but don’t forget the rule that you should have one test case for every conditional in your code. This rule is generally applied to unit testing, but in this case the appropriate test is an integration test, since you need to be sure that the LINQ provider will perform acceptably and not fail altogether when you fetch the data for the second page, which is now an entirely different query than that used to retrieve the data for the first page.

When I first observed the issue, I was happy to discover that I could make the query which fetches data for the first page of a result set (which happens to be the page which users most commonly request) 10 times faster. Of course, the other way to look at this is that the query that fetches data for the second and all subsequent pages is 10 times slower than the query which fetches data for the first page. Again, given that the first page is displayed much more frequently, that’s not necessarily a bad thing, as long as it doesn’t catch you by surprise. Indeed, calling Skip(n), where n > 0, and getting your results not quite as quickly as if you had not called Skip(n) at all is actually the best case; the worst possible case is that the LINQ provider will generate invalid SQL and the query will fail.

Note that it is especially difficult to ensure complete integration testing coverage as you introduce more and more ways that the user can shape a result set. If the user can change the sort order, enter a variety of search queries, etc., it’s a safe bet that every combination of these options will produce a different query with different performance characteristics. Avoiding calling Skip(0), in other words, does not increase the number of integration test cases for a particular list of data by one; it doubles it.

Share This | Email this page to a friend

Share This | Email this page to a friend

Share This | Email this page to a friend

Unfortunately, it is not just your own code you need to be concerned with. It turns out that some fairly popular LINQ helpers do this. For example, Rob Connery’s PagedList<T> type does it:

public PagedList(IQueryable<T> source, int index, int pageSize)
{
    this.TotalCount = source.Count();
    this.PageSize = pageSize;
    this.PageIndex = index;
    this.AddRange(source.Skip((index - 1) * pageSize).Take(pageSize).ToList());

Index, in this code, is the 1-based page you’d like to display. When index is 1, the code calls Skip(0), and you get less efficient SQL. Fixing the problem is quite simple:

    this.PageIndex = index;
    var pageData = index > 1 ?
        source.Skip((index - 1) * pageSize).Take(pageSize) : source.Take(pageSize);
    this.AddRange(pageData.ToList());

Naturally, this only optimizes SQL generation for the first page of results, but since this is by far the most common page a user might display, it can net you a substantial win in server performance.

Troy Goode’s updated PagedList type, which I have recommended in the past (and still do, with this tweak!), has the same problem. The solution is almost identical, although Troy’s type is 0-based rather than 1-based. Update: Troy has now posted an update containing this fix, among many other changes.

The root of the problem in the SQL generation is that when you call Skip(0), LINQ to Entities, in at least some cases, generates SQL like this:

)  AS [Project3]
WHERE [Project3].[row_number] > 0
ORDER BY [Project3].[TimeRecordDate] DESC

The reference to row_number can have a substantial performance impact. In one case I tried with a SQL Server table containing just over 1 million rows, a simple query ran with sub-second performance without this WHERE clause, and over 4 seconds with the WHERE clause, even though the result set was exactly the same.

Now, is this a bug in LINQ to Entities? Or LINQ itself? It’s hard to say. Both frameworks are doing exactly what you tell them to, even though it doesn’t make a lot of sense. It seems like a missed opportunity for optimization, but I’m open to the possibility that there might be some desirable effect of this that I have simply missed. For my own purposes, I am content to simply not call Skip(0) at all.

Share This | Email this page to a friend

It’s worth a listen if you’re interested in the history of the product which started Borland and evolved into Delphi.

Share This | Email this page to a friend

Download the demo solution: DayPilot ASP.NET MVC demo

We needed a calendar control for an ASP.NET MVC application. In particular, we needed a control which would show a number of events and when they occur within a particular day, "Outlook-style." I’ll include a screenshot of the demo application here, since it’s the best way to explain the type of control we were looking for:

DayPilot example

Possible Alternatives to DayPilot

In my experience, the type of control which works best with ASP.NET MVC is a control which is completely ignorant of ASP.NET, and all the lies such as ViewState and postbacks which accompany it. In particular, I’ve had a good deal of success with controls designed for the jQuery JavaScript framework, which is included, by default, with ASP.NET MVC applications. Unfortunately, I was unable to find a jQuery control which data what we needed. The closest I found was jMonthCalendar which cannot display a single day in the manner illustrated above. I would still prefer a jQuery-based solution if one ever turns up.

I did, however, find a couple of controls designed for "plain" ASP.NET which, from an end-user point of view, it exactly what we needed. One was the DayPilot control illustrated above, the other was the scheduler component included with Telerik’s control suite. Telerik’s control claims some support for MVC, but this is really limited to loading calendar data asynchronously, which was not actually what we wanted to do. Also, Telerik’s control can only be purchased as part of their suite, which is rather expensive if you only need the calendar/scheduler control. Paying for the entire suite would not be a problem for us if it was filled with controls we needed, designed for the environment (MVC) we use. While Telerik seems serious about supporting MVC, they are in the early stages right now, so it is difficult to justify paying for a large suite in order to get a single control, which I would have to use in a mostly-unsupported way. The Telerik control has many more features than the DayPilot control, and, in my opinion, looks a little nicer.

The DayPilot calendar is available in two different editions. There is an open source edition, which is free, and a non-open-source edition, which adds features. The principal difference is that the date version includes editing (via postbacks) and UpdatePanel support, which are not really useful in an ASP.NET MVC application anyway. As with Telerik’s control, I have to adapt the control for use within ASP.NET MVC. So we selected the open source version, even though the paid version is priced reasonably.

Adapting DayPilot to ASP.NET MVC

In order to use the DayPilot control in ASP.NET MVC, I wanted to use markup which would look familiar to ASP.NET MVC developers:

<%= Html.DayPilot(Model, new DayPilotViewOptions
                             {
                                 HourHeight = 30
                             }) %>

This means that I needed to write an Html helper which will render the control based on the data passed in Model and return the literal HTML markup. In order to make that easier, I created two new types, DayPilotData and DayPilotViewOptions, in order to encapsulate access to the properties of the control. The reason I created two different types was to separate the concerns of data for the events (which will be set up in the controller) and display of the control (which is the concern of the view). The types are trivial, so I’m not going to show them in this post, but you can examine them in the demo solution. There is an overload to this Html helper which does not include the options argument in case you are happy with the defaults.

        public static string DayPilot(
            this HtmlHelper helper,
            DayPilotData model,
            DayPilotViewOptions options)
        {
            var calendar = new DayPilotCalendar();
            if (model != null)
            {
                model.CopyTo(calendar);
            }
            if (options != null)
            {
                options.CopyTo(calendar);
            }
            var sb = new System.Text.StringBuilder();
            sb.Append("<div class=\"dayPilot\">"); // allows working around td cellpadding bug in css
            using (var sw = new System.IO.StringWriter(sb))
            {
                using (var tw = new HtmlTextWriter(sw))
                {
                    calendar.RenderControl(tw);
                }
            }
            sb.Append("</div>");
            return sb.ToString();
        }

Note the div. The current version of DayPilot astonishingly, as of this writing, does not put a class anywhere in its rendered markup. Since the stylesheet for the default ASP.NET MVC site conflicts with DayPilot, to some degree, I wrapped the whole control in a div so that I could fix the layout in CSS.

Another way which I could have used the control in ASP.NET MVC would be to register a tag prefix and use code behind/code generation, as with standard ASP.NET. I’ve registered a tag prefix in the demo solution in case you want to try that yourself. Be aware, however, that setting up codebehind and code generation files in an MVC application is somewhat less than simple; doing it right requires, among other things, manual editing of the csproj file. So I do recommend using the control in the way I’ve illustrated.

Finally, here’s the source code for the action which supplies data for the control. Since this is a demo solution, I’m generating random events:

        public ActionResult Index()
        {
            var r = new Random();
            var q = from i in Enumerable.Range(1, 4)
                    let start = DateTime.Today.AddHours(r.Next(47))
                    select new DayPilotDataItem
                    {
                        Id = i,
                        Start = start,
                        End = start.AddHours(1),
                        Text = string.Format("Event {0}", i)
                    };
            var model = new DayPilotData
            {
                StartDate = DateTime.Today,
                Days = 2,
                Data = q
            };
            return View(model);
        }

Of course, it’s just as easy to select data from a database.

The current version of DayPilot has a kind of a weird limitation: You cannot make it smaller, in the vertical dimension, than 30 pixels per hour. Based on some inspection of the source code, I think this is fixable if you are comfortable altering the source. But be aware of the limitation.

There are more DayPilot features which I have not covered, but that should be enough to get you on your way. If you have any questions, post them in comments, and I’ll see whdat I can do to help out.

Share This | Email this page to a friend

Finding your blogs very helpful - superbly written - just found out about jqgrid last night and have integrated your code into my project.

One thing that didn’t work for me though was the dates fix above…

adding

.ToString(System.Globalization.CultureInfo.CurrentUICulture)

onto the end of my DateTime values makes everything render Ok but when I try doing a search, it fails. Debugging shows the error is

Method ‘System.String ToString(System.IFormatProvider)’ has no supported translation to SQL.

at the TotalItemCount = source.Count(); line of the PagedList.cs file.

Any ideas?

In order to understand this error, we need to remember several important things about LINQ:

• LINQ is lazy. Creating or modifying an IQueryable with methods like Where, Take, OrderBy, etc. does not actually execute the query. The query will only be executed when you call a LINQ method which requires execution to produce its results — like Count.
• When a LINQ query expression includes something which looks like a method call, the method is not invoked directly. Instead, what is actually produced is an Expression representing that method. If the LINQ provider happens to be LINQ to Objects, the method will eventually be invoked when the query is executed. But if it happens to be LINQ to SQL or LINQ to Entities, the method may never be invoked, and instead might be replaced (if possible) with equivalent SQL.
• Most LINQ providers do not support every possible LINQ query expression. Most providers don’t even support the entire LINQ API, and only LINQ to Objects can be presumed to support all method calls on an object in a query. In some cases, as with LINQ to Entities, this is documented. With other providers, you may have to guess/experiment. Any LINQ provider which has to translate a LINQ query expression into SQL syntax can only really support methods which it is hard-coded to recognize. Worse, it is not even uncommon for providers to support one overload of a particular method but not a different overload of the same method. See, for example, the LINQ to Entities document I just linked or the ToString method, which is generally supported in LINQ to Entities/SQL, but has a special overload on the DateTime type which is not supported. Worse still, method calls may be supported in some contexts but not others. Graeme reports that LINQ to SQL accepted the ToString overload in the context of a Select, but not in the context of a Where. This makes sense from an implementation point of view, but means programmers have to be extraordinarily careful about what we include in our LINQ queries, especially when they are constructed dynamically.

By now it should be obvious what is going on here: The particular overload of ToString which I used in my example is supported in LINQ to Objects but not in LINQ to SQL in a Where expression. My demo solution used a repository based on LINQ to Objects, so the query I showed worked correctly there. But it does not work with LINQ to SQL. Because LINQ is lazy, you will not actually see an error about this until the query is translated into SQL by the LINQ provider, and that does not happen until the call to Count in PagedList.

This means that when you write a LINQ query expression using LINQ to SQL, LINQ to Entities, etc., you are essentially working in a partially dynamic language. Unless you study the documentation very, very carefully, you will not know for certain until runtime if your query will be accepted by the provider.

So how could we work around this problem?

• Use a different method or a different overload which is supported by the LINQ provider used in your project. This is fine if an appropriate method/overload exists.
• Give up on doing the formatting in C# and use JavaScript instead. This will work, but it’s a bit like using beheading to cure acne. C# is, after all, a general-purpose programming language. We should not have to give up on formatting a date.
• Bring the query into LINQ to Objects using a method such as AsEnumerable. Obviously, we would want to apply the paging and ordering first. In this method, we would execute one query (using LINQ to SQL or something) which would retrieve an ordered list of records for the specified page, using AsEnumerable to transform the results into a List. We would then execute a LINQ to Objects query against this list in order to perform the formatting. This will actually work OK, although it can end up being a good bit of code to write for what should be a fairly simple task.
• Write a new LINQ provider which understands the method in question. This seems like an enormous amount of work, but I’ll include it for completeness.
• Separate the concerns of serialization and querying. You can tell by the 50-cent words that this is the solution I prefer.

Remember, the original query worked just fine. It is only the serialization to JavaScript which caused unexpected results. One way to fix this is to hide the members which are problematic to serialize and replace them with a string version of same:

private class JavaScriptDateFixer
{
    [NonSerialized]
    internal DateTime _date;
    public int Id { get; set; }
    public int IntProperty { get; set; }
    public string StringProperty { get; set; }
    public string DateProperty { get { return _date.ToString(System.Globalization.CultureInfo.CurrentUICulture); } }
}

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new JavaScriptDateFixer
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    _date = entity.DateProperty
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that even though this code looks quite different, this does almost exactly the same thing as the code I’ve replaced. The only real difference is that I’ve delayed the call to ToString until after the query is executed. That’s enough to keep LINQ to SQL or LINQ to Entities happy.

So with this fix is everything fine? Maybe. I really like LINQ; I think it’s an extraordinarily powerful tool. But here we have hit upon what I consider to be the single biggest problem with LINQ. The combination of very limited support for all possible query expressions in common LINQ providers like LINQ to SQL and building queries dynamically can be very dangerous. Unless you test all possible query permutations, you cannot be completely confident that any query you build will actually execute successfully. In typical dynamic language programming, you mitigate this problem by writing unit tests. But unit tests are not supposed to connect to a database. Unfortunately, the limitation here is that the point of translation to SQL. So any unit test which mocked the LINQ provider, say, using LINQ to Objects or some other in-memory representation would not actually encounter the limitations we are trying to test.

I have yet to see a really good solution for unit testing LINQ to SQL or LINQ to Entities, one which would combine the desired attributes of not actually connecting to the database while surfacing any potential problems in SQL generation. I suspect that such a beast may never exist for LINQ to SQL, but for LINQ to Entities it seems at least possible. One could write an Entity Framework provider which returns mocked instances of entities. Any LINQ syntax not understood by the Entity Framework would be caught before the mock provider was ever invoked. This is a feature I would like to see in a future version of the Entity Framework.

Share This | Email this page to a friend

Share This | Email this page to a friend

The delete feature of jqGrid is, oddly, almost entirely undocumented, even though there quite a few examples of different methods of editing, and inserts are documented, to a lesser extent. But it does exist; I just had to read the source code to figure out how it works.

JqGrid has three different methods of editing data. The cell edit feature allows the user to edit a single cell the grid at a time, and saves the data when the cell loses focus. The row edit feature works similarly, but does not save the data until the row loses focus. The form edit feature shows a modal dialog instead of editing the values inline. Of these three, the form edit feature is the only one which supports deletes. However, you can still use one of the other "modes" for editing, if you like. Just be aware that when you do a delete, you will be in the form edit code. Also, deletes won’t work if you change the grid’s configuration to disable the form edit feature.

In order to allow the user to delete a row, I need to:

• Add a delete button to the grid’s toolbar. I could, of course, use a custom button or something instead of the grid’s toolbar if I chose to do so.
• Specify a URL for the delete action
• Implement a delete action on the controller

I can accomplish the first two tasks by making a slight change to the JavaScript that configures the grid:

}).navGrid(pager, { edit: false, add: false, del: true, search: false }, {}, {}, {url: "Delete"});

I’ve highlighted the parts I’ve changed in boldface. Changing the "del" property to true turns on the toolbutton. The fifth argument is an object specifying options for the delete action, of which I have only specified the URL. This is important: The only places you can specify the URL for delete are in the arguments to the delGridRow method or in the toolbar’s configuration. There is not a "delete URL" property of the grid itself.

If you don’t specify a delete URL in this way, the grid will presume that you want to send delete requests to the URL you have specified for editing rows. There will be an "oper” value in the submitted form indicating that the request is a delete rather than an edit. But I prefer to use a separate URL for delete, because I think that fits more naturally in the ASP.NET MVC paradigm.

Now let’s write an action to handle the delete:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Delete(int id)
{
    var repository = new Repository();
    var deleted = repository.Delete(id);
    if (!deleted)
    {
        Response.StatusCode = 500;
        return Content("Record not found.");
    }
    return Json(true);
}

Again, I’ll note that I’m using a mock repository for this demo so that you can compile and run the solution without needing to set up a database. Since the mock repository is based on IList, the Delete method of the repository returns a Boolean indicating whether or not the requested record was found, since that is how IList behaves. It would probably be more correct to return a partial view in the event of an error rather than specifying the contents directly, but error handling is a subject I hope to cover in more depth later on. For the time being, simply changing the StatusCode will tell the grid when something does not work.

If the delete was successful, I return "true." By default, the grid ignores returned data when the operation is successful. However, you can handle the grid’s afterSubmit event, which will be passed any data you do return. So what you choose to return, and what you do with it, is entirely up to you.

With all that in place, deletes work as expected. I’m not going to update the demo solution just yet, because I’ll be covering edits and inserts real soon now. But first I need to revisit formatting and explain more about LINQ".

Share This | Email this page to a friend

First, let’s fix the date display. The reason the date appeared so oddly is that there is no official support for Date types in JSON, even though JavaScript has a date type. Microsoft, therefore, invented their own way of encoding dates for their JSON serializer, which jqGrid does not understand. There are two possible ways to fix the problem. The first is to keep sending the data to the grid in the Microsoft format, and write a custom formatter for the grid to parse that into a human-readable string. The second is to change the data we send to the grid. Both ways work, but I’m going to demonstrate the second method, mostly because I find C#’s date handling features superior to (and less confusing than) JavaScript’s.

In order to change the date "column" in the results of the query to a formatted string, we simply need to project into an anonymous type. However, this would break the sort, because a text sort on a formatted date will not be in the correct order. Hence, it is necessary to apply the sort before projecting. So I add an OrderBy to the query, and pass null as the orderBy argument to the ToJqGridData method.

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    DateProperty = entity.DateProperty.ToString(
                        System.Globalization.CultureInfo.CurrentUICulture)
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that in this revised solution I’ve updated the mock repository to return more sensible dates.

Important update: This code works correctly with LINQ to Objects, as with my demo repository, but will not work with LINQ to SQL. I have addressed this issue in a subsequent post.

Now let’s add a search feature. One of the problems in making a "demo" search is that search is a highly application-specific feature, both in terms of implementation and in terms of user experience. JqGrid’s search feature presumes that you want a search which looks something like this:

Visually, there are lots of variations on this theme available, but in terms of the use case they all presume that the user will specify which fields she would like to search. However, I prefer a "Google-style" interface with just a search box and an implementation that can (hopefully) figure out what the user actually wants. So that’s what I’m going to demonstrate. The needs of your application may be different. The important thing to take away from this demonstration is that the jqGrid search feature can be co-opted to fit your design.

I want the search box to appear above the grid at all times, rather than requiring the user to press a toolbutton to see it. So first I need to add an empty div to the markup in order to indicate where the search box will be positioned, relative to the grid:

    <h2>GridDemo</h2>
    <div id="search"></div>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>

Next, I add a small bit of JavaScript to tell jqGrid to replace this div with a search control:

    search.filterGrid("#" + grid.attr("id"), {
        gridModel: false,
        filterModel: [{
            label: 'Search',
            name: 'search',
            stype: 'text'
        }]
    });

The filterModel tells the grid to show a search control for one column, called "Search." Note that the entities I’m returning do not have a property called "search." That’s OK. When the user types something in the search box and presses enter, the grid will add a query string parameter to its request for the data of the form "search=WhatTheUserTyped". That’s what I want, and the fact that there is no "search" column in the data does not adversely affect the grid in any way.

The first argument to filterGrid is a little odd. I’m required to pass a jQuery selector as a string rather than being able to pass a jQuery object directly. Since this is all inside a method which accepts jQuery objects as arguments (rather than hard-coding the IDs of the DOM objects in the method, for unit-testability), I have to "decode" the jQuery object into a selector string. But note that I did not need to do that when setting up the pager or the grid itself. Like I said before, the grid’s API is consistently inconsistent.

As you can see in the code at the beginning of this post, the GridDemoData action just passes the search query string parameter through to the ToJqGridData method unchanged. Inside that method, I use Dynamic LINQ to alter the IQueryable to implement the search. I’ve cleaned up this method substantially since I first posted the code. Here’s the revised method:

/// <summary>
/// Adds a Where to a Queryable list of entity instances.  In other words, filter the list
/// based on the search parameters passed.
/// </summary>
/// <typeparam name="T">Entity type contained within the list</typeparam>
/// <param name="baseList">Unfiltered list</param>
/// <param name="searchQuery">Whatever the user typed into the search box</param>
/// <param name="searchColumns">List of entity properties which should be included in the
/// search.  If any property in an entity instance begins with the search query, it will
/// be included in the result.</param>
/// <returns>Filtered list.  Note that the query will not actually be executed until the
/// IQueryable is enumerated.</returns>
private static IQueryable<T> ListAddSearchQuery<T>(
    IQueryable<T> baseList,
    string searchQuery,
    IEnumerable<string> searchColumns)
{
    if ((String.IsNullOrEmpty(searchQuery)) | (searchColumns == null)) return baseList;
    const string strpredicateFormat = "{0}.ToString().StartsWith(@0)";
    var searchExpression = new System.Text.StringBuilder();
    string orPart = String.Empty;
    foreach (string column in searchColumns)
    {
        searchExpression.Append(orPart);
        searchExpression.AppendFormat(strpredicateFormat, column, searchQuery);
        orPart = " OR ";
    }
    var filteredList = baseList.Where(searchExpression.ToString(), searchQuery);
    return filteredList;
}

Importantly, this method is simply adding to the expression represented by the IQueryable. It does not actually implement the search. So any search methodology which understands LINQ can deal with its results. At least, theoretically. In practice, LINQ providers vary greatly in terms of which LINQ features they support. So although this method works fine with LINQ to Objects (as used in the rest of the demo solution) I cannot guarantee that it will work with every LINQ provider in the world. You have to try, and revise it to fit the capabilities of the provider you’re using.

Putting all these pieces together, I now have a search feature which does what I want:

Be sure to download the updated solution in order to get the bug fixes and new features.

As a special, no-extra-charge bonus, the updated solution also includes a demonstration of using the grid’s tree view mode with client-side data from a JavaScript function. This has nothing to do with ASP.NET MVC; it’s just an example of how to use a grid feature which I wrote in response to a request in the jqGrid support forums.

That’s all for today. In the next post in the series, I’ll begin to demonstrate the grid’s editing features, and how to use them in an ASP.NET MVC application. But I will also take requests. If there’s a grid feature you’d like to see demonstrated query question regarding using the grid in ASP.NET MVC, please feel free to make a request in comments, and I will answer it as best I can.

Share This | Email this page to a friend

Share This | Email this page to a friend

On The Podcast at Delphi.org, Nick Hodges confirms my earlier speculation about reorganization in the "CodeGear" group at Embarcadero. The whole show is worth listening to. Congratulations are due to Chris Pattinson, who was promoted to Director of Quality for all of Embarcadero. Time to update the subtitle of your blog, Chris! (And maybe even write a post or two…) Sounds like good changes all around. You read it here first, folks!

I wrote a short demo showing how to use jqGrid in tree grid node with client-side data supplied by a JavaScript function.

Share This | Email this page to a friend

Rather than make you set up the Northwind database, I’ve written a quick, mock repository which supplies random data for the application. But the data persists for the lifetime of the app, so I’ll be able to demonstrate sorting, paging, etc. This way you can just run the application, without having to set up SQL Server.

So let’s write some code display a grid.

The data I’m going to display in the grid is a fairly useless type:

    public class SomeEntity
    {
        public int Id { get; set; }
        public int IntProperty { get; set; }
        public string StringProperty { get; set; }
        public DateTime DateProperty { get; set; }
    }

We need two actions. The first will display the page containing the grid. The second supplies data for one page of the grid. Remember, the grid can page through large datasets. It will only request one page of data at a time.

        public ActionResult GridDemo()
        {
            return View();
        }

        public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
        {
            var repository = new Repository();
            var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
                new[] { "IntProperty", "StringProperty", "DateProperty" });
            return Json(model);
        }

The first action simply returns a view containing the HTML table which will eventually become the grid. It doesn’t need a model, because the grid will request its data asynchronously. The second action handles the data requested by the grid.

In order to make this work, we need HTML elements for the grid and its pager in the view, along with a reference to the JavaScript which will turn those elements into the grid.

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <h2>GridDemo</h2>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>
</asp:Content>

<asp:Content ID="Script" ContentPlaceHolderID="ScriptContent" runat="server">
    <script language="javascript" type="text/javascript" src="<%= Url.Content("~/Scripts/Home.GridDemo.js") %>"></script>
</asp:Content>

The JavaScript file is where the grid itself is configured.

$(document).ready(function() {
    GridDemo.Home.GridDemo.setupGrid($("#grid"), $("#pager"));
});

GridDemo.Home.GridDemo = {
    setupGrid: function(grid, pager) {
        grid.jqGrid({
            colNames: ['Int', 'String', 'Date'],
            colModel: [
                        { name: 'IntProperty', index: 'IntProperty' },
                        { name: 'StringProperty', index: 'StringProperty' },
                        { name: 'DateProperty', index: 'DateProperty' },
                      ],
            pager: pager,
            sortname: 'IntProperty',
            rowNum: 10,
            rowList: [10, 20, 50],
            sortorder: "asc",
            url: "GridDemoData"
        }).navGrid(pager, { edit: false, add: false, del: false, search: false });
    }
};

Important to note here are:

• colNames, the grid column captions the user sees.
• colModel,which includes the property name, which is the field name in the returned JSON data which supplies the data for that column, and index, which is the name which will be sent to the GridDemoData action as the sidx (field name to sort on) argument when you click on a grid header. There are a number of other properties you can set here which control editing, with, formatting, etc. See the documentation for full details. Importantly, colNames and colModel must have the same number of entries, and be in the same order.
• url, which tells the grid where to get its data. You’ll notice that the value I’ve passed is the name of the second action.

As I mentioned in the last post, I set a number of the grid options in a script referenced by the Site.Master, in order to prevent having to set them on every grid I create. The configuration idea for the grid on an individual page should be only the things that are unique to that page. So there is a fair bit of configuration for the grid which is not included here. In addition, the Site.Master contains references to the grid’s JavaScript and CSS files.

Finally, it’s worth mentioning that when you add jqGrid to a project you need to edit the jQuery.jqGrid.js file to tell the grid where to find the rest of the JS files it needs. Once you look at this file, it is obvious what you need to change.

At this point, I’m going to make the demo application available for download, so that you can experiment with it yourself. However, I’ll caution that I intend to update it when I write the next post in this series. In other words, the current download is a work in progress. It will change in the next few days.

Download the Solution (587 KB)

So let’s run the solution and see what we get:

OK, the hideous and eyestrain-inducing clash between the default MVC theme and the jQuery UI "lightness" theme included with the grid download notwithstanding, this is actually not bad. Sorting and paging both work, and performance is very good.

But what’s the deal with the data in that date column? It turns out that there is no standard representation of a date in JSON, so Microsoft invented their own. The grid doesn’t know about this special format. We’ll clean that up in the next post in this series and also cover search. Stay tuned!

Share This | Email this page to a friend

Nick Hodges has a new role/job title.

Item 2:

During his presentation to the Columbus Architecture Group a couple of weeks ago, Mike Rozlog handed out business cards which gave his title as "Senior Project Manager, Java Tools." He told us that in a couple of days he would have new business cards, because his role was changing. He didn’t tell us precisely what the new role would be, but I got the feeling that, like Nick, he was happy with the change.

Now, it is of course possible that these two changes happening so close together is merely a coincidence. But my personal speculation is that there are bigger plans afoot, and I’m glad to see that Mike and Nick, two people I trust, seem to be happy with them.

Repeat: This is just my personal speculation. It is based on the evidence above, not inside information. I could be wrong.

Share This | Email this page to a friend

But it turns out that we’re using the grid in a different way, and I think the difference is important. So rather than just referring you to Phil’s post and skipping the rest of the series, I think I still have some important information to contribute. In particular, I’m using the grid in a way which allows me to write an extension method for IQueryable<T> which returns data suitable for the grid, without having to know anything about the type T. If you look closely at the code in Phil’s post, you will see that populating the cell array requires custom code (or use of reflection) every time you want to return data to a grid.

It’s important to me that the method should work without knowing anything about the type of the data in the list because I don’t want to have to write repetitious code in every action method which supplies data to a grid to shape the data according to the grid’s needs, and because I generally use anonymous types for JSON serialization. JSON cannot handle object graphs with cycles (i.e., a "circle" of objects which all reference each other). The .NET JSON serializer will throw an exception if you pass it an object which contains a circular reference, or refers to another object which does. Since this is largely data-dependent and, worse, can be occasionally masked by lazy loading and the like, the safest way to serialize to JSON is to pass it data which provably can never contain a cycle. Anonymous types work really well for this. I’ll probably elaborate on this point in a future post. For the time being, the important thing is that I need to be able to take any IQueryable and turn it into data suitable for the grid, without having to think about what type that list might contain.

In a nutshell, this is what I want to be able to do:

public JsonResult ListGridData(int page, int rows, string search, string sidx, string sord)
{
    var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
      new[] { "Column1", "Column2", "Column3" })
    return Json(model);
}

In this example, repository.SelectAll() returns an IQueryable<TSomething>. I’ve written a method, ToJqGridData, which can transform that IQueryable into data suitable for the grid without having to know what TSomething is. (What the heck is that string array? Don’t worry about that just yet; I’ll cover it in a little bit.) The argument names aren’t what I would have picked, but they come from the grid. There is probably a way to change them, but I’ve never bothered to look.

The key to this is a grid property called jsonReader.repeatitems. I’m not sure why it’s called repeatitems, but here’s what it does. When repeatitems is true (the default), data is returned to the grid in a "cell array," like in Phil’s post. That is, an individual row of data in the JSON looks like this:

{id:"1", cell:["cell11", "cell12", "cell13"]}

When repeatitems is false, the data looks like a regular object in JSON format:

{id:"45678",name:"Speakers",note:"note",stock:"false",ship:"4"}

Interestingly, it turns out that this is the format the .NET JSON serializer already uses. So I don’t have to write any code to get this portion of the data into the right shape. I just hand an object to the JSON serializer and let it do its thing. The only things I need to do to make the data work for the grid is to handle paging, sorting, search, and provide certain other bits of data, like the number of rows.

But, as you may have noticed, I’m lazy. Why write code when someone else has already done it? Rob Conery wrote a decent type for paging eons ago, and Troy Goode enhanced it. So I’m going to use their work instead of re-writing paging. It turns out this type already computes most of what the grid needs to know. And Microsoft has this thing called Dynamic LINQ which can deal with ordering and filtering by property names, mostly. All I have to do is tie this stuff together.

I wrote a type to hold the data which the grid needs. You can actually use an anonymous type for this; the JavaScript grid obviously doesn’t care if the data it receives comes from an anonymous type or not, but because I am writing extension methods to IQueryable, it was convenient for me to have a non-anonymous type so that the extension methods could build on each other. This type represents one page of data requested by the grid:

namespace GridDemo.Models
{
    /// <summary>
    /// This type is designed to conform to the structure required by the JqGrid JavaScript component.
    /// It has all of the properties required by the grid. When this type is serialized to JSON, the resulting
    /// JSON will be in the structure expected by the grid when it fetches pages of data via AJAX calls.
    /// </summary>
    [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId="Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    public class JqGridData
    {
        /// <summary>
        /// The number of pages which should be displayed in the paging controls at the bottom of the grid.
        /// </summary>
        public int Total { get; set; }
        /// <summary>
        /// The current page number which should be highlighted in the paging controls at the bottom of the grid.
        /// </summary>
        public int Page { get; set; }
        /// <summary>
        /// The total number of records in the entire data set, not just the portion returned in Rows.
        /// </summary>
        public int Records { get; set; }
        /// <summary>
        /// The data that will actually be displayed in the grid.
        /// </summary>
        public IEnumerable Rows { get; set; }
        /// <summary>
        /// Arbitrary data to be returned to the grid along with the row data. Leave null if not using. Must be serializable to JSON!
        /// </summary>
        public object UserData { get; set; }
    }
}

Note that I’ve used C# capitalization for the property names. This is not what the grid expects. But I have to tell the grid that I’m using a non-default value for repeatitems anyway, so it’s nice to follow C# conventions.

We have lots of grids in our application, so rather than configuring each grid individually, we set defaults for the grid in one place. In a JavaScript file referenced by the Site.Master, this method is called in the jQuery ready event:

    setDefaults: function() {
        $.jgrid.defaults = $.extend($.jgrid.defaults, {
            datatype: 'json',
            height: 'auto',
            imgpath: '/Content/jqGrid/themes/basic/images',
            jsonReader: {
                root: "Rows",
                page: "Page",
                total: "Total",
                records: "Records",
                repeatitems: false,
                userdata: "UserData",
                id: "Id"
            },
            loadui: "block",
            mtype: 'GET',
            multiboxonly: true,
            rowNum: 20,
            rowList: [10, 20, 50],
            url: "ListGridData.json",
            viewrecords: true
        });
    }

In addition to setting repeatitems, telling the grid to expect the capitalization I mentioned above, this code also sets the grid theme and other options we want to be consistent throughout the application. One important property set in the jsonReader property is id. The value supplied is the name of a column in the returned data which contains a unique ID for the row. You can supply any name you want to, and the type of the property can be just about anything serializable, but it is important that the data you return to the grid contains some unique ID. All of the types in our Entity Framework model contain a unique ID called "Id", so we set this in our defaults for the grid. If you don’t have a single operating name for any data you might supply to the grid, you will have to set this value on the individual grids, when you write the JavaScript for the view.

Now I can write a method which takes a PagedList and transforms it into jqGrid data:

        /// <summary>
        /// Converts a paged list into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="list">The list of records to display in the grid.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data.
        /// Leave null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this PagedList<T> list, object userData)
        {
            return new JqGridData()
            {
                Page = list.PageIndex,
                Records = list.TotalItemCount,
                Rows = from record in list
                       select record,
                Total = list.PageCount,
                UserData = userData
            };
        }

That’s getting somewhere, but it’s not yet enough to let me write the action method I showed above. For that, I need to wrap up the PagedList inside the action so that I can pass the arguments from the grid directly, without having to construct a new PagedList inside of every action method which supplies data to a grid. So I build on the method above:

        /// <summary>
        /// Converts a queryable expression into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="baseList">The list of records to display in the grid.</param>
        /// <param name="currentPage">A 1-based index indicating which page the grid is about to display.</param>
        /// <param name="rowsPerPage">The maximum number of rows which the grid can display at the moment.</param>
        /// <param name="orderBy">A string expression containing a column name and an optional ASC or
        /// DESC. You can, separate multiple column names as with SQL.</param>
        /// <param name="searchQuery">The value which the user typed into the search box, if any. Can be
        /// null/empty.</param>
        /// <param name="searchColumns">An array of strings containing the names of properties in the
        /// element type of baseList which should be considered when searching the data for searchQuery.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data. Leave
        /// null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this IQueryable<T> baseList,
            int currentPage,
            int rowsPerPage,
            string orderBy,
            string searchQuery,
            IEnumerable<string> searchColumns,
            object userData)
        {
            var filteredList = ListAddSearchQuery(baseList, searchQuery, searchColumns);
            var pagedModel = new PagedList<T>(filteredList.OrderBy(orderBy), currentPage, rowsPerPage);
            return pagedModel.ToJqGridData(userData);
        }

Now you see that string array (IEnumerable, actually) again. JqGrid supports sorting and searching, and so do I. Like Phil, I suggest using Microsoft Dynamic LINQ in order to take the field names that jqGrid passes and turn them into a sorted, filtered, result set. ListAddSearchQuery is a (longish) method which takes a search string and a list of columns to search and turns them into a Dynamic LINQ expression and applies it to the base list. Unfortunately, search is kind of application-specific. In some applications you might want to search different columns in different ways. Some applications might want "Google-style" searching. It kind of depends upon what you’re doing. So although I will supply the source code for this method when I make the demo project available, I cannot promise that it will be useful for your application. Feel free to substitute your own method here.

It turns out both methods (the one which takes a PagedList and the one which takes an IQueryable) are useful. Generally, I use the method which takes an IQueryable. But every once in a while I want to do something to the data I’m about to supply to the grid before the grid sees it, but after paging, like transform a single page of results in some way. In this case, the action builds up a PagedList, uses that to reduce to one page of data, modifies the results in whatever way is appropriate, and then calls the PagedList extension. I’ve also written additional overloads to both methods which do not have a userData argument, since it is very seldomly used. userData, as the name suggests, is an additional bit of data you can return to the grid with your result set for use in grid event handlers when loading a fresh page of data. Since it’s of type object, you can put as much data in there as you need to, so long as it’s serializable to JSON. In practice, though, we rarely use this feature of the grid.

With these methods, it is easy to write actions as concise as the code I showed at the beginning of this post. Just get a list from somewhere and call ToJqGridData, passing information about which page to display, how to sort, etc.

In a future post, I will tie this all together into a demo application.

Share This | Email this page to a friend

jqGrid provides a nice-looking grid with built-in features such as sorting, editing, search, drill-down, tree lists, and more. It is based upon jQuery and integrates well with ASP.NET MVC. The grid is dual-licensed under both the GPL and MIT licenses (just like jQuery), so it can fit well into both commercial and open-source projects.

If you want to display tabular (grid) data with paging in an ASP.NET MVC application, there are essentially three choices:

1. You can write your own grid components using HTML tables, links, manually-implemented paging, etc. This is actually less work than it might seem at first, and the first release of our ASP.NET MVC software did just this. But our own implementation lacked some of the visual flair of some of the third-party components, and avoided solving some of the harder problems, like in-line editing. Still, it was much easier than I expected, and worked well enough for our initial release.
2. You can use a commercial component designed for ASP.NET. However, many commercial grid components don’t work well with MVC. Telerik has MVC demos available for their grid, and Developer Express has said that they intend to support MVC to some degree in the future. Even when the commercial components do support MVC at all, they don’t tend to make all of their features available in MVC applications, owing to the fundamental difference between the way ASP.NET MVC and "plain" ASP.NET work. If you are building a Web application on ASP.NET MVC, I would never recommend buying an ASP.NET component unless the vendor for that component is fully committed to supporting MVC. You cannot presume that any given ASP.NET component will work at all with MVC unless the vendor explicitly supports this.
3. You can use a grid component which has no knowledge of ASP.NET at all. There are quite a number of these, generally built within the framework of one of the popular JavaScript libraries, and it is, ironically, easier to use one of them within ASP.NET MVC application than it is to use a grid component designed for ASP.NET (but not necessarily MVC).

As I said, we ended up choosing jqGrid. We were strongly influenced by the fact that we already use jQuery extensively in our application. There are several grid plug-ins available for jQuery. jqGrid has a number of advantages over the others I’ve seen, however:

• Supports three different models for editing data within the grid: Single-cell editing, in-line row editing, and editing within a pop-up form.
• Unlike some of the other jQuery grid plug-ins, jqGrid has decent documentation and a large number of demos with source code.
• You can display a simple grid, a "sub grid" (drill down), or a treeview.
• AJAX paging is supported in a number of different formats, including JSON, XML, or JavaScript objects via a custom function.
• The grid is updated very regularly, and a new version, with support for the jQuery UI theming library is under active development right now.
• Tracing through the grid source code is quite easy to do with Firebug, which is useful for figuring out things which aren’t quite covered by the documentation.

The grid also has a few downsides:

• The API is a bit of a mess. Capitalization (significant, in JavaScript) is so varied that it seems almost random. Portions of the code were clearly written by different people with limited collaboration with the original author of the grid. There is no consistent naming convention, and similar functionality is sometimes invoked in wildly different ways. For example, you set the grid height with a custom method, setGridHeight, instead of the jQuery method height. But you reload the grid’s data with a jQuery method, trigger, passing the string "reloadGrid" as an argument. It’s a good thing the grid has decent documentation and includes source code, because you’re going to need them. The API is just not discoverable.
• Although the grid has quite a number of options for editing, it doesn’t necessarily play well with some of the other tools you may want to use in the context of editing, such as jQuery validation. (For starters, the in-line row editing doesn’t actually use an HTML form.) The three different editing modes (single-cell, in-line row editing, and editing in a form) use different source modules, different configuration options, and work differently. Doing editing in a way not explicitly supported by the grid can be challenging.
• Variable names in the source code use a lot of single letter and two letter abbreviations. This is quite common in JavaScript libraries, but it does make the source code a bit challenging to read.
• Although there is both free and paid support available, there is not a commercial company standing behind the grid. So don’t expect round-the-clock support, a toll-free phone number, etc.

After a good deal of experimentation with several grids, we decided to use jqGrid because it has considerably more features than every other grid we examined (when used in MVC application), and because we didn’t find any of its limitations to be show stoppers.

The next post in this series is LINQ Extensions. There, I show an extension method for IQueryable<T> which returns data suitable for the grid, without having to know anything about the type T.

Share This | Email this page to a friend

On the other hand, I have not found in practice that programmers need to be mathematically inclined to become great software developers. Quite the opposite, in fact. This does depend heavily on what kind of code you’re writing, but the vast bulk of code that I’ve seen consists mostly of the "balancing your checkbook" sort of math, nothing remotely like what you’d find in the average college calculus textbook, even.

{
  i = j++ / (x + v);
}

Not exactly the stuff mathletes are made of.

Arithmetic is one kind of math. It is relatively common in elementary math classes, and relatively uncommon in programming. Does Jeff’s example look like the "vast bulk of code you’ve seen?" It certainly doesn’t look like the vast bulk of code I’ve seen.

On the other hand, I do see a good bit of:

• Relational database queries (relational algebra)
• LINQ (functors)
• Composition of lambda expressions (lambda calculus, of course)
• Set operations and logical operators (set theory)
• Etc.

That’s by no means a comprehensive list. For example, every general-purpose programming language I’ve ever used would not exist without the Church-Turing thesis. But I didn’t include that, since most people don’t do programming language design.

All programmers work with mathematical systems day in and day out, whether we recognize them or not. Perhaps the ability to recognize math when we see it will help us better evaluate the importance of understanding it.

Share This | Email this page to a friend

Update: Attendee Bill Melvin wrote a short summary of the presentation.

Share This | Email this page to a friend

1. Decide where you want to go.
2. Aim the car in that direction.
3. Get there.

It’s different when you ride a horse. In that case the process is more like:

1. Decide where you want to go.
2. Communicate your intention to the horse.
3. Arrive at some hybrid of where you want to go and where the horse wants to go.
4. Return to step (1). Repeat.

Now, which of those two would you find more analogous to software architecture? Does that sound like a silly question? The reason that I ask is that I read a paper last night which argues in a persuasive but decidedly unconventional manner that the things you design have, for lack of a better word, intention. In other words, building a cathedral, or a software application is more like riding a horse than driving a car.

The paper is called Designed as Designer, by Richard P. Gabriel. As technical papers go, this one is unusual to say the least. It was written as a response to an address by Fred Brooks at OOPSLA 2007, during his keynote. In the address, Brooks stated:

If we look back then at the 19th century and the things that happened—the cartwright and the textile machinery, Stephenson (the train), Brunel’s bridges and railway, Edison, Ford, the Wright brothers, etc.—these were very largely the designs of single designers or, in the case of the Wright brothers, pairs.

Gabriel disputes this idea, and chooses an unexpected way of making his case. Gabriel is a published author both in the fields of computer science and writing/poetry. The paper draws as much from his experience in the latter field as the former—there is considerably more poetry quoted in the paper than source code. He also manages to weave in, with some relevance, no less, baseball statistics.

The general outline of the paper is as follows: First, there is a running, fictional narrative about a child named Pippo who observes the construction of the cathedral of Firenze. Next, Gabriel addresses Brooks’s point directly. He discusses the relationship between poets T.S. Eliot and Ezra Pound in the evolution of Eliot’s masterpiece The Waste Land. The middle, and longest, section of the paper is an explanation and example of a theory of poetry he has developed based around an almost algorithmic approach to classifying a given text as either poetry or ordinary writing. Contrasting an early version of a published column with a later revision to same, he asserts that the structure of the earlier version leads the poet into the later revision. I should add here that even though I really like poetry, I find his analysis a bit academic for my taste. Returning to the Cathedral of Firenze, Gabriel shows that many people were involved in its design and construction, and that the Cathedral form itself had a bigger impact on the final design than the architect. Then he uses statistical analysis of the career of Roger Maris to show that luck, as much as skill, played a role in Maris’s exceeding Babe Ruth’s home run record.

The final section of the paper includes, almost shockingly at this point, discussion of computer science and one small example of (Lisp) source code. He relates the story of how Gerry Sussman and Guy Steele discovered how two mechanisms they were studying turned out to be exactly the same thing by implementing both of them separately and noticing later that the source code was identical.

If all this sounds a bit, well, crazy, it isn’t. The paper was presented at OOPSLA 2008. I’m not above a bit of schadenfreude, but this is definitely a "laugh with him" rather than "laugh at him" paper. If you have ever had the experience of designing a system, and then uncovering necessary changes in the system when you try to use your design, then you have some idea of what he’s getting at.

I am also reminded, more than a little bit, of Michael Pollan’s book, The Botany of Desire, which calls the evolution of the humans and plants "co-evolutionary," and questions who has domesticated whom. Again, we don’t tend to think of plants as having "intention," but maybe the question should be, "just what is intention, really?" So all in all, it’s a fascinating read.

That said, it seems to me that, to some extent, Brooks and Gabriel may be talking past each other. Surely, Brooks can’t be under the delusion that the Wright brothers invented the airplane in a vacuum? Surely, Gabriel wouldn’t presume that he is? Much of Brooks’s writing focuses on organizational support for creativity. Indeed, Brooks, to me, seems almost to take creativity as a given, more so than examining what it is and what makes it tick. Gabriel seems to be on the other side of this coin, examining the nature of creativity, rather than how to make it work in a large organization.

Both points of view are worth reading, I think.

Share This | Email this page to a friend

http://www.example.com/foo.html

…which is fine, because if you go to such a link, you probably really will get HTML back, to URIs which (often) look something like this:

http://www.example.com/foo.php

…which, as I noted in a previous post, is a lie. Actually, it’s worse than a lie; it’s poor encapsulation. What happens when you come to your senses and stop using PHP? Now you URIs have to change, which means you get to either create redirects for that abomination or break every link to your site. One of the things we’ve learned from the success of Google is that the information encoded in the links between sites is nearly as important as the information on the sites itself.

The web is dynamic. It has to be, because it is too big to compile. And so the same concerns we have when writing code and dynamic languages often apply to creating a web site, whether you use a dynamic language or not. You need to choose very good names, because if you ever need to change them you will either break stuff or have a substantial job on your hands correcting every reference to the name you’ve changed, including references you don’t personally control.

So I’m going to propose a general rule, which states, "Any URI which allows an informed developer to discern the technology used to serve the result is probably broken." The resources you are serving, if they are at all valuable to the user, will probably last longer than the implementation you use to serve them.

Of course, choosing good names is hard. I read a great quote the other day:

There are only two hard things in Computer Science: cache invalidation and naming things. -Phil Karlton

Also, even after you’ve chosen a good URI, you may have to do extra work to make it functional. If routing is not built into the technology stack you’re using, you may have to implement a URL rewriter to make your persistent URIs functional. It’s worth the effort to do so.

Late update: Thank you to commentor Stecki for the link to the following article: Cool URIs don’t change, by Tim Berners-Lee. Well said.

Share This | Email this page to a friend

Mason Wheeler noticed that TObject.InstanceSize returns 8 (bytes). It turns out that this is new in Delphi 2009; in previous releases, TObject.InstanceSize returned 4. But when you look at the definition of TObject in System.pas, you don’t see any fields declared at all.

Four out of the eight bytes are consumed by the VMT; this has been true since the first version of Delphi. You can read more about that in this chapter from Delphi In A Nutshell, which is old, but still accurate insofar as the VMT is concerned.

But what about the remaining four bytes? I didn’t remember the answer, but I enjoy a good puzzle, so I took a closer look at the source code for TObject. While poking around in System.pas, I happened to notice the following constants:

{ Hidden TObject field info }
hfFieldSize          = 4;
hfMonitorOffset      = 0;

OK, that explains the four bytes, but what is this used for? Searching for hfFieldSize yielded the answer:

class function TMonitor.GetFieldAddress(AObject: TObject): PPMonitor;
begin
  Result := PPMonitor(Integer(AObject) + AObject.InstanceSize - hfFieldSize + hfMonitorOffset);
end;

So any object can have a reference to a TMonitor instance, and that reference is as big as a pointer on the target CPU. That reminded me of Allen Bauer’s long series of blog posts last year on the "Delphi Parallel Library," which discuss the TMonitor class in detail. He notes:

The TMonitor class [...] is now tied directly to any TObject instance or derivative (which means any instance of a Delphi class).  [...] For Tiburón [a.k.a. Delphi 2009], you merely need to call System.TMonitor.Enter(<obj>); or System.TMonitor.Exit(<obj>); among the other related methods.

The ability to lock any object is a good feature to have, especially in a multi-CPU, multi-core world. So I’m happy to spend the extra four bytes to get this feature. But I do find the implementation a bit mysterious. Why not just declare an actual field in TObject? I guess one reason would be to reduce the chance of a conflict with existing code. Another possible reason would be to enforce TMonitor’s contract; by obfuscating access to the field, you reduce the chances that an uninformed programmer performs actions on the field which break the contract. But these are just guesses. I don’t know why the Delphi team settled on this implementation. I presume they have some good reason.

Update: In comments, Allen Bauer explains the reasons for this implementation:

The reason for always placing the monitor reference field at the end of the object instance is mainly for backward compatibility. I didn’t want to alter the layout of objects in case there was some code out there that depended upon it. Also, by hiding the implementation the chance of inadvertent mucking with the monitor data was reduced and therefore it increased the overall safety of using it.

If you look closely at what happens on descendant instances, the monitor field is always the last field of the instance. This happens because the compilers (Delphi & C++) ensure that the field is allocated as the last step in the process of laying out the class instance. By doing this, an object laid out in pre-Delphi 2009 will be laid out exactly the same in Delphi 2009 except there is now an extra trailing pointer-sized field.

Share This | Email this page to a friend

Share This | Email this page to a friend

Yes, it is true that ASP.NET MVC does not force you to use one particular ORM. Yes, that means that you’re going to have to expend some thought, and quite possibly even effort, deciding which one is appropriate for you. It will not be sufficient to read a blog post about the Entity Framework from somebody who tried it once and another blog post about NHibernate from somebody who saw a screencast about it and then conclude that there are no workable solutions in the world.

But it’s worth the effort, because you have a big reward in store: Whatever choice you make will work just fine with ASP.NET MVC. Because the framework has no dependencies whatsoever on any ORM, all ORMs work equally well. You get to choose the best tool for your data access needs without worrying that your web framework will reject it.

Share This | Email this page to a friend

Share This | Email this page to a friend

• Stores a list of routes.
• Translates an incoming URI into lists of values and tokens, and passes those lists to the handler specified on the route. In the MVC framework, this feature is used to call the appropriate Controller/Action.
• Translates a list of values into a URI suitable for inclusion within an HTML page. In the MVC framework, this feature is most commonly used in the Html.RouteLink/ActionLink methods you call to generate a link in your Views.

Let’s examine in more detail how MVC uses Routing, and distinguish between the features of Routing and the MVC framework.

Components of a URI

Consider this sample URI:

http://www.example.com:80/SiteRoot/Controller/Action?foo=A&bar=B#anchor

The individual parts of the URI are:

• The scheme (protocol), authority (host), and port ("http://www.example.com:80" in this sample URI). I will mostly ignore this part in the rest of the post, since Routing doesn’t deal with it, except to pass it along when necessary.
• The path ("/SiteRoot/Controller/Action" in this example). Per the URI RFC, this part "serves to identify a resource…" I’ll talk about what that means in a second.
• The query ("?foo=A&bar=B" in this example). My example uses name/value pairs is the query, because that seems to be the most common case. However, the RFC says that the query can be almost anything, and the purpose of the query is to provide a non-hierarchical means of identifying the resource requested by the user.
• The fragment ("#anchor" in this example). I’m going to ignore this, since Routing doesn’t deal with it.
• I’ve omitted some stuff like username/password which is almost never used today.

MapRoute

Now lets examine a simple case of adding a route to the routing system. This is from the default MVC application template:

routes.MapRoute(
    "Default",                                              // Route name
    "{controller}/{action}/{id}",                           // URL with parameters
    new { controller = "Home", action = "Index", id = "" }  // Parameter defaults
);

Note that MapRoute is actually not part of the routing system; it’s an extension method added to RouteCollection by ASP.NET MVC. Calling MapRoute will add a single route to the RouteCollection. Most web applications will have only one RouteCollection, stored in System.Web.Routing.RouteTable.Routes. The most important thing that MapRoute does is to create and pass an MvcRouteHandler instance when adding the route to the RouteCollection. This ensures that when ASP.NET handles an incoming URI which matches this route, control will be passed to the MVC framework for handling the request.

The second argument to MapRoute is named "url", but it’s not really a complete URL. Instead, it is a template for a fragment of a URL containing a portion of the path component of a URI. I say "a portion of the path" because it omits the site root, which I’ve called "/SiteRoot" in the example above. The site root is often empty, but it doesn’t have to be. The URIs produced by the routing system include quite a bit more than the path, however. They also include the scheme/authority (in other words, the protocol and host) and, sometimes, a query.

There are a number of overloads to MapRoute. In addition to the values passed in this simple example, you can also pass constraints (which I’ll discuss in a moment) and namespaces. The namespaces argument is for MVC only (it’s not used by routing) and is useful when you want to put a Controller in a place where MVC wouldn’t otherwise find it.

A route can also contain DataTokens, which are not used by the routing system, but are passed to the route handler. MapRoute doesn’t have an overload which includes these as an argument, and I’m not going to discuss them in this post. ASP.NET MVC uses them only to store the namespaces argument, discussed above.

Parsing a URI

Let’s examine what happens when the routing system is asked to interpret a URI submitted by the user. Importantly, the RouteCollection is ordered. So the routing system will attempt to match the routes in the collection in the order you added them. The first matching route "wins" even if a later route is a "better" match. The incoming URI is compared with the url portion of the route. If it matches, the entire URI, including the query, is tokenized. If there are any constraints on the route, these are considered.

In practice, this means that if you have a lot of routes, then you are almost guaranteed to encounter a bug, sooner or later, where the wrong route is matched to an incoming URI. The only good way to prevent this that I know of is to write unit tests for your routes. Unfortunately, the name of the route is not actually part of the route itself. It is stored in a private, internal dictionary in the RouteCollection. So there is no way to write a unit test which examines the name of the route returned.

Let’s use the sample URI and route from earlier in this post as an example. Out of the entire URI, http://www.example.com:80/SiteRoot/Controller/Action?foo=A&bar=B#anchor, the routing system is only going to examine this bit when processing an incoming URI:

/Controller/Action?foo=A&bar=B#anchor

Does this match the url portion of the route defined above? Recall that it is defined as:

"{controller}/{action}/{id}"

Yes, it matches, because "Controller" can be interpreted as the "{controller}" portion of the url, "Action" for the "{action}" part, and a default value exists for the "{id}" portions (empty string). Note that this means that having default values in a route makes that route much more likely to "match" an incoming URI. If there are defaults for every token in a URL pattern and there are no constraints, then that route will match every incoming URI. The route shown in this post is created by the MVC framework as a "catch all" route for a new application, which contains only that route. That is why it has defaults for all of its values and no constraints; it is intended to match every incoming URI. If, on the other hand, you write a special-purpose route intended only for one particular controller or action, and you provide default values for all of the tokens, then you need to implement constraints to prevent this route for matching other controllers or actions.

Because the url matches, we have five values, so far, to pass as the route data. "controller" will have the value Controller, and "action" will have the value "Action". "id" will have the default value. Also, from the query, "foo" will have the value "A", and "bar" will have the value "B". Now the routing system considers any constraints on the route, but in this case, there aren’t any. Therefore, this list of values is handed to the MvcRouteHandler which was created by MapRoute.

Importantly, note that all of these values are strings. The routing system only deals with URIs, which are always strings. Now, your actions on your controllers can have arguments which are not strings, and the MVC framework will attempt to coerce the strings supplied by the routing system into whatever type is specified in the method signature. But that’s a feature of the MVC framework, not of the routing system. (Yes, the Values property returns type object, but, when matching an incoming URI, the segments are always added as strings.)

Not all URI patterns are as simple as the default route from the standard MVC application template. You can use delimiters other than /, for example. If you have hyphens, parentheses, etc. in your url template, these will be matched against the incoming URI to determine if the route is a match. The following route, for example, would not match the sample URI at the top of the post, even though it contains all of the necessary tokens and has no constraints:

routes.MapRoute(
    "Parenthesized",                                        // Route name
    "{controller}({action})({id})",                         // URL with parameters
    new { id = "" }                                         // Parameter defaults
);

The incoming URI contains no parentheses, so this route can’t match, because the parentheses are needed to to delimit the values for controller and action, and there are no defaults for this values.

Matching a Route With an Action

After a route is matched to an incoming URI, and a list of route values is created, control goes to the MvcRouteHandler. Now the MVC framework can invoke the action on the controller specified by the "action" and "controller" route values. Note that there is nothing special about these names/values to the routing system; they’re just two more values in the list from the routing system’s point of view. But these two names, "action" and "controller" are very important to the MVC framework, for obvious reasons.

MvcRouteHandler (indirectly) calls MvcHandler.ProcessRequest, which constructs the controller using the ControllerFactory, and then calls Controller.Execute, which, in turn, invokes the action using the ControllerActionInvoker. It is at this point that the values in the route are matched against the arguments for the action. In other words, tokenizing the URI and matching those values with action arguments are two entirely separate operations. The tokenizing is done by the routing system, and the matching route values with controller action arguments is done by the MVC framework.

In order to invoke the action, the MVC framework needs to look at the individual arguments to the action, and match them with route values. This is performed by the model binding system.

Note that the name of the argument does not have to correspond to a name in the route values if the model binding system can instantiate an instance of the type of the argument from the values which do exist. For example, if the type of the argument is FormCollection, then this argument can always be passed, no matter what names are present in the route values, and no matter what the name of the argument. The model binding system knows that it can always create a FormCollection by rolling up any values in the form in the incoming POST, if there are any. If there are not, it will simply return an empty FormCollection.

If the action takes an argument which cannot be instantiated based on the data in the route values, that is fine, so long as the argument is nullable. If the action requires a non-nullable argument which is not in the route values, or if the route values contain a value for an argument which cannot be bound into the type of that argument, the MVC framework will throw an exception. Otherwise, the action will be invoked.

Creating a URI With RouteLink

Creating an "a href=" tag with Html.RouteLink is trivial. The routing system looks up the route by name, and populates the URI with values from the RouteCollection you passed. Any values in the RouteCollection not included in the string in the url argument to MapRoute will be added to the resulting link as query string parameters. In my view, you should always prefer RouteLink over ActionLink, whenever there is any ambiguity as to which route will be returned.

Again, let’s consider an example, with the route defined in the MapRoute call earlier in this post:

<%= Html.RouteLink("Link text to display on page", "Default",
    new RouteValueDictionary(new {controller = "Foo", action = "Bar", foo = "A", bar = "B" })) %>

With this example, the order of the routes in the RouteCollection, the constraints, etc., are not needed to find the matching route, because we have specified ("Default") directly. Since there are no tokens called "foo" and "bar" in the url argument we passed to MapRoute, these will be appended to the generated URI as query string parameters. So we end up with the sample URI at the top of this post, less the anchor, provided that the site is configured to live in /SiteRoot.

Creating a URI With ActionLink

ActionLink is more complicated, because the routing system must first find an appropriate route, and then use that route to generate the URI. In a way, the method of finding the route is similar to the method used when parsing a URI, insofar as the first matching route "wins." But in another way, it is very different, because there is no incoming URI to use as a pattern for matching the partial template passed as the url argument to MapRoute. The only thing that the url argument is used for in this scenario is determining which tokens must be included, either directly, or via defaults, in order for a route to match. Let’s re-examine the same scenario considered in the RouteLink section, above, only this time using ActionLink. The call looks like this:

<%= Html.ActionLink("Link text to display on page", "Bar", "Foo",
    new RouteValueDictionary(new {foo = "A", bar = "B" })) %>

Does this match the route defined at the top of this post? Yes it does, and here’s why: That route contains three tokens in its url argument. They are controller, action, and id. I have supplied values for two of these tokens in the call to ActionLink, namely controller and action, via the controller and action arguments in the overload of ActionLink I am using here. The remaining required token, id, is not supplied in the ActionLink call at all. However, there was a default for it (empty string) in the original call to MapRoute, so the route matches even without an explicit value for id. Because foo and bar are present in the RouteValueDictionary, but the url template in the route does not contain {foo} or {bar}, these will be added to the generated URI as query string parameters. Again, the generated URI will be the sample URI at the top of this post, less the anchor, provided that the site is configured to live in /SiteRoot.

Conclusions

I hope this long explanation has taken some of the mystery out of what routing does. Since there’s a lot of detail above, I’m going to summarize some of the information which is important for day-to-day work.

• Routing and the MVC framework are distinct features. MVC knows about Routing, but Routing does not know about MVC. Routing produces a list of values, but does not assign special meaning to them. MVC signs special meaning to the controller and action values.
• Routes include information about a portion of the path in the URI and also the query, since any token not in the path is treated as part of the query.
• The order of routes is very important, both for parsing incoming URIs and for generating URIs with ActionLink.
• Default values in a route will cause the route to match more incoming URIs or sets of values in a RouteValueDictionary. Constraints in a route will cause the route to match fewer incoming URI this or sets of values in a RouteValueDictionary. If there are defaults for every token in a URL pattern and there are no constraints, then that route will match every incoming URI or sets of values in a RouteValueDictionary. For a "catch all" route like the one in the default MVC application, it is fine for all of the tokens to have default values in for there to be no constraints. If you write a special-purpose route for a particular controller or action, however, then you should probably use constraints to prevent it from ever matching any other controller or action.
• If you have more than one route, then you simply must write unit tests to ensure that the correct route is selected for all incoming URIs or values used in your calls to ActionLink.

Share This | Email this page to a friend

Database Workbench supports InterBase and many other DB servers. You can buy the module for one (or more) DB server only if you don’t need all of them.

Share This | Email this page to a friend

I haven’t read this book, but someone in the newsgroups mentioned that it included InterBase. He writes:

"Note that they tested on IB6 — but I have still found most of their advice to be valid."

Share This | Email this page to a friend

When a web application misbehaves, one of the first things you’ll need to do is determine if the problem is in the browser/client (e.g., buggy JavaScript, attempting to use CSS on IE 6, etc.) or in the server application (e.g., returns incorrect HTML, inserts wrong data into database, etc.). In order to make this distinction, it is helpful to use a very simple conception of what a web application does. HTTP is a stateless protocol. Therefore, a web server is expected to always return the same results in response to a certain request, without regard to what has happened beforehand. So we can consider the server as a single function:

The illustration above may appear a bit strange, because of the close proximity of the browser and the database server. In reality, we know that the database server is likely closer to the web server than to the browser. But remember, I am attempting to treat the web server as a pure function here (I’ll explain why in a minute). The state of the database, if there is one, will certainly affect the page returned by the web server. Therefore, it is an input to the function. Similarly, authentication, which can also change the page returned by the web application is most often stored in a cookie. So cookies are also inputs to the server function.

Using this abstraction, it is now very easy to diagnose where the bug exists, by looking at the inputs to the function and its result:

• If the inputs are correct and the result is wrong, then the bug is in the server. But remember that "result" means the HTML returned by the server, not the manner in which the browser renders it. It is possible for correct HTML to be rendered incorrectly, especially if JavaScript or IE 6 are involved.
• If the inputs are incorrect, then the bug is obvious. But remember, inputs can include things like form data which, for some reason, people don’t always tend to examine.
• If the inputs and the result are both correct, but the page still looks wrong in the browser, then the bug is in the browser/client side.

The only circumstance in which this abstraction doesn’t really work is when you’re maintaining state on the client. An example would be when using frames. If you click a link which targets a frame, the result that you see in the browser will be different depending upon what you have done before, because you might have other pages open in other frames. This is an example of why I think that frames are not a good architectural fit with the rest of the web.

Once you’ve determined eactly where the bug lives, you are well on your way to fixing it. You can also describe the bug with more precision. For example, you will say things like "the server returns incorrect HTML (incorrect style on the div with ID ‘foo’) when…" instead of "the panel is on the wrong place in the page when…".

Colophon

It’s not really relevant to the rest of the post, but here is how I made the illustration above. I decided the post would be easier to understand if I included a diagram of the function. So I followed my standard process for using Microsoft Visio, which is something along the lines of the following:

1. Decide that I need a very simple block diagram, and remember that this is something which Visio is supposed to be good at.
2. Attempt to use Visio to produce the diagram.
3. "WTF, I’m not allowed to resize this? I thought that Visio was supposed to be for simple drawings."
4. Decide, "Wow, I’d forgotten how much I really despise Visio."
5. Produce diagram in some other piece of software.
6. Repeat, 6 months later. Some day I’ll learn.

I ended up using Inkscape, which has more than a few rough edges, but which was good enough to produce a simple diagram, which is more than I can personally say for Visio. The drawing includes an MPLed globe image, so I’m releasing the whole drawing under the terms of the MPL.

Share This | Email this page to a friend

Share This | Email this page to a friend

Share This | Email this page to a friend

They told us about the All Access product, which had been officially released that day. One of the most interesting things about this product is that it is installation-free, or so Embarcadero claims. It uses a virtualized registry, etc., so that you can just start the application directly. I’m thinking that this might be useful even outside of the context of All Access. Especially for something like a build machine, where you might want to run it in various different configurations. Sort of like a "lite" VM. It will be interesting to see if Embarcadero uses this technology outside of All Access.

One thing I wonder about right now: Perhaps the biggest differentiator between Delphi Enterprise and Delphi Architect is that with Architect you get Embarcadero ER/Studio. So with the way the SKUs are differentiated right now, All Access is sort of an alternative to Architect. That’s how it seems to me, anyway. I suspect that we will see a different feature mix in Architect in the future.

David expressed a lot of relief at not being part of Borland anymore. He said that since Embarcadero is private, they can talk about upcoming features more freely than when they were part of a publicly held corporation.

David also talked about how Delphi developers who do database work could really benefit from the Embarcadero database tools. He wants to bring the integration much closer. For example, he wants you to be able to step into the Rapid SQL stored procedure debugger directly when debugging through a Delphi application which called a stored procedure in the database. Embarcadero has profilers for Java applications and for database servers; you can see the potential for interaction there, too.

We are looking for a good solution for doing heterogeneous, bidirectional replication between InterBase and SQL Server. We have a working solution already using a SQL Server job and linked servers, but I would prefer a solution that we do not have to develop and maintain. I am aware of CopyTiger, which claims to support both, but doesn’t really seem flexible enough to do what we need. There are a lot of replicators which doesn’t explicitly support InterBase, that support JDBC or ODBC connections, and they could possibly be a solution. David suggested that Embarcadero’s Change Manager product would be a potential solution for us, so we’re going to look into that.

Share This | Email this page to a friend

var emptyInput = $("form input[value='']");

That’s a little hard to read, so note that the quotes around the entire selector are double quotes, but the quotes inside the square brackets are two single quotes. This says, "Give me any input where the value is equal to empty string."

This is broken in jQuery 1.3.1. Try it, and you will see (if you have a debugger attached):

Syntax error, unrecognized expression: value=""]

A fix for the bug has been checked in, so you can get a corrected version in the nightly build. If you prefer to stick with 1.3.1, I’ve found a workaround; do this instead:

var emptyInput = $("form input:not([value])");

This works because empty string is implicitly converted to false in JavaScript.

Update: The bug is fixed in jQuery 1.3.2.

Share This | Email this page to a friend

• HTML, a markup language for displaying rich documents
• HTTP, a stateless protocol for communicating with a web server
• The URI, a short, standardized reference to a network resource

Almost 20 years later, very little has changed, especially from the point of view of the server. To the list above, we have added client-side stuff like JavaScript and CSS, and HTML is now at version 4. But HTTP and the URI are close to unchanged.

And all three of the items in the list above are just text. Not just any text, mind you, but text which is formatted in such a way as to be mostly easy for humans to read.

This means that what web servers do is very simple. They accept text over a text-based protocol and return more text.

Despite this, people occasionally seem to find web servers confusing, in part because we like to lie about what they do. For example, you might see a URI which looks like this:

http://www.example.com/foo.php

This is a lie. If you point a web browser at this URI, you will almost certainly not receive a PHP source file; instead, you will probably get HTML. That the HTML was generated with PHP source code is an implementation detail unsuitable for inclusion in a Uniform Resource Identifier.

Some people might think I’m being pedantic here. How many end users actually care what the extension in the URI is? These are probably the same people who post questions on Stack Overflow wondering why their web application doesn’t work, but not including in the question the only things that the server really cares about, which you can find in the list at the beginning of this post. It doesn’t matter if you generated your HTML using aspx, rhtml, or PHP. It’s still just plain HTML to the browser. When you submit a form, you’re submitting name/value pairs of strings, not "integers" and "dates" and "classes."

That’s why this matters to a programmer: Web applications, as a class, are extremely easy to debug. Tools like Firebug and Fiddler allow you to see the text that is exchanged between the browser and the server, and even pretty it up a little to make it more readable than it already is. But already built-in to your web browser is the single most important tool that you need for debugging a web application; namely, the "View Source" command. Chances are very good that if you look at the HTML, the HTTP, and the URI, you can solve whatever issue you’re having with your web application. But if you can’t, including whichever of these three is relevant to the problem at hand will dramatically increase the chances of getting a solution when you ask someone else to help you out.

Perhaps it’s the fact that we’re used to debugging within an IDE which causes people to look at, for example, aspx source code instead of HTML when a web page doesn’t render the way we’d like, or to debug through model binders instead of looking at a form submission when the input to a controller action is not what we expect. But remember, the browser doesn’t see your aspx or PHP. The browser just sees the HTML you generate.

Perhaps another reason why this tends to confuse people is that certain web frameworks, such as ASP.NET, introduce fictions such as "event handlers" and "postbacks." No such concepts exist within HTTP. Instead, they are (extraordinarily leaky) abstractions built on top of HTTP within a certain framework. But under the hood, there is still just HTTP. If you want to understand the communication between your web application and a browser, look at the HTTP and the HTML.

I’m going to close this post with a recommendation: If you’ve never visited a web site via telnet, go do it right now. It only takes a second, and you’ll learn a lot in the process. This is an easy way to understand at a very fundamental level how web browsers and web servers talk to each other. If you need to debug your web application at a client site and can’t install Fiddler, you can probably use Telnet instead.

This is the first article in a series on the dynamic web.

Share This | Email this page to a friend

public ActionResult UpdateInts(IList<int> ints) {

The MVC framework, in this case, will look at all the inputs on your form, figure out which inputs correspond to which records on the form, find each record individually, and pass the whole list to your action as an IList<T>. Pretty convenient, that.

Note that the format for the name element requires square brackets and a period.

<input type="text" name="products[0].Name" value="Beer" />

There is no way around this required formatting; it is hard-coded into the framework source code. (See DefaultModelBinder.cs.)

Unfortunately, these are special characters in jQuery selectors. As a result, some plug-ins can become unhappy when you use them in an input name. For example, the ClockPick plugin has an option which is useful when you want an image that the user can click on in order to pop up the clock picker instead of clicking on the input itself:

$(".clockpick").clockpick({
    valuefield : 'myfieldname'
});

If the name attribute of your input happens to be myfieldname, this works fine. If, on the other hand, it is products[0].Name, it doesn’t work at all.

The solution is to include an extra set of quotes around the name with the brackets and the dot:

$(".clockpick").clockpick({
    valuefield : "'products[0].Name'"
});

With only one set of quotes, we just have a string containing products[0].Name. With two sets of quotes, the string contains the same thing, except surrounded by quotes. This causes jQuery to not treat the square brackets and the dots as special characters. As the jQuery documentation notes:

Quotes are optional in most cases, but should be used to avoid conflicts when the value contains characters like "]".

Share This | Email this page to a friend

type
  TFoo<TypeParam1; TypeParam2: ISomething> = class

This means that TypeParam2 must support ISomething. In C#, you would use where instead of the colon syntax.

To use (as opposed to declare) a generic type, however, you separate the type parameters with commas, like this:

  Foo := TFoo<integer, TBar>.Create;

Does that seem a little odd? It might to a C# programmer, I guess. But it should look very familiar to a Delphi programmer, because it exactly mirrors the syntax for procedure arguments:

  // declare:
  procedure Foo(ArgA: integer; ArgB: boolean);
  // invoke:
  Foo(A, B);

Indeed, the only reason this is really worth mentioning at all is that if you forget and you use a comma instead of a semicolon when declaring a list of type parameters in a generic type declaration, the message you get from the compiler is not especially helpful.

Share This | Email this page to a friend

Trust me, just leave that little bastard alone.

To understand why, we should review the Windows standards for interacting with radio buttons using the keyboard. If you drop several TRadioButtons onto a group box, then these buttons are treated as a group; although they aren’t quite the same thing as a TRadioGroup, they behave similarly. Namely, if you tab into the group box, then the checked radio button will receive input focus. If you press tab again, you will tab out of the group box. When the checked radio button has focus, pressing the arrow keys will check a different radio button in the group (and un-check all other radio buttons in the group box).

The VCL does not rely on Windows for this behavior; instead, it is implemented in the VCL source code, and you can trace through it.

The reason that this is on my mind is because of a curious bug report I received the other day. A tester reported that tabbing into a radio group on a certain form caused the selection to change (in other words, a different radio button was checked simply by tabbing into the group box). Based upon the explanation above, you can probably see where this is headed, but it took me a long time to figure out. The behavior was easy to reproduce, and I could even see a CN_COMMAND message of BN_CLICKED in the debugger, but it was near the top of the call stack, and I could not figure out where it was coming from.

It seems that if you tab to this particular radio button, and only this particular radio button, and only when you first enter the form, Windows will send BN_CLICKED. But why was I able to tab to a non-selected radio button at all? Shouldn’t focus go to the selected button?

No, not really. Again, Windows doesn’t choose which control is focused when you press tab; the Delphi VCL does. Glossing over a few details, it essentially looks for the next enabled control where TabStop is true. When you click a radio button, in addition to un-checking all of the other radio buttons in the group, the VCL sets their TabStop properties to False.

But only if they weren’t checked in the first place. If a radio button is not already checked, then its TabStop property will not be set false when another radio button in the radio group has its Checked property set true.

So here is what actually happened:

1. A few months ago, a well-meaning developer had done some tab order maintenance on the form in question. He changed the TabStop and TabIndex properties of a number of controls. The changes looked quite innocuous, and even passed code review.
2. Unfortunately, he had set TabStop true on a TRadioButton which was not Checked.
3. Some code in the form configured the radio group by setting the Checked property of a different radio button True. This caused the Checked and TabStop properties of the other radio buttons in the group to be set False, but only if their Checked properties were not true to begin with.
4. Now the tester tabbed into the group box. The first radio button in the tab order was not checked, so a BN_CLICKED message is sent. The tester is surprised that the "wrong" radio button is focused, and, worse, it is checked, as well.

The moral of the story? Just don’t touch the TRadioButton.TabStop property. The VCL is going to manipulate it in code. Leave it at its default value. If you want to change whether or not the user can tab into a group of radio buttons on a TGroupBox, then change the property of the container, instead.

Share This | Email this page to a friend

A first try

So here is a naïve attempt at creating metadata for this sort of data. I’m going to demonstrate this as a database table, but the issues involved are nearly identical when you create similar structures in code.

CREATE TABLE TimeRecords (
  Id INTEGER NOT NULL PRIMARY KEY,
  PersonId INTEGER [...],
  WorkDate DATE,
  StartTime TIME,
  StopTime TIME,
  TotalHours NUMERIC(18, 4));

This might look OK at first glance, but it turns out to be fairly badly broken. Before I discuss why, I should clarify the data types involved.

What isn’t broken

One thing you might notice when looking at this metadata is that I’ve made most of the columns nullable. In my experience, it is very important to allow users to save work in progress, and to only do validations (such as requiring that the Person is filled in) when you actually try to do something important that the time record, such as submit it for payroll. For example, an employee could begin a time record when they "punch in" at the beginning of a shift, and finish the time record when they "punch out" at the end of a shift.

DATE and TIME in this DDL refer to the SQL Standard data types which contain, respectively, a date and a time only. The SQL Standard data type for a date and time in a single column is TIMESTAMP. Not all databases (or programming languages/environments) support such types. In many cases, you will have to do with only TIMESTAMP, or its equivalent, such as TDateTime in Delphi.

Note that TotalHours is NUMERIC, not TIME. It is important not to confuse a time span with a time of day; they are different units. In .NET, there is a TimeSpan type which is appropriate for this.

Problems with the naïve approach

Presuming that the user has filled in all of the required information for a time record, what would we expect to see in a typical row?

If the user is recording only the date and the number of hours worked, it seems pretty obvious: both the WorkDate and the TotalHours fields will be non-NULL, and the StartTime and StopTime columns will be NULL.

What if the user needs to record the start and stop times? Well, the WorkDate, StartTime and StopTime fields almost be non-NULL. But what about TotalHours? We could fill that in, but wouldn’t it be redundant with the information in StartTime and StopTime? Or, worse, could we even be sure that all three fields are kept "in sync?" So we might be tempted to leave that NULL, but then, what if we want to do a SUM in SQL? By filling in TotalHours, we can compute an employee’s total work time, even if some of the records are filled in with start times and stop times, but others were filled in with only the total hours worked.

To compute the total hours on a record that has start and stop times, we might be tempted to subtract the start time from the stop time. But that doesn’t work. An employee could work a shift from 8 PM to 2 AM, yielding a negative result for this calculation.

We could make a rule that we add a day to the stop time before subtracting if it is less than the start time, but this presumes that the employee’s shift was less than 24 hours. Greater than 24 hour shifts may be uncommon, but they do happen; I once worked a 31 hour shift, fresh out of college.

What this means is that recording only the start and stop times without the start and stop dates does not give us sufficient information to compute the total hours worked in all cases, and makes it more difficult in many cases.

Some useful redundancy

Here’s another way to store this data. This is what I generally do. But I’ll forgive you if you’re skeptical when you see it. I’ll discuss some alternatives later.

CREATE TABLE TimeRecords (
  Id INTEGER NOT NULL PRIMARY KEY,
  PersonId INTEGER [...],
  WorkDate DATE,
  StartTime TIMESTAMP,
  StopTime TIMESTAMP,
  TotalHours NUMERIC(18, 4));

First, it is no longer a problem to compute TotalHours from the StopTime and StartTime, if they’re filled in. StopTime minus StartTime will always equal the correct total hours worked, when they are non-NULL. Also, it is no longer a problem to record a greater than 24 hour shift.

But isn’t there some redundancy here? WorkDate is always going to equal the date portion of StartTime, when StartTime is non-NULL. Somebody once suggested to me that it would be possible to just use the date portion of the StartTime TIMESTAMP for a "TotalHours-only"time record (and drop the WorkDate column altogether). The problem with this idea is that a TIMESTAMP always contains time information, whether or not you bother to fill in. If you assign only a date, then the time portion is presumed to be midnight. So there is no way to distinguish between a record where the user intended to fill in only total hours and one where the user intentionally filled in midnight as the StartTime but forgot to fill in the StopTime. That will be a problem when you validate the record prior to submitting it for payroll.

Another way

I found that the approach above works well for me, but those who find the duplication of one date completely untenable might be tempted to do something along the lines of this metadata:

CREATE TABLE TimeRecords (
  Id INTEGER NOT NULL PRIMARY KEY,
  PersonId INTEGER [...],
  WorkDate DATE,
  StartTime TIME,
  StopTime TIMESTAMP,
  TotalHours NUMERIC(18, 4));

There’s no redundancy in this version. You can store both time records with start times and stop times or time records with just elapsed times, and it’s easy to distinguish the two. Unfortunately, certain queries become more difficult. With the version which uses TIMESTAMP for the StartTime column, I can write a query like this to determine who was in the building at a particular point in time:

SELECT
  PersonId
FROM
  TimeRecords
WHERE
  :SomeParameter BETWEEN StartTime and StopTime

Records without start and stop times will be ignored by this query, since they do not contain enough information to answer the question. With a TIME instead of a TIMESTAMP data type for the StartTime column, I would have to use an expression in the where clause to write this query:

SELECT
  PersonId
FROM
  TimeRecords
WHERE
  :SomeParameter BETWEEN (WorkDate + StartTime) and StopTime

That might well mean that my query can no longer be indexed.

Conclusion

Although my preferred schema has a small bit of redundancy, I find it significantly more usable.It is considerably less work to handle copying the date between the WorkDate and StartTime fields, and computing the TotalHours from the StartTime and StopTime, than it is to not have the date in either column.And not having dates in the StartTime and StopTime columns at all is just unusable.

Share This | Email this page to a friend

Many people seem to be confused about the TempData feature in ASP.NET MVC. TempData behaves like the ViewData dictionary, except that it persists until the next request from the same browser.

Because of this, TempData should only be used when you know exactly what the next request is going to be. If you set TempData and your action then returns a ViewResult, then the next request, whatever it happens to be (an AJAX request, another page the user opened in a different tab, etc.), is going to see the TempData value you set, and no other request will see it. I’m guessing that this is never the behavior you want.

Realistically, this means that you should only use TempData immediately before you redirect (in other words, immediately before you return a RedirectResult), because this is the only time when you are sure what the next request from that browser is going to be. I actually think that RedirectData would be a better name for this property. Someone, perhaps Steve McConnell, once noted that you should never include "temp" in a variable name, since variables are, by definition, temporary. "Temp data," to me, is synonymous with "variable." The (functionally similar) "flash" feature in Rails is even more badly named.

If your controller action returns a ViewResult, and you are tempted to put data into TempData, don’t. Use ViewData, instead, in this case.

This may lead to a situation where you might need to have a view look into both ViewData and TempData for a key. One example would be if you have an area in your Site.Master to display error messages on the page. It is likely that one controller action will need to send a message to the view via ViewData (because that action returns a ViewResult), while another action will need to send a message to the view via TempData (because that action returns a RedirectResult, as part of a POST/redirect/GET cycle). In this case, you can use code like this in the Site.Master:

<%= ViewData["Message"] ?? TempData["Message"] %>

Under the hood, TempData works by using server-side Session state, so a distributed session cache like Velocity can be used if your application runs on a server farm. Since the lifetime of data put into TempData is limited to the next request, it is fairly easy to use without worrying about growing the session very large, especially since the most common use for TempData is to store a short error string. But the usual rules on what you can put into a Session apply. If you set the Session state mode to out of process, then the data you put in it must be serializable.

Share This | Email this page to a friend