Here, for example, is how we check the current SQL Server version:

        internal class SqlVersionInfo
        {
            public string Edition { get; set; }
            public string ProductLevel { get; set; }
            public string ProductVersion { get; set; }
        }

        private static SqlVersionInfo GetSqlServerVersion(ObjectContext context)
        {
            const string sql =
             @"SELECT SERVERPROPERTY('productversion') as [ProductVersion],
                      SERVERPROPERTY('productlevel') as [ProductLevel],
                      SERVERPROPERTY('edition') as [Edition];";
            var reader = context.ExecuteStoreQuery<SqlVersionInfo>(sql);
            return reader.Single();
        }

Note that no mapping step is required — the code will just run with any ObjectContext at all.

Share This | Email this page to a friend

A column has been specified more than once in the order by list. Columns in the order by list must be unique.

Seems clear enough, except that

1. There was no duplication in the ORDER BY. Our DBA discovered that the "problem" reference was [Project2].[C5] ASC, which is an alias, not a real DB column name. It certainly appeared only once, but removing it made the error go away. There is documentation which implies that this was an intentional change in SQL Server 2005, but:
2. The same query worked fine on other SQL Server 2005 servers. The query work correctly on a local QA environment. But it failed on the customer’s database.
3. We didn’t write the query; it was produced by the Entity Framework, and my experience tells me that the SQL Server provider for EF doesn’t tend to produce un-compilable SQL.

In order to isolate the problem, I ran the LINQ query in LINQPad. This allowed me to reproduce the problem outside the context of the application (useful, since the test case took a couple of minutes, whereas executing the query in LINQPad directly is almost instant), and also allowed me to switch DB servers easily.

I found a workaround — I could change the LINQ and produce SQL which would work on both servers. But I didn’t like this "fix" because I still didn’t understand the problem. I don’t want to "fix" one thing but cause another problem for some other customer. So I dug a little deeper.

Googling turned up information about the actual problem this error message is supposed to explain, but that wasn’t the case with our query. I did learn that we are not the first group to hit this problem, and that some people "fix" it by setting the database compatibility level to SQL Server 2000. But I don’t like this "fix" at all. Changing the database compatibility level would mean that I would also have to change the ProviderManifestToken in the application’s EDM ask, causing the Entity Framework to emit SQL Server 2000-compatible SQL. That would make the entire application less efficient. So I kept looking.

This Stack Overflow answer didn’t explain everything, but provided an important clue. We checked the server versions on both our local QA environment and the customer system. The customer server was running SQL Server 2005, SP 3. Our local QA environment was running SQL Server 2005, SP 4. Further experimentation showed that this was, in fact, the problem.

It’s generally true that "SELECT isn’t broken." But when you’ve ruled out every other possibility, it’s sometimes worth checking!

Share This | Email this page to a friend

When a to-do list item is marked as complete, the CompletedOn date time property shall be set to the current time.

That doesn’t sound so hard to implement, but how can I test it? I don’t know precisely what the value of the CompletedOn property "should be" since I don’t know precisely when the Complete setter will run.

Method #1: Fuzzy Logic

One option would be to see if the value is "close enough":

var todo = new TodoItem("Test item");

var testStart = DateTime.Now;
todo.Complete = true;
var testStop = DateTime.Now;

Assert.IsTrue(todo.CompletedOn >= testStart && todo.CompletedOn <= testStop,
    "CompletedOn not set to correct timestamp.");

There’s a lot to dislike about this technique. The test could fail if it runs when the system clock moves backwards, e.g., when daylight savings time ends. It’s imprecise, and clearly won’t work in cases where we need a total order across multiple timestamps.

On the other hand, it’s good enough to start writing an implementation, and with that in hand we can consider a better test.

Implementing TodoItem

Here’s a first pass at an implementation:

public class TodoItem
{
    public DateTime? CompletedOn { get; private set; }

    private bool _complete;
    public bool Complete
    {
        get { return _complete; }
        set
        {
             _complete = value;
             CompletedOn = value ? DateTime.Now ? null;
        }
    }
    // ... rest of class

Method #2: Ambient Context

In his excellent book, Dependency Injection in .NET (I’ll write a full review someday, but the book was just updated and I’m still reading that), Mark Seeman suggests replacing the call to DateTime.Now in the implementation with a separate singleton which I can control. The implementation would change to:

public class TodoItem
{
    public DateTime? CompletedOn { get; private set; }

    private bool _complete;
    public bool Complete
    {
        get { return _complete; }
        set
        {
             _complete = value;
             CompletedOn = value ? TimeProvider.Current.Now ? null;
        }
    }
    // ... rest of class

…and the test becomes:

var todo = new TodoItem("Test item");
var expectedTime = new DateTime(2011, 01, 01, 1, 2, 3);
TimeProvider.Current = new OneTimeOnlyProvider { Now = expectedTime } ;

todo.Complete = true;

Assert.AreEqual(expectedTime, todo.CompletedOn, "CompletedOn not set to correct timestamp.");

The test is quite a bit better than the first attempt. I’m now testing for a specific value.

On the other hand, the readability of the code suffers somewhat. Most .NET programmers immediately know what DateTime.Now means, whereas with TimeProvider.Current.Now they have to guess. Also, developers on the team must learn to use a "non-standard" method of obtaining the current time.

Should legacy code, which might not even require a specific value of DateTime.Now for its tests be updated to use this "new" method, just for the sake of consistency? What if that changes the performance characteristics of the code? It’s hard to dismiss this concern as "premature optimization" when we don’t want to change the functionality of the legacy code. This leaves me with the choice of a possibly (slightly) risky change to existing code or inconsistency in the code base.

At any rate, this seems like a good solution, but it’s not the only one.

Method #3: Moles

It’s pretty common to mock or stub methods not directly related to the system under test. Most mocking libraries don’t support this for static methods, but Moles, from Microsoft research, does.

Moles is a kind of monkeypatching for .NET; you can replace just about any method at runtime.

With Moles, we stick with the original implementation of the system under test and instead change the test:

var todo = new TodoItem("Test item");
var expectedTime = new DateTime(2011, 01, 01, 1, 2, 3);
MDateTime.NowGet = () => { return expectedTime; }; // this replaces all calls to DateTime.Now

todo.Complete = true;

Assert.AreEqual(expectedTime, todo.CompletedOn, "CompletedOn not set to correct timestamp.");

Again, it’s a better test than method #1.

Is it better than #2? Well, it’s certainly different. Moles replaces all calls to System.DateTime.Now in the code under test, whether or not I’ve intended this. In my sample code, there’s only one. But in the real world the code under test is generally far more complicated.

Interestingly, the technique in use here — replacing a standard method with one we supply — is the same for Ambient Context and Moles. However the place where we make this redirection is quite different. Although the new method is defined in the test project in both cases, the actual rewiring is done in the test with Moles, whereas with Ambient Context it’s a source code change.

Moles is clearly useful in cases where you need to change a dependency of code which is not your own. Imagine trying to test non-MVC ASP.NET code which depends on ASP.NET methods which internally refer to HttpContext.Current. Indeed, this was painful even in early versions of MVC, although not so much in MVC 3.

But that’s not the case with this code: I have a direct dependency on DateTime.Now, not an indirect dependency. So in this case I’m not forced to use something like Moles.

Moles is pre-release (though it is currently being "productized"), un-supported, and closed source. That, by itself, will disqualify its use from many projects. On the other hand, it’s required for Pex, which is useful in its own right.

Method #4: Constructor Injection

We could of course use normal DI for the time feature

public class TodoItem
{
    public DateTime? CompletedOn { get; private set; }

    private ITimeProvider _timeProvider;

    private bool _complete;
    public bool Complete
    {
        get { return _complete; }
        set
        {
             _complete = value;
             CompletedOn = value ? _timeProvider.Now ? null;
        }
    }
    // ... rest of class

…and the test becomes:

var expectedTime = new DateTime(2011, 01, 01, 1, 2, 3);
var todo = new TodoItem("Test item", new OneTimeOnlyProvider { Now = expectedTime });

todo.Complete = true;

Assert.AreEqual(expectedTime, todo.CompletedOn, "CompletedOn not set to correct timestamp.");

The test is simple and the code is clean. For this specific case it’s arguably the best option.

However, if a significant fraction of all types start requiring a time provider constructor argument, especially if it’s mostly a transient reference to pass along to another type’s constructor, then it’s not a good choice.

Conclusions?

The "fuzzy logic" method is better than no test at all, but that’s about the best I can say for it. Moles can do things which no other method can, but that’s not required for this case. The Ambient Context method works fine here, but raises questions about how to apply it consistently. I can’t tell you which method is best to use; I’m still trying to work that out for myself! Your comments would be welcome.

Thanks to Mark Seeman for answering my questions on Twitter.

Share This | Email this page to a friend

If you click on these, you see… nothing. A description would be nice. ("Application Request Routing? What’s that? EC-CUBE?")

But that’s not really the problem. The bigger problem is this: A "spotlighted installers" feature probably sounded great on the drawing board, but this tool is intended for public-facing web servers. It isn’t the App Store; public-facing web frameworks should not be impulse buys.

What should go here? I’m not certain. My first thought is security updates, but those should already come through Microsoft Update. Perhaps a link there, or a display of what is already installed, and whether it’s up to date?

Share This | Email this page to a friend

For example, I recently ordered a copy of Programming Language Pragmatics, by Michael L. Scott. It’s $63 new for the hardcover or $43 on a Kindle. I got a used copy of the (somewhat older) second edition for $3 + postage, for a total of $7. True, I don’t get the new chapter on VMs, but I can live with that. The third edition totally dried up the market for the second, so there are really good deals available!

The book itself is good so far, though I haven’t read enough to give a full review. It covers substantially similar territory as the dragon book, except more scope talking about programming languages and computer architecture and less depth talking about compiler internals. They’re both worth reading.

Share This | Email this page to a friend

has_secure_password: Dead-simple BCrypt-based passwords. Now there’s no excuse not to roll your own authentication scheme.

I will briefly provide an excuse.

"Simple BCrypt-based passwords" is a reasonable feature, but shouldn’t be mistaken for end-to-end authentication, or even a substantial subset of that problem. Web site authentication in the real world is a far harder problem than salting and hashing a password — which BCrypt does OK, as far as I know. You can prove this to yourself merely by observing that many frameworks which have correctly implemented salting and hashing have nonetheless had their authentication systems compromised by other means.

Having (correctly) validated a username and password, most web authentication frameworks use an encrypted authentication token stored in a cookie (or some other place) for future requests. This way the client (the browser) doesn’t need to remember the password or repeatedly prompt the user for it. However, once the token has been issued, having a copy of it is as good as having the password (for a short period of time, anyway — they typically expire). That’s how tools like Firesheep work on un-secured networks. If you produce or handle this token incorrectly then may as well not bother doing username and password authentication in the first place!

Remember last year when the ASP.NET membership provider used in large numbers of ASP.NET sites was discovered to be vulnerable to a padding oracle attack? ASP.NET, as far as anyone knows, is salting and hashing its passwords correctly. But that’s not enough to stop people from breaking into the system. The other links in the "security chain" have to be secure as well. In the case of ASP.NET, the server leaked information which was useful when attempting to break the encryption of the authentication token. This is sometimes called a side channel. Having succeeded in breaking the encryption, a client could then create a "fake" authentication token which the server would mistake for one it had issued itself. Hence, it was possible to break into a site without ever knowing the password. The authors of this exploit had formerly found similar vulnerabilities in JSF and other frameworks.

All of this adds up to old news: Security is hard. Even experts in the field make mistakes that other experts in the field overlook for years. Anyone can build a system which they themselves can’t break into. The best solution for developers of sites not specifically intended to prototype new and potentially vulnerable security systems is to use proven, off-the-shelf authentication systems, and to instead put their efforts into delivering value to their end users!

Share This | Email this page to a friend

public void Execute()
{
    DoStuff();        // breakpoint 1
}

public IEnumerable<Coordinate> DoStuff()
{
    LaunchMissiles(); // breakpoint 2
    // rest of method here
}

Note that the result of the function DoStuff is not used by Execute. That result actually exists only for testing purposes; it’s essentially a log we use to monitor changes the method makes to external state. The unit tests in question passed, so it was clear that DoStuff worked correctly, at least in a test context. The problem was that when the code ran outside of a test context (i.e., in the real application), the DoStuff method would never run. The debugger would stop at breakpoint 1, but not at breakpoint 2, but only in the "real" application. Similarly, attempting to step into DoStuff would not actually go into the method body. If we debugged the unit tests, the debugger would stop at both breakpoints, and the method worked.

Can you spot the bug?

Perhaps it would help if I showed more of the method:

public IEnumerable<Coordinate> DoStuff()
{
    LaunchMissiles(); // breakpoint 2
    yield return CurrentCoordinates();
}

Now do you see the bug? Remember, the unit tests pass. There is no special knowledge about our application needed to see the problem here; all of the information required to spot the bug is in the code snippets above. The problem is a code bug, not a setup or configuration issue.

Perhaps it would help if I showed you a version of DoStuff which "works."

public IEnumerable<Coordinate> DoStuff()
{
    LaunchMissiles(); // breakpoint 2
    return new List<Coordinate> { CurrentCoordinates() };
}

With this version, both the unit tests and the "real" application work correctly.

The Solution

At first glance, this might seem puzzling. I’ve changed only the last line, and both of those versions appear to do almost exactly the same thing. Why is the behavior of the breakpoint at the previous line different?

The answer is that using yield return causes the C# compiler to change the entire method, not just that single line. It surrounds the code with a state machine containing the rest of the method body. Importantly, the iterator returned from the "yield return" method is entirely lazy; it will not run the method body at all until you attempt to iterate the result of the method. But Execute ignores this result, so the method never runs at all.

Discussion

Some languages, like Haskell, go to great lengths to segregate expressions and side effects. C# isn’t one of them, but even so it’s common to try to improve quality by doing so. Eric Lippert, a member of the C# compiler team, once wrote:

I am philosophically opposed to providing [an IEnumerable<T>.ForEach() method], for two reasons.

The first reason is that doing so violates the functional programming principles that all the other sequence operators are based upon. Clearly the sole purpose of a call to this method is to cause side effects.

The purpose of an expression is to compute a value, not to cause a side effect. The purpose of a statement is to cause a side effect.

It is clear that causing side effects could cause an expression to change in mid-computation. This is problematic for debugging and quality, especially if some of the evaluations are lazy. But as this example demonstrates, the opposite is also true: Adding expressions to a computation can change the side effects, too.

Share This | Email this page to a friend

In my last post, I discussed some of the problems with existing grid components for ASP.NET MVC. Actually, there are a couple more design issues which I forgot to mention in that post. I’ll discuss them briefly before talking about View design.

• Many grids require two requests in order to display the first page of data: One for the page itself, then a second, AJAX request for the data. I can understand why the AJAX request is necessary for the second page of grid data, but shouldn’t the first page just come already populated? Making too many requests is one of the most common causes of performance problems in web applications.
• Many grids require executing a JavaScript method in order to set up the component. Unfortunately, they unnecessarily wait until the jQuery.ready event fires, resulting in unnecessary visual disruption and poor performances the page loads. It’s often unnecessary to wait for the ready event, so you can visibly improve the perceived performance of the page by simply calling that method sooner!
• The two problems above compound each other. If you wait until the ready event fires to even issue a request for data, and then wait some more for that data to return, that can be a lot of waiting!

Displaying Models in MVC 2

In ASP.NET MVC 1, you had to write specific markup in each view, in order to display each individual property of the view model correctly.

In ASP.NET MVC 2, there is a standard way to render the entire model for your page, no matter what kind of data you plan to show:

<%: Html.DisplayForModel() %>

When you do this, MVC will look up the correct template for your model, or its sub-properties, based on its type, and render it appropriately. You can easily customize this with user controls, at any level of your application. You can read about the details on Brad Wilson’s blog.

(Unfortunately, MVC 2’s VS tooling does not use this pattern when it generates new views. I believe this is fixed in MVC 3.)

If you want to render only a certain property of the view model, rather than the entire model (perhaps you have other properties on the view model which control page layout, and are not intended for the final, rendered page itself) then you use:

<%: Html.DisplayFor(m => m.MyCoolGrid) %>

Only if you want to render a control with customizations which cannot be inferred from the metadata on your view model should it be necessary to specify a specific control:

<%: Html.Grid(Model.MyCoolGrid, new { @class = "importantGrid" } ) %>

A Better Grid View API

Oddly, the demo code for most grids supporting MVC seems to favor the third form over the first two. To me, the third form is the exception, and the first two should be the standard methods for most pages. So the "ideal grid API" for views is really no API at all; you simply use the built-in methods which are already favored by MVC for every other kind of model.

Namespace Issues

In a non-trivial MVC application, it is very easy to go overboard with extensions to HtmlHelper. Some components, like Telerik’s suite for ASP.NET MVC, fight this by adding a single extension method, and then putting their components into this "namespace":

<%: Html.Telerik().Grid(Model) %>

I think that this attempt to fight clutter is admirably intentioned, but probably unnecessary. If you’re only going to use Telerik’s controls on a single page, then you can easily Import that namespace on just that one page, instead of specifying it in web.config. If, on the other hand, you intend to use them in most pages in your application, then the extra .Telerik() call in the code above is just noise. Yes, existing code might have a Grid extension method for HtmlHelper, but it is unlikely to specify Telerik’s model as its first argument, so it should not actually conflict.

So, in my initial design, I’ve opted to not do this. However, I remain open to arguments as to why I should. There doesn’t seem to be any clear right or wrong.

Custom Options

Of course, sometimes you do need to use the third form, because the model supplied by the controller is not enough to totally specify the custom needs of a particular page. MVC’s convention for this appears to be to supply large numbers of different overloads for each control, representing every possible combination of options imaginable. (There are, for example, 10 different overloads of Html.ActionLink, causing no small amount of confusion to MVC developers.)

Instead of overloads, I decided to use a strongly-typed options object. This requires a few more keystrokes, but the resulting code is much more readable, and, as I noted above, even calling the Grid method directly at all is the exception rather than the rule.

Here, for example, is how you might choose to override default options for one specific grid:

<%: Html.Grid(Model, new GridViewOptions
                         {
                             Multiselect = true,
                             Pager = false
                         } ) %>

This is much easier to understand than:

<%: Html.Grid(Model, 18, false, "Foo!" ) %>

I can also change the rendered grid to something other than the default grid for the project:

<%: Html.Grid(Model, new GridViewOptions
                         {
                             Renderer = new JqGridRenderer()
                         } ) %>

In many cases, this will be the only code change necessary in order to display a different grid, despite the fact that, for example, plain HTML tables and JqGrids work very differently!

What’s Not There

Please note that there is no need to define grid columns, headers, alignment, etc., in the view. This can all be inferred from model metadata!

A Fluent Interface?

Another possible approach, which some people like, is to use a so-called "fluent" interface. Telerik does this for their grid. With this approach, my example above would look like:

<%= Html.Grid(Model).EnableMultiselect().NoPager() %>

Note carefully, however, that I had to change the "safe" <%: syntax introduced in .NET 4 to the old, "unsafe" <%= syntax from .NET 2. The code above only works because the .ToString() method on the returned type is overridden to return markup — as a string, not as an MvcHtmlString! The <%: syntax handles encoding strings automatically, and, in my opinion, the <%= syntax should not be used at all in .NET 4. A "beautiful" API which requires using features which make your site vulnerable to XSS attacks is not a step forward, in my opinion!

I can’t think of a way to make the "fluent" approach work with the .NET 4 syntax. The Grid method is going to return HTML markup, so it should not be encoded. However, the "fluent" interface requires returning a type containing the fluent methods, which will not be MvcHtmlString. So a call like this:

<%: Html.Grid(Model).EnableMultiselect().NoPager() %>

…would return encoded output (e.g., "<table>"); not what we want! And this:

<%: Html.Grid(Model).EnableMultiselect().NoPager().ToHtmlString() %>

…is just ugly! If the user forgot the .ToHtmlString() bit, they would get no error, but get encoded output at runtime. Yuck!

I don’t want to close the door to a "fluent" interface permanently, since some people like them, and those who do not like them would not need to use it, but I’m not going to write it unless I can find a way to do it well, and I haven’t been impressed by what I’ve seen in other projects. If you’re reading this and you’re aware of a technique for overcoming the issues I’ve described here, please enlighten me!

The Part with the JavaScript

Many grids require calling a JavaScript method, often with a big JavaScript object specifying the configuration of the grid. Now, one of the original tenets of ASP.NET MVC, from day one, was "control over every angle bracket in your page."

Unfortunately, this dictum has not always been followed when it comes to the generated JavaScript. MVC 2’s validation support, for example, injects its script into the middle of your page, and you cannot change this. (MVC 3, however, fixes this.)

I have seen commercial grid components for MVC which inject a giant $.ready event handler right into the middle of the rendered markup! I’m not going to name names, but before you pay for a commercial grid, look at the markup on their demo site! In fairness, there are also commercial grid components which handle this better.

Anyway, in addition to an HtmlHelper extension for the HTML markup, we also need an extension for including the necessary JavaScript (if any; a "plain HTML table" grid renderer might not require any JavaScript at all) references in the rendered page:

<%: Html.GridScriptTag() %>

What this method really means is, "If any of the grids I have included in the page thus far require JavaScript code, please put that script right here." Because this method has a fairly low overhead and emits no markup if you haven’t actually used any grids on the page, you can put it in your Site.Master, if you prefer. Importantly, however, you can also put it somewhere else. This allows you to optimize the point at which the JavaScript method is invoked, should you need to do that.

This method has an overload which takes a strongly-typed options object. You won’t typically need to use this, but you might have JavaScript which you would like to include if and only if there is at least one grid present on the page:

<%: Html.GridScriptTag(new ViewOptions
                           {
                               AfterGridInitializationJavaScript = SomeScript
                           }) %>

"Unobtrusive" Scripting?

MVC 3 has an "unobtrusive" client validation option. Would this even make sense for a grid? I haven’t figured this out yet, but I’m thinking about it.

How Am I Doing?

I’m afraid that’s quite a lot of discussion for what is, in the end, design concerns over the signatures of two methods. If you’re still reading this, thanks!

At the same time, however, I really want to get this right. The controller and view APIs of this library are the parts which I imagine developers will have to grapple with day in and day out. If the internals of the library aren’t pretty in every corner, well, nothing is perfect. But the controller and view APIs are extremely visible, so they deserve a lot of scrutiny!

What’s Next?

In the future, I’d like to similarly examine controller design. But first I have some questions about testing this stuff; it’s tricky! Oh, yes, and I’ll be publishing the code!

Share This | Email this page to a friend

The obvious answer is, "I don’t know. I’m just getting started. Does it really matter, right now?" Don’t you wish!

There are many grids available for ASP.NET MVC. If you’re prepared to dedicate your project to a single grid at the outset of your project, and never change it, nor support alternate platforms, like mobile, then you can (almost) be a happy developer. But if you think you might want to support mobile devices, tablets, and desktop browsers with the same application, if you acknowledge the possibility that you might want to change your mind about which grid you will use in the future, or if you care about separation of concerns, then you may have a problem.

Most grids don’t support ASP.NET MVC very well. In particular, they often:

• Push presentation concerns into the controller. In my opinion, the specific grid you choose in the manner in which it is rendered (column headers, search features, paging, etc.) is a presentation concern, and belongs in the view portion of an application with MVC architecture. Controllers should be grid-agnostic, both in the datatypes they use and in the way you structure your actions.
• Render JavaScript inline. For best page rendering performance, JavaScript should be included at the very end of the body tag.
• Don’t support DataAnnotations and other features related to MVC 2’s templates. If I have a view model which is marked up with DataAnnotations attributes like [DataType(DataType.Date)], then I should not have to do anything further in order to get a grid to display correctly.
• Require too much code in too many places to get a decent grid on the page.
• Don’t support ASP.NET MVC at all. SlickGrid, for example, is a fairly popular JavaScript-based grid which suffers from being minimally documented and does not ship with an MVC interface. This is understandable, because writing a rich integration with ASP.NET MVC is a fair amount of work! What if there were an easier way…?

A while back, I published some simple examples of how to integrate jqGrid with ASP.NET MVC. I’ve used this general technique in real-world projects, but the lack of support for DataAnnotations/templated views in my code was becoming a maintenance issue; the technique I demonstrated there required writing too much JavaScript. I decided to go back and add support for MVC 2 templated views and generate the JavaScript. As I did so, however, I quickly realized that there was very little jqGrid-specific code in my project. I partitioned this off into its own namespace and added a feature allowing the user to specify a grid renderer at runtime. Mixing in fixes for all of the issues above, I now have the skeleton of a generic grid interface in place.

To be clear, I am not writing a new grid. Instead, I’ve written an interface which supports multiple grids, including jqGrid and even plain HTML tables, without requiring special, grid-specific code strewn throughout your application.

I’m releasing this as open source. Actually, it’s already out, but it’s not quite public yet. I have a few more pieces I need to put in place first, like a demo application.

In the meantime, I’d like to get some feedback on the API. My opinion is that most grids which have an MVC integration at all require the programmer to do far too much work in order to get a decently-formatted grid on the page. I admire the emphasis on API beauty which I see in the Rails community.

So I’ll be "releasing" this a little early. It’s not really ready for production use yet. Although the code is solid, I am still making fairly significant changes to the API. I want this to look right, and I’d appreciate the help of anyone reading this blog.

Or, continue on to the next post in this series, examining the API for views.

Share This | Email this page to a friend

Validating a ZIP Code

As an example, consider a model for an address.

public class EditModel
{
    public Guid Id { get; set; }
    public string Address1 { get; set; }
    public string Address2 { get; set; }
    public string City { get; set; }
    public string State { get; set; }
    public string ZIP { get; set; }
}

Now I would like to add a validation attribute in order to ensure that only valid ZIP codes can be entered. In the US, a valid ZIP code is either five digits, or five digits and then a hyphen and then four digits, e.g. 43082 or 43082-6999. I can use the built-in RegularExpressionAttribute for this:

public class EditModel
{
    public Guid Id { get; set; }
    public string Address1 { get; set; }
    public string Address2 { get; set; }
    public string City { get; set; }
    public string State { get; set; }
    [RegularExpression(@"\d{5}(-\d{4})?", ErrorMessageResourceName = "InvalidZIPCode", ErrorMessageResourceType = typeof(MyApp.Resources))]
    public string ZIP { get; set; }
}

This is enough to give me server-side validation when a model of this type is bound by the MVC framework. If I turn on client-side validation, then JavaScript will be generated to do the same validation on the client, as well.

//...
"ValidationParameters":{"pattern":"\\d{5}(-\\d{4})?"},"ValidationType":"regularExpression"}
// ..

So far, so good.

Don’t Repeat Yourself

If this is the only ZIP code which ever appears in my application, I’m done. But if I have several ZIP codes in the application, I would like to consolidate this code into one place.

Perhaps, in a future release of the application, I will support Canadian postal codes in addition to US ZIP codes. I would like to be able to make this change in one place.

I can create a dedicated ZIP code validation attribute by subtyping the RegularExpressionAttribute.

[AttributeUsageAttribute(AttributeTargets.Property | AttributeTargets.Field | AttributeTargets.Parameter, AllowMultiple = false)]
public sealed class ZipCodeAttribute : RegularExpressionAttribute
{
    public ZipCodeAttribute()
        : base(@"\d{5}(-\d{4})?")
    {
        ErrorMessageResourceName = "InvalidZIPCode";
        ErrorMessageResourceType = typeof(MyApp.Resources);
    }
}

…and update the edit model to use it:

public class EditModel
{
    public Guid Id { get; set; }
    public string Address1 { get; set; }
    public string Address2 { get; set; }
    public string City { get; set; }
    public string State { get; set; }
    [ZipCode]
    public string ZIP { get; set; }
}

Fixing Client-Side Validation

This looks good, and seems to work, but testing will reveal that MVC 2’s client-side validation feature has stopped working, even though server-side validation will continue to work as expected. Examining the rendered JavaScript will show that the expected regular expression has not been injected into the page.

Internally, MVC stores a dictionary of attributes, and their corresponding client validation adapters. When it comes time to render the page, it goes through the list of attributes, and finds corresponding entries in the dictionary. Because of the way that this is implemented, it will not find adapters for subtypes of known attribute types. This seems like a violation of the Liskov Substitution Principle.

We can work around the problem, though. There is an API for adding additional attributes to the internal dictionary. In Global.asax.cs, you can add the following code to the Application_Start event handler:

private void Application_Start(object sender, EventArgs e)
{
    RegisterRoutes(RouteTable.Routes);
    // ...
    DataAnnotationsModelValidatorProvider.RegisterAdapter(typeof(ZipCodeAttribute), typeof(RegularExpressionAttributeAdapter));

The only comprehensive "documentation" for this that I’m aware of is the MVC source code.

After adding this code, client-side validation will start working again.

Share This | Email this page to a friend

        public static MvcHtmlString MyTable(this HtmlHelper helper, MyModel model, IDictionary<string, object> htmlAttributes)
        {
            return MvcHtmlString.Empty;
        }

I’ve added just enough code here to get the prototype to compile. Now let’s write some unit tests. First we need to validate the arguments. I’m using MSTest here:

        [TestMethod]
        [ExpectedException(typeof(ArgumentNullException))]
        public void MyTable_should_throw_when_helper_argument_null()
        {
            var model = new MyModel();

            MyHelpers.MyTable(null, model, null);
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentNullException))]
        public void MyTable_should_throw_when_model_argument_null()
        {
            var hh = new HtmlHelper(null, null);

            MyHelpers.MyTable(hh, null, null);
        }

Note that I have elected to call this method as a regular static method rather than as an extension method. Either one is valid, but since I am unit testing the case where the HtmlHelper argument is null, it makes more sense to call the method under test as a regular static method.

Those tests will fail when run, so I’ll write some code to fix that:

        public static MvcHtmlString MyTable(this HtmlHelper helper, MyModel model, IDictionary<string, object> htmlAttributes)
        {
            if (helper == null)
            {
                throw new ArgumentNullException("helper");
            }
            if (model == null)
            {
                throw new ArgumentNullException("model");
            }
            return MvcHtmlString.Empty;
        }

Now let’s presume that the requirements of the new method specify that the table should get its HTML id from the HttpContext.Items. This is of course a very stupid place to store an HTML id, but I want to demonstrate how to fake the HttpContext, since that is something that people often find difficult. So let’s make a fake HttpContext. I’ll need a lightweight implementation of IViewDataContainer, as well, for reasons which will be clear later on:

        private class FakeHttpContext : HttpContextBase
        {
            private Dictionary<object, object> _items = new Dictionary<object, object>();
            public override IDictionary Items { get { return _items; } }
        }

        private class FakeViewDataContainer : IViewDataContainer
        {
            private ViewDataDictionary _viewData = new ViewDataDictionary();
            public ViewDataDictionary ViewData { get { return _viewData; } set { _viewData = value; } }
        }

I elected to only implement the Items property of HttpContextBase, since that is the only one I’ll need to reference in the implementation of my helper. Were I to reference other HttpContextBase properties which I did not override, I would expect to see a NotImplementedException at runtime.

With these two, small classes, I can write a test for the helper which will execute without throwing any exceptions. Well, except for the fact that the assertion fails:

        [TestMethod]
        public void MyTable_should_render_HTML_table_with_id_from_http_context()
        {
            var vc = new ViewContext();
            vc.HttpContext = new FakeHttpContext();
            vc.HttpContext.Items.Add(MyHelpers.IdKey, "foo");
            var hh = new HtmlHelper(vc, new FakeViewDataContainer());
            var model = new MyModel();

            var result = MyHelpers.MyTable(hh, model, null).ToString();

            Assert.AreEqual("<table id = \"foo\"></table>", result);
        }

I needed the IViewDataContainer because the two argument constructor for HtmlHelper will throw ArgumentNullException if you pass a null for the second argument.

Finally, I can implement the helper to make this unit test pass:

        public static MvcHtmlString MyTable(this HtmlHelper helper, MyModel model, IDictionary<string, object> htmlAttributes)
        {
            if (helper == null)
            {
                throw new ArgumentNullException("helper");
            }
            if (model == null)
            {
                throw new ArgumentNullException("model");
            }
            TagBuilder tagBuilder = new TagBuilder("table");
            tagBuilder.MergeAttributes(htmlAttributes);
            tagBuilder.GenerateId(helper.ViewContext.HttpContext.Items[MyHelpers.IdKey]);

            return MvcHtmlString.Create(tagBuilder.ToString(TagRenderMode.Normal));
        }

Share This | Email this page to a friend

The Html.EnableClientValidation method itself does almost nothing. It just sets a flag indicating that other methods which care about client validation should do their thing. The start of the sequence which concludes with client validation JavaScript being written into the HTML page begins with Html.EndForm. Most people don’t call this directly, instead calling Html.BeginForm inside of a using block, which amounts to the same thing. Html.EndForm, in MVC 2 does two things:

1. Writes a </form> closing tag. No surprise there.
2. Writes client-side validation JavaScript.

Therefore, client-side validation will not work if you manually write a <form> tag rather than directly or indirectly calling Html.EndForm. However, you can look at its source code and do the same thing yourself if you prefer to write your own <form> tag.

If this second action is a little surprising for a method called "EndForm" the manner in which it accomplishes this task is even more surprising: It calls a function on the ViewContext type which knows how to write client validation JavaScript onto the page. Perhaps, like me, you’ve always thought of the ViewContext type as simply an aggregation of properties containing information relevant to the (ahem) context in which the current page is being rendered. Well, in addition to this, ViewContext also contains a method which writes JavaScript for client validation. It also contains a const with the hard-coded template for the final <script> tag.

All in all, my conclusion is that the system used to write JavaScript for client validation in MVC 2 is completely hard-wired to that task and not extensible for other purposes.

Share This | Email this page to a friend

MetadataException: Unable to load the specified metadata resource

The cause of this error is a missing or malformed Entity Framework connection string. In particular, the cause of this error is the metadata parameter of the connection string. Let’s examine this more closely.

Which Config File?

Before you begin troubleshooting, make sure you are looking at the correct file.

Connection strings are loaded from the configuration file for the executing assembly, which may not be the assembly which contains your Entity Framework model. So if you have a data assembly containing an Entity Framework model and a web assembly containing a web application which references the data assembly, then the Entity Framework connection string needs to be in the Web.config file for the web assembly. The data assembly can have its own connection string, if you like (this is convenient, as it will be used by the Entity Framework designer), but it will not be used by the web application at runtime.

Metadata

MSDN documents EF connection strings generally. But the cause of the error above is specifically the metadata parameter. This parameter tells the Entity Framework where to find your EDMX at runtime. When your application is compiled, the EDMX is split into three parts: CSDL, MSL, and SSDL. For the purposes of this post, you don’t need to think about what they mean right now; it’s enough to know that you need all three of them at runtime.

The EDMX can be supplied to the application as embedded resources or files on disk. The default is embedded resources, so we’ll focus on that case.

The metadata parameter for an application with an Entity Framework model called Model.edmx in an assembly called Simple Mvc.Data.dll might look like this:

<connectionStrings>
	<add name="MyEntities" connectionString="metadata=
            res://Simple Mvc.Data.dll/Model.csdl|
            res://Simple Mvc.Data.dll/Model.ssdl|
            res://Simple Mvc.Data.dll/Model.msl;provider= <!-- ... -->

So you can see there is one reference for each of the three parts of the EDMX that we need at runtime. They all work in the same way, so let’s examine just the first more closely. The CSDL reference looks like this:

            res://Simple Mvc.Data.dll/Model.csdl

It specifies three things:

1. We’re loading the CSDL from a resource. That’s the "res://" part.
2. The name of the assembly which contains the resource, "Simple Mvc.Data.dll". If your assembly is strong named, you can specify a strong name, in all its verbose glory, here.
3. The name of the resource itself, "Model.csdl". Do not confuse this with the EDMX or model name. In this case they happen to be the same, except for the extension, but that’s not always true!

Let’s examine #2 and 3 more closely.

Assembly name

It’s really common to omit the assembly name in a connect string, and use * instead, like this:

<connectionStrings>
	<add name="MyEntities" connectionString="metadata=
            res://*/Model.csdl|
            res://*/Model.ssdl|
            res://*/Model.msl;provider= <!-- ... -->

This means that instead of loading the resource from one specific assembly, the EF will scan all loaded assemblies at runtime. Two obvious downsides to this are that it takes a bit of time to do this, and the assembly might not be loaded yet. The second case is one reason you might get the error at the start of this post. So specifying the assembly explicitly is always a good idea, I think.

Resource names

Remember how I said that the resource name isn’t necessarily the same as the model name? Here’s a real-life example of that. I opened one of our production assemblies in Reflector, and found this:

There’s actually a good reason that those resources have such bizarre names, but it’s a digression and not relevant to this post. The point is that after you’re sure the assembly name is right, the next step in solving the error at the top of this post is to double-check the resource names.

A simpler resource name will look like this:

Remember the "simpler" metadata where I used * instead of the assembly name? You can go even simpler. This metadata, perhaps surprisingly, actually works (with caveats):

<connectionStrings>
	<add name="MyEntities" connectionString="metadata=
            res://*/;provider= <!-- ... -->

This is the "throw everything against the wall and see what sticks" approach to a connect string. It will probably fail if your resources don’t happen to have the same name as your model, or if the assembly doesn’t happen to be loaded. But it (in simpler cases, anyway) frees the programmer from having to think about what any of this stuff means, so it’s a popular solution.

Share This | Email this page to a friend

If you produce a tool or component supporting InterBase and have not already received a notice about the Partner DVD, then please contact Anders Ohlsson (firstname.lastname@embarcadero.com) immediately.

Share This | Email this page to a friend

Share This | Email this page to a friend

jqGrid is a JavaScript grid component with many useful features. I have previously explained how to use it with ASP.NET MVC.

jQuery BBQ is a library which helps you manage proper functionality of the web browser’s back and forward buttons by changing the fragment (hash) of the URI. Using my integration means that you can have a jqGrid on a web page and the back and forward buttons will work as expected.

To be specific, when you first load a certain page, the grid will probably be on page 1. You could click the next page button on the grid to go to page 2. Without my integration, clicking the back button would not take you back to page 1. With my integration, it will. Similarly, I support the sort and records per page combo box. You can of course extend this yourself to track other grid features.

The image above shows a demo page I’ve prepared, which you try for yourself. The grid on the left does not use the history plug-in. The grid on the right does. Note that when you page, sort, or change the number of rows shown in the grid with the history, the URI fragment also changes.

I have contributed my plug-in back to the jqGrid project. You can get the source from github.

Share This | Email this page to a friend

public class MyBaseController : Controller
{
    // ...
    public new ISession Session { get; private set; }
    // ....
}

ISession is an interface type I wrote which exposes everything I store in the session at runtime. I use constructor injection to assign this to a simple type which maps to the "real" Session during application runtime, and a "mock" session for unit testing. Session, in ASP.NET, is a user-specific cache.

This hides Controller.Session (of type HttpSessionStateBase) and replaces it with a strongly-typed, mockable replacement. Normally I wouldn’t use public new, but in this specific case it seems like the best way. It stops developers from using the Web-dependent Session property in derived types and forces them to code to the strong-typed interface. But still, it’s public new, which seems yucky.

What do you think?

Share This | Email this page to a friend

If you’ve read Getting Real (you can sample it online for free at the link above, and in my opinion it’s worth the time to do so), or if you’ve seen one of DHH’s presentations, then you probably have a good idea of what to expect in Rework. It’s short, to the point, quotable, and uses 37 Signals as the model for everything that is good about how to run a business. I don’t fault the authors for being proud of their company, but if you are looking for a dispassionate examination of when it makes sense to run a business in the way that they do versus other ways, you won’t find it here. You quickly get the idea that the authors look at other methods of running a business about as fondly as DHH looks at WS-*. Again, this isn’t necessarily a bad thing. It’s clearly worked out pretty well for 37 Signals. One can’t fail to notice, however, that 37 Signals’s products fit within a pretty narrow spectrum. Will their techniques translate to other markets and other industries? Perhaps; perhaps not. Rework doesn’t really answer that question.

Did I say quotable? It’s full of snappy, almost pithy zingers which are so obviously right that they would seem like tautologies if they were not so commonly ignored:

• Workaholics aren’t heroes. They don’t save the day, they just use it up. The real hero is already home because she figured out a faster way to get things done.
• You’re better off with a kick-ass half than a half-assed whole.
• It’s also unfortunate that meetings are typically scheduled like TV shows. You set aside thirty minutes or an hour because that’s how scheduling software works (you’ll never see anyone schedule a seven-minute meeting with Outlook).

I don’t think these are empty words. I’m convinced the authors run their company this way. I’m also convinced it works for them. They took their own advice writing the book, as well:

We cut this book in half between the next-to-last and final drafts. From 57,000 words to about 27,000 words. Trust us, it’s better for it. So start chopping. Getting to great starts by cutting out stuff that’s merely good.

But I don’t agree with everything in the book. Sometimes the cuteness of the writing overtakes the point. It’s generally true that you should focus on the essentials. But they illustrate this with:

For example, if you’re opening a hot dog stand, you could worry about the condiments, the cart, the name, the decoration. But the first thing you should worry about is the hot dog. The hot dogs are the epicenter. Everything else is secondary.

Guys, next time you’re in Columbus, look me up, and I’ll buy you lunch at Dirty Franks. I respect what you have to say about business, but we have a thing or two to settle about hot dogs.

Indeed, later in the book the authors make this very point quite elegantly, with a more apropos example:

When we start designing something, we sketch out ideas with a big, thick Sharpie marker, instead of a ballpoint pen. Why? Pen points are too fine. They’re too high-resolution. They encourage you to worry about things that you shouldn’t worry about yet, like perfecting the shading or whether to use a dotted or dashed line. You end up focusing on things that should still be out of focus.

Avoid false precision.

The point here, after all, is not how to run a hot dog business in particular; it’s how to run your business. The best way to fail if that is to let details and directions steer you away from the ideas that only you can bring to the table. This is why "look inside yourself" can be a better approach to planning and business strategy than following an off-the-shelf formula. For most people, creating a new, free, application framework is the worst possible way to start a business selling web applications. I’ve seen many, many people fail at this. Except that for 37 Signals, it worked really well, perhaps because of the specific people who ran the company.

I strongly agree with the authors when they say:

Pour yourself into your product and everything around your product too: how you sell it, how you support it, how you explain it, and how you deliver it. Competitors can never copy the you in your product.

This is why it’s OK that Rework can be self-contradictory at times. In one chapter, the authors suggest that you "pick a fight." A good example of this is how Stack Overflow was to some degree a response to "hyphen-site." Only a few chapters later, however, the authors say:

Focus on competitors too much and you wind up diluting your own vision. Your chances of coming up with something fresh go way down when you keep feeding your brain other people’s ideas. You become reactionary instead of visionary. You wind up offering your competitor’s products with a different coat of paint. If you’re planning to build “the iPod killer” or “the next Pokemon,” you’re already dead. You’re allowing the competition to set the parameters.

Self-contradictory? Absolutely. Again, Rework is not a formula for how people with no interest in widgets can start a widget business. It’s written to provoke the reader, hopefully provoking them into keeping their business focused on what they can uniquely bring to the table.

Naturally, this presumes that the reader is, to some degree, interesting, and has ideas for things which could benefit other people. It’s not clear to me if everyone has good ideas, but many people don’t execute them, or if some people don’t execute good ideas because they don’t have them. What is clear is that having good ideas and not executing them is little better than not having them.

Share This | Email this page to a friend

The conference is held in a local movie theater. After a day of technical sessions, the attendees would be treated to a screening of Iron Man 2. I wasn’t able to stay long enough for that, but the technical content was more than worth the time and the measly $25 cost of the conference. The theater was still a great place to have a technical conference. The screens were huge and the seats were comfortable.

Keynote

Molly Holzschlag was scheduled to be the keynote speaker, but could not fly due to an ear infection and doctors orders. So she gave her keynote address, and a presentation on HTML 5 later on, via Skype, which was probably better than nothing, but only marginally so, given Internet bandwidth not really sufficient for real-time video. It at least provided some amusement when poorly-timed drop outs turned innocuous technical material into profanities. The HTML 5 presentation suffered from a lack of slides, which Molly made up for by asking us to imagine HTML DOCTYPEs and the like. I appreciate Molly going the extra mile to not leave the conference in the cold, but I didn’t get a lot out of these sessions, which is too bad, because HTML 5 is an ongoing concern for me right now.

Seeing Constraints, Kanban Explained

Generally, I rank conference presentations by whether or not I come out of them with useful information or ideas that I can put into practice back at work. The Kanban session did succeed in that respect. The speaker’s style was very conversational, almost off-the-cuff. He is a process consultant and related a lot of anecdotes about situations he has worked in.

One thing he emphasized which I do not see in much of the writing about Kanban is to use multiple boards when appropriate. For example, a development team can have "their own" board for non-functional changes, which might not ever appear on a board for company-wide items. You can go crazy with this, however; the speaker described one case where something like 36 boards were in use.

This session, like many presentations on development process, tended to presume that development throughput was a bottleneck in the company. This is often the case, but not always. What to do when development is not the bottleneck is a subject which could use fuller treatment.

Understanding User Experience Patterns

Since the original notion of a "design pattern" comes from Christopher Alexander’s writings on architecture (see Patterns of Software, by Richard P. Gabriel, for more info on this), it makes sense that software companies would want to apply design patterns to more than just code structure. Although there are a number of online user experience pattern catalogs, it still strikes me that this is an area where we, as an industry, have a lot to learn.

I very much enjoyed Ambrose Little’s presentation on UX patterns. He showed examples of good and bad interfaces, and discussed why they succeed or fail. He then tied those reasons into pattern catalogs, and demonstrated several of the online catalogs and discussed their strengths and weaknesses.

jQuery 1.4: What You Need to Know

For some reason, the lunchtime sessions were not in the Stir Trek catalog, so I will link Matt’s slides instead. Matt used a lightweight framework to display his slides using HTML and JavaScript instead of a tool like PowerPoint. This allowed him to demonstrate jQuery features right in the slides. It’s a very effective way of presenting his subject matter. There wasn’t a lot of new material in this presentation, given that jQuery 1.4 has been out for a while. It was a good overview, though.

Is Mobile Development Going to Follow the Book of Jobs (Steve That Is)?

This presentation was a lot like its title. Opinionated and verbose, but with an attempt to keep things fun the whole time. The presenter has been working in the mobile space for a long time, and has more than his share of war stories. Like a lot of people in mobile work, though, it seems to me that his opinions were colored by the specific nature of the work his company does. The speaker seemed to treat browser apps and web apps as the same thing. "Native" apps can be web apps. Browser apps can be local. Web app doesn’t necessarily mean "in the browser." HTML/JS isn’t necessarily a web app, especially in the mobile space. Overall, though, I enjoyed this a lot, though I’m not sure I’m going to take practical info home with me.

JQueryUI – The Magic From Behind The Curtain

Watching this presentation was a bit like playing with the jQuery UI demo page. Still, it exposed me to some features that I haven’t seen before. The presenter kept things fun, and I left the session feeling like I’ve learned something.

Conclusion

The Stir Trek organizers, who, as far as I know, are all volunteers, did an amazing job creating an event for 600 people that cost so little and delivered so much value. I haven’t even discussed the freebies like the O’Reilly e-book and the discounts on training, etc. Don’t miss this conference next year!

Share This | Email this page to a friend

if (q.Any())
{

Similarly, you can check to see if any elements in the list meet a certain condition:

if (q.Any(i => i.IsSpecial))
{

If the query provider is something like LINQ to Entities, this will be translated into fairly efficient SQL using EXISTS.

For some reason, I see a lot of people write this code using the Count() extension method instead (maybe they don’t know about Any()?), like this:

if (q.Count() > 0)
{

This is wrong for two reasons:

• It’s imperative instead of expressive. Using Any() tells the query provider, "Please determine if the list is non-empty using the most efficient way you can." Using Count() > 0 tells the query provider, "Please do exactly what I tell you, regardless of what I really need." Because LINQ is intended to support a variety of query providers, it is important to be expressive instead of imperative, and to let the provider specialize the implementation.
• Because we don’t actually care about the exact count of items in the list, only that it is greater than zero, using the Count() function can cause the provider to do considerably more work than necessary. Consider a linked list, or a SQL-based provider.

Share This | Email this page to a friend

Perhaps surprisingly, it’s actually very simple. (I say "surprisingly" because much of the math and technical papers on encryption is decidedly not simple, including that of Gentry’s first fully homomorphic scheme, which was based on ideal lattices.)

This scheme created by Marten van Dijk, Craig Gentry, Shai Halevi and Vinod Vaikuntanathan, however, uses basic arithmetic operations like division, with a few uncommon, but easy to understand, variations.

The nature of homomorphic encryption also dictates a programming style which will seem unfamiliar to users of most high-level languages, but very familiar to anyone who has ever taken a digital circuits class.

Integer Division

What is the integer quotient of 10 / 6? Most people would probably say "1, remainder 4." But the exact answer, in the domain of real numbers, is 1.666666 (repeating), which is closer to 2 than the previous answer of 1. So another way we could answer this question is "2, remainder -2." This quotient is closer to the exact answer, and, correspondingly, has a smaller remainder.

So which answer is correct? Arguably, both of them. As long as the following rule holds:

dividend = divisor * quotient + remainder (the division rule)

…then we have a valid answer.

Which method should you use? It depends on why you’re doing the division. If you’re computing how many cupcakes each child can have at a birthday party, then you’d better use the answer with the positive remainder. If you’re computing how many panels of fencing you need to surround your yard, then you’d better use the answer with the negative remainder.

In fact, there are at least five different, valid algorithms for selecting a quotient and remainder for a given integer division problem.

The homomorphic encryption scheme used in the van Dijk, et. al. paper and in Gentry’s CACM article uses "R-division":

1. Compute the real quotient QR.
2. Compute the integer quotient QZ by rounding QR to the closest integer.
3. Compute the remainder R = Dividend - Divisor * QZ

This is probably different than the method you learned in elementary school, but both are valid. Importantly, it’s probably different than what the integer division and remainder operators in your favorite programming language do.

Modular Arithmetic

If you’ve ever seen a 12 hour, analog clock, then you’ve worked with modular arithmetic. The time of 10:00 today is the same position on the clock as 10:00 p.m/22:00. Another way to say this is:

10 ≡ 22 mod 12

This reads: "10 is congruent to 22 modulo 12." In other words, we just ignore the "extra" 12 hours in the measurement of 22:00, because it’s the same clock dial position as 22:00, only with a circle around the dial.

Naturally, we can perform arithmetic operations like addition and subtraction and compare the congruence of the result. I can say "10:00 + 5 hours means that the hour hand will point to 3 on the clock," or:

10 + 5 ≡ 3 mod 12

Modulo 2 Arithmetic and Binary Operations

You can use any number as the modulus. When we make analog clocks, we use 12. A modulus which is particularly interesting in computer programming is 2. Arithmetic operations mod 2 correspond to binary operations, e.g.:

0 + 0 ≡ 0 mod 2
0 + 1 ≡ 1 mod 2
1 + 0 ≡ 1 mod 2
1 + 1 ≡ 0 mod 2

From this example, you can see that addition modulo 2 is the same as the binary operation XOR. It turns out that subtraction is exactly the same thing (for mod 2 only!):

0 - 0 ≡ 0 mod 2
0 - 1 ≡ 1 mod 2
1 - 0 ≡ 1 mod 2
1 - 1 ≡ 0 mod 2

Multiplication modulo 2, on the other hand, corresponds to the binary operation AND:

0 * 0 ≡ 0 mod 2
0 * 1 ≡ 0 mod 2
1 * 0 ≡ 0 mod 2
1 * 1 ≡ 1 mod 2

Mod in Programming

This is all very simple. However, a word of caution is in order here for anyone who has used a high-level programming language. Most languages have an operator like mod (in Delphi) or % (in C#, which is commonly pronounced "mod"). However, this operator is not strictly modular congruence. It is more like a remainder, although, as we’ve seen, different people and different programming languages might choose to compute a remainder in different (but hopefully valid) ways.

Remainders and congruence are often the same. In fact, the congruence relationship and the integer division remainder for the examples above are all the same. For example:

10 ≡ 22            mod 12
22 / 12 = 1 R 10

However, it’s easy to show examples where this is not true:

22 ≡ 34              mod 12       (1)
34 / 12 = 2 R 10                  (2)
-22 ≡ 2              mod 12       (3)
-22 / 12 = -1 R -10               (4)

(1) is probably not the most common choice; 10 would be a more common answer, as with (2). (1) is nevertheless correct as a statement of congruence. Now compare (3) with (4). The most common way to compute a modulus is to pick the smallest positive number. But the most common way to perform integer division is so-called "T-division", where you select the quotient closest to zero and then calculate the remainder, resulting in a negative remainder when either the dividend or the divisor is negative.

Programs as Digital Circuits

A homomorphic encryption scheme, in addition to the usual Encrypt and Decrypt and KeyGen functions, has an Evaluate function which performs operations on the ciphertext, resulting in the ciphertext of the result of the function.

Obviously, this necessitates a different style of programming than that afforded by the typical programming language. In particular, code like this:

bool a = b ? c : d

…(where a, b, c, and d are all bools) is impossible, because the value of b is encrypted, so the program cannot know what to assign to a. If, however, we can perform binary operations on the ciphertext of b, then we can rewrite the statement above as:

bool a = (b & c) | ((!b) & d)

…or its homomorphic encryption equivalent:

CypherText a = (b h_and c) h_or (h_not(b) h_and d)

…where the h_* operators are the homomorphic equivalents of the usual Boolean operations and a, b, c, and d are now of type CypherText.

Note that the version with the ternary operator and the version with the AND and OR operators are not strictly the same, although their results are; the ternary operator is lazy, whereas the AND and OR version uses complete evaluation. This is necessary when the values are encrypted. It also means that the function may be inefficient.

Intuitively, it’s easy to see that any referentially transparent function can be reduced to such operations; this is what compilers and operating systems do under the hood anyway. I’m not going to go into any detail on how this is done; get an introduction to digital circuits book if you are interested in the particulars.

Gentry’s article notes that other changes in programming style are necessary when performing operations within a homomorphic encryption scheme. For example, the size of the output of a function must be set in advance. Even if the plaintext is variable-length, the ciphertext must be of a known length, which corresponds to the number of "wires" in the circuit. The plaintext, therefore, would have "whitespace" at the end or be truncated, depending upon its size.

Minimizing Boolean Functions

In my first post on homomorphic encryption, I mentioned that Gentry’s encryption schemes can be considered fully homomorphic because they support two homomorphic operations, corresponding to Boolean AND and XOR. I’d like to go into a little bit more detail as to why that is true.

Many combinations of Boolean operations are equivalent. Perhaps the most famous are De Morgan’s laws. There are a variety of techniques for converting one function using Boolean operations into an equivalent function with different operations. This is often done in order to produce a simpler or more efficient function, but can also be done in order to use a specific type of Boolean gate.

It is possible to use combinations of NAND or NOR gates, for example, to produce circuits which perform a Boolean AND, OR, NOT, etc. Hence, a cryptosystem which supported a homomorphic NAND operation could be considered "fully homomorphic."

Similarly, having both AND and XOR gates available allows you to produce the same outputs as all other Boolean gates, as, for example, NOT p is the same as p XOR 1 and we can see by De Morgan’s laws that with NOT and AND we can implement OR.

Therefore, we can see that a cryptosystem can be considered fully homomorphic if it supports enough homomorphic operations to perform any logical Boolean operation. In particular, a cryptosystem which supports any of the following homomorphic operations would qualify:

• Just NAND
• Just NOR
• Both XOR and AND

What’s Next

If you understand the math above, you should now be able to follow along as I implement Gentry’s algorithm for "somewhat homomorphic" encryption. As you’ll see, the "somewhat homomorphic" encryption will be the first step towards a fully homomorphic cryptosystem.

Share This | Email this page to a friend

Share This | Email this page to a friend

Although these words may be unfamiliar to many, the subject matter is terribly important, because, like public-key encryption, which paved the way for secure transactions over the web, an efficient and fully homomorphic encryption scheme would enable new kinds of distributed computing. I’ll explain more about this in a little bit.

If you have an ACM membership, or can find the magazine in a library, I recommend the article. It is far and away more readable than the technical papers on the same subject which are available on the web. The article uses a running analogy of a jeweler who doesn’t quite trust her employees in order to help the reader understand the design of the system.

What Is Homomorphic Encryption?

"Homomorphic" is an adjective which describes a property of an encryption scheme. That property, in simple terms, is the ability to perform computations on the ciphertext without decrypting it first. Because this tends to sound either baffling or miraculous the first time you hear it, let’s begin with a very simple example.

The popular but wildly insecure cipher scheme rot-13 (a.k.a. "Caesar cipher") is partially homomorphic, specifically with respect to the concatenation operation. Imagine we write an Encrypt and Decrypt function using the rot-13 algorithm. The "secret key" will be 13, the number of characters each letter is shifted. Let’s encrypt two words and then concatenate the ciphertext, and finally decrypt the result. In psuedocode, this is:

var c1 = Encrypt(13, "HELLO");      // c1 = URYYB
var c2 = Encrypt(13, "WORLD");      // c2 = JBEYQ
var c3 = Concat (c1, c2);           // c3 = URYYBJBEYQ
var p  = Decrypt(13, c3);           // p  = HELLOWORLD

Because it was not necessary to first decrypt the two fragments of ciphertext before performing the concatenation operation, we can say that rot-13 is homomorphic with respect to concatenation.  In other words, it is possible to take two pieces of ciphertext and perform an operation on them which results in the ciphertext of the concatenation of the two respective pieces of plaintext.

Visually, it looks like this:

It just so happens that in this case the homomorphic concatenation (concatenating the two fragments of ciphertext) is the same operation as the non-homomorphic concatenation (concatenating two fragments of plaintext). This is not always the case. What is important is that we can perform some operation on the input ciphertext which will produce new ciphertext that, when decrypted, will produce output plaintext corresponding to a desired operation on the input plaintext.

Although rot-13 is not at all secure, it turns out that some cryptosystems widely believed to be secure have homomorphic properties. For example, "pure" and un-padded RSA is homomorphic with respect to multiplication.

Why Should I Care?

All of this may seem academically interesting, but one might wonder why I started this post claiming that "an efficient and fully homomorphic encryption scheme would enable new kinds of distributed computing."

I think that the scenario which best makes the benefits of homomorphic encryption clear is cloud computing. It is often much more economical to buy computing resources from a cloud provider then to build a data center yourself. This is especially true if your need for computing horsepower fluctuates. But what if you want to do this computing on private data? Maybe you trust your cloud provider, but what if trust is not enough? Is there a way to get the compelling economic benefits of cloud computing while keeping your data secure?

Imagine that you have developed a web application for income tax preparation. Your application collects personal and financial information from the user, creates a tax strategy resulting in the lowest possible (legal!) income tax payment, and submits the prepared return to the IRS. From a business point of view, this application might be compelling, but it’s a daunting task operationally. In the United States, most people file their taxes once a year, before April 15th. This means that your servers will see a huge spike in demand in the first quarter of the year, with relatively little demand other times. Also, potential customers might be understandably wary about uploading their personal information and financial data to the website of a new business.

Cloud computing offers an excellent answer to the first problem. Because you buy computing resources on as-needed basis, it is very easy to increase the number of available servers when tax season arrives, and reduce the number at other times. Unfortunately, it doesn’t do anything for the second problem.

Hard security in the cloud is not really possible today. It’s very easy to have secure transmission from a local machine to a cloud data store, and of course the stored data can be encrypted. But actually performing computations on that data in the cloud requires decrypting it first. Anyone with physical access to the machine, therefore, has access to your data.

What if it were possible for a user of your tax preparation software to upload their financial information encrypted under the IRS’s public key? Then their data would be secure. You can do this today, of course, but your software would be unable to select a tax strategy for the user or prepare forms. If, however, the encryption method used was also fully homomorphic, then your software could do all of this work without first decrypting the user’s information. The output of your software would be a tax return encrypted under the IRS’s public key. Only the IRS could decrypt it.

Fully Homomorphic Encryption

So a fully homomorphic encryption scheme would seem to solve the security problem while still being compatible with a cloud deployment scenario. Unfortunately, preparing a tax return requires more than just a Concat operation. We can say that an encryption scheme is "fully homomorphic" if it supports enough homomorphic operations to implement any function we need. I’m being purposely imprecise here, because the details are a bit of a digression at this point.

Although there are a number of encryption schemes with one homomorphic operation, until recently no one was really sure if fully homomorphic encryption was even possible. Rivest, Adleman, and Dertouzos suggested that fully homomorphic encryption may be possible in 1978, but were unable to find a secure scheme. Another scheme, from Boneh, et. al., supports two operations, but you can only perform one of them once.

When you consider the complexity of a typical programming language, two operations doesn’t sound like very many, but since we are only looking at operations which mutate data (as opposed to keeping track of the current instruction pointer, etc.) it turns out that supporting the and and xor operations is enough to consider a scheme fully homomorphic, if you can perform those operations an unlimited number of times.

So it was big news when Craig Gentry managed to create such a scheme as part of his Stanford doctoral thesis. Gentry, along with several other researchers from IBM and MIT later created yet another scheme using a different encryption technique, but a similar general approach, with similar homomorphic properties.

Some would say "too big." Bruce Schneier criticized the IBM press release for implying, in his reading, that Gentry’s scheme was practical for real applications, today. It isn’t; the computational and data storage overheads are far too high. However, this takes nothing away from Gentry’s achievement; he has shown that fully homomorphic encryption is actually possible. Indeed, Schneier concludes:

Visions of a fully homomorphic cryptosystem have been dancing in cryptographers’ heads for thirty years. I never expected to see one. It will be years before a sufficient number of cryptographers examine the algorithm that we can have any confidence that the scheme is secure, but — practicality be damned — this is an amazing piece of work.

Let’s Code It!

The second, fully homomorphic scheme proposed by Gentry, et al., and described in the CACM article is simple enough that I decided to implement it myself. I’ve learned quite a lot in the process of doing this. So in the near future I’ll be sharing my code and some of the insights I’ve made in this process.

Before I can do that, however, I need to describe some of the math involved. And I’d like to elaborate on why two operations is enough to consider a scheme "fully homomorphic."

(Continue reading…)

Share This | Email this page to a friend

From time to time, a developer will attempt to add the ASP.NET Membership/Forms Authentication tables to their Entity Framework model (or LINQ to SQL, NHibernate, etc.) model. Before doing this, they will often have created referential constraints against these tables. When the mapping doesn’t work out quite the way they planned, they will ask how to make the mapping work.

There’s only one correct answer to this question: Don’t do it at all!

There are a number of good reasons why you should not make your database and code depend upon the SQL Membership Provider database schema. In this post, I will focus on a few of the most important:

• Separation of concerns
• Membership and authentication providers are supposed to be interchangeable
• The SQL Membership Provider database schema is an implementation detail

Separation of Concerns

Your application’s data model is designed to fit your application domain. It will change based upon the needs of the end users. It should not have to change because Microsoft decides to update the SQL membership provider, as well. That would violate the single responsibility principle. It is often dangerous to combine data which is not closely related into a single data model. This danger is compounded when data from two separate domains, written by entirely different companies, and designed for orthogonal purposes is shoehorned into a single entity model.

Membership providers are supposed to be interchangeable

One of the most important design intentions of the ASP.NET authentication and membership provider model is to make it easy to interchange providers. If you decide to stop using the SQL membership provider in transition to Open ID, domain authentication, Facebook authentication, etc., this should be a matter of, at most, a couple of days work to migrate data from one provider to the other, rather than a complete rewrite of your application, starting with the database and moving out from there.

The SQL Membership Provider database schema is an implementation detail

The publicly-documented interfaces to membership and forms authentication are the Membership and FormsAuthentication types, respectively, as well as the relevant sections of the Web.config file. If you write your code around these types, you have a reasonable expectation that your code will continue to work when the .NET framework is next updated. On the other hand, if you query the database directly, there is no guarantee that the schema will not change when the next version of .NET ships. If Microsoft makes a security-related change to the SQL membership provider, then it is conceivable that the schema could even change with a service pack. The cost of relying on an implementation detail is that you never really know.

Share This | Email this page to a friend

Now when autoencode is set to true we encode the data coming from server and not only when we post it (secutity fix)

Prior to this, you were required to encode the data yourself.

Now personally, I think that should be the default. But it would have been a breaking change for the grid, since there are a few cases where you want to display unencoded data (I’ll discuss these exceptional cases in a second).

It’s really easy to make this the default for your application. Set the grid’s defaults before you create any grids:

    $.jgrid.defaults = $.extend($.jgrid.defaults, {
            autoencode: true,
            datatype: 'json',
            // etc....

You can override this in the rare cases when you don’t want encoded data by setting autoencode false when setting up your grid:

    $("#grid").jqGrid({
                    autoencode: false,
                    url: "/Some/Path",
                    // etc....,

So why wouldn’t you want to use this setting? I can think of two reasons:

1. The server is returning markup which you don’t want to encode. If you have a server method which is returning markup (e.g., "<img src=…"), then you won’t want the grid to encode it. It then becomes your responsibility to sanitize the rest of the grid data on the server.
2. There is a performance cost to doing the encoding in JavaScript. If you are absolutely certain that the server method you’re calling can return only sanitized data, then you can turn off the auto encoding and save this cost. Note that many JSON encoders, such as the "Json()" method in ASP.NET MVC will not encode HTML by default.

Note that you can only set autoencode at the grid level, not at the column level.

If you use custom formatters in your grid, then you should note that the strings they return will not be HTML encoded, even if you set autoencode true. This is probably a good thing, since it is common to use custom formatters to return HTML. However, it means that if you write a custom formatter, then you must ensure that any user data contained in the string and returns is encoded. You can use the grid’s encoder, which is little more than a string replace on angle brackets, in a custom formatter as follows:

    myFormatter: function(cellval, opts, action) {
        if (cellval) {
            return $.jgrid.htmlEncode(cellval.Name + "");
        };
        return "";
    }

Share This | Email this page to a friend

Obviously, if you use a source control tool which locks files on check out, then working concurrently on just about anything will be impossible. So I’m going to presume that your source control tool supports working concurrently, without locks, and has a decent merge tool to handle conflicts at check-in.

We tend to think of the EDMX files as having three sections:

• CSDL, which describes the client schema. This will be used when generating code for your entity classes.
• SSDL, which describes the storage schema, more commonly known as your database metadata.
• MSL, which describes the relationship between the first two.

In reality, however, the EDMX file has two main sections. The first, edmx:Runtime, contains the three sub-sections described above. The second, edmx:Designer, contains a couple of important properties, such as MetadataArtifactProcessing, plus a lot of information about the position of objects on the GUI designer, in a sub-node called edmx:Diagrams. The illustration below should make this clear:

Why is this important? My experience is that any decent merge tool doesn’t tend to have any problem merging what we typically think of as the "EDMX", namely the CSDL, SSDL, and MSL. It is well-formed XML with descriptive tags; it’s the sort of thing that text merge tools tend to do very well with. There are a couple of tips worth knowing, but before I describe them, I’d like to focus on the more common issue with EDMX merges.

In my experience, if two people change the layout of entities in the designer and then attempt to merge, the merge will probably fail. Moreover, you will find it difficult to impossible to fix this up manually, something which is generally fairly easy with merges of source code or other parts of the EDMX. When you look at what is in the edmx:Diagrams node, you will quickly understand why:

I strongly suspect that this "feature" is not specific to the Entity Framework designer, but rather is shared amongst all of Visual Studio’s various diagramming tools. At any rate, I think you will have a difficult time merging XML like this, so I have two recommendations regarding merging and EDMX files:

• If you are going to have two or more developers work on an EDMX file concurrently, then don’t change anything in the designer. If two or more people do this, it almost guarantees a conflict on check-in.
• The other side of the coin is that if you must change something in the designer, then you should give a heads-up to other developers, and exclusively lock the EDMX file while you work.

That said, we don’t generally use the GUI designer at all. It is too unwieldy to have more than a dozen or so types on a single design surface. On the other hand, the tree-style Model Browser is quite useful. I suggest using the Model Browser instead of the GUI designer to navigate your entity model.

Perhaps a future version of the Entity Framework designer will allow for multiple diagrams within a single entity model, somewhat like SQL Server diagrams. If each diagram had its own node in the XML, you might even be able to merge them.

If you steer clear of issues with the designer, then you will typically find that EDMX changes can be automatically merged by any decent merge tool (unless, of course, they actually conflict). In the unlikely event, however, that your merge tool reports a conflict, and you don’t think it’s an actual conflict between your changes and those made by the other developer, it does help to understand how the "Update Model from Database" wizard updates existing EDMX:

1. The wizard will generate new SSDL from scratch. It will then go through the newly generated SSDL, and replace each matching node in the existing SSDL with the newly generated version. There are a few SSDL features which are not supported by the designer/wizard, such as will, and these will be left alone, if they exist in your model.
2. Any new entities or new properties of existing entities will have corresponding nodes added to the CSDL and MSL. But existing entities and properties will not have their CSDL and MSL updated, because the wizard presumes that you want to keep any changes you may have made to them.

Therefore, when multiple developers are updating an EDMX file concurrently, they should use the same database. This will tend to make SSDL changes merge without conflict, and allows you to simply overwrite any false conflicts (merge tool failures) in the SSDL, because you can be confident that it was generated from the same object in the same database. But if you’ve manually customized your SSDL (i.e., edited the SSDL section of the EDMX as text), then you should keep an eye on your changes, to make sure that the wizard does not overwrite them. Most people never do this, though.

On the other hand, changes to the CSDL and MSL must be reviewed carefully in the unlikely event that there are conflicts in the merge, because you, or the other developer, may have made customizations to the mapping or entity types which you want to keep.

Finally, it is worth mentioning that the Entity Framework version 4 supports "code only" models which do not use EDMX files at all. If you prefer designing your entity model via source code, you can choose this option.

Share This | Email this page to a friend

Many people ask how to do a "left join" in LINQ to SQL, and unfortunately, the answer they nearly always get — "Use DefaultIfEmpty!" — is, in my opinion, terrible advice. Let’s implement the same query with and without the join keyword, and then compare the readability of the queries, the functionality, the knowledge required to write the query, and the maintainability of the code. I think you will find that using associations wins on every single criterion I examine.

I’m going to use the Northwind demo database for this example, since many people are familiar with its structure. I created a LINQ to SQL model for Northwind by simply dragging all of the tables in the database onto the LINQ to SQL designer. The only change I made to the model generated by the designer is to rename the "Employee1" property on the generated "Employee" type to the more descriptive name "Supervisor." I’m using LINQ to SQL for this demo, but everything I’m saying here applies equally to LINQ to Entities.

"Left" Joins

Let’s imagine that I am asked to produce a web page listing all employees in a company, along with their supervisor, if any. This requires a "left join," in SQL terms, because not all employees have a supervisor. I’ll project onto a presentation model, just like I do in LINQ to Entities. Using the association properties generated by LINQ to SQL, this is quite simple:

This is fairly readable. The one thing that you need to know is that both LINQ to SQL and LINQ to Entities coalesce nulls. This means that on a row where e.Supervisor is null, you will not get a NullReferenceException in the assignment to SupervisorName and SupervisorBirthDate, as you would with LINQ to Objects. Instead, null will be assigned. Therefore, it is important that EmployeeListItem.SupervisorBirthDate is of type DateTime? (a.k.a. Nullable<DateTime>) instead of the non-nullable DateTime.

Let’s compare that to the equivalent query using the join syntax, using the mysteriously popular DefaultIfEmpty trick:

Yuck! This is far less readable than the query above. Yet these two queries produce exactly the same results. If you don’t believe me, download the sample project attached to this post and try it yourself. Perhaps even worse than the general unreadability of the "join version" is the fact that this query requires knowledge of the structure of the database which is already present in the DBML (or EDMX, in the case of the Entity Framework) model. This is a problem for two reasons. First, it’s an opportunity for programmers to make a mistake, which the first query eliminates. Second, it’s a potential maintenance issue if the foreign key definition ever changes in the database.

"Inner" Joins

Now let’s compare an example of an "inner join" using both my recommended method of associations and the LINQ join keyword. Here’s the association version. It’s so readable that there is very little to say about it:

Here’s the join version:

Again, these two queries do exactly the same thing, as you can confirm for yourself by running the demo project. The join version here shares all of the faults of the "left join" version above.

API Consistency

Thus far, I’ve been discussing LINQ to SQL. But what if I have an Employee instance and that like to examine the employee’s supervisor? I might write code like this:

Now compare that code with the "association" and "join" query forms. You will see that using the associations makes the LINQ to SQL queries much more closely resemble how you work with the materialized entity objects in "regular" code. Again, I think this makes your code easier to read.

What About Performance?

Not surprisingly, the SQL generated by the equivalent association and join versions is close to identical. I would not expect to ever see a performance difference between the two syntaxes in terms of query execution time.

Is It Ever Correct to Use join?

The advantages of using associations are so strong that you may wonder why join exists in LINQ at all. Associations, however, are only helpful when they actually exist. There may be times when you need to "join" based on values which are not actually foreign keys. Or you may need to join between LINQ to SQL and LINQ to Objects.

Running the Demo Project

Here’s a the demo project. To build and run it, you’ll need to Visual Studio 2010 Beta, SQL Server (Express is fine), and the Northwind demo database I linked above. Open the project, find the Web.config file in Solution Explorer and open it. Change the connectionString to point to your SQL Server. Now you should be able to run the application.

Share This | Email this page to a friend

Share This | Email this page to a friend

Imagine that you’ve been asked to write a new web application to track employees for a customer, Chotchkies restaurant. The application must use ASP.NET MVC and the ADO.NET Entity Framework. The user interaction designers have mocked up the following interface for editing an employee:

Upon seeing this mockup, your first recommendation is to fire the user interface designers. But management declines to follow your recommendation. So how should you implement this? Since this is a new application, you have no database, no entity model, nothing. Where to begin?

Presentation Models

For a variety of reasons, I always use strongly-typed views in my ASP.NET MVC applications. Also, I use presentation models instead of using entity types directly as the model type for my views.This allows the user interface and data models to evolve independently.

Because I am likely to want to build user interfaces using ASP.NET MVC 2’s Dynamic Templated Views, I need to decorate the view model with presentation concerns, like noting that the "Flair count" field is read-only in this particular view.

Especially in view of the less-than-compelling user interface markup, it’s important to realize that the design of the user interface is likely to change wildly over the course of implementing the application. When actual users begin to test the application, they will request changes to the user interface, and you need to be able to adapt to this.

This has two important implications: You must get a prototype to testers and end-users as fast as possible, and your user interface must not be deeply coupled to the rest of the application, given the high likelihood of change.

So based upon the user interface prototype above, the following presentation model might be reasonable:

It’s now trivial to write an action which can be used to develop a view for editing the employee, and which matches the UI prototype above. Such an action might look like:

I’ll spare you the HTML. At this point, I have a running application which the user interaction designer can approve, and testers can try out. My total effort, thus far, is a few minutes of work. If I had started by designing a database and the data model, I would still have nothing to show for my efforts.

Entity Model

At some point however, you need to create a database and an entity model. That was, after all, part of the requirements you were given for the application. Knowing that there might be a need to reference people who are not employees, the entity model is going to have to look very different than the presentation model above. After some discussions with the business analyst, you learn that Chotchkies management is very serious about tracking employee flair, so a first attempt at an entity model might look like this:

Now you can generate a database. At this point, you have a presentation model and an entity model, and need only wire them together.

Projection

Taking a collection of instances of one type, and mapping their properties onto a collection of instances of another type is often called mapping or projection. With older ORMs, which have either no or very limited LINQ support, this can be quite tedious, requiring manual code, the use of tools like AutoMapper, and a good deal of thinking about eager loading, lazy loading, and optimizing situations like the fact that we don’t actually want to load all of the flair for an employee here; we just need the count.

The Entity Framework, on the other hand, makes this very easy, as we can just express the relationship between the entity model and the presentation model with a LINQ expression:

I’m glossing over some details here. In a real-world application, for example, we would use the repository pattern rather than grabbing the context directly in the controller. But the point of this post is shown in the code above:

By expressing the relationship between the entity model and the presentation model as a LINQ query, it is no longer necessary to worry about implementation details like eager loading versus lazy loading, optimizing the SQL for the Count and avoiding loading big properties not actually used here. All of this can be — and is — derived from the query above, automatically.

Share This | Email this page to a friend

There is no good reason not to use static analysis. If you are the sort of person who doesn’t allow hints and warnings in your code, and has configured your build to fail on any hint or warning (and I hope that you are), then you’ll love static analysis; it takes this kind of discipline to the next level. Sure, it’s useful to find existing errors in your code, but the most important benefit is that it prevents new errors from being introduced.

The only remotely valid argument I’ve heard against using static analysis is that when you first use a static analysis tool against a legacy code base, you will typically see hundreds of rules violations, many of them incorrect. Cleaning all this up will be a lot of work, which you usually can’t take on all at once.

Unlike compiler hints and warnings, static analysis rule violations often happen when there is no underlying bug. When you develop new code, you can analyze any newly introduced rule violation and exclude it if necessary. But that’s not practical to do with legacy code. My approach to this problem is very simple: Turn off rules in your static analysis tool until all the rule violations go away.

What’s the point of buying a static analysis tool and then turning off half the rules? Well, the other half are still turned on, and they are now a part of your automated build, making it impossible to introduce new code which violates them. Moreover, you can go back and turn on the other rules a reasonable pace, say, one or two a week, fixing real issues that you find or excluding the particular violation if it turns out to be a non-issue, until they are all turned back on.

Once you have even a few static analysis rules turned on in your automated build, you’re getting some degree of quality control for very little expense. Integration testing costs money, every time you do it. Unit testing costs money every time you write new tests. Static analysis just works and keeps working, and costs you nothing more than a software upgrade once every couple years or so.

So static analysis in general is a good thing. What about CodeHealer in particular? Well, we have been using it for around three years now. It has been stable, and I’ve seen new versions released regularly. Also, I have never seen anything better for Delphi.

Digression: Attributes

When Delphi 2010 was released with the enhanced RTTI and custom attributes feature, many Delphi developers wondered what this feature would be useful for. In general, your source code describes the behavior of your application, and attributes describe your mechanisms (thanks to Eric Lippert for this concise description). Static analysis is an excellent example. Imagine that I want to exclude a static analysis rule in a particular case. This has nothing to do with the desired functionality of the application; rather, it’s more like a parameter to the automated build process.

That may sound a bit abstract, so let’s look at a specific example. FxCop, a static analysis tool for .NET code, has rule that should not introduce private types which are never used. But FxCop cannot detect the instantiation of types inside of a LINQ query. So in this case the rule is making a mistake; it is failing to detect that a type is, in fact, instantiated in my code. One possible solution to this issue would be to try and write a better rule. But that’s a lot of work, and there are only about two cases of this false positive in our entire source tree. So it’s easier to just suppress this message in the case of a false positive, which you can do with an attribute:

[System.Diagnostics.CodeAnalysis.SuppressMessage(
    "Microsoft.Performance",
    "CA1812:AvoidUninstantiatedInternalClasses",
    Justification = "It is instantiated in a query, which FxCop can't see.")]
private class ConsumerRow //...

This is far easier (and, I think, a better design) than what CodeHealer currently offers. I hope that future versions of CodeHealer will use this method, now that Delphi has attribute support.

Share This | Email this page to a friend

Share This | Email this page to a friend

New Features

Some of the new features of preview 2 have been discussed elsewhere, so I won’t rehash them. But I’ve also noticed that there is a new attribute, [RequireHttps], which does what you would expect, when added to an action, and a new HTML helper, Html.HttpMethodOverride, which makes it easy to take a POST request and code as if the request were actually PUT or DELETE, by adding a hidden input containing a special value on the HTML form. This allows you to write your server in a more RESTful style, which will be suitable for user agents which know about the PUT and DELETE verbs, while maintaining compatibility with those (like browsers) which do not.

MvcHtmlString

After installing the new assembly into the GAC, I attempted to compile our existing projects. I got a compilation error on some of our custom HTML helpers, as the MVC extension methods like Html.RouteLink have been changed to return an instance of type MvcHtmlString instead of String. In many cases, I could just change the return type. However, the clear intention of the framework designers is that any HTML helpers which return HTML (with angle brackets) instead of "plain" text should return a MvcHtmlString instead of a String. So, in addition to changing the return types of methods where necessary to get those methods to compile, I also wanted to change the return types of any custom HTML helper which returned HTML containing angle brackets. It makes no difference today, but in ASP.NET 4, there will be a new syntax to guard against XSS attacks, and it makes sense to get ready for this by returning the correct type.

The trick here is that MvcHtmlString does not have a public constructor, so it’s not immediately obvious how to create one of these things. Perusing the MVC source code, I noticed that it does have a static Create method, and comments elsewhere in the source code indicate that this method should be used instead of the detected constructor. So you can now write a helper like this:

public static MvcHtmlString Fud(this HtmlHelper helper)
{
    return MvcHtmlString.Create("<acronym title=\"Fear, Uncertainty, and Doubt\">FUD</acronym>");
}

JsonRequestBehavior

ASP.NET MVC will now, by default, throw an exception when an action attempts to return JSON in response to a GET request. Unless you read the release notes carefully, you won’t discover this until runtime. This is in order to proactively defend against a particular cross-site attack. It is good to be safe by default, but you are only vulnerable to this attack under the following combination of circumstances:

• The data you’re returning is worth stealing.
• The root data object in the JSON response is an array.
• The requesting browser is not IE 8, or some other browser which doesn’t allow __defineSetter__

It turns out that in our application we almost never return JSON results with the root object as an array in a GET. So the best fix was generally just to tell the framework that we have examined the risk and determined it returning JSON in this case, by changing code like:

return Json(model);

…to:

return Json(model, JsonRequestBehavior.AllowGet);

Unfortunately, I had to make this change in a lot of places. Still, I agree with the framework designers: Better safe than sorry.

JavaScript Files

The MicrosoftAjax.js, MicrosoftAjax.debug.js, MicrosoftMvcAjax.debug.js, and MicrosoftMvcAjax.debug.js files have all changed for ASP.NET MVC 2. Existing ASP.NET MVC 1.0 projects will have old versions of these files. The upgrade instructions in the Release Notes don’t mention this, but (they do; I just missed it) you should replace these files with the newer versions when upgrading to MVC 2. To do this, create a new MVC 2 project, copy the JavaScript files from the Scripts folder in this new project, and paste them over the existing files in the project you are upgrading.

ModelBindingContext Changes

One of the new features of preview 2 is client-side validation and custom metadata providers for validations. In order to implement this feature, some of the implementation details of model binding have changed. This won’t affect most people, but the fix was a little tricky, and required a good bit of examination of the MVC source code, so I’ll list it anyway. Prior to preview 2, I had code in a unit test for a custom model binder like this:

        internal static T Bind<T>(string prefix, FormCollection collection, ModelStateDictionary modelState) where T:class
        {
            var mbc = new ModelBindingContext()
            {
                ModelName = prefix,
                ModelState = modelState,
                ModelType = typeof(T),
                ValueProvider = collection.ToValueProvider()
            };
            IModelBinder binder = new MyModelBinder();
            var cc = new ControllerContext();
            return binder.BindModel(cc, mbc) as T;
        }

With Preview 2, on the other hand, I have to do this:

        internal static T Bind<T>(string prefix, FormCollection collection, ModelStateDictionary modelState) where T:class
        {
            var mbc = new ModelBindingContext()
            {
                ModelMetadata = ModelMetadataProviders.Current.GetMetadataForType(null, typeof(T)),
                ModelName = prefix,
                ModelState = modelState,
                ValueProvider = collection.ToValueProvider()
            };
            IModelBinder binder = new MyModelBinder();
            var cc = new ControllerContext();
            return binder.BindModel(cc, mbc) as T;
        }

Not hard once you know the trick.

Share This | Email this page to a friend

Let me say at the outset that although all three systems work somewhat differently, I don’t see an overwhelming advantage to any one design. Generally, you can do what you need to do in all three environments. I’m writing this article not to claim that any one system is better than the others, but to point out some of the subtleties in the implementations.

Before I get started, I’d like to thank Barry Kelly for his useful feedback on my first draft of this article.

Compiling Instantiations

Every implementation of generic types works via a two-step process. First, you define a generic type or method with a "placeholder" for a specific type, which will be substituted later on. Later (exactly when depends upon the language), the type is "instantiated." Note that instantiating a generic type is very different from instantiating an object. The former happens within the compiler, whereas the latter happens at runtime.

Instantiation is triggered when some code uses a generic type or method with a specific type parameter, and means that based upon the generic definition and the types or values passed when the generic is used, a specific implementation is substituted in order to allow the generation of machine code. Instantiation is one of the most important differences between real generic types and using non-generic types with casts. In the end, different machine code is generated for instantiations for different type parameters.

In C# and Delphi, there is a language feature which is solely dedicated to implementing generic types and methods. In C++, on the other hand, the "templates" language feature can be used to implement generic types and methods, among many, many other things. It is even possible to do general-purpose programming using templates, which C++ programmers call "metaprogramming."

C++ templates require the template source code to be available when the code using the template is compiled. This is because the compiler does not actually compile the template as a separate entity, but rather instantiates it "in-place" and only compiles the instantiation. The C++ compiler is effectively doing code generation, substituting the type parameter (or value) for the placeholder for the type, and generating new code for the instantiation. Update: Moritz Beutel elaborates on this in his excellent comment on this post. You should read the full comment, but the short version is that the manner in which templates are compiled can result in errors in the code which uses the template appearing (from compiler error messages), incorrectly, to be errors in the template itself. Moreover, the implementation of most C++ compilers makes this problem even worse than what is necessary in order to implement the C++ standard.

In Delphi and C#, on the other hand, the generic type or method in the code which uses the generic type or method can be compiled separately. Therefore, you can compile a library which contains a generic type, and later on compile an executable which uses a instantiation of that type and has a reference to the binary library, rather than to the source code for the library.

Another way to think of this difference is that in C++, a template will not be compiled at all until it is used. In Delphi and C#, on the other hand, a generic type or method must be compiled before it can be used.

In Delphi, the compiler uses a feature closely related to the method inlining feature. This causes the compiler to store the relevant bits of the abstract syntax tree for the generic type parameter in the compiled DCU. When the code which uses the generic type is compiled, this bit of the abstract syntax tree is read and included in the abstract syntax tree for the code which uses the generic type, so that when machine code is produced, based on the new, “compound” abstract syntax tree, it looks, to the code emitter, like the type was defined with the type parameter "hard coded." Instead of linking to compiled code in the DCU, the code which uses the generic type emits new code for the instantiation into its own DCU.

Because generic instantiation is performed in the same area of the Delphi compiler which does method inlining, there are some limitations on what you can do in a generic method, or a method of a generic type. Like inlined methods, these methods cannot contain ASM. Also, calls to these methods cannot be inlined. These restrictions are limitations of the implementation, not of the language design, and could theoretically be removed in a future version of the compiler.

C# generics use the .NET Framework 2.0+, which has native support for generic types. The C# compiler emits IL which specifies that a generic type should be used, with certain type parameters. The .NET framework implements these types using one instantiation for any reference type, and custom instantiations for value types. (Don’t confuse “instantiation” with “instance” in the preceding sentence; they mean entirely different things in this context. There are usually many instances of one instantiation.) This is because a reference to a reference type is always the same size, whereas value types can be many different sizes. Later, the IL will be JITted into machine code, and, as with compiled C++ or Delphi code, types don’t really exist at the machine code level. In .NET, generic type instantiation and JITting are two distinct operations.

So one important difference in generics implementations is when the instantiation occurs. It occurs very early in C++ compilation, somewhat later for Delphi compilation, and as late as possible for .NET compilation.

Custom Specializations

Another very important difference is that C++ allows custom instantiations, called specializations, including specializations by value. With C# and Delphi, on the other hand, the only way to instantiate a generic type is to use that type with an explicit type parameter. The implementation will always be the same, with the exception of the type of the type parameter. Because C++ allows custom instantiations, it is easy for a programmer to write different implementations of a method, for example, for different integer values. Like operator overloading, this is a powerful feature which requires considerable self-restraint to avoid abuse.

Constraints

Delphi and C# both have a generic constraint feature, which allows/requires the developer of a generic or method type to limit which type parameter values can be passed. For example, a generic type which needs to iterate over some list of data could require that the type parameter support IEnumerable, in either language. This allows the developer of the generic type to make her intentions for the use of the type very clear. It also allows the IDE to provide code completion/IntelliSense on the type parameter, within the definition of the generic type. Also, it allows a user of the generic type to be confident that they are passing a legal value for the type parameter without having to compile their code to find out.

In C++, on the other hand, there is not presently any such feature. A more powerful/complex feature called "concepts" was considered for, but ultimately removed from, C++0x.

An implication of the lack of constraints is that C++ templates are duck typed. If a generic method calls some method, Foo on a type passed as the generic type parameter, then the template is going to compile just fine so long as the type parameter passed contains some method called Foo with the appropriate signature, no matter where or how it is defined.

Covariance and Contravariance

Let’s say I have a function which takes an argument of type IEnumerable<TParent>. Can I pass an argument of type IEnumerable<TChild>; to that function? What if the argument type were List<TParent>; instead of IEnumerable<TParent>? Or what if the generic type was the function result rather than the function argument? The formal names for these problems are covariance and contravariance. The precise details are too complicated to explain in this article, but the examples above summarize the most common times you run into the problem.

Delphi generics and C++ templates do not support covariance and contravariance. So the answers to the questions above are no, no, and no, although there are, of course, workarounds, like copying the data into a new list. In C# 4.0, function arguments and results can be declared covariant or contravariant, so the examples above can be made to work where appropriate. "Where appropriate" involves non-trivial subtleties hinted at above, and exemplified by the fact that arrays in .NET have (intentionally) broken covariance. However, the BCL routines in the .NET Framework 4.0 have been annotated to support covariance and contravariance when appropriate, so developers will benefit from the feature without having to fully understand it.

Share This | Email this page to a friend

• Most tools tend to work on either compiled assemblies or source code. NDepend uses both.
• Most tools implement their tests in compiled, executable code. NDepend, on the other hand, uses a proprietary, domain-specific language called CQL.
• NDepend can enhance its analysis by importing analysis data from related builds, like NCover, Team System coverage analysis, and can compare multiple builds of the same project.

As with other static analysis tools, however, NDepend is designed to be used both within Visual Studio and as part of an automated build/continuous integration process.

NDepend claims to support "any .NET language including C#, VB.NET, MC++ and C++\CLI." To put this claim to the test, I ran it against a Delphi Prism application. Indeed, it did produce useful results in the analysis, although there were some quirks. It couldn’t find the assembly from the solution, but after I located it manually analysis could run. I suspect NDepend doesn’t parse the source code of Prism applications as it does with C#.

What NDepend Does

The name "NDepend" presumably comes from the dependency analysis feature. NDepend will display dependencies between assemblies either as a cross tab or a directed graph. See the website for illustrations of this feature.

NDepend also does static analysis along the same lines as Microsoft FxCop and StyleCop, only in a much more interactive manner. By combining the features I’ve already mentioned, you can easily ask queries like, "What are the most complex methods I’ve changed recently which have no unit test coverage?"

All of NDepend’s features are available in its own GUI application, from the command line, within Visual Studio, and in a custom MS build task. So, as with most static analysis tools, you can integrate NDepend into a continuous build environment in order to set enforceable standards on code complexity and the like.

Installation

NDepend does not have an installer. You need to create a folder for the application, and unzip an archive file into that folder. To install the integration with Visual Studio, you click a button in the GUI application, once it’s running. I find point and click installers a bit more convenient, but it was certainly easy to get the application working.

Default Analysis of a Do-Nothing Application

NDepend is quite strict even in its default configuration. For example, I ran it on a do-nothing, "File->New" ASP.NET MVC 2 Preview 1 application, with no code of my own added. It reported quite a number of warnings.

Many of the warnings were incorrect, because NDepend did not automatically find the System.Web.Mvc assembly, which is not stored along with the rest of the .NET framework, even though it is in the GAC. In general, the current version of NDepend will only find assemblies which are in the solution folder or are in the .NET Framework folders automatically. You should add other folders for referenced assemblies manually, even if those assemblies are GACced. To fix this, I had to do the following steps:

1. Go to the Project Properties tab in the GUI.
2. Click the "View Folders" button.
3. Click the "Browse" button and manually locate the folder containing the required assembly.

With these steps completed, the quality of the analysis improved quite a lot.

NDepend shows the results in its GUI and also produces an HTML report. The HTML report for the do-nothing application was 14 pages long, and the report for our production web application ran over 50 pages. This can be kind of overwhelming for a new user. The issues flagged in the do-nothing application included (with my comments in parentheses):

• Methods with > 10 lines of code should be commented. (I disagree; methods should be commented when their intention cannot be made clear in code. This has little relationship to method length and is difficult to impossible to flag with static analysis.)
• Types with no state should probably be static. (Worth considering.)
• Types with no subtypes could be sealed. (NDepend makes it easy to turn this off for public types.)
• Classes with small instance sizes, etc., could be structs. (Maybe, if you want value semantics.)
• Avoid namespaces with few types. (This is expected on a File->New application; would work better in a "real" app. FxCop has the same rule.)
• You should’t have types outside a declared namespace. (I agree.)
• Avoid [un] boxing. (I agree; great test!)
• Types which "could be internal." (NDepend can’t pick up instantiation via reflection, making this less useful in the specific case of MVC.)
• Fields which are assigned only by the constructors of their types should be marked read-only. (I agree; this is a really useful test.)
• Type names should begin with an uppercase letter. The default MVC application includes a type called "_Default".

For the sake of comparison, I analyzed the same application using FxCop. This produced 11 warnings, about half correct, and about half incorrect. Incorrect warnings included not recognizing the TLA MVC, asserting that Application_Start should be static, and claiming that the application should target .NET Framework 3.5 SP 1 (as opposed to just 3.5), something which Visual Studio 2008 does not allow. Probably correct warnings included that the application should have the CLSCompliant attribute, and that the _Default type was oddly named.

Customizing the Analysis For a Real Application

NDepend’s use of its CQL language to implement its test makes individual test very easy to customize. For example, our production web application includes the Microsoft DynamicQuery library, which is distributed as a single CS file. Because we included this in one of our assemblies instead of creating a separate assembly for it, NDepend analyzes it along with the rest of our assembly. It contains many long methods, and other things which NDepend does not like, but because this is Microsoft’s code, and not ours, I would like exclude it from the analysis. So I can, for example, change the test for "method too long" from:

// <Name>Methods too big (NbLinesOfCode)</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE NbLinesOfCode > 30 ORDER BY NbLinesOfCode DESC
// METHODS WHERE NbLinesOfCode > 30 are extremely complex and
// should be split in smaller methods
// (except if they are automatically generated by a tool).
// See the definition of the NbLinesOfCode metric here http://www.ndepend.com/Metrics.aspx#NbLinesOfCode

to:

WARN IF Count > 0 IN SELECT TOP 10 METHODS
    WHERE NbLinesOfCode > 30
        AND !(FullNameLike "^System.Linq.Dynamic") 
    ORDER BY NbLinesOfCode DESC

What if you don’t know how to write a regular expression? Well, then you need to learn how to write a regular expression. Regular expressions are the only option for this kind of name matching in NDepend.

It might seem at first that using a proprietary language would be a barrier to using the tool. After all, other tools, like FxCop can be customized using C# and other .NET languages. Doesn’t a proprietary syntax like CQL discourage customization? Not really, in my experience. The real win of NDepend is that you can customize the rules interactively, within the GUI. In contrast to building a custom FxCop rule and restarting FxCop, this is a huge win, because it encourages the use of ad hoc queries and trial and error to build the ideal heuristic for your application. Coupled with auto complete in the CQL editor, it makes the relatively minor inconvenience of dealing with a proprietary language almost irrelevant. Yes, I would prefer to use LINQ, but the interactivity of CQL is a much bigger win.

However, you don’t get the same kind of customizability at a global level. You can include and exclude assemblies from analysis globally, but that’s about it. So while you can include and exclude code based on almost any imaginable criterion at the level of an individual test, I have not found any way to do an exclusion like the one I demonstrate above globally. (Note, however, that in a comment on this blog a week ago, a member of the NDepend team suggested that features along this line may be coming next year.)

I should add that most static analysis tools seem to work this way. They typically make it fairly easy to include and exclude entire tests or specific tests on specific cases, but don’t typically include a way to say, "Don’t run any test on any bit of code which meets the following criteria, ever." So NDepend is no worse than other tools I’ve used in this regard, and it’s features to customize individual tests are considerably better than most other tools, although the learning curve is a bit steeper.

Like every static analysis tool I’ve ever tried, NDepend does not produce useful results for generated code. Since this is not an NDepend-specific issue, I discussed it in an earlier post, along with some tips for excluding generated code from NDepend’s analysis.

Documentation

The documentation in the NDepend online help is a bit thin. But the authors have written a number of informative blog posts on how to use specific features and tips for getting the most out of the tool. Once you realize that these posts exist and know where to find them, it is quite a bit easier to read up on how to use the tool effectively.

Usability

I’m of two, contrasting opinions regarding the usability of NDepend. I found the initial learning curve to be quite steep, and elements of the user interface to be not very discoverable. On the other hand, once I figured out how to use the application, which took me a few weeks (maybe I’m just slow), I found the application to be very usable, and I could do what I needed to do very quickly. Watching the videos on the NDepend website helps quite a lot in getting over this initial hump. I’ve also had some e-mail discussions with the makers of the software on how they might improve some of the things this new user found initially confusing in their next release.

For those new to the application, here are some things which may help you get started:

• The CQL query editor has auto complete, so it’s fairly easy to customize a query even if you’re unfamiliar with the language. However, if you intend to do any serious work in CQL, it is worth seeking out and spending some time reviewing the CQL examples on the NDepend website.
• After you do an initial analysis, look at the error list before considering the analysis results. Adjust the configuration (by adding assemblies or search directories) to clean up any of the errors before moving on to looking at the analysis results. There are a few valid C# language constructs that NDepend’s source code parser won’t recognize. I got a warning from a class constraint on a generic type. I understand this will be fixed in a future version. You can ignore these warnings; they’re harmless.
• Whatever you want to do within the application, the place to start is by selecting the appropriate view configuration from the View menu. So if you want to alter a query to return more specific results, you would select View -> Reset Views to work with the CQL language.
• The CQL Query Result tab will display a query, and has been Edit button, you can never added a query in that window. To edit a query, go to the CQL Queries tab, select the query you’d like to edit, and click the Edit Query button. A new tab will appear which allows you to actually edit query.
• When you click on a query in the CQL Queries window, you will only see the results if the query returns any rows. If the query returns no rows, then the results window will not appear.
• Whenever you analyze the project in the NDepend GUI, NDepend creates a lengthy HTML report, and opens it in your web browser. But all of the information in the report is also available within the GUI application, as well. So you can see the same information in whichever place is most convenient for you to navigate.

Performance

NDepend is fast, sometimes surprisingly so, given that it is looking at both binaries and source code. It generally takes less time to run an analysis than it does to compile a solution. I never experienced any performance problems in the NDepend GUI, either.

Conclusion

Do you need NDepend? It depends. NDepend will be useful to developers and teams who have a functioning continuous integration process (and you should) and want to impose standards on this sort of code which can be checked in to the repository. But if you’re going to go down this road, you should probably start with FxCop, which is free and better known. NDepend, however, can take this kind of analysis much, much farther than would be possible even with custom FxCop rules, like comparing multiple versions of an application and mandating unit test coverage. I also find it much easier to customize.

If you do have an automated build procedure in place and you’re not already doing static analysis, it’s time to start! It can be a bit overwhelming to run static analysis for the first time on a legacy project and see hundreds of warnings or errors. But that’s easy to get over; simply disable all the warnings but one, and start cleaning things up a bit at a time. You can make the analysis stricter as time goes on. You don’t have to start by fixing everything. Even a few rules in place will prevent new such errors being introduced to the source base.

It will be also useful to developers confronting a large source base for the first time, particularly if there are more than a few assemblies involved.

While NDepend might be useful to a single developer reviewing their own code, and not part of an automated process, I’ve found that static analysis will really only be used regularly when it is automated. Few developers have the discipline to run static analysis and cleanup any problems uncovered without integrating it into build automation.

Disclaimer

This review is based on a review copy supplied by the makers of NDepend. If you think that has colored my impressions of the tool, well, I disagree, but at least you know.

Share This | Email this page to a friend

I think I know what the answer is. Very often, someone will suggest an approach to a problem which strikes me as technically wrong or otherwise misguided. Like a lot of geeks, I have an almost compulsive desire to do things correctly, especially in areas where (to me, anyway) there is a clear-cut difference between "the right way" and "the wrong way" of doing it. Over and over again, I have said "no" in a way which ranged somewhere between insufficiently respectful and inappropriate.

I think I can explain why I do this (and I’ll do so below, for anyone who actually cares), but the important part is this: I need to teach myself to say "No" in a much more constructive way.

When I hear (what I feel is) a bad idea, I think I have this little moment of panic about the bad idea getting into our code, and having to maintain it for 10 years, because goodness knows I have a lot of bad ideas to maintain in our VCS. It’s like I’m worried that simply mentioning the idea will doom it (or us) to implementation, so I went to put the barrier in front of it before it can go any further.

This may accomplish the technical goal, but saying "No" in an unconstructive way is poor leadership. I don’t want to create a climate where people hesitate to suggest solutions to a problem, because they might be incorrect. Indeed, I learned as least as much, if not more, from my mistakes is from my successes, so discussing a bad idea may contribute as much to a project is discussing a good idea. Rather than just saying, "That won’t work because…" I need to help the person who suggested the idea explore its implications and work out the ups and downs on their own.

Share This | Email this page to a friend

Therefore, I’ve updated the About page to include formal licensing terms. In particular, any source code I’ve written and posted here is available under the MIT license, which should allow you to do almost anything you want to with it, but read the About page for the particulars.

Thanks to Robert Claypool for pointing this out.

One other bit of news about this site: Readers who prefer to be informed of new posts via Twitter can now follow the blog there.

Share This | Email this page to a friend

A Useful, But Imperfect, Tool

Static analysis can be incredibly useful. It finds errors in much the same way that compilers report hints and warnings, but at a significantly deeper level. Compilers typically report a hint or a warning only in cases where the compiler is completely certain that there is a problem with the code, like a completely unused variable. Static analysis tools, on the other hand, can report errors in cases where, for example, the code itself might function, but appears to be too complex to be maintainable. Obviously, there is a gray area between maintainable and unmaintainable, and every static analysis tool I have ever tried has returned false positives. FxCop, for example, will complain that an internal type is never instantiated if the only time it is ever instantiated is inside of a LINQ to Entities query; it doesn’t seem to be able to "see into the Expression."

For Code Written By Humans Only

Static analysis is close to useless for generated code, because it is generated by a tool, tends to always function correctly, and is presumably never maintained by humans, at least directly. But generated code tends to really set off alarms on static analysis tools, due to high line counts, cyclomatic complexities, and numbers of parameters, no unit tests, unusual type and member names, etc. In other words, in generated code functionality is much more important than maintainability. But in non-generated code, maintainability is at least as important as functionality. The best solution is to convince your static analysis tool to ignore the generated code altogether, but this is often easier said than done.

When I run FxCop or NDepend against the source code for our commercial applications, for example, they report a large number of errors, with the majority coming from generated code. The reality is that no tool I have used can consistently distinguish between unmaintainable code written by code generators and unmaintainable code written by programmers, automatically. Therefore, in order to get the maximum benefit from static analysis, we have some additional work to do in order to help the tools look at only the code where their analysis will return useful information for developers.

Excluding Generated Code From Static Analysis

The .NET framework includes an attribute, GeneratedCode, which is intended to tell tools such as FxCop to ignore parts of your code. Also, FxCop has its own attribute, SuppressMessage which instructs the tool not to report individual messages in specific cases, and individual tests can be turned off. CodeHealer can be told to suppress a certain number of messages, and you can turn individual tests on and off.

Even with these tools, however, the line between generated and non-generated code is not always clear. FxCop, for example, looks only at compiled assemblies, not at source code. If you have a partial class containing both generated code and human-written code, it’s important to recall that FxCop cannot tell the difference between the parts of the class which came from the generated code file and the parts which came from the non-generated file. The distinction simply does not exist in the compiled assembly, because "partial" classes are combined at compile time by the C# compiler. So you’re left with the choice of putting the GeneratedCode attribute on the entire class, causing static analysis to miss code you wrote, or not putting it on the entire class, resulting in static analysis errors from the generated code. The best solution is probably to keep any code you write as part of a partial class as simple as possible, such as calls into another class which does the real work. Considering that partial classes are typically used for things like forms, or LINQ to SQL types, this is a good practice anyway.

Code generation can also work by producing IL or assembly directly, without a source file at all. Obviously, tools which look only at source code, like CodeHealer or Microsoft StyleCop will miss this sort of code (which is a good thing) by their very design. But tools which reflect into compiled code will see this sort of code as no different from any other code in the assembly unless they are specifically designed to exclude it.

For example, when you have an anonymous type with a large number of members in C#, the compiler will generate (in IL) a constructor with an equivalent number of arguments. NDepend flags these as excessive numbers of arguments, and, indeed, it would be correct if this were a real method which I had written.

Improving Heuristics to Avoid False Positives

NDepend offers an additional means of filtering data. The tool has a domain specific language for querying your source code, called CQL, and you can use this to filter the specific areas analyzed by NDepend. I can modify the NDepend rules by editing the CQL, so I can, for example, exclude the cases I’ve just described (generated constructors with large numbers of parameters) by changing the default rule for "too many arguments," which looks like this:

// <Name>Methods with too many parameters (NbParameters)</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE NbParameters > 5 ORDER BY NbParameters DESC
// METHODS WHERE NbParameters > 5 might be painful to call and might degrade performance.
// You should prefer using additional properties/fields to the declaring type to handle
// numerous states. Another alternative is to provide a class or structure dedicated to
// handle arguments passing (for example see the class System.Diagnostics.ProcessStartInfo
// and the method System.Diagnostics.Process.Start(ProcessStartInfo))
// See the definition of the NbParameters metric here http://www.ndepend.com/Metrics.aspx#NbParameters

to this:

WARN IF Count > 0 IN SELECT TOP 10 METHODS 
    WHERE NbParameters > 5 
        AND !IsGeneratedByCompiler
    ORDER BY NbParameters DESC

…and the false positives go away. Unfortunately, this only fixes one test. Similar CQL appears in other default NDepend metrics, like this one:

// <Name>Quick summary of methods to refactor</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS /*OUT OF "YourGeneratedCode" */ WHERE 

                                           // Metrics' definitions
     (  NbLinesOfCode > 30 OR              // http://www.ndepend.com/Metrics.aspx#NbLinesOfCode
        NbILInstructions > 200 OR          // http://www.ndepend.com/Metrics.aspx#NbILInstructions
        CyclomaticComplexity > 20 OR       // http://www.ndepend.com/Metrics.aspx#CC
        ILCyclomaticComplexity > 50 OR     // http://www.ndepend.com/Metrics.aspx#ILCC
        ILNestingDepth > 4 OR              // http://www.ndepend.com/Metrics.aspx#ILNestingDepth
        NbParameters > 5 OR                // http://www.ndepend.com/Metrics.aspx#NbParameters
        NbVariables > 8 OR                 // http://www.ndepend.com/Metrics.aspx#NbVariables
        NbOverloads > 6 )                  // http://www.ndepend.com/Metrics.aspx#NbOverloads
     AND 

     // Here are some ways to avoid taking account of generated methods.
     !( NameIs "InitializeComponent()" OR
        // NDepend.CQL.GeneratedAttribute is defined in the redistributable assembly $NDependInstallDir$\Lib\NDepend.CQL.dll
        // You can define your own attribute to mark "Generated".
        HasAttribute "OPTIONAL:NDepend.CQL.GeneratedAttribute")

These must all be fixed individually. On the other hand, at least they can be fixed by easily changing the heuristics of the rule, instead of by excluding specific cases, as with many other static analysis tools.

One Time Pain

The real win from static analysis tools comes from integrating them into an automated build process. Tools like FinalBuilder and MSBuild can easily be configured to run all of the tools mentioned so far. This allows you to fail the build if potentially unmaintainable code is checked in. The first time you use any static analysis tool, you will probably need to spend a substantial effort in order to get the project to the point where it will not fail the build with existing code. This may involve turning off individual rules (you can gradually turn them on later), fixing specific violations, and excluding code such as generated code. You will then need to integrate the tool into your automated build process. This is, in all honesty, a fair bit of work, but it’s really a one-time pain. After that, the analysis will pay you back for your effort, day in and day out.

Share This | Email this page to a friend

Share This | Email this page to a friend

These are very bad approaches, for two important reasons:

1. They don’t work. If your action result is cached by ASP.NET, then the action will not even run the next time it is requested. AuthorizeAttribute handles this correctly. Code you write in a Controller cannot handle this. Code you write in a custom action filter could work, if you cut and paste the implementation from AuthorizeAttribute. But AuthorizeAttribute is unsealed, so why cut and paste, when you can subtype?
2. The modularity is wrong. You should aim to develop MVC sites which can be used with any authentication (or role) provider, whether it is ASP.NET membership, domain authentication, OpenId, or a custom membership provider. Wiring authentication concerns into a Controller makes this extremely difficult.

I’ve already hinted at the correct solution for the problem: If you cannot find an out of the box membership provider which does what you need, then look harder. If you still can’t find one, write your own. Here is a video and an article on how to do it. Similarly, the role provider can also be customized. In other words, the correct extensibility point for membership in an MVC site is exactly the same as the extensibility point for membership in a non-MVC ASP.NET site.

AuthorizeAttribute does a very simple thing in a somewhat complicated way. The simple thing it does is to tie the MVC framework into the existing ASP.NET membership system, which is quite customizable. That’s why I am skeptical about claims that AuthorizeAttribute can never serve someone’s needs. The implementation is more complex than one might imagine is required on first consideration because it must both be thread safe and cooperate with caching, which makes attempting to re-invent it at a different level a doubly bad idea.

Although AuthorizeAttribute doesn’t do very much, you may decide that it’s specific behavior is not correct for your project. In that case, it is generally a better idea to subtype AuthorizeAttribute instead of re-inventing your own. This allows you to preserve desirable behavior (like cooperating with caching) while customizing the behavior you need. You will need to override the AuthorizeCore method, taking careful note of the comments (in the MVC source code for the inherited method) on thread-safety.

One final word about customizing security: Your general approach should be to use proven, off the shelf, actively maintained solutions and customize as little as possible. If the problem you are trying to solve is not strictly security-related (like adding a custom field to a user profile), then do not alter (and potentially compromise) your security in order to solve it. Instead, find a more appropriate solution, like storing the information elsewhere, such as in an ASP.NET profile or your own DB. Similarly, don’t use something which is not intended to have hard security, like Session, for security purposes. Attackers can and do hijack session information. Doing security correctly is hard, and the consequences of small mistakes can be high. Even experts in the field make mistakes which can take years to uncover. Even if you follow to the letter everything which is known about implementing good security, today, your implementation may be obsolete tomorrow.

Share This | Email this page to a friend

How to Think Like the ADO.NET Entity Framework

(and why you might want to bother learning to do so in the first place)

In order to use the ADO.NET Entity Framework effectively, you must understand its value-based data model, which is distinctly different from the SQL and ORM approaches to representing and querying data. Programmers who approach the Entity Framework as "just another ORM" often find themselves frustrated or disappointed, a common reaction to trying to force a square peg into a round hole. This presentation examines the Entity Data Model in-depth, explaining where it fits into the context of a larger application, and demonstrates different techniques for retrieving and altering data. Also covered are performance considerations and future directions.

Sound like something you’d like to learn about? The conference is free, but you must register to attend. I’ll be on at 5:00 PDT; you can convert that to your local time.

Embarcadero has me in the Delphi Prism track, but my examples will be in C#.

If .NET’s not your cup of tea, there are plenty of other sessions you can join.

Update: The conference is over, but you can download a recording of my session.

Share This | Email this page to a friend

If you can persuade someone to use your modified compiler, or if you can modify the compiler on their machine, then you can infect the executables they produce, even if they recompile the compiler itself from source code! If that seems unlikely or difficult, read the whole paper. It’s easier than you think. Thompson concludes:

The moral is obvious. You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untested code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.

Now it seems that somebody has actually gone and done this with old versions of Delphi. Frankly, I’m surprised it took so long. Perhaps this is just the first time we’ve noticed. But the problem described by Thompson is so insidious that in the quarter century since he gave his address (which really just gave additional prominence to an issue discovered decades before that) it’s almost remarkable that this technique is not more widespread. It’s also interesting that this specific case is relatively innocuous; the virus does nothing more than reproduce itself, and only affects a very old compiler. Is this a test shot? Someone whose curiosity got the better of them?

I don’t know. But the historical perspective is clear: This is not about Delphi, and it’s not about the malware of the week. It’s an old question of trust becoming new again.

Here is one proposal for countering these attacks. Interestingly, the technique can sometimes find unintended compiler bugs, as well. They can be hard to distinguish sometimes!

Share This | Email this page to a friend

Share This | Email this page to a friend

Change Manager 5.1.1 will add support for InterBase 2007 and 2009 databases to: compare, version, and synchronize schemas; compare, synchronize, move, and mask data; and compare, version, and monitor version configurations for compliance and tracking.

And for any Firebird SQL users out there, you can join the party too because Change Manager will support versions 1.5 and 2.0 of that database. Move your data to and from any platform we support and move and compare schemas between Firebird and InterBase.

Also, it’s worth noting you can get Change Manager for free with the purchase of another Embarcadero tool, including Delphi or InterBase.

Share This | Email this page to a friend

Warning: The following comic has been demonstrated to severely lower developer productivity upon first introduction.

Sydney Padua has done the impossible: Created a programming comic which I like as much as XKCD. You can start at the beginning or read the latest, thrilling episode. She also did a special for the BBC. Navigation is a bit awkward (look for the "next" and "back" links above each post), but entirely worth the effort.

Share This | Email this page to a friend

The Give Camp organizers found charities who needed web development donations and split the registered developers into teams based upon skill sets and organization needs. I was assigned to the team for the Shiloh Family Institute College Choir Concert Series, a small organization which sponsors a series of gospel choir performances once a year. They needed a web site to promote the organization, list upcoming events, allow their patrons to register for e-mail alerts of scheduled concerts, and think their donors. Critically, the site had to be easy to maintain for administrators without computer expertise. My team included Brahma Ghosh, Brian Jackett, Joel Broughton, Joseph Ours, and Peter Vasys. Prior to the start of Give Camp, Brahma, the project manager, met with representatives of the organization and produced a mockup of each page we would build in the site.

We decided to build the sites using (non-MVC) ASP.NET, because that was the "least common denominator" of the skill sets of our team members. The requirements for the site specified displaying 10-20 concerts and around 10 sponsors. In view of the tiny data set and the emphasis on ease-of-use for the folks who will be maintaining the site, we decided against using a database or off-the-shelf content management system. Instead, we built the site such that the content areas would simply be editable in a WYSIWYG editor if you are logged into the site. So the site maintainers only needed to be instructed as to how to log in, and the rest is obvious.

Here is a part of what a logged-in user sees on the site homepage; you can compare that with the production site in order to see the difference.

WYSIWYG editor on site home page

At the beginning of Friday night, we met our clients, discussed the requirements for the project, settled on which technology we would use to build the site, and got to work. One of the first things we needed to do was to register a domain name and secure hosting so that the site could go live on Sunday. Also, we set up a Subversion repository on assembla.com. Then we split up the work and started writing code. The project manager created a solution for the project and checked it into subversion. I started creating repositories for the data which would be editable on the site. Because each dataset would only hold 10 through 20 records maximum, we decided to use XML files instead of a database to hold the data. Performance is not a problem with such a small dataset, and using files instead of a database allows non-technical users to easily move the site from one hosting provider to another. Also, we used the repository pattern, so it’s possible to switch out the storage without changing a line of web client code in the event the site ever grows beyond its current scale. The first thing I did was check in the public interfaces for the repositories so that the people creating the web pages could start work before I was done with the implementation. Other team members created web pages, started the site design, and began implementing a Google Maps feature on the "upcoming concert" page.

We wrote unit tests for all of the repositories in the site. In view of the very short amount of time we had to develop the site, that might seem surprising. In fact, however, writing unit tests allowed us to test the repository functionality before the web pages which used the repositories were finished, so even though it was a little extra work, it improved our ability to work concurrently.

I went home to get some sleep around 2 a.m. Friday night, but some people worked through the whole night. The Give Camp organizers supplied us with food, caffeine, and entertainment. I spent most of Saturday implementing a feature to allow the site maintainers to add and remove sponsors, including a feature to upload sponsor logos and thumbnail them to the correct size for the site. By the end of Saturday, just after midnight, all of the major features of the site except for the Google Maps were working, and we were down to fixing bugs and typos. A pleasant surprise for me was that the food provided was consistently good and occasionally even healthy; the Give Camp organizers had recruited some local restaurants as sponsors, and a grateful charity brought in some large containers of fruit.

Incidentally, there are some photos of Give Camp on Flickr.

We spent the first half of Sunday cleaning up defects, and the second half testing and fixing typos in the content. Some of the other team members implemented the Google Maps feature in two different ways in order to be sure that we would have at least one which worked before we deployed the site to production.

At 4:00 p.m. on Sunday, all eight teams presented their work. There was a real diversity of projects. In our case, we built a simple site from scratch and finished it the same weekend. Other teams added new features to existing sites. One team was given three or four different sites using different technologies with the instructions to make them have a common look and feel. Every one of the clients appeared completely delighted with the end results.

Whenever I read "studies" which claim that the majority of software projects fail, I have to balance these claims against my own personal experience, which demonstrates that projects can be successful even with very little time and team members who have never spoken to each other before the instant the project begins. So why did this project work out so well?

• Our project manager did excellent work up front with the client, so we knew exactly what we needed to build.
• The client had a good sense of priorities for the project.
• The team was highly motivated. We were all participating by choice. The Give Camp organizers kept the emphasis on fun. The deadline was challenging, but realistic.
• The Give Camp organizers did a great job splitting up the teams. We had people with project management skills, design skills, and development skills.
• We used off-the-shelf functionality where appropriate. For example, the mailing list registration is handled via the MailChimp API. The map feature is done with Google Maps. Event photos are hosted on Flickr. We used their API to show a single image when there was only one image of an event available, and a slideshow when multiple images were available.
• We built only what was needed. Using an off-the-shelf CMS would have delivered more features, but at the cost of being difficult for the client to maintain, both in terms of the user interface and a database on the back end. The client made it clear that ease-of-use was paramount, and we stuck to that.
• There was no dead weight on the team. A single, unmotivated developer can slow down an entire team. Since everyone was there by choice, and everyone was qualified for the work, we didn’t have to deal with any problem team members.
• We had our choice of technologies, so we could choose a platform with which everyone on the team had at least some familiarity.

Give Camp was the most fun I’ve had coding in some time, even though (non-MVC) ASP.NET is not my preferred platform. The emphasis on fun, the opportunity to try things outside of my comfort zone, and the excellent team atmosphere made it a great experience.

Share This | Email this page to a friend

Given that calling Skip(0) introduces a performance penalty, but there is a reason for introducing that penalty, should we avoid making this call? The short answer is yes, the performance improvement is worth it, but don’t forget the rule that you should have one test case for every conditional in your code. This rule is generally applied to unit testing, but in this case the appropriate test is an integration test, since you need to be sure that the LINQ provider will perform acceptably and not fail altogether when you fetch the data for the second page, which is now an entirely different query than that used to retrieve the data for the first page.

When I first observed the issue, I was happy to discover that I could make the query which fetches data for the first page of a result set (which happens to be the page which users most commonly request) 10 times faster. Of course, the other way to look at this is that the query that fetches data for the second and all subsequent pages is 10 times slower than the query which fetches data for the first page. Again, given that the first page is displayed much more frequently, that’s not necessarily a bad thing, as long as it doesn’t catch you by surprise. Indeed, calling Skip(n), where n > 0, and getting your results not quite as quickly as if you had not called Skip(n) at all is actually the best case; the worst possible case is that the LINQ provider will generate invalid SQL and the query will fail.

Note that it is especially difficult to ensure complete integration testing coverage as you introduce more and more ways that the user can shape a result set. If the user can change the sort order, enter a variety of search queries, etc., it’s a safe bet that every combination of these options will produce a different query with different performance characteristics. Avoiding calling Skip(0), in other words, does not increase the number of integration test cases for a particular list of data by one; it doubles it.

Share This | Email this page to a friend

Share This | Email this page to a friend

Share This | Email this page to a friend

Unfortunately, it is not just your own code you need to be concerned with. It turns out that some fairly popular LINQ helpers do this. For example, Rob Connery’s PagedList<T> type does it:

public PagedList(IQueryable<T> source, int index, int pageSize)
{
    this.TotalCount = source.Count();
    this.PageSize = pageSize;
    this.PageIndex = index;
    this.AddRange(source.Skip((index - 1) * pageSize).Take(pageSize).ToList());

Index, in this code, is the 1-based page you’d like to display. When index is 1, the code calls Skip(0), and you get less efficient SQL. Fixing the problem is quite simple:

    this.PageIndex = index;
    var pageData = index > 1 ?
        source.Skip((index - 1) * pageSize).Take(pageSize) : source.Take(pageSize);
    this.AddRange(pageData.ToList());

Naturally, this only optimizes SQL generation for the first page of results, but since this is by far the most common page a user might display, it can net you a substantial win in server performance.

Troy Goode’s updated PagedList type, which I have recommended in the past (and still do, with this tweak!), has the same problem. The solution is almost identical, although Troy’s type is 0-based rather than 1-based. Update: Troy has now posted an update containing this fix, among many other changes.

The root of the problem in the SQL generation is that when you call Skip(0), LINQ to Entities, in at least some cases, generates SQL like this:

)  AS [Project3]
WHERE [Project3].[row_number] > 0
ORDER BY [Project3].[TimeRecordDate] DESC

The reference to row_number can have a substantial performance impact. In one case I tried with a SQL Server table containing just over 1 million rows, a simple query ran with sub-second performance without this WHERE clause, and over 4 seconds with the WHERE clause, even though the result set was exactly the same.

Now, is this a bug in LINQ to Entities? Or LINQ itself? It’s hard to say. Both frameworks are doing exactly what you tell them to, even though it doesn’t make a lot of sense. It seems like a missed opportunity for optimization, but I’m open to the possibility that there might be some desirable effect of this that I have simply missed. For my own purposes, I am content to simply not call Skip(0) at all.

Share This | Email this page to a friend

It’s worth a listen if you’re interested in the history of the product which started Borland and evolved into Delphi.

Share This | Email this page to a friend

Download the demo solution: DayPilot ASP.NET MVC demo

We needed a calendar control for an ASP.NET MVC application. In particular, we needed a control which would show a number of events and when they occur within a particular day, "Outlook-style." I’ll include a screenshot of the demo application here, since it’s the best way to explain the type of control we were looking for:

DayPilot example

Possible Alternatives to DayPilot

In my experience, the type of control which works best with ASP.NET MVC is a control which is completely ignorant of ASP.NET, and all the lies such as ViewState and postbacks which accompany it. In particular, I’ve had a good deal of success with controls designed for the jQuery JavaScript framework, which is included, by default, with ASP.NET MVC applications. Unfortunately, I was unable to find a jQuery control which data what we needed. The closest I found was jMonthCalendar which cannot display a single day in the manner illustrated above. I would still prefer a jQuery-based solution if one ever turns up.

I did, however, find a couple of controls designed for "plain" ASP.NET which, from an end-user point of view, it exactly what we needed. One was the DayPilot control illustrated above, the other was the scheduler component included with Telerik’s control suite. Telerik’s control claims some support for MVC, but this is really limited to loading calendar data asynchronously, which was not actually what we wanted to do. Also, Telerik’s control can only be purchased as part of their suite, which is rather expensive if you only need the calendar/scheduler control. Paying for the entire suite would not be a problem for us if it was filled with controls we needed, designed for the environment (MVC) we use. While Telerik seems serious about supporting MVC, they are in the early stages right now, so it is difficult to justify paying for a large suite in order to get a single control, which I would have to use in a mostly-unsupported way. The Telerik control has many more features than the DayPilot control, and, in my opinion, looks a little nicer.

The DayPilot calendar is available in two different editions. There is an open source edition, which is free, and a non-open-source edition, which adds features. The principal difference is that the date version includes editing (via postbacks) and UpdatePanel support, which are not really useful in an ASP.NET MVC application anyway. As with Telerik’s control, I have to adapt the control for use within ASP.NET MVC. So we selected the open source version, even though the paid version is priced reasonably.

Adapting DayPilot to ASP.NET MVC

In order to use the DayPilot control in ASP.NET MVC, I wanted to use markup which would look familiar to ASP.NET MVC developers:

<%= Html.DayPilot(Model, new DayPilotViewOptions
                             {
                                 HourHeight = 30
                             }) %>

This means that I needed to write an Html helper which will render the control based on the data passed in Model and return the literal HTML markup. In order to make that easier, I created two new types, DayPilotData and DayPilotViewOptions, in order to encapsulate access to the properties of the control. The reason I created two different types was to separate the concerns of data for the events (which will be set up in the controller) and display of the control (which is the concern of the view). The types are trivial, so I’m not going to show them in this post, but you can examine them in the demo solution. There is an overload to this Html helper which does not include the options argument in case you are happy with the defaults.

        public static string DayPilot(
            this HtmlHelper helper,
            DayPilotData model,
            DayPilotViewOptions options)
        {
            var calendar = new DayPilotCalendar();
            if (model != null)
            {
                model.CopyTo(calendar);
            }
            if (options != null)
            {
                options.CopyTo(calendar);
            }
            var sb = new System.Text.StringBuilder();
            sb.Append("<div class=\"dayPilot\">"); // allows working around td cellpadding bug in css
            using (var sw = new System.IO.StringWriter(sb))
            {
                using (var tw = new HtmlTextWriter(sw))
                {
                    calendar.RenderControl(tw);
                }
            }
            sb.Append("</div>");
            return sb.ToString();
        }

Note the div. The current version of DayPilot astonishingly, as of this writing, does not put a class anywhere in its rendered markup. Since the stylesheet for the default ASP.NET MVC site conflicts with DayPilot, to some degree, I wrapped the whole control in a div so that I could fix the layout in CSS.

Another way which I could have used the control in ASP.NET MVC would be to register a tag prefix and use code behind/code generation, as with standard ASP.NET. I’ve registered a tag prefix in the demo solution in case you want to try that yourself. Be aware, however, that setting up codebehind and code generation files in an MVC application is somewhat less than simple; doing it right requires, among other things, manual editing of the csproj file. So I do recommend using the control in the way I’ve illustrated.

Finally, here’s the source code for the action which supplies data for the control. Since this is a demo solution, I’m generating random events:

        public ActionResult Index()
        {
            var r = new Random();
            var q = from i in Enumerable.Range(1, 4)
                    let start = DateTime.Today.AddHours(r.Next(47))
                    select new DayPilotDataItem
                    {
                        Id = i,
                        Start = start,
                        End = start.AddHours(1),
                        Text = string.Format("Event {0}", i)
                    };
            var model = new DayPilotData
            {
                StartDate = DateTime.Today,
                Days = 2,
                Data = q
            };
            return View(model);
        }

Of course, it’s just as easy to select data from a database.

The current version of DayPilot has a kind of a weird limitation: You cannot make it smaller, in the vertical dimension, than 30 pixels per hour. Based on some inspection of the source code, I think this is fixable if you are comfortable altering the source. But be aware of the limitation.

There are more DayPilot features which I have not covered, but that should be enough to get you on your way. If you have any questions, post them in comments, and I’ll see whdat I can do to help out.

Share This | Email this page to a friend

Finding your blogs very helpful - superbly written - just found out about jqgrid last night and have integrated your code into my project.

One thing that didn’t work for me though was the dates fix above…

adding

.ToString(System.Globalization.CultureInfo.CurrentUICulture)

onto the end of my DateTime values makes everything render Ok but when I try doing a search, it fails. Debugging shows the error is

Method ‘System.String ToString(System.IFormatProvider)’ has no supported translation to SQL.

at the TotalItemCount = source.Count(); line of the PagedList.cs file.

Any ideas?

In order to understand this error, we need to remember several important things about LINQ:

• LINQ is lazy. Creating or modifying an IQueryable with methods like Where, Take, OrderBy, etc. does not actually execute the query. The query will only be executed when you call a LINQ method which requires execution to produce its results — like Count.
• When a LINQ query expression includes something which looks like a method call, the method is not invoked directly. Instead, what is actually produced is an Expression representing that method. If the LINQ provider happens to be LINQ to Objects, the method will eventually be invoked when the query is executed. But if it happens to be LINQ to SQL or LINQ to Entities, the method may never be invoked, and instead might be replaced (if possible) with equivalent SQL.
• Most LINQ providers do not support every possible LINQ query expression. Most providers don’t even support the entire LINQ API, and only LINQ to Objects can be presumed to support all method calls on an object in a query. In some cases, as with LINQ to Entities, this is documented. With other providers, you may have to guess/experiment. Any LINQ provider which has to translate a LINQ query expression into SQL syntax can only really support methods which it is hard-coded to recognize. Worse, it is not even uncommon for providers to support one overload of a particular method but not a different overload of the same method. See, for example, the LINQ to Entities document I just linked or the ToString method, which is generally supported in LINQ to Entities/SQL, but has a special overload on the DateTime type which is not supported. Worse still, method calls may be supported in some contexts but not others. Graeme reports that LINQ to SQL accepted the ToString overload in the context of a Select, but not in the context of a Where. This makes sense from an implementation point of view, but means programmers have to be extraordinarily careful about what we include in our LINQ queries, especially when they are constructed dynamically.

By now it should be obvious what is going on here: The particular overload of ToString which I used in my example is supported in LINQ to Objects but not in LINQ to SQL in a Where expression. My demo solution used a repository based on LINQ to Objects, so the query I showed worked correctly there. But it does not work with LINQ to SQL. Because LINQ is lazy, you will not actually see an error about this until the query is translated into SQL by the LINQ provider, and that does not happen until the call to Count in PagedList.

This means that when you write a LINQ query expression using LINQ to SQL, LINQ to Entities, etc., you are essentially working in a partially dynamic language. Unless you study the documentation very, very carefully, you will not know for certain until runtime if your query will be accepted by the provider.

So how could we work around this problem?

• Use a different method or a different overload which is supported by the LINQ provider used in your project. This is fine if an appropriate method/overload exists.
• Give up on doing the formatting in C# and use JavaScript instead. This will work, but it’s a bit like using beheading to cure acne. C# is, after all, a general-purpose programming language. We should not have to give up on formatting a date.
• Bring the query into LINQ to Objects using a method such as AsEnumerable. Obviously, we would want to apply the paging and ordering first. In this method, we would execute one query (using LINQ to SQL or something) which would retrieve an ordered list of records for the specified page, using AsEnumerable to transform the results into a List. We would then execute a LINQ to Objects query against this list in order to perform the formatting. This will actually work OK, although it can end up being a good bit of code to write for what should be a fairly simple task.
• Write a new LINQ provider which understands the method in question. This seems like an enormous amount of work, but I’ll include it for completeness.
• Separate the concerns of serialization and querying. You can tell by the 50-cent words that this is the solution I prefer.

Remember, the original query worked just fine. It is only the serialization to JavaScript which caused unexpected results. One way to fix this is to hide the members which are problematic to serialize and replace them with a string version of same:

private class JavaScriptDateFixer
{
    [NonSerialized]
    internal DateTime _date;
    public int Id { get; set; }
    public int IntProperty { get; set; }
    public string StringProperty { get; set; }
    public string DateProperty { get { return _date.ToString(System.Globalization.CultureInfo.CurrentUICulture); } }
}

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new JavaScriptDateFixer
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    _date = entity.DateProperty
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that even though this code looks quite different, this does almost exactly the same thing as the code I’ve replaced. The only real difference is that I’ve delayed the call to ToString until after the query is executed. That’s enough to keep LINQ to SQL or LINQ to Entities happy.

So with this fix is everything fine? Maybe. I really like LINQ; I think it’s an extraordinarily powerful tool. But here we have hit upon what I consider to be the single biggest problem with LINQ. The combination of very limited support for all possible query expressions in common LINQ providers like LINQ to SQL and building queries dynamically can be very dangerous. Unless you test all possible query permutations, you cannot be completely confident that any query you build will actually execute successfully. In typical dynamic language programming, you mitigate this problem by writing unit tests. But unit tests are not supposed to connect to a database. Unfortunately, the limitation here is that the point of translation to SQL. So any unit test which mocked the LINQ provider, say, using LINQ to Objects or some other in-memory representation would not actually encounter the limitations we are trying to test.

I have yet to see a really good solution for unit testing LINQ to SQL or LINQ to Entities, one which would combine the desired attributes of not actually connecting to the database while surfacing any potential problems in SQL generation. I suspect that such a beast may never exist for LINQ to SQL, but for LINQ to Entities it seems at least possible. One could write an Entity Framework provider which returns mocked instances of entities. Any LINQ syntax not understood by the Entity Framework would be caught before the mock provider was ever invoked. This is a feature I would like to see in a future version of the Entity Framework.

Share This | Email this page to a friend

Share This | Email this page to a friend

The delete feature of jqGrid is, oddly, almost entirely undocumented, even though there quite a few examples of different methods of editing, and inserts are documented, to a lesser extent. But it does exist; I just had to read the source code to figure out how it works.

JqGrid has three different methods of editing data. The cell edit feature allows the user to edit a single cell the grid at a time, and saves the data when the cell loses focus. The row edit feature works similarly, but does not save the data until the row loses focus. The form edit feature shows a modal dialog instead of editing the values inline. Of these three, the form edit feature is the only one which supports deletes. However, you can still use one of the other "modes" for editing, if you like. Just be aware that when you do a delete, you will be in the form edit code. Also, deletes won’t work if you change the grid’s configuration to disable the form edit feature.

In order to allow the user to delete a row, I need to:

• Add a delete button to the grid’s toolbar. I could, of course, use a custom button or something instead of the grid’s toolbar if I chose to do so.
• Specify a URL for the delete action
• Implement a delete action on the controller

I can accomplish the first two tasks by making a slight change to the JavaScript that configures the grid:

}).navGrid(pager, { edit: false, add: false, del: true, search: false }, {}, {}, {url: "Delete"});

I’ve highlighted the parts I’ve changed in boldface. Changing the "del" property to true turns on the toolbutton. The fifth argument is an object specifying options for the delete action, of which I have only specified the URL. This is important: The only places you can specify the URL for delete are in the arguments to the delGridRow method or in the toolbar’s configuration. There is not a "delete URL" property of the grid itself.

If you don’t specify a delete URL in this way, the grid will presume that you want to send delete requests to the URL you have specified for editing rows. There will be an "oper” value in the submitted form indicating that the request is a delete rather than an edit. But I prefer to use a separate URL for delete, because I think that fits more naturally in the ASP.NET MVC paradigm.

Now let’s write an action to handle the delete:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Delete(int id)
{
    var repository = new Repository();
    var deleted = repository.Delete(id);
    if (!deleted)
    {
        Response.StatusCode = 500;
        return Content("Record not found.");
    }
    return Json(true);
}

Again, I’ll note that I’m using a mock repository for this demo so that you can compile and run the solution without needing to set up a database. Since the mock repository is based on IList, the Delete method of the repository returns a Boolean indicating whether or not the requested record was found, since that is how IList behaves. It would probably be more correct to return a partial view in the event of an error rather than specifying the contents directly, but error handling is a subject I hope to cover in more depth later on. For the time being, simply changing the StatusCode will tell the grid when something does not work.

If the delete was successful, I return "true." By default, the grid ignores returned data when the operation is successful. However, you can handle the grid’s afterSubmit event, which will be passed any data you do return. So what you choose to return, and what you do with it, is entirely up to you.

With all that in place, deletes work as expected. I’m not going to update the demo solution just yet, because I’ll be covering edits and inserts real soon now. But first I need to revisit formatting and explain more about LINQ".

Share This | Email this page to a friend

First, let’s fix the date display. The reason the date appeared so oddly is that there is no official support for Date types in JSON, even though JavaScript has a date type. Microsoft, therefore, invented their own way of encoding dates for their JSON serializer, which jqGrid does not understand. There are two possible ways to fix the problem. The first is to keep sending the data to the grid in the Microsoft format, and write a custom formatter for the grid to parse that into a human-readable string. The second is to change the data we send to the grid. Both ways work, but I’m going to demonstrate the second method, mostly because I find C#’s date handling features superior to (and less confusing than) JavaScript’s.

In order to change the date "column" in the results of the query to a formatted string, we simply need to project into an anonymous type. However, this would break the sort, because a text sort on a formatted date will not be in the correct order. Hence, it is necessary to apply the sort before projecting. So I add an OrderBy to the query, and pass null as the orderBy argument to the ToJqGridData method.

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    DateProperty = entity.DateProperty.ToString(
                        System.Globalization.CultureInfo.CurrentUICulture)
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that in this revised solution I’ve updated the mock repository to return more sensible dates.

Important update: This code works correctly with LINQ to Objects, as with my demo repository, but will not work with LINQ to SQL. I have addressed this issue in a subsequent post.

Now let’s add a search feature. One of the problems in making a "demo" search is that search is a highly application-specific feature, both in terms of implementation and in terms of user experience. JqGrid’s search feature presumes that you want a search which looks something like this:

Visually, there are lots of variations on this theme available, but in terms of the use case they all presume that the user will specify which fields she would like to search. However, I prefer a "Google-style" interface with just a search box and an implementation that can (hopefully) figure out what the user actually wants. So that’s what I’m going to demonstrate. The needs of your application may be different. The important thing to take away from this demonstration is that the jqGrid search feature can be co-opted to fit your design.

I want the search box to appear above the grid at all times, rather than requiring the user to press a toolbutton to see it. So first I need to add an empty div to the markup in order to indicate where the search box will be positioned, relative to the grid:

    <h2>GridDemo</h2>
    <div id="search"></div>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>

Next, I add a small bit of JavaScript to tell jqGrid to replace this div with a search control:

    search.filterGrid("#" + grid.attr("id"), {
        gridModel: false,
        filterModel: [{
            label: 'Search',
            name: 'search',
            stype: 'text'
        }]
    });

The filterModel tells the grid to show a search control for one column, called "Search." Note that the entities I’m returning do not have a property called "search." That’s OK. When the user types something in the search box and presses enter, the grid will add a query string parameter to its request for the data of the form "search=WhatTheUserTyped". That’s what I want, and the fact that there is no "search" column in the data does not adversely affect the grid in any way.

The first argument to filterGrid is a little odd. I’m required to pass a jQuery selector as a string rather than being able to pass a jQuery object directly. Since this is all inside a method which accepts jQuery objects as arguments (rather than hard-coding the IDs of the DOM objects in the method, for unit-testability), I have to "decode" the jQuery object into a selector string. But note that I did not need to do that when setting up the pager or the grid itself. Like I said before, the grid’s API is consistently inconsistent.

As you can see in the code at the beginning of this post, the GridDemoData action just passes the search query string parameter through to the ToJqGridData method unchanged. Inside that method, I use Dynamic LINQ to alter the IQueryable to implement the search. I’ve cleaned up this method substantially since I first posted the code. Here’s the revised method:

/// <summary>
/// Adds a Where to a Queryable list of entity instances.  In other words, filter the list
/// based on the search parameters passed.
/// </summary>
/// <typeparam name="T">Entity type contained within the list</typeparam>
/// <param name="baseList">Unfiltered list</param>
/// <param name="searchQuery">Whatever the user typed into the search box</param>
/// <param name="searchColumns">List of entity properties which should be included in the
/// search.  If any property in an entity instance begins with the search query, it will
/// be included in the result.</param>
/// <returns>Filtered list.  Note that the query will not actually be executed until the
/// IQueryable is enumerated.</returns>
private static IQueryable<T> ListAddSearchQuery<T>(
    IQueryable<T> baseList,
    string searchQuery,
    IEnumerable<string> searchColumns)
{
    if ((String.IsNullOrEmpty(searchQuery)) | (searchColumns == null)) return baseList;
    const string strpredicateFormat = "{0}.ToString().StartsWith(@0)";
    var searchExpression = new System.Text.StringBuilder();
    string orPart = String.Empty;
    foreach (string column in searchColumns)
    {
        searchExpression.Append(orPart);
        searchExpression.AppendFormat(strpredicateFormat, column, searchQuery);
        orPart = " OR ";
    }
    var filteredList = baseList.Where(searchExpression.ToString(), searchQuery);
    return filteredList;
}

Importantly, this method is simply adding to the expression represented by the IQueryable. It does not actually implement the search. So any search methodology which understands LINQ can deal with its results. At least, theoretically. In practice, LINQ providers vary greatly in terms of which LINQ features they support. So although this method works fine with LINQ to Objects (as used in the rest of the demo solution) I cannot guarantee that it will work with every LINQ provider in the world. You have to try, and revise it to fit the capabilities of the provider you’re using.

Putting all these pieces together, I now have a search feature which does what I want:

Be sure to download the updated solution in order to get the bug fixes and new features.

As a special, no-extra-charge bonus, the updated solution also includes a demonstration of using the grid’s tree view mode with client-side data from a JavaScript function. This has nothing to do with ASP.NET MVC; it’s just an example of how to use a grid feature which I wrote in response to a request in the jqGrid support forums.

That’s all for today. In the next post in the series, I’ll begin to demonstrate the grid’s editing features, and how to use them in an ASP.NET MVC application. But I will also take requests. If there’s a grid feature you’d like to see demonstrated query question regarding using the grid in ASP.NET MVC, please feel free to make a request in comments, and I will answer it as best I can.

Share This | Email this page to a friend

Share This | Email this page to a friend

On The Podcast at Delphi.org, Nick Hodges confirms my earlier speculation about reorganization in the "CodeGear" group at Embarcadero. The whole show is worth listening to. Congratulations are due to Chris Pattinson, who was promoted to Director of Quality for all of Embarcadero. Time to update the subtitle of your blog, Chris! (And maybe even write a post or two…) Sounds like good changes all around. You read it here first, folks!

I wrote a short demo showing how to use jqGrid in tree grid node with client-side data supplied by a JavaScript function.

Share This | Email this page to a friend

Rather than make you set up the Northwind database, I’ve written a quick, mock repository which supplies random data for the application. But the data persists for the lifetime of the app, so I’ll be able to demonstrate sorting, paging, etc. This way you can just run the application, without having to set up SQL Server.

So let’s write some code display a grid.

The data I’m going to display in the grid is a fairly useless type:

    public class SomeEntity
    {
        public int Id { get; set; }
        public int IntProperty { get; set; }
        public string StringProperty { get; set; }
        public DateTime DateProperty { get; set; }
    }

We need two actions. The first will display the page containing the grid. The second supplies data for one page of the grid. Remember, the grid can page through large datasets. It will only request one page of data at a time.

        public ActionResult GridDemo()
        {
            return View();
        }

        public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
        {
            var repository = new Repository();
            var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
                new[] { "IntProperty", "StringProperty", "DateProperty" });
            return Json(model);
        }

The first action simply returns a view containing the HTML table which will eventually become the grid. It doesn’t need a model, because the grid will request its data asynchronously. The second action handles the data requested by the grid.

In order to make this work, we need HTML elements for the grid and its pager in the view, along with a reference to the JavaScript which will turn those elements into the grid.

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <h2>GridDemo</h2>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>
</asp:Content>

<asp:Content ID="Script" ContentPlaceHolderID="ScriptContent" runat="server">
    <script language="javascript" type="text/javascript" src="<%= Url.Content("~/Scripts/Home.GridDemo.js") %>"></script>
</asp:Content>

The JavaScript file is where the grid itself is configured.

$(document).ready(function() {
    GridDemo.Home.GridDemo.setupGrid($("#grid"), $("#pager"));
});

GridDemo.Home.GridDemo = {
    setupGrid: function(grid, pager) {
        grid.jqGrid({
            colNames: ['Int', 'String', 'Date'],
            colModel: [
                        { name: 'IntProperty', index: 'IntProperty' },
                        { name: 'StringProperty', index: 'StringProperty' },
                        { name: 'DateProperty', index: 'DateProperty' },
                      ],
            pager: pager,
            sortname: 'IntProperty',
            rowNum: 10,
            rowList: [10, 20, 50],
            sortorder: "asc",
            url: "GridDemoData"
        }).navGrid(pager, { edit: false, add: false, del: false, search: false });
    }
};

Important to note here are:

• colNames, the grid column captions the user sees.
• colModel,which includes the property name, which is the field name in the returned JSON data which supplies the data for that column, and index, which is the name which will be sent to the GridDemoData action as the sidx (field name to sort on) argument when you click on a grid header. There are a number of other properties you can set here which control editing, with, formatting, etc. See the documentation for full details. Importantly, colNames and colModel must have the same number of entries, and be in the same order.
• url, which tells the grid where to get its data. You’ll notice that the value I’ve passed is the name of the second action.

As I mentioned in the last post, I set a number of the grid options in a script referenced by the Site.Master, in order to prevent having to set them on every grid I create. The configuration idea for the grid on an individual page should be only the things that are unique to that page. So there is a fair bit of configuration for the grid which is not included here. In addition, the Site.Master contains references to the grid’s JavaScript and CSS files.

Finally, it’s worth mentioning that when you add jqGrid to a project you need to edit the jQuery.jqGrid.js file to tell the grid where to find the rest of the JS files it needs. Once you look at this file, it is obvious what you need to change.

At this point, I’m going to make the demo application available for download, so that you can experiment with it yourself. However, I’ll caution that I intend to update it when I write the next post in this series. In other words, the current download is a work in progress. It will change in the next few days.

Download the Solution (587 KB)

So let’s run the solution and see what we get:

OK, the hideous and eyestrain-inducing clash between the default MVC theme and the jQuery UI "lightness" theme included with the grid download notwithstanding, this is actually not bad. Sorting and paging both work, and performance is very good.

But what’s the deal with the data in that date column? It turns out that there is no standard representation of a date in JSON, so Microsoft invented their own. The grid doesn’t know about this special format. We’ll clean that up in the next post in this series and also cover search. Stay tuned!

Share This | Email this page to a friend

Nick Hodges has a new role/job title.

Item 2:

During his presentation to the Columbus Architecture Group a couple of weeks ago, Mike Rozlog handed out business cards which gave his title as "Senior Project Manager, Java Tools." He told us that in a couple of days he would have new business cards, because his role was changing. He didn’t tell us precisely what the new role would be, but I got the feeling that, like Nick, he was happy with the change.

Now, it is of course possible that these two changes happening so close together is merely a coincidence. But my personal speculation is that there are bigger plans afoot, and I’m glad to see that Mike and Nick, two people I trust, seem to be happy with them.

Repeat: This is just my personal speculation. It is based on the evidence above, not inside information. I could be wrong.

Share This | Email this page to a friend

But it turns out that we’re using the grid in a different way, and I think the difference is important. So rather than just referring you to Phil’s post and skipping the rest of the series, I think I still have some important information to contribute. In particular, I’m using the grid in a way which allows me to write an extension method for IQueryable<T> which returns data suitable for the grid, without having to know anything about the type T. If you look closely at the code in Phil’s post, you will see that populating the cell array requires custom code (or use of reflection) every time you want to return data to a grid.

It’s important to me that the method should work without knowing anything about the type of the data in the list because I don’t want to have to write repetitious code in every action method which supplies data to a grid to shape the data according to the grid’s needs, and because I generally use anonymous types for JSON serialization. JSON cannot handle object graphs with cycles (i.e., a "circle" of objects which all reference each other). The .NET JSON serializer will throw an exception if you pass it an object which contains a circular reference, or refers to another object which does. Since this is largely data-dependent and, worse, can be occasionally masked by lazy loading and the like, the safest way to serialize to JSON is to pass it data which provably can never contain a cycle. Anonymous types work really well for this. I’ll probably elaborate on this point in a future post. For the time being, the important thing is that I need to be able to take any IQueryable and turn it into data suitable for the grid, without having to think about what type that list might contain.

In a nutshell, this is what I want to be able to do:

public JsonResult ListGridData(int page, int rows, string search, string sidx, string sord)
{
    var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
      new[] { "Column1", "Column2", "Column3" })
    return Json(model);
}

In this example, repository.SelectAll() returns an IQueryable<TSomething>. I’ve written a method, ToJqGridData, which can transform that IQueryable into data suitable for the grid without having to know what TSomething is. (What the heck is that string array? Don’t worry about that just yet; I’ll cover it in a little bit.) The argument names aren’t what I would have picked, but they come from the grid. There is probably a way to change them, but I’ve never bothered to look.

The key to this is a grid property called jsonReader.repeatitems. I’m not sure why it’s called repeatitems, but here’s what it does. When repeatitems is true (the default), data is returned to the grid in a "cell array," like in Phil’s post. That is, an individual row of data in the JSON looks like this:

{id:"1", cell:["cell11", "cell12", "cell13"]}

When repeatitems is false, the data looks like a regular object in JSON format:

{id:"45678",name:"Speakers",note:"note",stock:"false",ship:"4"}

Interestingly, it turns out that this is the format the .NET JSON serializer already uses. So I don’t have to write any code to get this portion of the data into the right shape. I just hand an object to the JSON serializer and let it do its thing. The only things I need to do to make the data work for the grid is to handle paging, sorting, search, and provide certain other bits of data, like the number of rows.

But, as you may have noticed, I’m lazy. Why write code when someone else has already done it? Rob Conery wrote a decent type for paging eons ago, and Troy Goode enhanced it. So I’m going to use their work instead of re-writing paging. It turns out this type already computes most of what the grid needs to know. And Microsoft has this thing called Dynamic LINQ which can deal with ordering and filtering by property names, mostly. All I have to do is tie this stuff together.

I wrote a type to hold the data which the grid needs. You can actually use an anonymous type for this; the JavaScript grid obviously doesn’t care if the data it receives comes from an anonymous type or not, but because I am writing extension methods to IQueryable, it was convenient for me to have a non-anonymous type so that the extension methods could build on each other. This type represents one page of data requested by the grid:

namespace GridDemo.Models
{
    /// <summary>
    /// This type is designed to conform to the structure required by the JqGrid JavaScript component.
    /// It has all of the properties required by the grid. When this type is serialized to JSON, the resulting
    /// JSON will be in the structure expected by the grid when it fetches pages of data via AJAX calls.
    /// </summary>
    [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId="Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    public class JqGridData
    {
        /// <summary>
        /// The number of pages which should be displayed in the paging controls at the bottom of the grid.
        /// </summary>
        public int Total { get; set; }
        /// <summary>
        /// The current page number which should be highlighted in the paging controls at the bottom of the grid.
        /// </summary>
        public int Page { get; set; }
        /// <summary>
        /// The total number of records in the entire data set, not just the portion returned in Rows.
        /// </summary>
        public int Records { get; set; }
        /// <summary>
        /// The data that will actually be displayed in the grid.
        /// </summary>
        public IEnumerable Rows { get; set; }
        /// <summary>
        /// Arbitrary data to be returned to the grid along with the row data. Leave null if not using. Must be serializable to JSON!
        /// </summary>
        public object UserData { get; set; }
    }
}

Note that I’ve used C# capitalization for the property names. This is not what the grid expects. But I have to tell the grid that I’m using a non-default value for repeatitems anyway, so it’s nice to follow C# conventions.

We have lots of grids in our application, so rather than configuring each grid individually, we set defaults for the grid in one place. In a JavaScript file referenced by the Site.Master, this method is called in the jQuery ready event:

    setDefaults: function() {
        $.jgrid.defaults = $.extend($.jgrid.defaults, {
            datatype: 'json',
            height: 'auto',
            imgpath: '/Content/jqGrid/themes/basic/images',
            jsonReader: {
                root: "Rows",
                page: "Page",
                total: "Total",
                records: "Records",
                repeatitems: false,
                userdata: "UserData",
                id: "Id"
            },
            loadui: "block",
            mtype: 'GET',
            multiboxonly: true,
            rowNum: 20,
            rowList: [10, 20, 50],
            url: "ListGridData.json",
            viewrecords: true
        });
    }

In addition to setting repeatitems, telling the grid to expect the capitalization I mentioned above, this code also sets the grid theme and other options we want to be consistent throughout the application. One important property set in the jsonReader property is id. The value supplied is the name of a column in the returned data which contains a unique ID for the row. You can supply any name you want to, and the type of the property can be just about anything serializable, but it is important that the data you return to the grid contains some unique ID. All of the types in our Entity Framework model contain a unique ID called "Id", so we set this in our defaults for the grid. If you don’t have a single operating name for any data you might supply to the grid, you will have to set this value on the individual grids, when you write the JavaScript for the view.

Now I can write a method which takes a PagedList and transforms it into jqGrid data:

        /// <summary>
        /// Converts a paged list into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="list">The list of records to display in the grid.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data.
        /// Leave null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this PagedList<T> list, object userData)
        {
            return new JqGridData()
            {
                Page = list.PageIndex,
                Records = list.TotalItemCount,
                Rows = from record in list
                       select record,
                Total = list.PageCount,
                UserData = userData
            };
        }

That’s getting somewhere, but it’s not yet enough to let me write the action method I showed above. For that, I need to wrap up the PagedList inside the action so that I can pass the arguments from the grid directly, without having to construct a new PagedList inside of every action method which supplies data to a grid. So I build on the method above:

        /// <summary>
        /// Converts a queryable expression into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="baseList">The list of records to display in the grid.</param>
        /// <param name="currentPage">A 1-based index indicating which page the grid is about to display.</param>
        /// <param name="rowsPerPage">The maximum number of rows which the grid can display at the moment.</param>
        /// <param name="orderBy">A string expression containing a column name and an optional ASC or
        /// DESC. You can, separate multiple column names as with SQL.</param>
        /// <param name="searchQuery">The value which the user typed into the search box, if any. Can be
        /// null/empty.</param>
        /// <param name="searchColumns">An array of strings containing the names of properties in the
        /// element type of baseList which should be considered when searching the data for searchQuery.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data. Leave
        /// null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this IQueryable<T> baseList,
            int currentPage,
            int rowsPerPage,
            string orderBy,
            string searchQuery,
            IEnumerable<string> searchColumns,
            object userData)
        {
            var filteredList = ListAddSearchQuery(baseList, searchQuery, searchColumns);
            var pagedModel = new PagedList<T>(filteredList.OrderBy(orderBy), currentPage, rowsPerPage);
            return pagedModel.ToJqGridData(userData);
        }

Now you see that string array (IEnumerable, actually) again. JqGrid supports sorting and searching, and so do I. Like Phil, I suggest using Microsoft Dynamic LINQ in order to take the field names that jqGrid passes and turn them into a sorted, filtered, result set. ListAddSearchQuery is a (longish) method which takes a search string and a list of columns to search and turns them into a Dynamic LINQ expression and applies it to the base list. Unfortunately, search is kind of application-specific. In some applications you might want to search different columns in different ways. Some applications might want "Google-style" searching. It kind of depends upon what you’re doing. So although I will supply the source code for this method when I make the demo project available, I cannot promise that it will be useful for your application. Feel free to substitute your own method here.

It turns out both methods (the one which takes a PagedList and the one which takes an IQueryable) are useful. Generally, I use the method which takes an IQueryable. But every once in a while I want to do something to the data I’m about to supply to the grid before the grid sees it, but after paging, like transform a single page of results in some way. In this case, the action builds up a PagedList, uses that to reduce to one page of data, modifies the results in whatever way is appropriate, and then calls the PagedList extension. I’ve also written additional overloads to both methods which do not have a userData argument, since it is very seldomly used. userData, as the name suggests, is an additional bit of data you can return to the grid with your result set for use in grid event handlers when loading a fresh page of data. Since it’s of type object, you can put as much data in there as you need to, so long as it’s serializable to JSON. In practice, though, we rarely use this feature of the grid.

With these methods, it is easy to write actions as concise as the code I showed at the beginning of this post. Just get a list from somewhere and call ToJqGridData, passing information about which page to display, how to sort, etc.

In a future post, I will tie this all together into a demo application.

Share This | Email this page to a friend