MetadataException: Unable to load the specified metadata resource

The cause of this error is a missing or malformed Entity Framework connection string. In particular, the cause of this error is the metadata parameter of the connection string. Let’s examine this more closely.

Which Config File?

Before you begin troubleshooting, make sure you are looking at the correct file.

Connection strings are loaded from the configuration file for the executing assembly, which may not be the assembly which contains your Entity Framework model. So if you have a data assembly containing an Entity Framework model and a web assembly containing a web application which references the data assembly, then the Entity Framework connection string needs to be in the Web.config file for the web assembly. The data assembly can have its own connection string, if you like (this is convenient, as it will be used by the Entity Framework designer), but it will not be used by the web application at runtime.

Metadata

MSDN documents EF connection strings generally. But the cause of the error above is specifically the metadata parameter. This parameter tells the Entity Framework where to find your EDMX at runtime. When your application is compiled, the EDMX is split into three parts: CSDL, MSL, and SSDL. For the purposes of this post, you don’t need to think about what they mean right now; it’s enough to know that you need all three of them at runtime.

The EDMX can be supplied to the application as embedded resources or files on disk. The default is embedded resources, so we’ll focus on that case.

The metadata parameter for an application with an Entity Framework model called Model.edmx in an assembly called Simple Mvc.Data.dll might look like this:

<connectionStrings>
	<add name="MyEntities" connectionString="metadata=
            res://Simple Mvc.Data.dll/Model.csdl|
            res://Simple Mvc.Data.dll/Model.ssdl|
            res://Simple Mvc.Data.dll/Model.msl;provider= <!-- ... -->

So you can see there is one reference for each of the three parts of the EDMX that we need at runtime. They all work in the same way, so let’s examine just the first more closely. The CSDL reference looks like this:

            res://Simple Mvc.Data.dll/Model.csdl

It specifies three things:

1. We’re loading the CSDL from a resource. That’s the "res://" part.
2. The name of the assembly which contains the resource, "Simple Mvc.Data.dll". If your assembly is strong named, you can specify a strong name, in all its verbose glory, here.
3. The name of the resource itself, "Model.csdl". Do not confuse this with the EDMX or model name. In this case they happen to be the same, except for the extension, but that’s not always true!

Let’s examine #2 and 3 more closely.

Assembly name

It’s really common to omit the assembly name in a connect string, and use * instead, like this:

<connectionStrings>
	<add name="MyEntities" connectionString="metadata=
            res://*/Model.csdl|
            res://*/Model.ssdl|
            res://*/Model.msl;provider= <!-- ... -->

This means that instead of loading the resource from one specific assembly, the EF will scan all loaded assemblies at runtime. Two obvious downsides to this are that it takes a bit of time to do this, and the assembly might not be loaded yet. The second case is one reason you might get the error at the start of this post. So specifying the assembly explicitly is always a good idea, I think.

Resource names

Remember how I said that the resource name isn’t necessarily the same as the model name? Here’s a real-life example of that. I opened one of our production assemblies in Reflector, and found this:

There’s actually a good reason that those resources have such bizarre names, but it’s a digression and not relevant to this post. The point is that after you’re sure the assembly name is right, the next step in solving the error at the top of this post is to double-check the resource names.

A simpler resource name will look like this:

Remember the "simpler" metadata where I used * instead of the assembly name? You can go even simpler. This metadata, perhaps surprisingly, actually works (with caveats):

<connectionStrings>
	<add name="MyEntities" connectionString="metadata=
            res://*/;provider= <!-- ... -->

This is the "throw everything against the wall and see what sticks" approach to a connect string. It will probably fail if your resources don’t happen to have the same name as your model, or if the assembly doesn’t happen to be loaded. But it (in simpler cases, anyway) frees the programmer from having to think about what any of this stuff means, so it’s a popular solution.

Share This | Email this page to a friend

If you produce a tool or component supporting InterBase and have not already received a notice about the Partner DVD, then please contact Anders Ohlsson (firstname.lastname@embarcadero.com) immediately.

Share This | Email this page to a friend

Share This | Email this page to a friend

jqGrid is a JavaScript grid component with many useful features. I have previously explained how to use it with ASP.NET MVC.

jQuery BBQ is a library which helps you manage proper functionality of the web browser’s back and forward buttons by changing the fragment (hash) of the URI. Using my integration means that you can have a jqGrid on a web page and the back and forward buttons will work as expected.

To be specific, when you first load a certain page, the grid will probably be on page 1. You could click the next page button on the grid to go to page 2. Without my integration, clicking the back button would not take you back to page 1. With my integration, it will. Similarly, I support the sort and records per page combo box. You can of course extend this yourself to track other grid features.

The image above shows a demo page I’ve prepared, which you try for yourself. The grid on the left does not use the history plug-in. The grid on the right does. Note that when you page, sort, or change the number of rows shown in the grid with the history, the URI fragment also changes.

I have contributed my plug-in back to the jqGrid project. You can get the source from github.

Share This | Email this page to a friend

public class MyBaseController : Controller
{
    // ...
    public new ISession Session { get; private set; }
    // ....
}

ISession is an interface type I wrote which exposes everything I store in the session at runtime. I use constructor injection to assign this to a simple type which maps to the "real" Session during application runtime, and a "mock" session for unit testing. Session, in ASP.NET, is a user-specific cache.

This hides Controller.Session (of type HttpSessionStateBase) and replaces it with a strongly-typed, mockable replacement. Normally I wouldn’t use public new, but in this specific case it seems like the best way. It stops developers from using the Web-dependent Session property in derived types and forces them to code to the strong-typed interface. But still, it’s public new, which seems yucky.

What do you think?

Share This | Email this page to a friend

If you’ve read Getting Real (you can sample it online for free at the link above, and in my opinion it’s worth the time to do so), or if you’ve seen one of DHH’s presentations, then you probably have a good idea of what to expect in Rework. It’s short, to the point, quotable, and uses 37 Signals as the model for everything that is good about how to run a business. I don’t fault the authors for being proud of their company, but if you are looking for a dispassionate examination of when it makes sense to run a business in the way that they do versus other ways, you won’t find it here. You quickly get the idea that the authors look at other methods of running a business about as fondly as DHH looks at WS-*. Again, this isn’t necessarily a bad thing. It’s clearly worked out pretty well for 37 Signals. One can’t fail to notice, however, that 37 Signals’s products fit within a pretty narrow spectrum. Will their techniques translate to other markets and other industries? Perhaps; perhaps not. Rework doesn’t really answer that question.

Did I say quotable? It’s full of snappy, almost pithy zingers which are so obviously right that they would seem like tautologies if they were not so commonly ignored:

• Workaholics aren’t heroes. They don’t save the day, they just use it up. The real hero is already home because she figured out a faster way to get things done.
• You’re better off with a kick-ass half than a half-assed whole.
• It’s also unfortunate that meetings are typically scheduled like TV shows. You set aside thirty minutes or an hour because that’s how scheduling software works (you’ll never see anyone schedule a seven-minute meeting with Outlook).

I don’t think these are empty words. I’m convinced the authors run their company this way. I’m also convinced it works for them. They took their own advice writing the book, as well:

We cut this book in half between the next-to-last and final drafts. From 57,000 words to about 27,000 words. Trust us, it’s better for it. So start chopping. Getting to great starts by cutting out stuff that’s merely good.

But I don’t agree with everything in the book. Sometimes the cuteness of the writing overtakes the point. It’s generally true that you should focus on the essentials. But they illustrate this with:

For example, if you’re opening a hot dog stand, you could worry about the condiments, the cart, the name, the decoration. But the first thing you should worry about is the hot dog. The hot dogs are the epicenter. Everything else is secondary.

Guys, next time you’re in Columbus, look me up, and I’ll buy you lunch at Dirty Franks. I respect what you have to say about business, but we have a thing or two to settle about hot dogs.

Indeed, later in the book the authors make this very point quite elegantly, with a more apropos example:

When we start designing something, we sketch out ideas with a big, thick Sharpie marker, instead of a ballpoint pen. Why? Pen points are too fine. They’re too high-resolution. They encourage you to worry about things that you shouldn’t worry about yet, like perfecting the shading or whether to use a dotted or dashed line. You end up focusing on things that should still be out of focus.

Avoid false precision.

The point here, after all, is not how to run a hot dog business in particular; it’s how to run your business. The best way to fail if that is to let details and directions steer you away from the ideas that only you can bring to the table. This is why "look inside yourself" can be a better approach to planning and business strategy than following an off-the-shelf formula. For most people, creating a new, free, application framework is the worst possible way to start a business selling web applications. I’ve seen many, many people fail at this. Except that for 37 Signals, it worked really well, perhaps because of the specific people who ran the company.

I strongly agree with the authors when they say:

Pour yourself into your product and everything around your product too: how you sell it, how you support it, how you explain it, and how you deliver it. Competitors can never copy the you in your product.

This is why it’s OK that Rework can be self-contradictory at times. In one chapter, the authors suggest that you "pick a fight." A good example of this is how Stack Overflow was to some degree a response to "hyphen-site." Only a few chapters later, however, the authors say:

Focus on competitors too much and you wind up diluting your own vision. Your chances of coming up with something fresh go way down when you keep feeding your brain other people’s ideas. You become reactionary instead of visionary. You wind up offering your competitor’s products with a different coat of paint. If you’re planning to build “the iPod killer” or “the next Pokemon,” you’re already dead. You’re allowing the competition to set the parameters.

Self-contradictory? Absolutely. Again, Rework is not a formula for how people with no interest in widgets can start a widget business. It’s written to provoke the reader, hopefully provoking them into keeping their business focused on what they can uniquely bring to the table.

Naturally, this presumes that the reader is, to some degree, interesting, and has ideas for things which could benefit other people. It’s not clear to me if everyone has good ideas, but many people don’t execute them, or if some people don’t execute good ideas because they don’t have them. What is clear is that having good ideas and not executing them is little better than not having them.

Share This | Email this page to a friend

The conference is held in a local movie theater. After a day of technical sessions, the attendees would be treated to a screening of Iron Man 2. I wasn’t able to stay long enough for that, but the technical content was more than worth the time and the measly $25 cost of the conference. The theater was still a great place to have a technical conference. The screens were huge and the seats were comfortable.

Keynote

Molly Holzschlag was scheduled to be the keynote speaker, but could not fly due to an ear infection and doctors orders. So she gave her keynote address, and a presentation on HTML 5 later on, via Skype, which was probably better than nothing, but only marginally so, given Internet bandwidth not really sufficient for real-time video. It at least provided some amusement when poorly-timed drop outs turned innocuous technical material into profanities. The HTML 5 presentation suffered from a lack of slides, which Molly made up for by asking us to imagine HTML DOCTYPEs and the like. I appreciate Molly going the extra mile to not leave the conference in the cold, but I didn’t get a lot out of these sessions, which is too bad, because HTML 5 is an ongoing concern for me right now.

Seeing Constraints, Kanban Explained

Generally, I rank conference presentations by whether or not I come out of them with useful information or ideas that I can put into practice back at work. The Kanban session did succeed in that respect. The speaker’s style was very conversational, almost off-the-cuff. He is a process consultant and related a lot of anecdotes about situations he has worked in.

One thing he emphasized which I do not see in much of the writing about Kanban is to use multiple boards when appropriate. For example, a development team can have "their own" board for non-functional changes, which might not ever appear on a board for company-wide items. You can go crazy with this, however; the speaker described one case where something like 36 boards were in use.

This session, like many presentations on development process, tended to presume that development throughput was a bottleneck in the company. This is often the case, but not always. What to do when development is not the bottleneck is a subject which could use fuller treatment.

Understanding User Experience Patterns

Since the original notion of a "design pattern" comes from Christopher Alexander’s writings on architecture (see Patterns of Software, by Richard P. Gabriel, for more info on this), it makes sense that software companies would want to apply design patterns to more than just code structure. Although there are a number of online user experience pattern catalogs, it still strikes me that this is an area where we, as an industry, have a lot to learn.

I very much enjoyed Ambrose Little’s presentation on UX patterns. He showed examples of good and bad interfaces, and discussed why they succeed or fail. He then tied those reasons into pattern catalogs, and demonstrated several of the online catalogs and discussed their strengths and weaknesses.

jQuery 1.4: What You Need to Know

For some reason, the lunchtime sessions were not in the Stir Trek catalog, so I will link Matt’s slides instead. Matt used a lightweight framework to display his slides using HTML and JavaScript instead of a tool like PowerPoint. This allowed him to demonstrate jQuery features right in the slides. It’s a very effective way of presenting his subject matter. There wasn’t a lot of new material in this presentation, given that jQuery 1.4 has been out for a while. It was a good overview, though.

Is Mobile Development Going to Follow the Book of Jobs (Steve That Is)?

This presentation was a lot like its title. Opinionated and verbose, but with an attempt to keep things fun the whole time. The presenter has been working in the mobile space for a long time, and has more than his share of war stories. Like a lot of people in mobile work, though, it seems to me that his opinions were colored by the specific nature of the work his company does. The speaker seemed to treat browser apps and web apps as the same thing. "Native" apps can be web apps. Browser apps can be local. Web app doesn’t necessarily mean "in the browser." HTML/JS isn’t necessarily a web app, especially in the mobile space. Overall, though, I enjoyed this a lot, though I’m not sure I’m going to take practical info home with me.

JQueryUI – The Magic From Behind The Curtain

Watching this presentation was a bit like playing with the jQuery UI demo page. Still, it exposed me to some features that I haven’t seen before. The presenter kept things fun, and I left the session feeling like I’ve learned something.

Conclusion

The Stir Trek organizers, who, as far as I know, are all volunteers, did an amazing job creating an event for 600 people that cost so little and delivered so much value. I haven’t even discussed the freebies like the O’Reilly e-book and the discounts on training, etc. Don’t miss this conference next year!

Share This | Email this page to a friend

if (q.Any())
{

Similarly, you can check to see if any elements in the list meet a certain condition:

if (q.Any(i => i.IsSpecial))
{

If the query provider is something like LINQ to Entities, this will be translated into fairly efficient SQL using EXISTS.

For some reason, I see a lot of people write this code using the Count() extension method instead (maybe they don’t know about Any()?), like this:

if (q.Count() > 0)
{

This is wrong for two reasons:

• It’s imperative instead of expressive. Using Any() tells the query provider, "Please determine if the list is non-empty using the most efficient way you can." Using Count() > 0 tells the query provider, "Please do exactly what I tell you, regardless of what I really need." Because LINQ is intended to support a variety of query providers, it is important to be expressive instead of imperative, and to let the provider specialize the implementation.
• Because we don’t actually care about the exact count of items in the list, only that it is greater than zero, using the Count() function can cause the provider to do considerably more work than necessary. Consider a linked list, or a SQL-based provider.

Share This | Email this page to a friend

Perhaps surprisingly, it’s actually very simple. (I say "surprisingly" because much of the math and technical papers on encryption is decidedly not simple, including that of Gentry’s first fully homomorphic scheme, which was based on ideal lattices.)

This scheme created by Marten van Dijk, Craig Gentry, Shai Halevi and Vinod Vaikuntanathan, however, uses basic arithmetic operations like division, with a few uncommon, but easy to understand, variations.

The nature of homomorphic encryption also dictates a programming style which will seem unfamiliar to users of most high-level languages, but very familiar to anyone who has ever taken a digital circuits class.

Integer Division

What is the integer quotient of 10 / 6? Most people would probably say "1, remainder 4." But the exact answer, in the domain of real numbers, is 1.666666 (repeating), which is closer to 2 than the previous answer of 1. So another way we could answer this question is "2, remainder -2." This quotient is closer to the exact answer, and, correspondingly, has a smaller remainder.

So which answer is correct? Arguably, both of them. As long as the following rule holds:

dividend = divisor * quotient + remainder (the division rule)

…then we have a valid answer.

Which method should you use? It depends on why you’re doing the division. If you’re computing how many cupcakes each child can have at a birthday party, then you’d better use the answer with the positive remainder. If you’re computing how many panels of fencing you need to surround your yard, then you’d better use the answer with the negative remainder.

In fact, there are at least five different, valid algorithms for selecting a quotient and remainder for a given integer division problem.

The homomorphic encryption scheme used in the van Dijk, et. al. paper and in Gentry’s CACM article uses "R-division":

1. Compute the real quotient QR.
2. Compute the integer quotient QZ by rounding QR to the closest integer.
3. Compute the remainder R = Dividend - Divisor * QZ

This is probably different than the method you learned in elementary school, but both are valid. Importantly, it’s probably different than what the integer division and remainder operators in your favorite programming language do.

Modular Arithmetic

If you’ve ever seen a 12 hour, analog clock, then you’ve worked with modular arithmetic. The time of 10:00 today is the same position on the clock as 10:00 p.m/22:00. Another way to say this is:

10 ≡ 22 mod 12

This reads: "10 is congruent to 22 modulo 12." In other words, we just ignore the "extra" 12 hours in the measurement of 22:00, because it’s the same clock dial position as 22:00, only with a circle around the dial.

Naturally, we can perform arithmetic operations like addition and subtraction and compare the congruence of the result. I can say "10:00 + 5 hours means that the hour hand will point to 3 on the clock," or:

10 + 5 ≡ 3 mod 12

Modulo 2 Arithmetic and Binary Operations

You can use any number as the modulus. When we make analog clocks, we use 12. A modulus which is particularly interesting in computer programming is 2. Arithmetic operations mod 2 correspond to binary operations, e.g.:

0 + 0 ≡ 0 mod 2
0 + 1 ≡ 1 mod 2
1 + 0 ≡ 1 mod 2
1 + 1 ≡ 0 mod 2

From this example, you can see that addition modulo 2 is the same as the binary operation XOR. It turns out that subtraction is exactly the same thing (for mod 2 only!):

0 - 0 ≡ 0 mod 2
0 - 1 ≡ 1 mod 2
1 - 0 ≡ 1 mod 2
1 - 1 ≡ 0 mod 2

Multiplication modulo 2, on the other hand, corresponds to the binary operation AND:

0 * 0 ≡ 0 mod 2
0 * 1 ≡ 0 mod 2
1 * 0 ≡ 0 mod 2
1 * 1 ≡ 1 mod 2

Mod in Programming

This is all very simple. However, a word of caution is in order here for anyone who has used a high-level programming language. Most languages have an operator like mod (in Delphi) or % (in C#, which is commonly pronounced "mod"). However, this operator is not strictly modular congruence. It is more like a remainder, although, as we’ve seen, different people and different programming languages might choose to compute a remainder in different (but hopefully valid) ways.

Remainders and congruence are often the same. In fact, the congruence relationship and the integer division remainder for the examples above are all the same. For example:

10 ≡ 22            mod 12
22 / 12 = 1 R 10

However, it’s easy to show examples where this is not true:

22 ≡ 34              mod 12       (1)
34 / 12 = 2 R 10                  (2)
-22 ≡ 2              mod 12       (3)
-22 / 12 = -1 R -10               (4)

(1) is probably not the most common choice; 10 would be a more common answer, as with (2). (1) is nevertheless correct as a statement of congruence. Now compare (3) with (4). The most common way to compute a modulus is to pick the smallest positive number. But the most common way to perform integer division is so-called "T-division", where you select the quotient closest to zero and then calculate the remainder, resulting in a negative remainder when either the dividend or the divisor is negative.

Programs as Digital Circuits

A homomorphic encryption scheme, in addition to the usual Encrypt and Decrypt and KeyGen functions, has an Evaluate function which performs operations on the ciphertext, resulting in the ciphertext of the result of the function.

Obviously, this necessitates a different style of programming than that afforded by the typical programming language. In particular, code like this:

bool a = b ? c : d

…(where a, b, c, and d are all bools) is impossible, because the value of b is encrypted, so the program cannot know what to assign to a. If, however, we can perform binary operations on the ciphertext of b, then we can rewrite the statement above as:

bool a = (b & c) | ((!b) & d)

…or its homomorphic encryption equivalent:

CypherText a = (b h_and c) h_or (h_not(b) h_and d)

…where the h_* operators are the homomorphic equivalents of the usual Boolean operations and a, b, c, and d are now of type CypherText.

Note that the version with the ternary operator and the version with the AND and OR operators are not strictly the same, although their results are; the ternary operator is lazy, whereas the AND and OR version uses complete evaluation. This is necessary when the values are encrypted. It also means that the function may be inefficient.

Intuitively, it’s easy to see that any referentially transparent function can be reduced to such operations; this is what compilers and operating systems do under the hood anyway. I’m not going to go into any detail on how this is done; get an introduction to digital circuits book if you are interested in the particulars.

Gentry’s article notes that other changes in programming style are necessary when performing operations within a homomorphic encryption scheme. For example, the size of the output of a function must be set in advance. Even if the plaintext is variable-length, the ciphertext must be of a known length, which corresponds to the number of "wires" in the circuit. The plaintext, therefore, would have "whitespace" at the end or be truncated, depending upon its size.

Minimizing Boolean Functions

In my first post on homomorphic encryption, I mentioned that Gentry’s encryption schemes can be considered fully homomorphic because they support two homomorphic operations, corresponding to Boolean AND and XOR. I’d like to go into a little bit more detail as to why that is true.

Many combinations of Boolean operations are equivalent. Perhaps the most famous are De Morgan’s laws. There are a variety of techniques for converting one function using Boolean operations into an equivalent function with different operations. This is often done in order to produce a simpler or more efficient function, but can also be done in order to use a specific type of Boolean gate.

It is possible to use combinations of NAND or NOR gates, for example, to produce circuits which perform a Boolean AND, OR, NOT, etc. Hence, a cryptosystem which supported a homomorphic NAND operation could be considered "fully homomorphic."

Similarly, having both AND and XOR gates available allows you to produce the same outputs as all other Boolean gates, as, for example, NOT p is the same as p XOR 1 and we can see by De Morgan’s laws that with NOT and AND we can implement OR.

Therefore, we can see that a cryptosystem can be considered fully homomorphic if it supports enough homomorphic operations to perform any logical Boolean operation. In particular, a cryptosystem which supports any of the following homomorphic operations would qualify:

• Just NAND
• Just NOR
• Both XOR and AND

What’s Next

If you understand the math above, you should now be able to follow along as I implement Gentry’s algorithm for "somewhat homomorphic" encryption. As you’ll see, the "somewhat homomorphic" encryption will be the first step towards a fully homomorphic cryptosystem.

Share This | Email this page to a friend

Share This | Email this page to a friend

Although these words may be unfamiliar to many, the subject matter is terribly important, because, like public-key encryption, which paved the way for secure transactions over the web, an efficient and fully homomorphic encryption scheme would enable new kinds of distributed computing. I’ll explain more about this in a little bit.

If you have an ACM membership, or can find the magazine in a library, I recommend the article. It is far and away more readable than the technical papers on the same subject which are available on the web. The article uses a running analogy of a jeweler who doesn’t quite trust her employees in order to help the reader understand the design of the system.

What Is Homomorphic Encryption?

"Homomorphic" is an adjective which describes a property of an encryption scheme. That property, in simple terms, is the ability to perform computations on the ciphertext without decrypting it first. Because this tends to sound either baffling or miraculous the first time you hear it, let’s begin with a very simple example.

The popular but wildly insecure cipher scheme rot-13 (a.k.a. "Caesar cipher") is partially homomorphic, specifically with respect to the concatenation operation. Imagine we write an Encrypt and Decrypt function using the rot-13 algorithm. The "secret key" will be 13, the number of characters each letter is shifted. Let’s encrypt two words and then concatenate the ciphertext, and finally decrypt the result. In psuedocode, this is:

var c1 = Encrypt(13, "HELLO");      // c1 = URYYB
var c2 = Encrypt(13, "WORLD");      // c2 = JBEYQ
var c3 = Concat (c1, c2);           // c3 = URYYBJBEYQ
var p  = Decrypt(13, c3);           // p  = HELLOWORLD

Because it was not necessary to first decrypt the two fragments of ciphertext before performing the concatenation operation, we can say that rot-13 is homomorphic with respect to concatenation.  In other words, it is possible to take two pieces of ciphertext and perform an operation on them which results in the ciphertext of the concatenation of the two respective pieces of plaintext.

Visually, it looks like this:

It just so happens that in this case the homomorphic concatenation (concatenating the two fragments of ciphertext) is the same operation as the non-homomorphic concatenation (concatenating two fragments of plaintext). This is not always the case. What is important is that we can perform some operation on the input ciphertext which will produce new ciphertext that, when decrypted, will produce output plaintext corresponding to a desired operation on the input plaintext.

Although rot-13 is not at all secure, it turns out that some cryptosystems widely believed to be secure have homomorphic properties. For example, "pure" and un-padded RSA is homomorphic with respect to multiplication.

Why Should I Care?

All of this may seem academically interesting, but one might wonder why I started this post claiming that "an efficient and fully homomorphic encryption scheme would enable new kinds of distributed computing."

I think that the scenario which best makes the benefits of homomorphic encryption clear is cloud computing. It is often much more economical to buy computing resources from a cloud provider then to build a data center yourself. This is especially true if your need for computing horsepower fluctuates. But what if you want to do this computing on private data? Maybe you trust your cloud provider, but what if trust is not enough? Is there a way to get the compelling economic benefits of cloud computing while keeping your data secure?

Imagine that you have developed a web application for income tax preparation. Your application collects personal and financial information from the user, creates a tax strategy resulting in the lowest possible (legal!) income tax payment, and submits the prepared return to the IRS. From a business point of view, this application might be compelling, but it’s a daunting task operationally. In the United States, most people file their taxes once a year, before April 15th. This means that your servers will see a huge spike in demand in the first quarter of the year, with relatively little demand other times. Also, potential customers might be understandably wary about uploading their personal information and financial data to the website of a new business.

Cloud computing offers an excellent answer to the first problem. Because you buy computing resources on as-needed basis, it is very easy to increase the number of available servers when tax season arrives, and reduce the number at other times. Unfortunately, it doesn’t do anything for the second problem.

Hard security in the cloud is not really possible today. It’s very easy to have secure transmission from a local machine to a cloud data store, and of course the stored data can be encrypted. But actually performing computations on that data in the cloud requires decrypting it first. Anyone with physical access to the machine, therefore, has access to your data.

What if it were possible for a user of your tax preparation software to upload their financial information encrypted under the IRS’s public key? Then their data would be secure. You can do this today, of course, but your software would be unable to select a tax strategy for the user or prepare forms. If, however, the encryption method used was also fully homomorphic, then your software could do all of this work without first decrypting the user’s information. The output of your software would be a tax return encrypted under the IRS’s public key. Only the IRS could decrypt it.

Fully Homomorphic Encryption

So a fully homomorphic encryption scheme would seem to solve the security problem while still being compatible with a cloud deployment scenario. Unfortunately, preparing a tax return requires more than just a Concat operation. We can say that an encryption scheme is "fully homomorphic" if it supports enough homomorphic operations to implement any function we need. I’m being purposely imprecise here, because the details are a bit of a digression at this point.

Although there are a number of encryption schemes with one homomorphic operation, until recently no one was really sure if fully homomorphic encryption was even possible. Rivest, Adleman, and Dertouzos suggested that fully homomorphic encryption may be possible in 1978, but were unable to find a secure scheme. Another scheme, from Boneh, et. al., supports two operations, but you can only perform one of them once.

When you consider the complexity of a typical programming language, two operations doesn’t sound like very many, but since we are only looking at operations which mutate data (as opposed to keeping track of the current instruction pointer, etc.) it turns out that supporting the and and xor operations is enough to consider a scheme fully homomorphic, if you can perform those operations an unlimited number of times.

So it was big news when Craig Gentry managed to create such a scheme as part of his Stanford doctoral thesis. Gentry, along with several other researchers from IBM and MIT later created yet another scheme using a different encryption technique, but a similar general approach, with similar homomorphic properties.

Some would say "too big." Bruce Schneier criticized the IBM press release for implying, in his reading, that Gentry’s scheme was practical for real applications, today. It isn’t; the computational and data storage overheads are far too high. However, this takes nothing away from Gentry’s achievement; he has shown that fully homomorphic encryption is actually possible. Indeed, Schneier concludes:

Visions of a fully homomorphic cryptosystem have been dancing in cryptographers’ heads for thirty years. I never expected to see one. It will be years before a sufficient number of cryptographers examine the algorithm that we can have any confidence that the scheme is secure, but — practicality be damned — this is an amazing piece of work.

Let’s Code It!

The second, fully homomorphic scheme proposed by Gentry, et al., and described in the CACM article is simple enough that I decided to implement it myself. I’ve learned quite a lot in the process of doing this. So in the near future I’ll be sharing my code and some of the insights I’ve made in this process.

Before I can do that, however, I need to describe some of the math involved. And I’d like to elaborate on why two operations is enough to consider a scheme "fully homomorphic."

(Continue reading…)

Share This | Email this page to a friend

From time to time, a developer will attempt to add the ASP.NET Membership/Forms Authentication tables to their Entity Framework model (or LINQ to SQL, NHibernate, etc.) model. Before doing this, they will often have created referential constraints against these tables. When the mapping doesn’t work out quite the way they planned, they will ask how to make the mapping work.

There’s only one correct answer to this question: Don’t do it at all!

There are a number of good reasons why you should not make your database and code depend upon the SQL Membership Provider database schema. In this post, I will focus on a few of the most important:

• Separation of concerns
• Membership and authentication providers are supposed to be interchangeable
• The SQL Membership Provider database schema is an implementation detail

Separation of Concerns

Your application’s data model is designed to fit your application domain. It will change based upon the needs of the end users. It should not have to change because Microsoft decides to update the SQL membership provider, as well. That would violate the single responsibility principle. It is often dangerous to combine data which is not closely related into a single data model. This danger is compounded when data from two separate domains, written by entirely different companies, and designed for orthogonal purposes is shoehorned into a single entity model.

Membership providers are supposed to be interchangeable

One of the most important design intentions of the ASP.NET authentication and membership provider model is to make it easy to interchange providers. If you decide to stop using the SQL membership provider in transition to Open ID, domain authentication, Facebook authentication, etc., this should be a matter of, at most, a couple of days work to migrate data from one provider to the other, rather than a complete rewrite of your application, starting with the database and moving out from there.

The SQL Membership Provider database schema is an implementation detail

The publicly-documented interfaces to membership and forms authentication are the Membership and FormsAuthentication types, respectively, as well as the relevant sections of the Web.config file. If you write your code around these types, you have a reasonable expectation that your code will continue to work when the .NET framework is next updated. On the other hand, if you query the database directly, there is no guarantee that the schema will not change when the next version of .NET ships. If Microsoft makes a security-related change to the SQL membership provider, then it is conceivable that the schema could even change with a service pack. The cost of relying on an implementation detail is that you never really know.

Share This | Email this page to a friend

Now when autoencode is set to true we encode the data coming from server and not only when we post it (secutity fix)

Prior to this, you were required to encode the data yourself.

Now personally, I think that should be the default. But it would have been a breaking change for the grid, since there are a few cases where you want to display unencoded data (I’ll discuss these exceptional cases in a second).

It’s really easy to make this the default for your application. Set the grid’s defaults before you create any grids:

    $.jgrid.defaults = $.extend($.jgrid.defaults, {
            autoencode: true,
            datatype: 'json',
            // etc....

You can override this in the rare cases when you don’t want encoded data by setting autoencode false when setting up your grid:

    $("#grid").jqGrid({
                    autoencode: false,
                    url: "/Some/Path",
                    // etc....,

So why wouldn’t you want to use this setting? I can think of two reasons:

1. The server is returning markup which you don’t want to encode. If you have a server method which is returning markup (e.g., "<img src=…"), then you won’t want the grid to encode it. It then becomes your responsibility to sanitize the rest of the grid data on the server.
2. There is a performance cost to doing the encoding in JavaScript. If you are absolutely certain that the server method you’re calling can return only sanitized data, then you can turn off the auto encoding and save this cost. Note that many JSON encoders, such as the "Json()" method in ASP.NET MVC will not encode HTML by default.

Note that you can only set autoencode at the grid level, not at the column level.

If you use custom formatters in your grid, then you should note that the strings they return will not be HTML encoded, even if you set autoencode true. This is probably a good thing, since it is common to use custom formatters to return HTML. However, it means that if you write a custom formatter, then you must ensure that any user data contained in the string and returns is encoded. You can use the grid’s encoder, which is little more than a string replace on angle brackets, in a custom formatter as follows:

    myFormatter: function(cellval, opts, action) {
        if (cellval) {
            return $.jgrid.htmlEncode(cellval.Name + "");
        };
        return "";
    }

Share This | Email this page to a friend

Obviously, if you use a source control tool which locks files on check out, then working concurrently on just about anything will be impossible. So I’m going to presume that your source control tool supports working concurrently, without locks, and has a decent merge tool to handle conflicts at check-in.

We tend to think of the EDMX files as having three sections:

• CSDL, which describes the client schema. This will be used when generating code for your entity classes.
• SSDL, which describes the storage schema, more commonly known as your database metadata.
• MSL, which describes the relationship between the first two.

In reality, however, the EDMX file has two main sections. The first, edmx:Runtime, contains the three sub-sections described above. The second, edmx:Designer, contains a couple of important properties, such as MetadataArtifactProcessing, plus a lot of information about the position of objects on the GUI designer, in a sub-node called edmx:Diagrams. The illustration below should make this clear:

Why is this important? My experience is that any decent merge tool doesn’t tend to have any problem merging what we typically think of as the "EDMX", namely the CSDL, SSDL, and MSL. It is well-formed XML with descriptive tags; it’s the sort of thing that text merge tools tend to do very well with. There are a couple of tips worth knowing, but before I describe them, I’d like to focus on the more common issue with EDMX merges.

In my experience, if two people change the layout of entities in the designer and then attempt to merge, the merge will probably fail. Moreover, you will find it difficult to impossible to fix this up manually, something which is generally fairly easy with merges of source code or other parts of the EDMX. When you look at what is in the edmx:Diagrams node, you will quickly understand why:

I strongly suspect that this "feature" is not specific to the Entity Framework designer, but rather is shared amongst all of Visual Studio’s various diagramming tools. At any rate, I think you will have a difficult time merging XML like this, so I have two recommendations regarding merging and EDMX files:

• If you are going to have two or more developers work on an EDMX file concurrently, then don’t change anything in the designer. If two or more people do this, it almost guarantees a conflict on check-in.
• The other side of the coin is that if you must change something in the designer, then you should give a heads-up to other developers, and exclusively lock the EDMX file while you work.

That said, we don’t generally use the GUI designer at all. It is too unwieldy to have more than a dozen or so types on a single design surface. On the other hand, the tree-style Model Browser is quite useful. I suggest using the Model Browser instead of the GUI designer to navigate your entity model.

Perhaps a future version of the Entity Framework designer will allow for multiple diagrams within a single entity model, somewhat like SQL Server diagrams. If each diagram had its own node in the XML, you might even be able to merge them.

If you steer clear of issues with the designer, then you will typically find that EDMX changes can be automatically merged by any decent merge tool (unless, of course, they actually conflict). In the unlikely event, however, that your merge tool reports a conflict, and you don’t think it’s an actual conflict between your changes and those made by the other developer, it does help to understand how the "Update Model from Database" wizard updates existing EDMX:

1. The wizard will generate new SSDL from scratch. It will then go through the newly generated SSDL, and replace each matching node in the existing SSDL with the newly generated version. There are a few SSDL features which are not supported by the designer/wizard, such as will, and these will be left alone, if they exist in your model.
2. Any new entities or new properties of existing entities will have corresponding nodes added to the CSDL and MSL. But existing entities and properties will not have their CSDL and MSL updated, because the wizard presumes that you want to keep any changes you may have made to them.

Therefore, when multiple developers are updating an EDMX file concurrently, they should use the same database. This will tend to make SSDL changes merge without conflict, and allows you to simply overwrite any false conflicts (merge tool failures) in the SSDL, because you can be confident that it was generated from the same object in the same database. But if you’ve manually customized your SSDL (i.e., edited the SSDL section of the EDMX as text), then you should keep an eye on your changes, to make sure that the wizard does not overwrite them. Most people never do this, though.

On the other hand, changes to the CSDL and MSL must be reviewed carefully in the unlikely event that there are conflicts in the merge, because you, or the other developer, may have made customizations to the mapping or entity types which you want to keep.

Finally, it is worth mentioning that the Entity Framework version 4 supports "code only" models which do not use EDMX files at all. If you prefer designing your entity model via source code, you can choose this option.

Share This | Email this page to a friend

Many people ask how to do a "left join" in LINQ to SQL, and unfortunately, the answer they nearly always get — "Use DefaultIfEmpty!" — is, in my opinion, terrible advice. Let’s implement the same query with and without the join keyword, and then compare the readability of the queries, the functionality, the knowledge required to write the query, and the maintainability of the code. I think you will find that using associations wins on every single criterion I examine.

I’m going to use the Northwind demo database for this example, since many people are familiar with its structure. I created a LINQ to SQL model for Northwind by simply dragging all of the tables in the database onto the LINQ to SQL designer. The only change I made to the model generated by the designer is to rename the "Employee1" property on the generated "Employee" type to the more descriptive name "Supervisor." I’m using LINQ to SQL for this demo, but everything I’m saying here applies equally to LINQ to Entities.

"Left" Joins

Let’s imagine that I am asked to produce a web page listing all employees in a company, along with their supervisor, if any. This requires a "left join," in SQL terms, because not all employees have a supervisor. I’ll project onto a presentation model, just like I do in LINQ to Entities. Using the association properties generated by LINQ to SQL, this is quite simple:

This is fairly readable. The one thing that you need to know is that both LINQ to SQL and LINQ to Entities coalesce nulls. This means that on a row where e.Supervisor is null, you will not get a NullReferenceException in the assignment to SupervisorName and SupervisorBirthDate, as you would with LINQ to Objects. Instead, null will be assigned. Therefore, it is important that EmployeeListItem.SupervisorBirthDate is of type DateTime? (a.k.a. Nullable<DateTime>) instead of the non-nullable DateTime.

Let’s compare that to the equivalent query using the join syntax, using the mysteriously popular DefaultIfEmpty trick:

Yuck! This is far less readable than the query above. Yet these two queries produce exactly the same results. If you don’t believe me, download the sample project attached to this post and try it yourself. Perhaps even worse than the general unreadability of the "join version" is the fact that this query requires knowledge of the structure of the database which is already present in the DBML (or EDMX, in the case of the Entity Framework) model. This is a problem for two reasons. First, it’s an opportunity for programmers to make a mistake, which the first query eliminates. Second, it’s a potential maintenance issue if the foreign key definition ever changes in the database.

"Inner" Joins

Now let’s compare an example of an "inner join" using both my recommended method of associations and the LINQ join keyword. Here’s the association version. It’s so readable that there is very little to say about it:

Here’s the join version:

Again, these two queries do exactly the same thing, as you can confirm for yourself by running the demo project. The join version here shares all of the faults of the "left join" version above.

API Consistency

Thus far, I’ve been discussing LINQ to SQL. But what if I have an Employee instance and that like to examine the employee’s supervisor? I might write code like this:

Now compare that code with the "association" and "join" query forms. You will see that using the associations makes the LINQ to SQL queries much more closely resemble how you work with the materialized entity objects in "regular" code. Again, I think this makes your code easier to read.

What About Performance?

Not surprisingly, the SQL generated by the equivalent association and join versions is close to identical. I would not expect to ever see a performance difference between the two syntaxes in terms of query execution time.

Is It Ever Correct to Use join?

The advantages of using associations are so strong that you may wonder why join exists in LINQ at all. Associations, however, are only helpful when they actually exist. There may be times when you need to "join" based on values which are not actually foreign keys. Or you may need to join between LINQ to SQL and LINQ to Objects.

Running the Demo Project

Here’s a the demo project. To build and run it, you’ll need to Visual Studio 2010 Beta, SQL Server (Express is fine), and the Northwind demo database I linked above. Open the project, find the Web.config file in Solution Explorer and open it. Change the connectionString to point to your SQL Server. Now you should be able to run the application.

Share This | Email this page to a friend

Share This | Email this page to a friend

Imagine that you’ve been asked to write a new web application to track employees for a customer, Chotchkies restaurant. The application must use ASP.NET MVC and the ADO.NET Entity Framework. The user interaction designers have mocked up the following interface for editing an employee:

Upon seeing this mockup, your first recommendation is to fire the user interface designers. But management declines to follow your recommendation. So how should you implement this? Since this is a new application, you have no database, no entity model, nothing. Where to begin?

Presentation Models

For a variety of reasons, I always use strongly-typed views in my ASP.NET MVC applications. Also, I use presentation models instead of using entity types directly as the model type for my views.This allows the user interface and data models to evolve independently.

Because I am likely to want to build user interfaces using ASP.NET MVC 2’s Dynamic Templated Views, I need to decorate the view model with presentation concerns, like noting that the "Flair count" field is read-only in this particular view.

Especially in view of the less-than-compelling user interface markup, it’s important to realize that the design of the user interface is likely to change wildly over the course of implementing the application. When actual users begin to test the application, they will request changes to the user interface, and you need to be able to adapt to this.

This has two important implications: You must get a prototype to testers and end-users as fast as possible, and your user interface must not be deeply coupled to the rest of the application, given the high likelihood of change.

So based upon the user interface prototype above, the following presentation model might be reasonable:

It’s now trivial to write an action which can be used to develop a view for editing the employee, and which matches the UI prototype above. Such an action might look like:

I’ll spare you the HTML. At this point, I have a running application which the user interaction designer can approve, and testers can try out. My total effort, thus far, is a few minutes of work. If I had started by designing a database and the data model, I would still have nothing to show for my efforts.

Entity Model

At some point however, you need to create a database and an entity model. That was, after all, part of the requirements you were given for the application. Knowing that there might be a need to reference people who are not employees, the entity model is going to have to look very different than the presentation model above. After some discussions with the business analyst, you learn that Chotchkies management is very serious about tracking employee flair, so a first attempt at an entity model might look like this:

Now you can generate a database. At this point, you have a presentation model and an entity model, and need only wire them together.

Projection

Taking a collection of instances of one type, and mapping their properties onto a collection of instances of another type is often called mapping or projection. With older ORMs, which have either no or very limited LINQ support, this can be quite tedious, requiring manual code, the use of tools like AutoMapper, and a good deal of thinking about eager loading, lazy loading, and optimizing situations like the fact that we don’t actually want to load all of the flair for an employee here; we just need the count.

The Entity Framework, on the other hand, makes this very easy, as we can just express the relationship between the entity model and the presentation model with a LINQ expression:

I’m glossing over some details here. In a real-world application, for example, we would use the repository pattern rather than grabbing the context directly in the controller. But the point of this post is shown in the code above:

By expressing the relationship between the entity model and the presentation model as a LINQ query, it is no longer necessary to worry about implementation details like eager loading versus lazy loading, optimizing the SQL for the Count and avoiding loading big properties not actually used here. All of this can be — and is — derived from the query above, automatically.

Share This | Email this page to a friend

There is no good reason not to use static analysis. If you are the sort of person who doesn’t allow hints and warnings in your code, and has configured your build to fail on any hint or warning (and I hope that you are), then you’ll love static analysis; it takes this kind of discipline to the next level. Sure, it’s useful to find existing errors in your code, but the most important benefit is that it prevents new errors from being introduced.

The only remotely valid argument I’ve heard against using static analysis is that when you first use a static analysis tool against a legacy code base, you will typically see hundreds of rules violations, many of them incorrect. Cleaning all this up will be a lot of work, which you usually can’t take on all at once.

Unlike compiler hints and warnings, static analysis rule violations often happen when there is no underlying bug. When you develop new code, you can analyze any newly introduced rule violation and exclude it if necessary. But that’s not practical to do with legacy code. My approach to this problem is very simple: Turn off rules in your static analysis tool until all the rule violations go away.

What’s the point of buying a static analysis tool and then turning off half the rules? Well, the other half are still turned on, and they are now a part of your automated build, making it impossible to introduce new code which violates them. Moreover, you can go back and turn on the other rules a reasonable pace, say, one or two a week, fixing real issues that you find or excluding the particular violation if it turns out to be a non-issue, until they are all turned back on.

Once you have even a few static analysis rules turned on in your automated build, you’re getting some degree of quality control for very little expense. Integration testing costs money, every time you do it. Unit testing costs money every time you write new tests. Static analysis just works and keeps working, and costs you nothing more than a software upgrade once every couple years or so.

So static analysis in general is a good thing. What about CodeHealer in particular? Well, we have been using it for around three years now. It has been stable, and I’ve seen new versions released regularly. Also, I have never seen anything better for Delphi.

Digression: Attributes

When Delphi 2010 was released with the enhanced RTTI and custom attributes feature, many Delphi developers wondered what this feature would be useful for. In general, your source code describes the behavior of your application, and attributes describe your mechanisms (thanks to Eric Lippert for this concise description). Static analysis is an excellent example. Imagine that I want to exclude a static analysis rule in a particular case. This has nothing to do with the desired functionality of the application; rather, it’s more like a parameter to the automated build process.

That may sound a bit abstract, so let’s look at a specific example. FxCop, a static analysis tool for .NET code, has rule that should not introduce private types which are never used. But FxCop cannot detect the instantiation of types inside of a LINQ query. So in this case the rule is making a mistake; it is failing to detect that a type is, in fact, instantiated in my code. One possible solution to this issue would be to try and write a better rule. But that’s a lot of work, and there are only about two cases of this false positive in our entire source tree. So it’s easier to just suppress this message in the case of a false positive, which you can do with an attribute:

[System.Diagnostics.CodeAnalysis.SuppressMessage(
    "Microsoft.Performance",
    "CA1812:AvoidUninstantiatedInternalClasses",
    Justification = "It is instantiated in a query, which FxCop can't see.")]
private class ConsumerRow //...

This is far easier (and, I think, a better design) than what CodeHealer currently offers. I hope that future versions of CodeHealer will use this method, now that Delphi has attribute support.

Share This | Email this page to a friend

Share This | Email this page to a friend

New Features

Some of the new features of preview 2 have been discussed elsewhere, so I won’t rehash them. But I’ve also noticed that there is a new attribute, [RequireHttps], which does what you would expect, when added to an action, and a new HTML helper, Html.HttpMethodOverride, which makes it easy to take a POST request and code as if the request were actually PUT or DELETE, by adding a hidden input containing a special value on the HTML form. This allows you to write your server in a more RESTful style, which will be suitable for user agents which know about the PUT and DELETE verbs, while maintaining compatibility with those (like browsers) which do not.

MvcHtmlString

After installing the new assembly into the GAC, I attempted to compile our existing projects. I got a compilation error on some of our custom HTML helpers, as the MVC extension methods like Html.RouteLink have been changed to return an instance of type MvcHtmlString instead of String. In many cases, I could just change the return type. However, the clear intention of the framework designers is that any HTML helpers which return HTML (with angle brackets) instead of "plain" text should return a MvcHtmlString instead of a String. So, in addition to changing the return types of methods where necessary to get those methods to compile, I also wanted to change the return types of any custom HTML helper which returned HTML containing angle brackets. It makes no difference today, but in ASP.NET 4, there will be a new syntax to guard against XSS attacks, and it makes sense to get ready for this by returning the correct type.

The trick here is that MvcHtmlString does not have a public constructor, so it’s not immediately obvious how to create one of these things. Perusing the MVC source code, I noticed that it does have a static Create method, and comments elsewhere in the source code indicate that this method should be used instead of the detected constructor. So you can now write a helper like this:

public static MvcHtmlString Fud(this HtmlHelper helper)
{
    return MvcHtmlString.Create("<acronym title=\"Fear, Uncertainty, and Doubt\">FUD</acronym>");
}

JsonRequestBehavior

ASP.NET MVC will now, by default, throw an exception when an action attempts to return JSON in response to a GET request. Unless you read the release notes carefully, you won’t discover this until runtime. This is in order to proactively defend against a particular cross-site attack. It is good to be safe by default, but you are only vulnerable to this attack under the following combination of circumstances:

• The data you’re returning is worth stealing.
• The root data object in the JSON response is an array.
• The requesting browser is not IE 8, or some other browser which doesn’t allow __defineSetter__

It turns out that in our application we almost never return JSON results with the root object as an array in a GET. So the best fix was generally just to tell the framework that we have examined the risk and determined it returning JSON in this case, by changing code like:

return Json(model);

…to:

return Json(model, JsonRequestBehavior.AllowGet);

Unfortunately, I had to make this change in a lot of places. Still, I agree with the framework designers: Better safe than sorry.

JavaScript Files

The MicrosoftAjax.js, MicrosoftAjax.debug.js, MicrosoftMvcAjax.debug.js, and MicrosoftMvcAjax.debug.js files have all changed for ASP.NET MVC 2. Existing ASP.NET MVC 1.0 projects will have old versions of these files. The upgrade instructions in the Release Notes don’t mention this, but (they do; I just missed it) you should replace these files with the newer versions when upgrading to MVC 2. To do this, create a new MVC 2 project, copy the JavaScript files from the Scripts folder in this new project, and paste them over the existing files in the project you are upgrading.

ModelBindingContext Changes

One of the new features of preview 2 is client-side validation and custom metadata providers for validations. In order to implement this feature, some of the implementation details of model binding have changed. This won’t affect most people, but the fix was a little tricky, and required a good bit of examination of the MVC source code, so I’ll list it anyway. Prior to preview 2, I had code in a unit test for a custom model binder like this:

        internal static T Bind<T>(string prefix, FormCollection collection, ModelStateDictionary modelState) where T:class
        {
            var mbc = new ModelBindingContext()
            {
                ModelName = prefix,
                ModelState = modelState,
                ModelType = typeof(T),
                ValueProvider = collection.ToValueProvider()
            };
            IModelBinder binder = new MyModelBinder();
            var cc = new ControllerContext();
            return binder.BindModel(cc, mbc) as T;
        }

With Preview 2, on the other hand, I have to do this:

        internal static T Bind<T>(string prefix, FormCollection collection, ModelStateDictionary modelState) where T:class
        {
            var mbc = new ModelBindingContext()
            {
                ModelMetadata = ModelMetadataProviders.Current.GetMetadataForType(null, typeof(T)),
                ModelName = prefix,
                ModelState = modelState,
                ValueProvider = collection.ToValueProvider()
            };
            IModelBinder binder = new MyModelBinder();
            var cc = new ControllerContext();
            return binder.BindModel(cc, mbc) as T;
        }

Not hard once you know the trick.

Share This | Email this page to a friend

Let me say at the outset that although all three systems work somewhat differently, I don’t see an overwhelming advantage to any one design. Generally, you can do what you need to do in all three environments. I’m writing this article not to claim that any one system is better than the others, but to point out some of the subtleties in the implementations.

Before I get started, I’d like to thank Barry Kelly for his useful feedback on my first draft of this article.

Compiling Instantiations

Every implementation of generic types works via a two-step process. First, you define a generic type or method with a "placeholder" for a specific type, which will be substituted later on. Later (exactly when depends upon the language), the type is "instantiated." Note that instantiating a generic type is very different from instantiating an object. The former happens within the compiler, whereas the latter happens at runtime.

Instantiation is triggered when some code uses a generic type or method with a specific type parameter, and means that based upon the generic definition and the types or values passed when the generic is used, a specific implementation is substituted in order to allow the generation of machine code. Instantiation is one of the most important differences between real generic types and using non-generic types with casts. In the end, different machine code is generated for instantiations for different type parameters.

In C# and Delphi, there is a language feature which is solely dedicated to implementing generic types and methods. In C++, on the other hand, the "templates" language feature can be used to implement generic types and methods, among many, many other things. It is even possible to do general-purpose programming using templates, which C++ programmers call "metaprogramming."

C++ templates require the template source code to be available when the code using the template is compiled. This is because the compiler does not actually compile the template as a separate entity, but rather instantiates it "in-place" and only compiles the instantiation. The C++ compiler is effectively doing code generation, substituting the type parameter (or value) for the placeholder for the type, and generating new code for the instantiation. Update: Moritz Beutel elaborates on this in his excellent comment on this post. You should read the full comment, but the short version is that the manner in which templates are compiled can result in errors in the code which uses the template appearing (from compiler error messages), incorrectly, to be errors in the template itself. Moreover, the implementation of most C++ compilers makes this problem even worse than what is necessary in order to implement the C++ standard.

In Delphi and C#, on the other hand, the generic type or method in the code which uses the generic type or method can be compiled separately. Therefore, you can compile a library which contains a generic type, and later on compile an executable which uses a instantiation of that type and has a reference to the binary library, rather than to the source code for the library.

Another way to think of this difference is that in C++, a template will not be compiled at all until it is used. In Delphi and C#, on the other hand, a generic type or method must be compiled before it can be used.

In Delphi, the compiler uses a feature closely related to the method inlining feature. This causes the compiler to store the relevant bits of the abstract syntax tree for the generic type parameter in the compiled DCU. When the code which uses the generic type is compiled, this bit of the abstract syntax tree is read and included in the abstract syntax tree for the code which uses the generic type, so that when machine code is produced, based on the new, “compound” abstract syntax tree, it looks, to the code emitter, like the type was defined with the type parameter "hard coded." Instead of linking to compiled code in the DCU, the code which uses the generic type emits new code for the instantiation into its own DCU.

Because generic instantiation is performed in the same area of the Delphi compiler which does method inlining, there are some limitations on what you can do in a generic method, or a method of a generic type. Like inlined methods, these methods cannot contain ASM. Also, calls to these methods cannot be inlined. These restrictions are limitations of the implementation, not of the language design, and could theoretically be removed in a future version of the compiler.

C# generics use the .NET Framework 2.0+, which has native support for generic types. The C# compiler emits IL which specifies that a generic type should be used, with certain type parameters. The .NET framework implements these types using one instantiation for any reference type, and custom instantiations for value types. (Don’t confuse “instantiation” with “instance” in the preceding sentence; they mean entirely different things in this context. There are usually many instances of one instantiation.) This is because a reference to a reference type is always the same size, whereas value types can be many different sizes. Later, the IL will be JITted into machine code, and, as with compiled C++ or Delphi code, types don’t really exist at the machine code level. In .NET, generic type instantiation and JITting are two distinct operations.

So one important difference in generics implementations is when the instantiation occurs. It occurs very early in C++ compilation, somewhat later for Delphi compilation, and as late as possible for .NET compilation.

Custom Specializations

Another very important difference is that C++ allows custom instantiations, called specializations, including specializations by value. With C# and Delphi, on the other hand, the only way to instantiate a generic type is to use that type with an explicit type parameter. The implementation will always be the same, with the exception of the type of the type parameter. Because C++ allows custom instantiations, it is easy for a programmer to write different implementations of a method, for example, for different integer values. Like operator overloading, this is a powerful feature which requires considerable self-restraint to avoid abuse.

Constraints

Delphi and C# both have a generic constraint feature, which allows/requires the developer of a generic or method type to limit which type parameter values can be passed. For example, a generic type which needs to iterate over some list of data could require that the type parameter support IEnumerable, in either language. This allows the developer of the generic type to make her intentions for the use of the type very clear. It also allows the IDE to provide code completion/IntelliSense on the type parameter, within the definition of the generic type. Also, it allows a user of the generic type to be confident that they are passing a legal value for the type parameter without having to compile their code to find out.

In C++, on the other hand, there is not presently any such feature. A more powerful/complex feature called "concepts" was considered for, but ultimately removed from, C++0x.

An implication of the lack of constraints is that C++ templates are duck typed. If a generic method calls some method, Foo on a type passed as the generic type parameter, then the template is going to compile just fine so long as the type parameter passed contains some method called Foo with the appropriate signature, no matter where or how it is defined.

Covariance and Contravariance

Let’s say I have a function which takes an argument of type IEnumerable<TParent>. Can I pass an argument of type IEnumerable<TChild>; to that function? What if the argument type were List<TParent>; instead of IEnumerable<TParent>? Or what if the generic type was the function result rather than the function argument? The formal names for these problems are covariance and contravariance. The precise details are too complicated to explain in this article, but the examples above summarize the most common times you run into the problem.

Delphi generics and C++ templates do not support covariance and contravariance. So the answers to the questions above are no, no, and no, although there are, of course, workarounds, like copying the data into a new list. In C# 4.0, function arguments and results can be declared covariant or contravariant, so the examples above can be made to work where appropriate. "Where appropriate" involves non-trivial subtleties hinted at above, and exemplified by the fact that arrays in .NET have (intentionally) broken covariance. However, the BCL routines in the .NET Framework 4.0 have been annotated to support covariance and contravariance when appropriate, so developers will benefit from the feature without having to fully understand it.

Share This | Email this page to a friend

• Most tools tend to work on either compiled assemblies or source code. NDepend uses both.
• Most tools implement their tests in compiled, executable code. NDepend, on the other hand, uses a proprietary, domain-specific language called CQL.
• NDepend can enhance its analysis by importing analysis data from related builds, like NCover, Team System coverage analysis, and can compare multiple builds of the same project.

As with other static analysis tools, however, NDepend is designed to be used both within Visual Studio and as part of an automated build/continuous integration process.

NDepend claims to support "any .NET language including C#, VB.NET, MC++ and C++\CLI." To put this claim to the test, I ran it against a Delphi Prism application. Indeed, it did produce useful results in the analysis, although there were some quirks. It couldn’t find the assembly from the solution, but after I located it manually analysis could run. I suspect NDepend doesn’t parse the source code of Prism applications as it does with C#.

What NDepend Does

The name "NDepend" presumably comes from the dependency analysis feature. NDepend will display dependencies between assemblies either as a cross tab or a directed graph. See the website for illustrations of this feature.

NDepend also does static analysis along the same lines as Microsoft FxCop and StyleCop, only in a much more interactive manner. By combining the features I’ve already mentioned, you can easily ask queries like, "What are the most complex methods I’ve changed recently which have no unit test coverage?"

All of NDepend’s features are available in its own GUI application, from the command line, within Visual Studio, and in a custom MS build task. So, as with most static analysis tools, you can integrate NDepend into a continuous build environment in order to set enforceable standards on code complexity and the like.

Installation

NDepend does not have an installer. You need to create a folder for the application, and unzip an archive file into that folder. To install the integration with Visual Studio, you click a button in the GUI application, once it’s running. I find point and click installers a bit more convenient, but it was certainly easy to get the application working.

Default Analysis of a Do-Nothing Application

NDepend is quite strict even in its default configuration. For example, I ran it on a do-nothing, "File->New" ASP.NET MVC 2 Preview 1 application, with no code of my own added. It reported quite a number of warnings.

Many of the warnings were incorrect, because NDepend did not automatically find the System.Web.Mvc assembly, which is not stored along with the rest of the .NET framework, even though it is in the GAC. In general, the current version of NDepend will only find assemblies which are in the solution folder or are in the .NET Framework folders automatically. You should add other folders for referenced assemblies manually, even if those assemblies are GACced. To fix this, I had to do the following steps:

1. Go to the Project Properties tab in the GUI.
2. Click the "View Folders" button.
3. Click the "Browse" button and manually locate the folder containing the required assembly.

With these steps completed, the quality of the analysis improved quite a lot.

NDepend shows the results in its GUI and also produces an HTML report. The HTML report for the do-nothing application was 14 pages long, and the report for our production web application ran over 50 pages. This can be kind of overwhelming for a new user. The issues flagged in the do-nothing application included (with my comments in parentheses):

• Methods with > 10 lines of code should be commented. (I disagree; methods should be commented when their intention cannot be made clear in code. This has little relationship to method length and is difficult to impossible to flag with static analysis.)
• Types with no state should probably be static. (Worth considering.)
• Types with no subtypes could be sealed. (NDepend makes it easy to turn this off for public types.)
• Classes with small instance sizes, etc., could be structs. (Maybe, if you want value semantics.)
• Avoid namespaces with few types. (This is expected on a File->New application; would work better in a "real" app. FxCop has the same rule.)
• You should’t have types outside a declared namespace. (I agree.)
• Avoid [un] boxing. (I agree; great test!)
• Types which "could be internal." (NDepend can’t pick up instantiation via reflection, making this less useful in the specific case of MVC.)
• Fields which are assigned only by the constructors of their types should be marked read-only. (I agree; this is a really useful test.)
• Type names should begin with an uppercase letter. The default MVC application includes a type called "_Default".

For the sake of comparison, I analyzed the same application using FxCop. This produced 11 warnings, about half correct, and about half incorrect. Incorrect warnings included not recognizing the TLA MVC, asserting that Application_Start should be static, and claiming that the application should target .NET Framework 3.5 SP 1 (as opposed to just 3.5), something which Visual Studio 2008 does not allow. Probably correct warnings included that the application should have the CLSCompliant attribute, and that the _Default type was oddly named.

Customizing the Analysis For a Real Application

NDepend’s use of its CQL language to implement its test makes individual test very easy to customize. For example, our production web application includes the Microsoft DynamicQuery library, which is distributed as a single CS file. Because we included this in one of our assemblies instead of creating a separate assembly for it, NDepend analyzes it along with the rest of our assembly. It contains many long methods, and other things which NDepend does not like, but because this is Microsoft’s code, and not ours, I would like exclude it from the analysis. So I can, for example, change the test for "method too long" from:

// <Name>Methods too big (NbLinesOfCode)</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE NbLinesOfCode > 30 ORDER BY NbLinesOfCode DESC
// METHODS WHERE NbLinesOfCode > 30 are extremely complex and
// should be split in smaller methods
// (except if they are automatically generated by a tool).
// See the definition of the NbLinesOfCode metric here http://www.ndepend.com/Metrics.aspx#NbLinesOfCode

to:

WARN IF Count > 0 IN SELECT TOP 10 METHODS
    WHERE NbLinesOfCode > 30
        AND !(FullNameLike "^System.Linq.Dynamic") 
    ORDER BY NbLinesOfCode DESC

What if you don’t know how to write a regular expression? Well, then you need to learn how to write a regular expression. Regular expressions are the only option for this kind of name matching in NDepend.

It might seem at first that using a proprietary language would be a barrier to using the tool. After all, other tools, like FxCop can be customized using C# and other .NET languages. Doesn’t a proprietary syntax like CQL discourage customization? Not really, in my experience. The real win of NDepend is that you can customize the rules interactively, within the GUI. In contrast to building a custom FxCop rule and restarting FxCop, this is a huge win, because it encourages the use of ad hoc queries and trial and error to build the ideal heuristic for your application. Coupled with auto complete in the CQL editor, it makes the relatively minor inconvenience of dealing with a proprietary language almost irrelevant. Yes, I would prefer to use LINQ, but the interactivity of CQL is a much bigger win.

However, you don’t get the same kind of customizability at a global level. You can include and exclude assemblies from analysis globally, but that’s about it. So while you can include and exclude code based on almost any imaginable criterion at the level of an individual test, I have not found any way to do an exclusion like the one I demonstrate above globally. (Note, however, that in a comment on this blog a week ago, a member of the NDepend team suggested that features along this line may be coming next year.)

I should add that most static analysis tools seem to work this way. They typically make it fairly easy to include and exclude entire tests or specific tests on specific cases, but don’t typically include a way to say, "Don’t run any test on any bit of code which meets the following criteria, ever." So NDepend is no worse than other tools I’ve used in this regard, and it’s features to customize individual tests are considerably better than most other tools, although the learning curve is a bit steeper.

Like every static analysis tool I’ve ever tried, NDepend does not produce useful results for generated code. Since this is not an NDepend-specific issue, I discussed it in an earlier post, along with some tips for excluding generated code from NDepend’s analysis.

Documentation

The documentation in the NDepend online help is a bit thin. But the authors have written a number of informative blog posts on how to use specific features and tips for getting the most out of the tool. Once you realize that these posts exist and know where to find them, it is quite a bit easier to read up on how to use the tool effectively.

Usability

I’m of two, contrasting opinions regarding the usability of NDepend. I found the initial learning curve to be quite steep, and elements of the user interface to be not very discoverable. On the other hand, once I figured out how to use the application, which took me a few weeks (maybe I’m just slow), I found the application to be very usable, and I could do what I needed to do very quickly. Watching the videos on the NDepend website helps quite a lot in getting over this initial hump. I’ve also had some e-mail discussions with the makers of the software on how they might improve some of the things this new user found initially confusing in their next release.

For those new to the application, here are some things which may help you get started:

• The CQL query editor has auto complete, so it’s fairly easy to customize a query even if you’re unfamiliar with the language. However, if you intend to do any serious work in CQL, it is worth seeking out and spending some time reviewing the CQL examples on the NDepend website.
• After you do an initial analysis, look at the error list before considering the analysis results. Adjust the configuration (by adding assemblies or search directories) to clean up any of the errors before moving on to looking at the analysis results. There are a few valid C# language constructs that NDepend’s source code parser won’t recognize. I got a warning from a class constraint on a generic type. I understand this will be fixed in a future version. You can ignore these warnings; they’re harmless.
• Whatever you want to do within the application, the place to start is by selecting the appropriate view configuration from the View menu. So if you want to alter a query to return more specific results, you would select View -> Reset Views to work with the CQL language.
• The CQL Query Result tab will display a query, and has been Edit button, you can never added a query in that window. To edit a query, go to the CQL Queries tab, select the query you’d like to edit, and click the Edit Query button. A new tab will appear which allows you to actually edit query.
• When you click on a query in the CQL Queries window, you will only see the results if the query returns any rows. If the query returns no rows, then the results window will not appear.
• Whenever you analyze the project in the NDepend GUI, NDepend creates a lengthy HTML report, and opens it in your web browser. But all of the information in the report is also available within the GUI application, as well. So you can see the same information in whichever place is most convenient for you to navigate.

Performance

NDepend is fast, sometimes surprisingly so, given that it is looking at both binaries and source code. It generally takes less time to run an analysis than it does to compile a solution. I never experienced any performance problems in the NDepend GUI, either.

Conclusion

Do you need NDepend? It depends. NDepend will be useful to developers and teams who have a functioning continuous integration process (and you should) and want to impose standards on this sort of code which can be checked in to the repository. But if you’re going to go down this road, you should probably start with FxCop, which is free and better known. NDepend, however, can take this kind of analysis much, much farther than would be possible even with custom FxCop rules, like comparing multiple versions of an application and mandating unit test coverage. I also find it much easier to customize.

If you do have an automated build procedure in place and you’re not already doing static analysis, it’s time to start! It can be a bit overwhelming to run static analysis for the first time on a legacy project and see hundreds of warnings or errors. But that’s easy to get over; simply disable all the warnings but one, and start cleaning things up a bit at a time. You can make the analysis stricter as time goes on. You don’t have to start by fixing everything. Even a few rules in place will prevent new such errors being introduced to the source base.

It will be also useful to developers confronting a large source base for the first time, particularly if there are more than a few assemblies involved.

While NDepend might be useful to a single developer reviewing their own code, and not part of an automated process, I’ve found that static analysis will really only be used regularly when it is automated. Few developers have the discipline to run static analysis and cleanup any problems uncovered without integrating it into build automation.

Disclaimer

This review is based on a review copy supplied by the makers of NDepend. If you think that has colored my impressions of the tool, well, I disagree, but at least you know.

Share This | Email this page to a friend

I think I know what the answer is. Very often, someone will suggest an approach to a problem which strikes me as technically wrong or otherwise misguided. Like a lot of geeks, I have an almost compulsive desire to do things correctly, especially in areas where (to me, anyway) there is a clear-cut difference between "the right way" and "the wrong way" of doing it. Over and over again, I have said "no" in a way which ranged somewhere between insufficiently respectful and inappropriate.

I think I can explain why I do this (and I’ll do so below, for anyone who actually cares), but the important part is this: I need to teach myself to say "No" in a much more constructive way.

When I hear (what I feel is) a bad idea, I think I have this little moment of panic about the bad idea getting into our code, and having to maintain it for 10 years, because goodness knows I have a lot of bad ideas to maintain in our VCS. It’s like I’m worried that simply mentioning the idea will doom it (or us) to implementation, so I went to put the barrier in front of it before it can go any further.

This may accomplish the technical goal, but saying "No" in an unconstructive way is poor leadership. I don’t want to create a climate where people hesitate to suggest solutions to a problem, because they might be incorrect. Indeed, I learned as least as much, if not more, from my mistakes is from my successes, so discussing a bad idea may contribute as much to a project is discussing a good idea. Rather than just saying, "That won’t work because…" I need to help the person who suggested the idea explore its implications and work out the ups and downs on their own.

Share This | Email this page to a friend

Therefore, I’ve updated the About page to include formal licensing terms. In particular, any source code I’ve written and posted here is available under the MIT license, which should allow you to do almost anything you want to with it, but read the About page for the particulars.

Thanks to Robert Claypool for pointing this out.

One other bit of news about this site: Readers who prefer to be informed of new posts via Twitter can now follow the blog there.

Share This | Email this page to a friend

A Useful, But Imperfect, Tool

Static analysis can be incredibly useful. It finds errors in much the same way that compilers report hints and warnings, but at a significantly deeper level. Compilers typically report a hint or a warning only in cases where the compiler is completely certain that there is a problem with the code, like a completely unused variable. Static analysis tools, on the other hand, can report errors in cases where, for example, the code itself might function, but appears to be too complex to be maintainable. Obviously, there is a gray area between maintainable and unmaintainable, and every static analysis tool I have ever tried has returned false positives. FxCop, for example, will complain that an internal type is never instantiated if the only time it is ever instantiated is inside of a LINQ to Entities query; it doesn’t seem to be able to "see into the Expression."

For Code Written By Humans Only

Static analysis is close to useless for generated code, because it is generated by a tool, tends to always function correctly, and is presumably never maintained by humans, at least directly. But generated code tends to really set off alarms on static analysis tools, due to high line counts, cyclomatic complexities, and numbers of parameters, no unit tests, unusual type and member names, etc. In other words, in generated code functionality is much more important than maintainability. But in non-generated code, maintainability is at least as important as functionality. The best solution is to convince your static analysis tool to ignore the generated code altogether, but this is often easier said than done.

When I run FxCop or NDepend against the source code for our commercial applications, for example, they report a large number of errors, with the majority coming from generated code. The reality is that no tool I have used can consistently distinguish between unmaintainable code written by code generators and unmaintainable code written by programmers, automatically. Therefore, in order to get the maximum benefit from static analysis, we have some additional work to do in order to help the tools look at only the code where their analysis will return useful information for developers.

Excluding Generated Code From Static Analysis

The .NET framework includes an attribute, GeneratedCode, which is intended to tell tools such as FxCop to ignore parts of your code. Also, FxCop has its own attribute, SuppressMessage which instructs the tool not to report individual messages in specific cases, and individual tests can be turned off. CodeHealer can be told to suppress a certain number of messages, and you can turn individual tests on and off.

Even with these tools, however, the line between generated and non-generated code is not always clear. FxCop, for example, looks only at compiled assemblies, not at source code. If you have a partial class containing both generated code and human-written code, it’s important to recall that FxCop cannot tell the difference between the parts of the class which came from the generated code file and the parts which came from the non-generated file. The distinction simply does not exist in the compiled assembly, because "partial" classes are combined at compile time by the C# compiler. So you’re left with the choice of putting the GeneratedCode attribute on the entire class, causing static analysis to miss code you wrote, or not putting it on the entire class, resulting in static analysis errors from the generated code. The best solution is probably to keep any code you write as part of a partial class as simple as possible, such as calls into another class which does the real work. Considering that partial classes are typically used for things like forms, or LINQ to SQL types, this is a good practice anyway.

Code generation can also work by producing IL or assembly directly, without a source file at all. Obviously, tools which look only at source code, like CodeHealer or Microsoft StyleCop will miss this sort of code (which is a good thing) by their very design. But tools which reflect into compiled code will see this sort of code as no different from any other code in the assembly unless they are specifically designed to exclude it.

For example, when you have an anonymous type with a large number of members in C#, the compiler will generate (in IL) a constructor with an equivalent number of arguments. NDepend flags these as excessive numbers of arguments, and, indeed, it would be correct if this were a real method which I had written.

Improving Heuristics to Avoid False Positives

NDepend offers an additional means of filtering data. The tool has a domain specific language for querying your source code, called CQL, and you can use this to filter the specific areas analyzed by NDepend. I can modify the NDepend rules by editing the CQL, so I can, for example, exclude the cases I’ve just described (generated constructors with large numbers of parameters) by changing the default rule for "too many arguments," which looks like this:

// <Name>Methods with too many parameters (NbParameters)</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE NbParameters > 5 ORDER BY NbParameters DESC
// METHODS WHERE NbParameters > 5 might be painful to call and might degrade performance.
// You should prefer using additional properties/fields to the declaring type to handle
// numerous states. Another alternative is to provide a class or structure dedicated to
// handle arguments passing (for example see the class System.Diagnostics.ProcessStartInfo
// and the method System.Diagnostics.Process.Start(ProcessStartInfo))
// See the definition of the NbParameters metric here http://www.ndepend.com/Metrics.aspx#NbParameters

to this:

WARN IF Count > 0 IN SELECT TOP 10 METHODS 
    WHERE NbParameters > 5 
        AND !IsGeneratedByCompiler
    ORDER BY NbParameters DESC

…and the false positives go away. Unfortunately, this only fixes one test. Similar CQL appears in other default NDepend metrics, like this one:

// <Name>Quick summary of methods to refactor</Name>
WARN IF Count > 0 IN SELECT TOP 10 METHODS /*OUT OF "YourGeneratedCode" */ WHERE 

                                           // Metrics' definitions
     (  NbLinesOfCode > 30 OR              // http://www.ndepend.com/Metrics.aspx#NbLinesOfCode
        NbILInstructions > 200 OR          // http://www.ndepend.com/Metrics.aspx#NbILInstructions
        CyclomaticComplexity > 20 OR       // http://www.ndepend.com/Metrics.aspx#CC
        ILCyclomaticComplexity > 50 OR     // http://www.ndepend.com/Metrics.aspx#ILCC
        ILNestingDepth > 4 OR              // http://www.ndepend.com/Metrics.aspx#ILNestingDepth
        NbParameters > 5 OR                // http://www.ndepend.com/Metrics.aspx#NbParameters
        NbVariables > 8 OR                 // http://www.ndepend.com/Metrics.aspx#NbVariables
        NbOverloads > 6 )                  // http://www.ndepend.com/Metrics.aspx#NbOverloads
     AND 

     // Here are some ways to avoid taking account of generated methods.
     !( NameIs "InitializeComponent()" OR
        // NDepend.CQL.GeneratedAttribute is defined in the redistributable assembly $NDependInstallDir$\Lib\NDepend.CQL.dll
        // You can define your own attribute to mark "Generated".
        HasAttribute "OPTIONAL:NDepend.CQL.GeneratedAttribute")

These must all be fixed individually. On the other hand, at least they can be fixed by easily changing the heuristics of the rule, instead of by excluding specific cases, as with many other static analysis tools.

One Time Pain

The real win from static analysis tools comes from integrating them into an automated build process. Tools like FinalBuilder and MSBuild can easily be configured to run all of the tools mentioned so far. This allows you to fail the build if potentially unmaintainable code is checked in. The first time you use any static analysis tool, you will probably need to spend a substantial effort in order to get the project to the point where it will not fail the build with existing code. This may involve turning off individual rules (you can gradually turn them on later), fixing specific violations, and excluding code such as generated code. You will then need to integrate the tool into your automated build process. This is, in all honesty, a fair bit of work, but it’s really a one-time pain. After that, the analysis will pay you back for your effort, day in and day out.

Share This | Email this page to a friend

Share This | Email this page to a friend

These are very bad approaches, for two important reasons:

1. They don’t work. If your action result is cached by ASP.NET, then the action will not even run the next time it is requested. AuthorizeAttribute handles this correctly. Code you write in a Controller cannot handle this. Code you write in a custom action filter could work, if you cut and paste the implementation from AuthorizeAttribute. But AuthorizeAttribute is unsealed, so why cut and paste, when you can subtype?
2. The modularity is wrong. You should aim to develop MVC sites which can be used with any authentication (or role) provider, whether it is ASP.NET membership, domain authentication, OpenId, or a custom membership provider. Wiring authentication concerns into a Controller makes this extremely difficult.

I’ve already hinted at the correct solution for the problem: If you cannot find an out of the box membership provider which does what you need, then look harder. If you still can’t find one, write your own. Here is a video and an article on how to do it. Similarly, the role provider can also be customized. In other words, the correct extensibility point for membership in an MVC site is exactly the same as the extensibility point for membership in a non-MVC ASP.NET site.

AuthorizeAttribute does a very simple thing in a somewhat complicated way. The simple thing it does is to tie the MVC framework into the existing ASP.NET membership system, which is quite customizable. That’s why I am skeptical about claims that AuthorizeAttribute can never serve someone’s needs. The implementation is more complex than one might imagine is required on first consideration because it must both be thread safe and cooperate with caching, which makes attempting to re-invent it at a different level a doubly bad idea.

Although AuthorizeAttribute doesn’t do very much, you may decide that it’s specific behavior is not correct for your project. In that case, it is generally a better idea to subtype AuthorizeAttribute instead of re-inventing your own. This allows you to preserve desirable behavior (like cooperating with caching) while customizing the behavior you need. You will need to override the AuthorizeCore method, taking careful note of the comments (in the MVC source code for the inherited method) on thread-safety.

One final word about customizing security: Your general approach should be to use proven, off the shelf, actively maintained solutions and customize as little as possible. If the problem you are trying to solve is not strictly security-related (like adding a custom field to a user profile), then do not alter (and potentially compromise) your security in order to solve it. Instead, find a more appropriate solution, like storing the information elsewhere, such as in an ASP.NET profile or your own DB. Similarly, don’t use something which is not intended to have hard security, like Session, for security purposes. Attackers can and do hijack session information. Doing security correctly is hard, and the consequences of small mistakes can be high. Even experts in the field make mistakes which can take years to uncover. Even if you follow to the letter everything which is known about implementing good security, today, your implementation may be obsolete tomorrow.

Share This | Email this page to a friend

How to Think Like the ADO.NET Entity Framework

(and why you might want to bother learning to do so in the first place)

In order to use the ADO.NET Entity Framework effectively, you must understand its value-based data model, which is distinctly different from the SQL and ORM approaches to representing and querying data. Programmers who approach the Entity Framework as "just another ORM" often find themselves frustrated or disappointed, a common reaction to trying to force a square peg into a round hole. This presentation examines the Entity Data Model in-depth, explaining where it fits into the context of a larger application, and demonstrates different techniques for retrieving and altering data. Also covered are performance considerations and future directions.

Sound like something you’d like to learn about? The conference is free, but you must register to attend. I’ll be on at 5:00 PDT; you can convert that to your local time.

Embarcadero has me in the Delphi Prism track, but my examples will be in C#.

If .NET’s not your cup of tea, there are plenty of other sessions you can join.

Update: The conference is over, but you can download a recording of my session.

Share This | Email this page to a friend

If you can persuade someone to use your modified compiler, or if you can modify the compiler on their machine, then you can infect the executables they produce, even if they recompile the compiler itself from source code! If that seems unlikely or difficult, read the whole paper. It’s easier than you think. Thompson concludes:

The moral is obvious. You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untested code. In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.

Now it seems that somebody has actually gone and done this with old versions of Delphi. Frankly, I’m surprised it took so long. Perhaps this is just the first time we’ve noticed. But the problem described by Thompson is so insidious that in the quarter century since he gave his address (which really just gave additional prominence to an issue discovered decades before that) it’s almost remarkable that this technique is not more widespread. It’s also interesting that this specific case is relatively innocuous; the virus does nothing more than reproduce itself, and only affects a very old compiler. Is this a test shot? Someone whose curiosity got the better of them?

I don’t know. But the historical perspective is clear: This is not about Delphi, and it’s not about the malware of the week. It’s an old question of trust becoming new again.

Here is one proposal for countering these attacks. Interestingly, the technique can sometimes find unintended compiler bugs, as well. They can be hard to distinguish sometimes!

Share This | Email this page to a friend

Share This | Email this page to a friend

Change Manager 5.1.1 will add support for InterBase 2007 and 2009 databases to: compare, version, and synchronize schemas; compare, synchronize, move, and mask data; and compare, version, and monitor version configurations for compliance and tracking.

And for any Firebird SQL users out there, you can join the party too because Change Manager will support versions 1.5 and 2.0 of that database. Move your data to and from any platform we support and move and compare schemas between Firebird and InterBase.

Also, it’s worth noting you can get Change Manager for free with the purchase of another Embarcadero tool, including Delphi or InterBase.

Share This | Email this page to a friend

Warning: The following comic has been demonstrated to severely lower developer productivity upon first introduction.

Sydney Padua has done the impossible: Created a programming comic which I like as much as XKCD. You can start at the beginning or read the latest, thrilling episode. She also did a special for the BBC. Navigation is a bit awkward (look for the "next" and "back" links above each post), but entirely worth the effort.

Share This | Email this page to a friend

The Give Camp organizers found charities who needed web development donations and split the registered developers into teams based upon skill sets and organization needs. I was assigned to the team for the Shiloh Family Institute College Choir Concert Series, a small organization which sponsors a series of gospel choir performances once a year. They needed a web site to promote the organization, list upcoming events, allow their patrons to register for e-mail alerts of scheduled concerts, and think their donors. Critically, the site had to be easy to maintain for administrators without computer expertise. My team included Brahma Ghosh, Brian Jackett, Joel Broughton, Joseph Ours, and Peter Vasys. Prior to the start of Give Camp, Brahma, the project manager, met with representatives of the organization and produced a mockup of each page we would build in the site.

We decided to build the sites using (non-MVC) ASP.NET, because that was the "least common denominator" of the skill sets of our team members. The requirements for the site specified displaying 10-20 concerts and around 10 sponsors. In view of the tiny data set and the emphasis on ease-of-use for the folks who will be maintaining the site, we decided against using a database or off-the-shelf content management system. Instead, we built the site such that the content areas would simply be editable in a WYSIWYG editor if you are logged into the site. So the site maintainers only needed to be instructed as to how to log in, and the rest is obvious.

Here is a part of what a logged-in user sees on the site homepage; you can compare that with the production site in order to see the difference.

WYSIWYG editor on site home page

At the beginning of Friday night, we met our clients, discussed the requirements for the project, settled on which technology we would use to build the site, and got to work. One of the first things we needed to do was to register a domain name and secure hosting so that the site could go live on Sunday. Also, we set up a Subversion repository on assembla.com. Then we split up the work and started writing code. The project manager created a solution for the project and checked it into subversion. I started creating repositories for the data which would be editable on the site. Because each dataset would only hold 10 through 20 records maximum, we decided to use XML files instead of a database to hold the data. Performance is not a problem with such a small dataset, and using files instead of a database allows non-technical users to easily move the site from one hosting provider to another. Also, we used the repository pattern, so it’s possible to switch out the storage without changing a line of web client code in the event the site ever grows beyond its current scale. The first thing I did was check in the public interfaces for the repositories so that the people creating the web pages could start work before I was done with the implementation. Other team members created web pages, started the site design, and began implementing a Google Maps feature on the "upcoming concert" page.

We wrote unit tests for all of the repositories in the site. In view of the very short amount of time we had to develop the site, that might seem surprising. In fact, however, writing unit tests allowed us to test the repository functionality before the web pages which used the repositories were finished, so even though it was a little extra work, it improved our ability to work concurrently.

I went home to get some sleep around 2 a.m. Friday night, but some people worked through the whole night. The Give Camp organizers supplied us with food, caffeine, and entertainment. I spent most of Saturday implementing a feature to allow the site maintainers to add and remove sponsors, including a feature to upload sponsor logos and thumbnail them to the correct size for the site. By the end of Saturday, just after midnight, all of the major features of the site except for the Google Maps were working, and we were down to fixing bugs and typos. A pleasant surprise for me was that the food provided was consistently good and occasionally even healthy; the Give Camp organizers had recruited some local restaurants as sponsors, and a grateful charity brought in some large containers of fruit.

Incidentally, there are some photos of Give Camp on Flickr.

We spent the first half of Sunday cleaning up defects, and the second half testing and fixing typos in the content. Some of the other team members implemented the Google Maps feature in two different ways in order to be sure that we would have at least one which worked before we deployed the site to production.

At 4:00 p.m. on Sunday, all eight teams presented their work. There was a real diversity of projects. In our case, we built a simple site from scratch and finished it the same weekend. Other teams added new features to existing sites. One team was given three or four different sites using different technologies with the instructions to make them have a common look and feel. Every one of the clients appeared completely delighted with the end results.

Whenever I read "studies" which claim that the majority of software projects fail, I have to balance these claims against my own personal experience, which demonstrates that projects can be successful even with very little time and team members who have never spoken to each other before the instant the project begins. So why did this project work out so well?

• Our project manager did excellent work up front with the client, so we knew exactly what we needed to build.
• The client had a good sense of priorities for the project.
• The team was highly motivated. We were all participating by choice. The Give Camp organizers kept the emphasis on fun. The deadline was challenging, but realistic.
• The Give Camp organizers did a great job splitting up the teams. We had people with project management skills, design skills, and development skills.
• We used off-the-shelf functionality where appropriate. For example, the mailing list registration is handled via the MailChimp API. The map feature is done with Google Maps. Event photos are hosted on Flickr. We used their API to show a single image when there was only one image of an event available, and a slideshow when multiple images were available.
• We built only what was needed. Using an off-the-shelf CMS would have delivered more features, but at the cost of being difficult for the client to maintain, both in terms of the user interface and a database on the back end. The client made it clear that ease-of-use was paramount, and we stuck to that.
• There was no dead weight on the team. A single, unmotivated developer can slow down an entire team. Since everyone was there by choice, and everyone was qualified for the work, we didn’t have to deal with any problem team members.
• We had our choice of technologies, so we could choose a platform with which everyone on the team had at least some familiarity.

Give Camp was the most fun I’ve had coding in some time, even though (non-MVC) ASP.NET is not my preferred platform. The emphasis on fun, the opportunity to try things outside of my comfort zone, and the excellent team atmosphere made it a great experience.

Share This | Email this page to a friend

Given that calling Skip(0) introduces a performance penalty, but there is a reason for introducing that penalty, should we avoid making this call? The short answer is yes, the performance improvement is worth it, but don’t forget the rule that you should have one test case for every conditional in your code. This rule is generally applied to unit testing, but in this case the appropriate test is an integration test, since you need to be sure that the LINQ provider will perform acceptably and not fail altogether when you fetch the data for the second page, which is now an entirely different query than that used to retrieve the data for the first page.

When I first observed the issue, I was happy to discover that I could make the query which fetches data for the first page of a result set (which happens to be the page which users most commonly request) 10 times faster. Of course, the other way to look at this is that the query that fetches data for the second and all subsequent pages is 10 times slower than the query which fetches data for the first page. Again, given that the first page is displayed much more frequently, that’s not necessarily a bad thing, as long as it doesn’t catch you by surprise. Indeed, calling Skip(n), where n > 0, and getting your results not quite as quickly as if you had not called Skip(n) at all is actually the best case; the worst possible case is that the LINQ provider will generate invalid SQL and the query will fail.

Note that it is especially difficult to ensure complete integration testing coverage as you introduce more and more ways that the user can shape a result set. If the user can change the sort order, enter a variety of search queries, etc., it’s a safe bet that every combination of these options will produce a different query with different performance characteristics. Avoiding calling Skip(0), in other words, does not increase the number of integration test cases for a particular list of data by one; it doubles it.

Share This | Email this page to a friend

Share This | Email this page to a friend

Share This | Email this page to a friend

Unfortunately, it is not just your own code you need to be concerned with. It turns out that some fairly popular LINQ helpers do this. For example, Rob Connery’s PagedList<T> type does it:

public PagedList(IQueryable<T> source, int index, int pageSize)
{
    this.TotalCount = source.Count();
    this.PageSize = pageSize;
    this.PageIndex = index;
    this.AddRange(source.Skip((index - 1) * pageSize).Take(pageSize).ToList());

Index, in this code, is the 1-based page you’d like to display. When index is 1, the code calls Skip(0), and you get less efficient SQL. Fixing the problem is quite simple:

    this.PageIndex = index;
    var pageData = index > 1 ?
        source.Skip((index - 1) * pageSize).Take(pageSize) : source.Take(pageSize);
    this.AddRange(pageData.ToList());

Naturally, this only optimizes SQL generation for the first page of results, but since this is by far the most common page a user might display, it can net you a substantial win in server performance.

Troy Goode’s updated PagedList type, which I have recommended in the past (and still do, with this tweak!), has the same problem. The solution is almost identical, although Troy’s type is 0-based rather than 1-based. Update: Troy has now posted an update containing this fix, among many other changes.

The root of the problem in the SQL generation is that when you call Skip(0), LINQ to Entities, in at least some cases, generates SQL like this:

)  AS [Project3]
WHERE [Project3].[row_number] > 0
ORDER BY [Project3].[TimeRecordDate] DESC

The reference to row_number can have a substantial performance impact. In one case I tried with a SQL Server table containing just over 1 million rows, a simple query ran with sub-second performance without this WHERE clause, and over 4 seconds with the WHERE clause, even though the result set was exactly the same.

Now, is this a bug in LINQ to Entities? Or LINQ itself? It’s hard to say. Both frameworks are doing exactly what you tell them to, even though it doesn’t make a lot of sense. It seems like a missed opportunity for optimization, but I’m open to the possibility that there might be some desirable effect of this that I have simply missed. For my own purposes, I am content to simply not call Skip(0) at all.

Share This | Email this page to a friend

It’s worth a listen if you’re interested in the history of the product which started Borland and evolved into Delphi.

Share This | Email this page to a friend

Download the demo solution: DayPilot ASP.NET MVC demo

We needed a calendar control for an ASP.NET MVC application. In particular, we needed a control which would show a number of events and when they occur within a particular day, "Outlook-style." I’ll include a screenshot of the demo application here, since it’s the best way to explain the type of control we were looking for:

DayPilot example

Possible Alternatives to DayPilot

In my experience, the type of control which works best with ASP.NET MVC is a control which is completely ignorant of ASP.NET, and all the lies such as ViewState and postbacks which accompany it. In particular, I’ve had a good deal of success with controls designed for the jQuery JavaScript framework, which is included, by default, with ASP.NET MVC applications. Unfortunately, I was unable to find a jQuery control which data what we needed. The closest I found was jMonthCalendar which cannot display a single day in the manner illustrated above. I would still prefer a jQuery-based solution if one ever turns up.

I did, however, find a couple of controls designed for "plain" ASP.NET which, from an end-user point of view, it exactly what we needed. One was the DayPilot control illustrated above, the other was the scheduler component included with Telerik’s control suite. Telerik’s control claims some support for MVC, but this is really limited to loading calendar data asynchronously, which was not actually what we wanted to do. Also, Telerik’s control can only be purchased as part of their suite, which is rather expensive if you only need the calendar/scheduler control. Paying for the entire suite would not be a problem for us if it was filled with controls we needed, designed for the environment (MVC) we use. While Telerik seems serious about supporting MVC, they are in the early stages right now, so it is difficult to justify paying for a large suite in order to get a single control, which I would have to use in a mostly-unsupported way. The Telerik control has many more features than the DayPilot control, and, in my opinion, looks a little nicer.

The DayPilot calendar is available in two different editions. There is an open source edition, which is free, and a non-open-source edition, which adds features. The principal difference is that the date version includes editing (via postbacks) and UpdatePanel support, which are not really useful in an ASP.NET MVC application anyway. As with Telerik’s control, I have to adapt the control for use within ASP.NET MVC. So we selected the open source version, even though the paid version is priced reasonably.

Adapting DayPilot to ASP.NET MVC

In order to use the DayPilot control in ASP.NET MVC, I wanted to use markup which would look familiar to ASP.NET MVC developers:

<%= Html.DayPilot(Model, new DayPilotViewOptions
                             {
                                 HourHeight = 30
                             }) %>

This means that I needed to write an Html helper which will render the control based on the data passed in Model and return the literal HTML markup. In order to make that easier, I created two new types, DayPilotData and DayPilotViewOptions, in order to encapsulate access to the properties of the control. The reason I created two different types was to separate the concerns of data for the events (which will be set up in the controller) and display of the control (which is the concern of the view). The types are trivial, so I’m not going to show them in this post, but you can examine them in the demo solution. There is an overload to this Html helper which does not include the options argument in case you are happy with the defaults.

        public static string DayPilot(
            this HtmlHelper helper,
            DayPilotData model,
            DayPilotViewOptions options)
        {
            var calendar = new DayPilotCalendar();
            if (model != null)
            {
                model.CopyTo(calendar);
            }
            if (options != null)
            {
                options.CopyTo(calendar);
            }
            var sb = new System.Text.StringBuilder();
            sb.Append("<div class=\"dayPilot\">"); // allows working around td cellpadding bug in css
            using (var sw = new System.IO.StringWriter(sb))
            {
                using (var tw = new HtmlTextWriter(sw))
                {
                    calendar.RenderControl(tw);
                }
            }
            sb.Append("</div>");
            return sb.ToString();
        }

Note the div. The current version of DayPilot astonishingly, as of this writing, does not put a class anywhere in its rendered markup. Since the stylesheet for the default ASP.NET MVC site conflicts with DayPilot, to some degree, I wrapped the whole control in a div so that I could fix the layout in CSS.

Another way which I could have used the control in ASP.NET MVC would be to register a tag prefix and use code behind/code generation, as with standard ASP.NET. I’ve registered a tag prefix in the demo solution in case you want to try that yourself. Be aware, however, that setting up codebehind and code generation files in an MVC application is somewhat less than simple; doing it right requires, among other things, manual editing of the csproj file. So I do recommend using the control in the way I’ve illustrated.

Finally, here’s the source code for the action which supplies data for the control. Since this is a demo solution, I’m generating random events:

        public ActionResult Index()
        {
            var r = new Random();
            var q = from i in Enumerable.Range(1, 4)
                    let start = DateTime.Today.AddHours(r.Next(47))
                    select new DayPilotDataItem
                    {
                        Id = i,
                        Start = start,
                        End = start.AddHours(1),
                        Text = string.Format("Event {0}", i)
                    };
            var model = new DayPilotData
            {
                StartDate = DateTime.Today,
                Days = 2,
                Data = q
            };
            return View(model);
        }

Of course, it’s just as easy to select data from a database.

The current version of DayPilot has a kind of a weird limitation: You cannot make it smaller, in the vertical dimension, than 30 pixels per hour. Based on some inspection of the source code, I think this is fixable if you are comfortable altering the source. But be aware of the limitation.

There are more DayPilot features which I have not covered, but that should be enough to get you on your way. If you have any questions, post them in comments, and I’ll see whdat I can do to help out.

Share This | Email this page to a friend

Finding your blogs very helpful - superbly written - just found out about jqgrid last night and have integrated your code into my project.

One thing that didn’t work for me though was the dates fix above…

adding

.ToString(System.Globalization.CultureInfo.CurrentUICulture)

onto the end of my DateTime values makes everything render Ok but when I try doing a search, it fails. Debugging shows the error is

Method ‘System.String ToString(System.IFormatProvider)’ has no supported translation to SQL.

at the TotalItemCount = source.Count(); line of the PagedList.cs file.

Any ideas?

In order to understand this error, we need to remember several important things about LINQ:

• LINQ is lazy. Creating or modifying an IQueryable with methods like Where, Take, OrderBy, etc. does not actually execute the query. The query will only be executed when you call a LINQ method which requires execution to produce its results — like Count.
• When a LINQ query expression includes something which looks like a method call, the method is not invoked directly. Instead, what is actually produced is an Expression representing that method. If the LINQ provider happens to be LINQ to Objects, the method will eventually be invoked when the query is executed. But if it happens to be LINQ to SQL or LINQ to Entities, the method may never be invoked, and instead might be replaced (if possible) with equivalent SQL.
• Most LINQ providers do not support every possible LINQ query expression. Most providers don’t even support the entire LINQ API, and only LINQ to Objects can be presumed to support all method calls on an object in a query. In some cases, as with LINQ to Entities, this is documented. With other providers, you may have to guess/experiment. Any LINQ provider which has to translate a LINQ query expression into SQL syntax can only really support methods which it is hard-coded to recognize. Worse, it is not even uncommon for providers to support one overload of a particular method but not a different overload of the same method. See, for example, the LINQ to Entities document I just linked or the ToString method, which is generally supported in LINQ to Entities/SQL, but has a special overload on the DateTime type which is not supported. Worse still, method calls may be supported in some contexts but not others. Graeme reports that LINQ to SQL accepted the ToString overload in the context of a Select, but not in the context of a Where. This makes sense from an implementation point of view, but means programmers have to be extraordinarily careful about what we include in our LINQ queries, especially when they are constructed dynamically.

By now it should be obvious what is going on here: The particular overload of ToString which I used in my example is supported in LINQ to Objects but not in LINQ to SQL in a Where expression. My demo solution used a repository based on LINQ to Objects, so the query I showed worked correctly there. But it does not work with LINQ to SQL. Because LINQ is lazy, you will not actually see an error about this until the query is translated into SQL by the LINQ provider, and that does not happen until the call to Count in PagedList.

This means that when you write a LINQ query expression using LINQ to SQL, LINQ to Entities, etc., you are essentially working in a partially dynamic language. Unless you study the documentation very, very carefully, you will not know for certain until runtime if your query will be accepted by the provider.

So how could we work around this problem?

• Use a different method or a different overload which is supported by the LINQ provider used in your project. This is fine if an appropriate method/overload exists.
• Give up on doing the formatting in C# and use JavaScript instead. This will work, but it’s a bit like using beheading to cure acne. C# is, after all, a general-purpose programming language. We should not have to give up on formatting a date.
• Bring the query into LINQ to Objects using a method such as AsEnumerable. Obviously, we would want to apply the paging and ordering first. In this method, we would execute one query (using LINQ to SQL or something) which would retrieve an ordered list of records for the specified page, using AsEnumerable to transform the results into a List. We would then execute a LINQ to Objects query against this list in order to perform the formatting. This will actually work OK, although it can end up being a good bit of code to write for what should be a fairly simple task.
• Write a new LINQ provider which understands the method in question. This seems like an enormous amount of work, but I’ll include it for completeness.
• Separate the concerns of serialization and querying. You can tell by the 50-cent words that this is the solution I prefer.

Remember, the original query worked just fine. It is only the serialization to JavaScript which caused unexpected results. One way to fix this is to hide the members which are problematic to serialize and replace them with a string version of same:

private class JavaScriptDateFixer
{
    [NonSerialized]
    internal DateTime _date;
    public int Id { get; set; }
    public int IntProperty { get; set; }
    public string StringProperty { get; set; }
    public string DateProperty { get { return _date.ToString(System.Globalization.CultureInfo.CurrentUICulture); } }
}

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new JavaScriptDateFixer
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    _date = entity.DateProperty
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that even though this code looks quite different, this does almost exactly the same thing as the code I’ve replaced. The only real difference is that I’ve delayed the call to ToString until after the query is executed. That’s enough to keep LINQ to SQL or LINQ to Entities happy.

So with this fix is everything fine? Maybe. I really like LINQ; I think it’s an extraordinarily powerful tool. But here we have hit upon what I consider to be the single biggest problem with LINQ. The combination of very limited support for all possible query expressions in common LINQ providers like LINQ to SQL and building queries dynamically can be very dangerous. Unless you test all possible query permutations, you cannot be completely confident that any query you build will actually execute successfully. In typical dynamic language programming, you mitigate this problem by writing unit tests. But unit tests are not supposed to connect to a database. Unfortunately, the limitation here is that the point of translation to SQL. So any unit test which mocked the LINQ provider, say, using LINQ to Objects or some other in-memory representation would not actually encounter the limitations we are trying to test.

I have yet to see a really good solution for unit testing LINQ to SQL or LINQ to Entities, one which would combine the desired attributes of not actually connecting to the database while surfacing any potential problems in SQL generation. I suspect that such a beast may never exist for LINQ to SQL, but for LINQ to Entities it seems at least possible. One could write an Entity Framework provider which returns mocked instances of entities. Any LINQ syntax not understood by the Entity Framework would be caught before the mock provider was ever invoked. This is a feature I would like to see in a future version of the Entity Framework.

Share This | Email this page to a friend

Share This | Email this page to a friend

The delete feature of jqGrid is, oddly, almost entirely undocumented, even though there quite a few examples of different methods of editing, and inserts are documented, to a lesser extent. But it does exist; I just had to read the source code to figure out how it works.

JqGrid has three different methods of editing data. The cell edit feature allows the user to edit a single cell the grid at a time, and saves the data when the cell loses focus. The row edit feature works similarly, but does not save the data until the row loses focus. The form edit feature shows a modal dialog instead of editing the values inline. Of these three, the form edit feature is the only one which supports deletes. However, you can still use one of the other "modes" for editing, if you like. Just be aware that when you do a delete, you will be in the form edit code. Also, deletes won’t work if you change the grid’s configuration to disable the form edit feature.

In order to allow the user to delete a row, I need to:

• Add a delete button to the grid’s toolbar. I could, of course, use a custom button or something instead of the grid’s toolbar if I chose to do so.
• Specify a URL for the delete action
• Implement a delete action on the controller

I can accomplish the first two tasks by making a slight change to the JavaScript that configures the grid:

}).navGrid(pager, { edit: false, add: false, del: true, search: false }, {}, {}, {url: "Delete"});

I’ve highlighted the parts I’ve changed in boldface. Changing the "del" property to true turns on the toolbutton. The fifth argument is an object specifying options for the delete action, of which I have only specified the URL. This is important: The only places you can specify the URL for delete are in the arguments to the delGridRow method or in the toolbar’s configuration. There is not a "delete URL" property of the grid itself.

If you don’t specify a delete URL in this way, the grid will presume that you want to send delete requests to the URL you have specified for editing rows. There will be an "oper” value in the submitted form indicating that the request is a delete rather than an edit. But I prefer to use a separate URL for delete, because I think that fits more naturally in the ASP.NET MVC paradigm.

Now let’s write an action to handle the delete:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Delete(int id)
{
    var repository = new Repository();
    var deleted = repository.Delete(id);
    if (!deleted)
    {
        Response.StatusCode = 500;
        return Content("Record not found.");
    }
    return Json(true);
}

Again, I’ll note that I’m using a mock repository for this demo so that you can compile and run the solution without needing to set up a database. Since the mock repository is based on IList, the Delete method of the repository returns a Boolean indicating whether or not the requested record was found, since that is how IList behaves. It would probably be more correct to return a partial view in the event of an error rather than specifying the contents directly, but error handling is a subject I hope to cover in more depth later on. For the time being, simply changing the StatusCode will tell the grid when something does not work.

If the delete was successful, I return "true." By default, the grid ignores returned data when the operation is successful. However, you can handle the grid’s afterSubmit event, which will be passed any data you do return. So what you choose to return, and what you do with it, is entirely up to you.

With all that in place, deletes work as expected. I’m not going to update the demo solution just yet, because I’ll be covering edits and inserts real soon now. But first I need to revisit formatting and explain more about LINQ".

Share This | Email this page to a friend

First, let’s fix the date display. The reason the date appeared so oddly is that there is no official support for Date types in JSON, even though JavaScript has a date type. Microsoft, therefore, invented their own way of encoding dates for their JSON serializer, which jqGrid does not understand. There are two possible ways to fix the problem. The first is to keep sending the data to the grid in the Microsoft format, and write a custom formatter for the grid to parse that into a human-readable string. The second is to change the data we send to the grid. Both ways work, but I’m going to demonstrate the second method, mostly because I find C#’s date handling features superior to (and less confusing than) JavaScript’s.

In order to change the date "column" in the results of the query to a formatted string, we simply need to project into an anonymous type. However, this would break the sort, because a text sort on a formatted date will not be in the correct order. Hence, it is necessary to apply the sort before projecting. So I add an OrderBy to the query, and pass null as the orderBy argument to the ToJqGridData method.

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    DateProperty = entity.DateProperty.ToString(
                        System.Globalization.CultureInfo.CurrentUICulture)
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that in this revised solution I’ve updated the mock repository to return more sensible dates.

Important update: This code works correctly with LINQ to Objects, as with my demo repository, but will not work with LINQ to SQL. I have addressed this issue in a subsequent post.

Now let’s add a search feature. One of the problems in making a "demo" search is that search is a highly application-specific feature, both in terms of implementation and in terms of user experience. JqGrid’s search feature presumes that you want a search which looks something like this:

Visually, there are lots of variations on this theme available, but in terms of the use case they all presume that the user will specify which fields she would like to search. However, I prefer a "Google-style" interface with just a search box and an implementation that can (hopefully) figure out what the user actually wants. So that’s what I’m going to demonstrate. The needs of your application may be different. The important thing to take away from this demonstration is that the jqGrid search feature can be co-opted to fit your design.

I want the search box to appear above the grid at all times, rather than requiring the user to press a toolbutton to see it. So first I need to add an empty div to the markup in order to indicate where the search box will be positioned, relative to the grid:

    <h2>GridDemo</h2>
    <div id="search"></div>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>

Next, I add a small bit of JavaScript to tell jqGrid to replace this div with a search control:

    search.filterGrid("#" + grid.attr("id"), {
        gridModel: false,
        filterModel: [{
            label: 'Search',
            name: 'search',
            stype: 'text'
        }]
    });

The filterModel tells the grid to show a search control for one column, called "Search." Note that the entities I’m returning do not have a property called "search." That’s OK. When the user types something in the search box and presses enter, the grid will add a query string parameter to its request for the data of the form "search=WhatTheUserTyped". That’s what I want, and the fact that there is no "search" column in the data does not adversely affect the grid in any way.

The first argument to filterGrid is a little odd. I’m required to pass a jQuery selector as a string rather than being able to pass a jQuery object directly. Since this is all inside a method which accepts jQuery objects as arguments (rather than hard-coding the IDs of the DOM objects in the method, for unit-testability), I have to "decode" the jQuery object into a selector string. But note that I did not need to do that when setting up the pager or the grid itself. Like I said before, the grid’s API is consistently inconsistent.

As you can see in the code at the beginning of this post, the GridDemoData action just passes the search query string parameter through to the ToJqGridData method unchanged. Inside that method, I use Dynamic LINQ to alter the IQueryable to implement the search. I’ve cleaned up this method substantially since I first posted the code. Here’s the revised method:

/// <summary>
/// Adds a Where to a Queryable list of entity instances.  In other words, filter the list
/// based on the search parameters passed.
/// </summary>
/// <typeparam name="T">Entity type contained within the list</typeparam>
/// <param name="baseList">Unfiltered list</param>
/// <param name="searchQuery">Whatever the user typed into the search box</param>
/// <param name="searchColumns">List of entity properties which should be included in the
/// search.  If any property in an entity instance begins with the search query, it will
/// be included in the result.</param>
/// <returns>Filtered list.  Note that the query will not actually be executed until the
/// IQueryable is enumerated.</returns>
private static IQueryable<T> ListAddSearchQuery<T>(
    IQueryable<T> baseList,
    string searchQuery,
    IEnumerable<string> searchColumns)
{
    if ((String.IsNullOrEmpty(searchQuery)) | (searchColumns == null)) return baseList;
    const string strpredicateFormat = "{0}.ToString().StartsWith(@0)";
    var searchExpression = new System.Text.StringBuilder();
    string orPart = String.Empty;
    foreach (string column in searchColumns)
    {
        searchExpression.Append(orPart);
        searchExpression.AppendFormat(strpredicateFormat, column, searchQuery);
        orPart = " OR ";
    }
    var filteredList = baseList.Where(searchExpression.ToString(), searchQuery);
    return filteredList;
}

Importantly, this method is simply adding to the expression represented by the IQueryable. It does not actually implement the search. So any search methodology which understands LINQ can deal with its results. At least, theoretically. In practice, LINQ providers vary greatly in terms of which LINQ features they support. So although this method works fine with LINQ to Objects (as used in the rest of the demo solution) I cannot guarantee that it will work with every LINQ provider in the world. You have to try, and revise it to fit the capabilities of the provider you’re using.

Putting all these pieces together, I now have a search feature which does what I want:

Be sure to download the updated solution in order to get the bug fixes and new features.

As a special, no-extra-charge bonus, the updated solution also includes a demonstration of using the grid’s tree view mode with client-side data from a JavaScript function. This has nothing to do with ASP.NET MVC; it’s just an example of how to use a grid feature which I wrote in response to a request in the jqGrid support forums.

That’s all for today. In the next post in the series, I’ll begin to demonstrate the grid’s editing features, and how to use them in an ASP.NET MVC application. But I will also take requests. If there’s a grid feature you’d like to see demonstrated query question regarding using the grid in ASP.NET MVC, please feel free to make a request in comments, and I will answer it as best I can.

Share This | Email this page to a friend

Share This | Email this page to a friend

On The Podcast at Delphi.org, Nick Hodges confirms my earlier speculation about reorganization in the "CodeGear" group at Embarcadero. The whole show is worth listening to. Congratulations are due to Chris Pattinson, who was promoted to Director of Quality for all of Embarcadero. Time to update the subtitle of your blog, Chris! (And maybe even write a post or two…) Sounds like good changes all around. You read it here first, folks!

I wrote a short demo showing how to use jqGrid in tree grid node with client-side data supplied by a JavaScript function.

Share This | Email this page to a friend

Rather than make you set up the Northwind database, I’ve written a quick, mock repository which supplies random data for the application. But the data persists for the lifetime of the app, so I’ll be able to demonstrate sorting, paging, etc. This way you can just run the application, without having to set up SQL Server.

So let’s write some code display a grid.

The data I’m going to display in the grid is a fairly useless type:

    public class SomeEntity
    {
        public int Id { get; set; }
        public int IntProperty { get; set; }
        public string StringProperty { get; set; }
        public DateTime DateProperty { get; set; }
    }

We need two actions. The first will display the page containing the grid. The second supplies data for one page of the grid. Remember, the grid can page through large datasets. It will only request one page of data at a time.

        public ActionResult GridDemo()
        {
            return View();
        }

        public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
        {
            var repository = new Repository();
            var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
                new[] { "IntProperty", "StringProperty", "DateProperty" });
            return Json(model);
        }

The first action simply returns a view containing the HTML table which will eventually become the grid. It doesn’t need a model, because the grid will request its data asynchronously. The second action handles the data requested by the grid.

In order to make this work, we need HTML elements for the grid and its pager in the view, along with a reference to the JavaScript which will turn those elements into the grid.

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <h2>GridDemo</h2>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>
</asp:Content>

<asp:Content ID="Script" ContentPlaceHolderID="ScriptContent" runat="server">
    <script language="javascript" type="text/javascript" src="<%= Url.Content("~/Scripts/Home.GridDemo.js") %>"></script>
</asp:Content>

The JavaScript file is where the grid itself is configured.

$(document).ready(function() {
    GridDemo.Home.GridDemo.setupGrid($("#grid"), $("#pager"));
});

GridDemo.Home.GridDemo = {
    setupGrid: function(grid, pager) {
        grid.jqGrid({
            colNames: ['Int', 'String', 'Date'],
            colModel: [
                        { name: 'IntProperty', index: 'IntProperty' },
                        { name: 'StringProperty', index: 'StringProperty' },
                        { name: 'DateProperty', index: 'DateProperty' },
                      ],
            pager: pager,
            sortname: 'IntProperty',
            rowNum: 10,
            rowList: [10, 20, 50],
            sortorder: "asc",
            url: "GridDemoData"
        }).navGrid(pager, { edit: false, add: false, del: false, search: false });
    }
};

Important to note here are:

• colNames, the grid column captions the user sees.
• colModel,which includes the property name, which is the field name in the returned JSON data which supplies the data for that column, and index, which is the name which will be sent to the GridDemoData action as the sidx (field name to sort on) argument when you click on a grid header. There are a number of other properties you can set here which control editing, with, formatting, etc. See the documentation for full details. Importantly, colNames and colModel must have the same number of entries, and be in the same order.
• url, which tells the grid where to get its data. You’ll notice that the value I’ve passed is the name of the second action.

As I mentioned in the last post, I set a number of the grid options in a script referenced by the Site.Master, in order to prevent having to set them on every grid I create. The configuration idea for the grid on an individual page should be only the things that are unique to that page. So there is a fair bit of configuration for the grid which is not included here. In addition, the Site.Master contains references to the grid’s JavaScript and CSS files.

Finally, it’s worth mentioning that when you add jqGrid to a project you need to edit the jQuery.jqGrid.js file to tell the grid where to find the rest of the JS files it needs. Once you look at this file, it is obvious what you need to change.

At this point, I’m going to make the demo application available for download, so that you can experiment with it yourself. However, I’ll caution that I intend to update it when I write the next post in this series. In other words, the current download is a work in progress. It will change in the next few days.

Download the Solution (587 KB)

So let’s run the solution and see what we get:

OK, the hideous and eyestrain-inducing clash between the default MVC theme and the jQuery UI "lightness" theme included with the grid download notwithstanding, this is actually not bad. Sorting and paging both work, and performance is very good.

But what’s the deal with the data in that date column? It turns out that there is no standard representation of a date in JSON, so Microsoft invented their own. The grid doesn’t know about this special format. We’ll clean that up in the next post in this series and also cover search. Stay tuned!

Share This | Email this page to a friend

Nick Hodges has a new role/job title.

Item 2:

During his presentation to the Columbus Architecture Group a couple of weeks ago, Mike Rozlog handed out business cards which gave his title as "Senior Project Manager, Java Tools." He told us that in a couple of days he would have new business cards, because his role was changing. He didn’t tell us precisely what the new role would be, but I got the feeling that, like Nick, he was happy with the change.

Now, it is of course possible that these two changes happening so close together is merely a coincidence. But my personal speculation is that there are bigger plans afoot, and I’m glad to see that Mike and Nick, two people I trust, seem to be happy with them.

Repeat: This is just my personal speculation. It is based on the evidence above, not inside information. I could be wrong.

Share This | Email this page to a friend

But it turns out that we’re using the grid in a different way, and I think the difference is important. So rather than just referring you to Phil’s post and skipping the rest of the series, I think I still have some important information to contribute. In particular, I’m using the grid in a way which allows me to write an extension method for IQueryable<T> which returns data suitable for the grid, without having to know anything about the type T. If you look closely at the code in Phil’s post, you will see that populating the cell array requires custom code (or use of reflection) every time you want to return data to a grid.

It’s important to me that the method should work without knowing anything about the type of the data in the list because I don’t want to have to write repetitious code in every action method which supplies data to a grid to shape the data according to the grid’s needs, and because I generally use anonymous types for JSON serialization. JSON cannot handle object graphs with cycles (i.e., a "circle" of objects which all reference each other). The .NET JSON serializer will throw an exception if you pass it an object which contains a circular reference, or refers to another object which does. Since this is largely data-dependent and, worse, can be occasionally masked by lazy loading and the like, the safest way to serialize to JSON is to pass it data which provably can never contain a cycle. Anonymous types work really well for this. I’ll probably elaborate on this point in a future post. For the time being, the important thing is that I need to be able to take any IQueryable and turn it into data suitable for the grid, without having to think about what type that list might contain.

In a nutshell, this is what I want to be able to do:

public JsonResult ListGridData(int page, int rows, string search, string sidx, string sord)
{
    var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
      new[] { "Column1", "Column2", "Column3" })
    return Json(model);
}

In this example, repository.SelectAll() returns an IQueryable<TSomething>. I’ve written a method, ToJqGridData, which can transform that IQueryable into data suitable for the grid without having to know what TSomething is. (What the heck is that string array? Don’t worry about that just yet; I’ll cover it in a little bit.) The argument names aren’t what I would have picked, but they come from the grid. There is probably a way to change them, but I’ve never bothered to look.

The key to this is a grid property called jsonReader.repeatitems. I’m not sure why it’s called repeatitems, but here’s what it does. When repeatitems is true (the default), data is returned to the grid in a "cell array," like in Phil’s post. That is, an individual row of data in the JSON looks like this:

{id:"1", cell:["cell11", "cell12", "cell13"]}

When repeatitems is false, the data looks like a regular object in JSON format:

{id:"45678",name:"Speakers",note:"note",stock:"false",ship:"4"}

Interestingly, it turns out that this is the format the .NET JSON serializer already uses. So I don’t have to write any code to get this portion of the data into the right shape. I just hand an object to the JSON serializer and let it do its thing. The only things I need to do to make the data work for the grid is to handle paging, sorting, search, and provide certain other bits of data, like the number of rows.

But, as you may have noticed, I’m lazy. Why write code when someone else has already done it? Rob Conery wrote a decent type for paging eons ago, and Troy Goode enhanced it. So I’m going to use their work instead of re-writing paging. It turns out this type already computes most of what the grid needs to know. And Microsoft has this thing called Dynamic LINQ which can deal with ordering and filtering by property names, mostly. All I have to do is tie this stuff together.

I wrote a type to hold the data which the grid needs. You can actually use an anonymous type for this; the JavaScript grid obviously doesn’t care if the data it receives comes from an anonymous type or not, but because I am writing extension methods to IQueryable, it was convenient for me to have a non-anonymous type so that the extension methods could build on each other. This type represents one page of data requested by the grid:

namespace GridDemo.Models
{
    /// <summary>
    /// This type is designed to conform to the structure required by the JqGrid JavaScript component.
    /// It has all of the properties required by the grid. When this type is serialized to JSON, the resulting
    /// JSON will be in the structure expected by the grid when it fetches pages of data via AJAX calls.
    /// </summary>
    [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId="Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    public class JqGridData
    {
        /// <summary>
        /// The number of pages which should be displayed in the paging controls at the bottom of the grid.
        /// </summary>
        public int Total { get; set; }
        /// <summary>
        /// The current page number which should be highlighted in the paging controls at the bottom of the grid.
        /// </summary>
        public int Page { get; set; }
        /// <summary>
        /// The total number of records in the entire data set, not just the portion returned in Rows.
        /// </summary>
        public int Records { get; set; }
        /// <summary>
        /// The data that will actually be displayed in the grid.
        /// </summary>
        public IEnumerable Rows { get; set; }
        /// <summary>
        /// Arbitrary data to be returned to the grid along with the row data. Leave null if not using. Must be serializable to JSON!
        /// </summary>
        public object UserData { get; set; }
    }
}

Note that I’ve used C# capitalization for the property names. This is not what the grid expects. But I have to tell the grid that I’m using a non-default value for repeatitems anyway, so it’s nice to follow C# conventions.

We have lots of grids in our application, so rather than configuring each grid individually, we set defaults for the grid in one place. In a JavaScript file referenced by the Site.Master, this method is called in the jQuery ready event:

    setDefaults: function() {
        $.jgrid.defaults = $.extend($.jgrid.defaults, {
            datatype: 'json',
            height: 'auto',
            imgpath: '/Content/jqGrid/themes/basic/images',
            jsonReader: {
                root: "Rows",
                page: "Page",
                total: "Total",
                records: "Records",
                repeatitems: false,
                userdata: "UserData",
                id: "Id"
            },
            loadui: "block",
            mtype: 'GET',
            multiboxonly: true,
            rowNum: 20,
            rowList: [10, 20, 50],
            url: "ListGridData.json",
            viewrecords: true
        });
    }

In addition to setting repeatitems, telling the grid to expect the capitalization I mentioned above, this code also sets the grid theme and other options we want to be consistent throughout the application. One important property set in the jsonReader property is id. The value supplied is the name of a column in the returned data which contains a unique ID for the row. You can supply any name you want to, and the type of the property can be just about anything serializable, but it is important that the data you return to the grid contains some unique ID. All of the types in our Entity Framework model contain a unique ID called "Id", so we set this in our defaults for the grid. If you don’t have a single operating name for any data you might supply to the grid, you will have to set this value on the individual grids, when you write the JavaScript for the view.

Now I can write a method which takes a PagedList and transforms it into jqGrid data:

        /// <summary>
        /// Converts a paged list into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="list">The list of records to display in the grid.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data.
        /// Leave null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this PagedList<T> list, object userData)
        {
            return new JqGridData()
            {
                Page = list.PageIndex,
                Records = list.TotalItemCount,
                Rows = from record in list
                       select record,
                Total = list.PageCount,
                UserData = userData
            };
        }

That’s getting somewhere, but it’s not yet enough to let me write the action method I showed above. For that, I need to wrap up the PagedList inside the action so that I can pass the arguments from the grid directly, without having to construct a new PagedList inside of every action method which supplies data to a grid. So I build on the method above:

        /// <summary>
        /// Converts a queryable expression into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="baseList">The list of records to display in the grid.</param>
        /// <param name="currentPage">A 1-based index indicating which page the grid is about to display.</param>
        /// <param name="rowsPerPage">The maximum number of rows which the grid can display at the moment.</param>
        /// <param name="orderBy">A string expression containing a column name and an optional ASC or
        /// DESC. You can, separate multiple column names as with SQL.</param>
        /// <param name="searchQuery">The value which the user typed into the search box, if any. Can be
        /// null/empty.</param>
        /// <param name="searchColumns">An array of strings containing the names of properties in the
        /// element type of baseList which should be considered when searching the data for searchQuery.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data. Leave
        /// null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this IQueryable<T> baseList,
            int currentPage,
            int rowsPerPage,
            string orderBy,
            string searchQuery,
            IEnumerable<string> searchColumns,
            object userData)
        {
            var filteredList = ListAddSearchQuery(baseList, searchQuery, searchColumns);
            var pagedModel = new PagedList<T>(filteredList.OrderBy(orderBy), currentPage, rowsPerPage);
            return pagedModel.ToJqGridData(userData);
        }

Now you see that string array (IEnumerable, actually) again. JqGrid supports sorting and searching, and so do I. Like Phil, I suggest using Microsoft Dynamic LINQ in order to take the field names that jqGrid passes and turn them into a sorted, filtered, result set. ListAddSearchQuery is a (longish) method which takes a search string and a list of columns to search and turns them into a Dynamic LINQ expression and applies it to the base list. Unfortunately, search is kind of application-specific. In some applications you might want to search different columns in different ways. Some applications might want "Google-style" searching. It kind of depends upon what you’re doing. So although I will supply the source code for this method when I make the demo project available, I cannot promise that it will be useful for your application. Feel free to substitute your own method here.

It turns out both methods (the one which takes a PagedList and the one which takes an IQueryable) are useful. Generally, I use the method which takes an IQueryable. But every once in a while I want to do something to the data I’m about to supply to the grid before the grid sees it, but after paging, like transform a single page of results in some way. In this case, the action builds up a PagedList, uses that to reduce to one page of data, modifies the results in whatever way is appropriate, and then calls the PagedList extension. I’ve also written additional overloads to both methods which do not have a userData argument, since it is very seldomly used. userData, as the name suggests, is an additional bit of data you can return to the grid with your result set for use in grid event handlers when loading a fresh page of data. Since it’s of type object, you can put as much data in there as you need to, so long as it’s serializable to JSON. In practice, though, we rarely use this feature of the grid.

With these methods, it is easy to write actions as concise as the code I showed at the beginning of this post. Just get a list from somewhere and call ToJqGridData, passing information about which page to display, how to sort, etc.

In a future post, I will tie this all together into a demo application.

Share This | Email this page to a friend

jqGrid provides a nice-looking grid with built-in features such as sorting, editing, search, drill-down, tree lists, and more. It is based upon jQuery and integrates well with ASP.NET MVC. The grid is dual-licensed under both the GPL and MIT licenses (just like jQuery), so it can fit well into both commercial and open-source projects.

If you want to display tabular (grid) data with paging in an ASP.NET MVC application, there are essentially three choices:

1. You can write your own grid components using HTML tables, links, manually-implemented paging, etc. This is actually less work than it might seem at first, and the first release of our ASP.NET MVC software did just this. But our own implementation lacked some of the visual flair of some of the third-party components, and avoided solving some of the harder problems, like in-line editing. Still, it was much easier than I expected, and worked well enough for our initial release.
2. You can use a commercial component designed for ASP.NET. However, many commercial grid components don’t work well with MVC. Telerik has MVC demos available for their grid, and Developer Express has said that they intend to support MVC to some degree in the future. Even when the commercial components do support MVC at all, they don’t tend to make all of their features available in MVC applications, owing to the fundamental difference between the way ASP.NET MVC and "plain" ASP.NET work. If you are building a Web application on ASP.NET MVC, I would never recommend buying an ASP.NET component unless the vendor for that component is fully committed to supporting MVC. You cannot presume that any given ASP.NET component will work at all with MVC unless the vendor explicitly supports this.
3. You can use a grid component which has no knowledge of ASP.NET at all. There are quite a number of these, generally built within the framework of one of the popular JavaScript libraries, and it is, ironically, easier to use one of them within ASP.NET MVC application than it is to use a grid component designed for ASP.NET (but not necessarily MVC).

As I said, we ended up choosing jqGrid. We were strongly influenced by the fact that we already use jQuery extensively in our application. There are several grid plug-ins available for jQuery. jqGrid has a number of advantages over the others I’ve seen, however:

• Supports three different models for editing data within the grid: Single-cell editing, in-line row editing, and editing within a pop-up form.
• Unlike some of the other jQuery grid plug-ins, jqGrid has decent documentation and a large number of demos with source code.
• You can display a simple grid, a "sub grid" (drill down), or a treeview.
• AJAX paging is supported in a number of different formats, including JSON, XML, or JavaScript objects via a custom function.
• The grid is updated very regularly, and a new version, with support for the jQuery UI theming library is under active development right now.
• Tracing through the grid source code is quite easy to do with Firebug, which is useful for figuring out things which aren’t quite covered by the documentation.

The grid also has a few downsides:

• The API is a bit of a mess. Capitalization (significant, in JavaScript) is so varied that it seems almost random. Portions of the code were clearly written by different people with limited collaboration with the original author of the grid. There is no consistent naming convention, and similar functionality is sometimes invoked in wildly different ways. For example, you set the grid height with a custom method, setGridHeight, instead of the jQuery method height. But you reload the grid’s data with a jQuery method, trigger, passing the string "reloadGrid" as an argument. It’s a good thing the grid has decent documentation and includes source code, because you’re going to need them. The API is just not discoverable.
• Although the grid has quite a number of options for editing, it doesn’t necessarily play well with some of the other tools you may want to use in the context of editing, such as jQuery validation. (For starters, the in-line row editing doesn’t actually use an HTML form.) The three different editing modes (single-cell, in-line row editing, and editing in a form) use different source modules, different configuration options, and work differently. Doing editing in a way not explicitly supported by the grid can be challenging.
• Variable names in the source code use a lot of single letter and two letter abbreviations. This is quite common in JavaScript libraries, but it does make the source code a bit challenging to read.
• Although there is both free and paid support available, there is not a commercial company standing behind the grid. So don’t expect round-the-clock support, a toll-free phone number, etc.

After a good deal of experimentation with several grids, we decided to use jqGrid because it has considerably more features than every other grid we examined (when used in MVC application), and because we didn’t find any of its limitations to be show stoppers.

The next post in this series is LINQ Extensions. There, I show an extension method for IQueryable<T> which returns data suitable for the grid, without having to know anything about the type T.

Share This | Email this page to a friend

On the other hand, I have not found in practice that programmers need to be mathematically inclined to become great software developers. Quite the opposite, in fact. This does depend heavily on what kind of code you’re writing, but the vast bulk of code that I’ve seen consists mostly of the "balancing your checkbook" sort of math, nothing remotely like what you’d find in the average college calculus textbook, even.

{
  i = j++ / (x + v);
}

Not exactly the stuff mathletes are made of.

Arithmetic is one kind of math. It is relatively common in elementary math classes, and relatively uncommon in programming. Does Jeff’s example look like the "vast bulk of code you’ve seen?" It certainly doesn’t look like the vast bulk of code I’ve seen.

On the other hand, I do see a good bit of:

• Relational database queries (relational algebra)
• LINQ (functors)
• Composition of lambda expressions (lambda calculus, of course)
• Set operations and logical operators (set theory)
• Etc.

That’s by no means a comprehensive list. For example, every general-purpose programming language I’ve ever used would not exist without the Church-Turing thesis. But I didn’t include that, since most people don’t do programming language design.

All programmers work with mathematical systems day in and day out, whether we recognize them or not. Perhaps the ability to recognize math when we see it will help us better evaluate the importance of understanding it.

Share This | Email this page to a friend

Update: Attendee Bill Melvin wrote a short summary of the presentation.

Share This | Email this page to a friend

1. Decide where you want to go.
2. Aim the car in that direction.
3. Get there.

It’s different when you ride a horse. In that case the process is more like:

1. Decide where you want to go.
2. Communicate your intention to the horse.
3. Arrive at some hybrid of where you want to go and where the horse wants to go.
4. Return to step (1). Repeat.

Now, which of those two would you find more analogous to software architecture? Does that sound like a silly question? The reason that I ask is that I read a paper last night which argues in a persuasive but decidedly unconventional manner that the things you design have, for lack of a better word, intention. In other words, building a cathedral, or a software application is more like riding a horse than driving a car.

The paper is called Designed as Designer, by Richard P. Gabriel. As technical papers go, this one is unusual to say the least. It was written as a response to an address by Fred Brooks at OOPSLA 2007, during his keynote. In the address, Brooks stated:

If we look back then at the 19th century and the things that happened—the cartwright and the textile machinery, Stephenson (the train), Brunel’s bridges and railway, Edison, Ford, the Wright brothers, etc.—these were very largely the designs of single designers or, in the case of the Wright brothers, pairs.

Gabriel disputes this idea, and chooses an unexpected way of making his case. Gabriel is a published author both in the fields of computer science and writing/poetry. The paper draws as much from his experience in the latter field as the former—there is considerably more poetry quoted in the paper than source code. He also manages to weave in, with some relevance, no less, baseball statistics.

The general outline of the paper is as follows: First, there is a running, fictional narrative about a child named Pippo who observes the construction of the cathedral of Firenze. Next, Gabriel addresses Brooks’s point directly. He discusses the relationship between poets T.S. Eliot and Ezra Pound in the evolution of Eliot’s masterpiece The Waste Land. The middle, and longest, section of the paper is an explanation and example of a theory of poetry he has developed based around an almost algorithmic approach to classifying a given text as either poetry or ordinary writing. Contrasting an early version of a published column with a later revision to same, he asserts that the structure of the earlier version leads the poet into the later revision. I should add here that even though I really like poetry, I find his analysis a bit academic for my taste. Returning to the Cathedral of Firenze, Gabriel shows that many people were involved in its design and construction, and that the Cathedral form itself had a bigger impact on the final design than the architect. Then he uses statistical analysis of the career of Roger Maris to show that luck, as much as skill, played a role in Maris’s exceeding Babe Ruth’s home run record.

The final section of the paper includes, almost shockingly at this point, discussion of computer science and one small example of (Lisp) source code. He relates the story of how Gerry Sussman and Guy Steele discovered how two mechanisms they were studying turned out to be exactly the same thing by implementing both of them separately and noticing later that the source code was identical.

If all this sounds a bit, well, crazy, it isn’t. The paper was presented at OOPSLA 2008. I’m not above a bit of schadenfreude, but this is definitely a "laugh with him" rather than "laugh at him" paper. If you have ever had the experience of designing a system, and then uncovering necessary changes in the system when you try to use your design, then you have some idea of what he’s getting at.

I am also reminded, more than a little bit, of Michael Pollan’s book, The Botany of Desire, which calls the evolution of the humans and plants "co-evolutionary," and questions who has domesticated whom. Again, we don’t tend to think of plants as having "intention," but maybe the question should be, "just what is intention, really?" So all in all, it’s a fascinating read.

That said, it seems to me that, to some extent, Brooks and Gabriel may be talking past each other. Surely, Brooks can’t be under the delusion that the Wright brothers invented the airplane in a vacuum? Surely, Gabriel wouldn’t presume that he is? Much of Brooks’s writing focuses on organizational support for creativity. Indeed, Brooks, to me, seems almost to take creativity as a given, more so than examining what it is and what makes it tick. Gabriel seems to be on the other side of this coin, examining the nature of creativity, rather than how to make it work in a large organization.

Both points of view are worth reading, I think.

Share This | Email this page to a friend

http://www.example.com/foo.html

…which is fine, because if you go to such a link, you probably really will get HTML back, to URIs which (often) look something like this:

http://www.example.com/foo.php

…which, as I noted in a previous post, is a lie. Actually, it’s worse than a lie; it’s poor encapsulation. What happens when you come to your senses and stop using PHP? Now you URIs have to change, which means you get to either create redirects for that abomination or break every link to your site. One of the things we’ve learned from the success of Google is that the information encoded in the links between sites is nearly as important as the information on the sites itself.

The web is dynamic. It has to be, because it is too big to compile. And so the same concerns we have when writing code and dynamic languages often apply to creating a web site, whether you use a dynamic language or not. You need to choose very good names, because if you ever need to change them you will either break stuff or have a substantial job on your hands correcting every reference to the name you’ve changed, including references you don’t personally control.

So I’m going to propose a general rule, which states, "Any URI which allows an informed developer to discern the technology used to serve the result is probably broken." The resources you are serving, if they are at all valuable to the user, will probably last longer than the implementation you use to serve them.

Of course, choosing good names is hard. I read a great quote the other day:

There are only two hard things in Computer Science: cache invalidation and naming things. -Phil Karlton

Also, even after you’ve chosen a good URI, you may have to do extra work to make it functional. If routing is not built into the technology stack you’re using, you may have to implement a URL rewriter to make your persistent URIs functional. It’s worth the effort to do so.

Late update: Thank you to commentor Stecki for the link to the following article: Cool URIs don’t change, by Tim Berners-Lee. Well said.

Share This | Email this page to a friend

Mason Wheeler noticed that TObject.InstanceSize returns 8 (bytes). It turns out that this is new in Delphi 2009; in previous releases, TObject.InstanceSize returned 4. But when you look at the definition of TObject in System.pas, you don’t see any fields declared at all.

Four out of the eight bytes are consumed by the VMT; this has been true since the first version of Delphi. You can read more about that in this chapter from Delphi In A Nutshell, which is old, but still accurate insofar as the VMT is concerned.

But what about the remaining four bytes? I didn’t remember the answer, but I enjoy a good puzzle, so I took a closer look at the source code for TObject. While poking around in System.pas, I happened to notice the following constants:

{ Hidden TObject field info }
hfFieldSize          = 4;
hfMonitorOffset      = 0;

OK, that explains the four bytes, but what is this used for? Searching for hfFieldSize yielded the answer:

class function TMonitor.GetFieldAddress(AObject: TObject): PPMonitor;
begin
  Result := PPMonitor(Integer(AObject) + AObject.InstanceSize - hfFieldSize + hfMonitorOffset);
end;

So any object can have a reference to a TMonitor instance, and that reference is as big as a pointer on the target CPU. That reminded me of Allen Bauer’s long series of blog posts last year on the "Delphi Parallel Library," which discuss the TMonitor class in detail. He notes:

The TMonitor class [...] is now tied directly to any TObject instance or derivative (which means any instance of a Delphi class).  [...] For Tiburón [a.k.a. Delphi 2009], you merely need to call System.TMonitor.Enter(<obj>); or System.TMonitor.Exit(<obj>); among the other related methods.

The ability to lock any object is a good feature to have, especially in a multi-CPU, multi-core world. So I’m happy to spend the extra four bytes to get this feature. But I do find the implementation a bit mysterious. Why not just declare an actual field in TObject? I guess one reason would be to reduce the chance of a conflict with existing code. Another possible reason would be to enforce TMonitor’s contract; by obfuscating access to the field, you reduce the chances that an uninformed programmer performs actions on the field which break the contract. But these are just guesses. I don’t know why the Delphi team settled on this implementation. I presume they have some good reason.

Update: In comments, Allen Bauer explains the reasons for this implementation:

The reason for always placing the monitor reference field at the end of the object instance is mainly for backward compatibility. I didn’t want to alter the layout of objects in case there was some code out there that depended upon it. Also, by hiding the implementation the chance of inadvertent mucking with the monitor data was reduced and therefore it increased the overall safety of using it.

If you look closely at what happens on descendant instances, the monitor field is always the last field of the instance. This happens because the compilers (Delphi & C++) ensure that the field is allocated as the last step in the process of laying out the class instance. By doing this, an object laid out in pre-Delphi 2009 will be laid out exactly the same in Delphi 2009 except there is now an extra trailing pointer-sized field.

Share This | Email this page to a friend

Share This | Email this page to a friend

Yes, it is true that ASP.NET MVC does not force you to use one particular ORM. Yes, that means that you’re going to have to expend some thought, and quite possibly even effort, deciding which one is appropriate for you. It will not be sufficient to read a blog post about the Entity Framework from somebody who tried it once and another blog post about NHibernate from somebody who saw a screencast about it and then conclude that there are no workable solutions in the world.

But it’s worth the effort, because you have a big reward in store: Whatever choice you make will work just fine with ASP.NET MVC. Because the framework has no dependencies whatsoever on any ORM, all ORMs work equally well. You get to choose the best tool for your data access needs without worrying that your web framework will reject it.

Share This | Email this page to a friend

Share This | Email this page to a friend

• Stores a list of routes.
• Translates an incoming URI into lists of values and tokens, and passes those lists to the handler specified on the route. In the MVC framework, this feature is used to call the appropriate Controller/Action.
• Translates a list of values into a URI suitable for inclusion within an HTML page. In the MVC framework, this feature is most commonly used in the Html.RouteLink/ActionLink methods you call to generate a link in your Views.

Let’s examine in more detail how MVC uses Routing, and distinguish between the features of Routing and the MVC framework.

Components of a URI

Consider this sample URI:

http://www.example.com:80/SiteRoot/Controller/Action?foo=A&bar=B#anchor

The individual parts of the URI are:

• The scheme (protocol), authority (host), and port ("http://www.example.com:80" in this sample URI). I will mostly ignore this part in the rest of the post, since Routing doesn’t deal with it, except to pass it along when necessary.
• The path ("/SiteRoot/Controller/Action" in this example). Per the URI RFC, this part "serves to identify a resource…" I’ll talk about what that means in a second.
• The query ("?foo=A&bar=B" in this example). My example uses name/value pairs is the query, because that seems to be the most common case. However, the RFC says that the query can be almost anything, and the purpose of the query is to provide a non-hierarchical means of identifying the resource requested by the user.
• The fragment ("#anchor" in this example). I’m going to ignore this, since Routing doesn’t deal with it.
• I’ve omitted some stuff like username/password which is almost never used today.

MapRoute

Now lets examine a simple case of adding a route to the routing system. This is from the default MVC application template:

routes.MapRoute(
    "Default",                                              // Route name
    "{controller}/{action}/{id}",                           // URL with parameters
    new { controller = "Home", action = "Index", id = "" }  // Parameter defaults
);

Note that MapRoute is actually not part of the routing system; it’s an extension method added to RouteCollection by ASP.NET MVC. Calling MapRoute will add a single route to the RouteCollection. Most web applications will have only one RouteCollection, stored in System.Web.Routing.RouteTable.Routes. The most important thing that MapRoute does is to create and pass an MvcRouteHandler instance when adding the route to the RouteCollection. This ensures that when ASP.NET handles an incoming URI which matches this route, control will be passed to the MVC framework for handling the request.

The second argument to MapRoute is named "url", but it’s not really a complete URL. Instead, it is a template for a fragment of a URL containing a portion of the path component of a URI. I say "a portion of the path" because it omits the site root, which I’ve called "/SiteRoot" in the example above. The site root is often empty, but it doesn’t have to be. The URIs produced by the routing system include quite a bit more than the path, however. They also include the scheme/authority (in other words, the protocol and host) and, sometimes, a query.

There are a number of overloads to MapRoute. In addition to the values passed in this simple example, you can also pass constraints (which I’ll discuss in a moment) and namespaces. The namespaces argument is for MVC only (it’s not used by routing) and is useful when you want to put a Controller in a place where MVC wouldn’t otherwise find it.

A route can also contain DataTokens, which are not used by the routing system, but are passed to the route handler. MapRoute doesn’t have an overload which includes these as an argument, and I’m not going to discuss them in this post. ASP.NET MVC uses them only to store the namespaces argument, discussed above.

Parsing a URI

Let’s examine what happens when the routing system is asked to interpret a URI submitted by the user. Importantly, the RouteCollection is ordered. So the routing system will attempt to match the routes in the collection in the order you added them. The first matching route "wins" even if a later route is a "better" match. The incoming URI is compared with the url portion of the route. If it matches, the entire URI, including the query, is tokenized. If there are any constraints on the route, these are considered.

In practice, this means that if you have a lot of routes, then you are almost guaranteed to encounter a bug, sooner or later, where the wrong route is matched to an incoming URI. The only good way to prevent this that I know of is to write unit tests for your routes. Unfortunately, the name of the route is not actually part of the route itself. It is stored in a private, internal dictionary in the RouteCollection. So there is no way to write a unit test which examines the name of the route returned.

Let’s use the sample URI and route from earlier in this post as an example. Out of the entire URI, http://www.example.com:80/SiteRoot/Controller/Action?foo=A&bar=B#anchor, the routing system is only going to examine this bit when processing an incoming URI:

/Controller/Action?foo=A&bar=B#anchor

Does this match the url portion of the route defined above? Recall that it is defined as:

"{controller}/{action}/{id}"

Yes, it matches, because "Controller" can be interpreted as the "{controller}" portion of the url, "Action" for the "{action}" part, and a default value exists for the "{id}" portions (empty string). Note that this means that having default values in a route makes that route much more likely to "match" an incoming URI. If there are defaults for every token in a URL pattern and there are no constraints, then that route will match every incoming URI. The route shown in this post is created by the MVC framework as a "catch all" route for a new application, which contains only that route. That is why it has defaults for all of its values and no constraints; it is intended to match every incoming URI. If, on the other hand, you write a special-purpose route intended only for one particular controller or action, and you provide default values for all of the tokens, then you need to implement constraints to prevent this route for matching other controllers or actions.

Because the url matches, we have five values, so far, to pass as the route data. "controller" will have the value Controller, and "action" will have the value "Action". "id" will have the default value. Also, from the query, "foo" will have the value "A", and "bar" will have the value "B". Now the routing system considers any constraints on the route, but in this case, there aren’t any. Therefore, this list of values is handed to the MvcRouteHandler which was created by MapRoute.

Importantly, note that all of these values are strings. The routing system only deals with URIs, which are always strings. Now, your actions on your controllers can have arguments which are not strings, and the MVC framework will attempt to coerce the strings supplied by the routing system into whatever type is specified in the method signature. But that’s a feature of the MVC framework, not of the routing system. (Yes, the Values property returns type object, but, when matching an incoming URI, the segments are always added as strings.)

Not all URI patterns are as simple as the default route from the standard MVC application template. You can use delimiters other than /, for example. If you have hyphens, parentheses, etc. in your url template, these will be matched against the incoming URI to determine if the route is a match. The following route, for example, would not match the sample URI at the top of the post, even though it contains all of the necessary tokens and has no constraints:

routes.MapRoute(
    "Parenthesized",                                        // Route name
    "{controller}({action})({id})",                         // URL with parameters
    new { id = "" }                                         // Parameter defaults
);

The incoming URI contains no parentheses, so this route can’t match, because the parentheses are needed to to delimit the values for controller and action, and there are no defaults for this values.

Matching a Route With an Action

After a route is matched to an incoming URI, and a list of route values is created, control goes to the MvcRouteHandler. Now the MVC framework can invoke the action on the controller specified by the "action" and "controller" route values. Note that there is nothing special about these names/values to the routing system; they’re just two more values in the list from the routing system’s point of view. But these two names, "action" and "controller" are very important to the MVC framework, for obvious reasons.

MvcRouteHandler (indirectly) calls MvcHandler.ProcessRequest, which constructs the controller using the ControllerFactory, and then calls Controller.Execute, which, in turn, invokes the action using the ControllerActionInvoker. It is at this point that the values in the route are matched against the arguments for the action. In other words, tokenizing the URI and matching those values with action arguments are two entirely separate operations. The tokenizing is done by the routing system, and the matching route values with controller action arguments is done by the MVC framework.

In order to invoke the action, the MVC framework needs to look at the individual arguments to the action, and match them with route values. This is performed by the model binding system.

Note that the name of the argument does not have to correspond to a name in the route values if the model binding system can instantiate an instance of the type of the argument from the values which do exist. For example, if the type of the argument is FormCollection, then this argument can always be passed, no matter what names are present in the route values, and no matter what the name of the argument. The model binding system knows that it can always create a FormCollection by rolling up any values in the form in the incoming POST, if there are any. If there are not, it will simply return an empty FormCollection.

If the action takes an argument which cannot be instantiated based on the data in the route values, that is fine, so long as the argument is nullable. If the action requires a non-nullable argument which is not in the route values, or if the route values contain a value for an argument which cannot be bound into the type of that argument, the MVC framework will throw an exception. Otherwise, the action will be invoked.

Creating a URI With RouteLink

Creating an "a href=" tag with Html.RouteLink is trivial. The routing system looks up the route by name, and populates the URI with values from the RouteCollection you passed. Any values in the RouteCollection not included in the string in the url argument to MapRoute will be added to the resulting link as query string parameters. In my view, you should always prefer RouteLink over ActionLink, whenever there is any ambiguity as to which route will be returned.

Again, let’s consider an example, with the route defined in the MapRoute call earlier in this post:

<%= Html.RouteLink("Link text to display on page", "Default",
    new RouteValueDictionary(new {controller = "Foo", action = "Bar", foo = "A", bar = "B" })) %>

With this example, the order of the routes in the RouteCollection, the constraints, etc., are not needed to find the matching route, because we have specified ("Default") directly. Since there are no tokens called "foo" and "bar" in the url argument we passed to MapRoute, these will be appended to the generated URI as query string parameters. So we end up with the sample URI at the top of this post, less the anchor, provided that the site is configured to live in /SiteRoot.

Creating a URI With ActionLink

ActionLink is more complicated, because the routing system must first find an appropriate route, and then use that route to generate the URI. In a way, the method of finding the route is similar to the method used when parsing a URI, insofar as the first matching route "wins." But in another way, it is very different, because there is no incoming URI to use as a pattern for matching the partial template passed as the url argument to MapRoute. The only thing that the url argument is used for in this scenario is determining which tokens must be included, either directly, or via defaults, in order for a route to match. Let’s re-examine the same scenario considered in the RouteLink section, above, only this time using ActionLink. The call looks like this:

<%= Html.ActionLink("Link text to display on page", "Bar", "Foo",
    new RouteValueDictionary(new {foo = "A", bar = "B" })) %>

Does this match the route defined at the top of this post? Yes it does, and here’s why: That route contains three tokens in its url argument. They are controller, action, and id. I have supplied values for two of these tokens in the call to ActionLink, namely controller and action, via the controller and action arguments in the overload of ActionLink I am using here. The remaining required token, id, is not supplied in the ActionLink call at all. However, there was a default for it (empty string) in the original call to MapRoute, so the route matches even without an explicit value for id. Because foo and bar are present in the RouteValueDictionary, but the url template in the route does not contain {foo} or {bar}, these will be added to the generated URI as query string parameters. Again, the generated URI will be the sample URI at the top of this post, less the anchor, provided that the site is configured to live in /SiteRoot.

Conclusions

I hope this long explanation has taken some of the mystery out of what routing does. Since there’s a lot of detail above, I’m going to summarize some of the information which is important for day-to-day work.

• Routing and the MVC framework are distinct features. MVC knows about Routing, but Routing does not know about MVC. Routing produces a list of values, but does not assign special meaning to them. MVC signs special meaning to the controller and action values.
• Routes include information about a portion of the path in the URI and also the query, since any token not in the path is treated as part of the query.
• The order of routes is very important, both for parsing incoming URIs and for generating URIs with ActionLink.
• Default values in a route will cause the route to match more incoming URIs or sets of values in a RouteValueDictionary. Constraints in a route will cause the route to match fewer incoming URI this or sets of values in a RouteValueDictionary. If there are defaults for every token in a URL pattern and there are no constraints, then that route will match every incoming URI or sets of values in a RouteValueDictionary. For a "catch all" route like the one in the default MVC application, it is fine for all of the tokens to have default values in for there to be no constraints. If you write a special-purpose route for a particular controller or action, however, then you should probably use constraints to prevent it from ever matching any other controller or action.
• If you have more than one route, then you simply must write unit tests to ensure that the correct route is selected for all incoming URIs or values used in your calls to ActionLink.

Share This | Email this page to a friend

Database Workbench supports InterBase and many other DB servers. You can buy the module for one (or more) DB server only if you don’t need all of them.

Share This | Email this page to a friend

I haven’t read this book, but someone in the newsgroups mentioned that it included InterBase. He writes:

"Note that they tested on IB6 — but I have still found most of their advice to be valid."

Share This | Email this page to a friend