Share This | Email this page to a friend

Unfortunately, it is not just your own code you need to be concerned with. It turns out that some fairly popular LINQ helpers do this. For example, Rob Connery’s PagedList<T> type does it:

public PagedList(IQueryable<T> source, int index, int pageSize)
{
    this.TotalCount = source.Count();
    this.PageSize = pageSize;
    this.PageIndex = index;
    this.AddRange(source.Skip((index - 1) * pageSize).Take(pageSize).ToList());

Index, in this code, is the 1-based page you’d like to display. When index is 1, the code calls Skip(0), and you get less efficient SQL. Fixing the problem is quite simple:

    this.PageIndex = index;
    var pageData = index > 1 ?
        source.Skip((index - 1) * pageSize).Take(pageSize) : source.Take(pageSize);
    this.AddRange(pageData.ToList());

Naturally, this only optimizes SQL generation for the first page of results, but since this is by far the most common page a user might display, it can net you a substantial win in server performance.

Troy Goode’s updated PagedList type, which I have recommended in the past (and still do, with this tweak!), has the same problem. The solution is almost identical, although Troy’s type is 0-based rather than 1-based. Update: Troy has now posted an update containing this fix, among many other changes.

The root of the problem in the SQL generation is that when you call Skip(0), LINQ to Entities, in at least some cases, generates SQL like this:

)  AS [Project3]
WHERE [Project3].[row_number] > 0
ORDER BY [Project3].[TimeRecordDate] DESC

The reference to row_number can have a substantial performance impact. In one case I tried with a SQL Server table containing just over 1 million rows, a simple query ran with sub-second performance without this WHERE clause, and over 4 seconds with the WHERE clause, even though the result set was exactly the same.

Now, is this a bug in LINQ to Entities? Or LINQ itself? It’s hard to say. Both frameworks are doing exactly what you tell them to, even though it doesn’t make a lot of sense. It seems like a missed opportunity for optimization, but I’m open to the possibility that there might be some desirable effect of this that I have simply missed. For my own purposes, I am content to simply not call Skip(0) at all.

Share This | Email this page to a friend

It’s worth a listen if you’re interested in the history of the product which started Borland and evolved into Delphi.

Share This | Email this page to a friend

Download the demo solution: DayPilot ASP.NET MVC demo

We needed a calendar control for an ASP.NET MVC application. In particular, we needed a control which would show a number of events and when they occur within a particular day, "Outlook-style." I’ll include a screenshot of the demo application here, since it’s the best way to explain the type of control we were looking for:

DayPilot example

Possible Alternatives to DayPilot

In my experience, the type of control which works best with ASP.NET MVC is a control which is completely ignorant of ASP.NET, and all the lies such as ViewState and postbacks which accompany it. In particular, I’ve had a good deal of success with controls designed for the jQuery JavaScript framework, which is included, by default, with ASP.NET MVC applications. Unfortunately, I was unable to find a jQuery control which data what we needed. The closest I found was jMonthCalendar which cannot display a single day in the manner illustrated above. I would still prefer a jQuery-based solution if one ever turns up.

I did, however, find a couple of controls designed for "plain" ASP.NET which, from an end-user point of view, it exactly what we needed. One was the DayPilot control illustrated above, the other was the scheduler component included with Telerik’s control suite. Telerik’s control claims some support for MVC, but this is really limited to loading calendar data asynchronously, which was not actually what we wanted to do. Also, Telerik’s control can only be purchased as part of their suite, which is rather expensive if you only need the calendar/scheduler control. Paying for the entire suite would not be a problem for us if it was filled with controls we needed, designed for the environment (MVC) we use. While Telerik seems serious about supporting MVC, they are in the early stages right now, so it is difficult to justify paying for a large suite in order to get a single control, which I would have to use in a mostly-unsupported way. The Telerik control has many more features than the DayPilot control, and, in my opinion, looks a little nicer.

The DayPilot calendar is available in two different editions. There is an open source edition, which is free, and a non-open-source edition, which adds features. The principal difference is that the date version includes editing (via postbacks) and UpdatePanel support, which are not really useful in an ASP.NET MVC application anyway. As with Telerik’s control, I have to adapt the control for use within ASP.NET MVC. So we selected the open source version, even though the paid version is priced reasonably.

Adapting DayPilot to ASP.NET MVC

In order to use the DayPilot control in ASP.NET MVC, I wanted to use markup which would look familiar to ASP.NET MVC developers:

<%= Html.DayPilot(Model, new DayPilotViewOptions
                             {
                                 HourHeight = 30
                             }) %>

This means that I needed to write an Html helper which will render the control based on the data passed in Model and return the literal HTML markup. In order to make that easier, I created two new types, DayPilotData and DayPilotViewOptions, in order to encapsulate access to the properties of the control. The reason I created two different types was to separate the concerns of data for the events (which will be set up in the controller) and display of the control (which is the concern of the view). The types are trivial, so I’m not going to show them in this post, but you can examine them in the demo solution. There is an overload to this Html helper which does not include the options argument in case you are happy with the defaults.

        public static string DayPilot(
            this HtmlHelper helper,
            DayPilotData model,
            DayPilotViewOptions options)
        {
            var calendar = new DayPilotCalendar();
            if (model != null)
            {
                model.CopyTo(calendar);
            }
            if (options != null)
            {
                options.CopyTo(calendar);
            }
            var sb = new System.Text.StringBuilder();
            sb.Append("<div class=\"dayPilot\">"); // allows working around td cellpadding bug in css
            using (var sw = new System.IO.StringWriter(sb))
            {
                using (var tw = new HtmlTextWriter(sw))
                {
                    calendar.RenderControl(tw);
                }
            }
            sb.Append("</div>");
            return sb.ToString();
        }

Note the div. The current version of DayPilot astonishingly, as of this writing, does not put a class anywhere in its rendered markup. Since the stylesheet for the default ASP.NET MVC site conflicts with DayPilot, to some degree, I wrapped the whole control in a div so that I could fix the layout in CSS.

Another way which I could have used the control in ASP.NET MVC would be to register a tag prefix and use code behind/code generation, as with standard ASP.NET. I’ve registered a tag prefix in the demo solution in case you want to try that yourself. Be aware, however, that setting up codebehind and code generation files in an MVC application is somewhat less than simple; doing it right requires, among other things, manual editing of the csproj file. So I do recommend using the control in the way I’ve illustrated.

Finally, here’s the source code for the action which supplies data for the control. Since this is a demo solution, I’m generating random events:

        public ActionResult Index()
        {
            var r = new Random();
            var q = from i in Enumerable.Range(1, 4)
                    let start = DateTime.Today.AddHours(r.Next(47))
                    select new DayPilotDataItem
                    {
                        Id = i,
                        Start = start,
                        End = start.AddHours(1),
                        Text = string.Format("Event {0}", i)
                    };
            var model = new DayPilotData
            {
                StartDate = DateTime.Today,
                Days = 2,
                Data = q
            };
            return View(model);
        }

Of course, it’s just as easy to select data from a database.

The current version of DayPilot has a kind of a weird limitation: You cannot make it smaller, in the vertical dimension, than 30 pixels per hour. Based on some inspection of the source code, I think this is fixable if you are comfortable altering the source. But be aware of the limitation.

There are more DayPilot features which I have not covered, but that should be enough to get you on your way. If you have any questions, post them in comments, and I’ll see whdat I can do to help out.

Share This | Email this page to a friend

Finding your blogs very helpful - superbly written - just found out about jqgrid last night and have integrated your code into my project.

One thing that didn’t work for me though was the dates fix above…

adding

.ToString(System.Globalization.CultureInfo.CurrentUICulture)

onto the end of my DateTime values makes everything render Ok but when I try doing a search, it fails. Debugging shows the error is

Method ‘System.String ToString(System.IFormatProvider)’ has no supported translation to SQL.

at the TotalItemCount = source.Count(); line of the PagedList.cs file.

Any ideas?

In order to understand this error, we need to remember several important things about LINQ:

• LINQ is lazy. Creating or modifying an IQueryable with methods like Where, Take, OrderBy, etc. does not actually execute the query. The query will only be executed when you call a LINQ method which requires execution to produce its results — like Count.
• When a LINQ query expression includes something which looks like a method call, the method is not invoked directly. Instead, what is actually produced is an Expression representing that method. If the LINQ provider happens to be LINQ to Objects, the method will eventually be invoked when the query is executed. But if it happens to be LINQ to SQL or LINQ to Entities, the method may never be invoked, and instead might be replaced (if possible) with equivalent SQL.
• Most LINQ providers do not support every possible LINQ query expression. Most providers don’t even support the entire LINQ API, and only LINQ to Objects can be presumed to support all method calls on an object in a query. In some cases, as with LINQ to Entities, this is documented. With other providers, you may have to guess/experiment. Any LINQ provider which has to translate a LINQ query expression into SQL syntax can only really support methods which it is hard-coded to recognize. Worse, it is not even uncommon for providers to support one overload of a particular method but not a different overload of the same method. See, for example, the LINQ to Entities document I just linked or the ToString method, which is generally supported in LINQ to Entities/SQL, but has a special overload on the DateTime type which is not supported. Worse still, method calls may be supported in some contexts but not others. Graeme reports that LINQ to SQL accepted the ToString overload in the context of a Select, but not in the context of a Where. This makes sense from an implementation point of view, but means programmers have to be extraordinarily careful about what we include in our LINQ queries, especially when they are constructed dynamically.

By now it should be obvious what is going on here: The particular overload of ToString which I used in my example is supported in LINQ to Objects but not in LINQ to SQL in a Where expression. My demo solution used a repository based on LINQ to Objects, so the query I showed worked correctly there. But it does not work with LINQ to SQL. Because LINQ is lazy, you will not actually see an error about this until the query is translated into SQL by the LINQ provider, and that does not happen until the call to Count in PagedList.

This means that when you write a LINQ query expression using LINQ to SQL, LINQ to Entities, etc., you are essentially working in a partially dynamic language. Unless you study the documentation very, very carefully, you will not know for certain until runtime if your query will be accepted by the provider.

So how could we work around this problem?

• Use a different method or a different overload which is supported by the LINQ provider used in your project. This is fine if an appropriate method/overload exists.
• Give up on doing the formatting in C# and use JavaScript instead. This will work, but it’s a bit like using beheading to cure acne. C# is, after all, a general-purpose programming language. We should not have to give up on formatting a date.
• Bring the query into LINQ to Objects using a method such as AsEnumerable. Obviously, we would want to apply the paging and ordering first. In this method, we would execute one query (using LINQ to SQL or something) which would retrieve an ordered list of records for the specified page, using AsEnumerable to transform the results into a List. We would then execute a LINQ to Objects query against this list in order to perform the formatting. This will actually work OK, although it can end up being a good bit of code to write for what should be a fairly simple task.
• Write a new LINQ provider which understands the method in question. This seems like an enormous amount of work, but I’ll include it for completeness.
• Separate the concerns of serialization and querying. You can tell by the 50-cent words that this is the solution I prefer.

Remember, the original query worked just fine. It is only the serialization to JavaScript which caused unexpected results. One way to fix this is to hide the members which are problematic to serialize and replace them with a string version of same:

private class JavaScriptDateFixer
{
    [NonSerialized]
    internal DateTime _date;
    public int Id { get; set; }
    public int IntProperty { get; set; }
    public string StringProperty { get; set; }
    public string DateProperty { get { return _date.ToString(System.Globalization.CultureInfo.CurrentUICulture); } }
}

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new JavaScriptDateFixer
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    _date = entity.DateProperty
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that even though this code looks quite different, this does almost exactly the same thing as the code I’ve replaced. The only real difference is that I’ve delayed the call to ToString until after the query is executed. That’s enough to keep LINQ to SQL or LINQ to Entities happy.

So with this fix is everything fine? Maybe. I really like LINQ; I think it’s an extraordinarily powerful tool. But here we have hit upon what I consider to be the single biggest problem with LINQ. The combination of very limited support for all possible query expressions in common LINQ providers like LINQ to SQL and building queries dynamically can be very dangerous. Unless you test all possible query permutations, you cannot be completely confident that any query you build will actually execute successfully. In typical dynamic language programming, you mitigate this problem by writing unit tests. But unit tests are not supposed to connect to a database. Unfortunately, the limitation here is that the point of translation to SQL. So any unit test which mocked the LINQ provider, say, using LINQ to Objects or some other in-memory representation would not actually encounter the limitations we are trying to test.

I have yet to see a really good solution for unit testing LINQ to SQL or LINQ to Entities, one which would combine the desired attributes of not actually connecting to the database while surfacing any potential problems in SQL generation. I suspect that such a beast may never exist for LINQ to SQL, but for LINQ to Entities it seems at least possible. One could write an Entity Framework provider which returns mocked instances of entities. Any LINQ syntax not understood by the Entity Framework would be caught before the mock provider was ever invoked. This is a feature I would like to see in a future version of the Entity Framework.

Share This | Email this page to a friend

Share This | Email this page to a friend

The delete feature of jqGrid is, oddly, almost entirely undocumented, even though there quite a few examples of different methods of editing, and inserts are documented, to a lesser extent. But it does exist; I just had to read the source code to figure out how it works.

JqGrid has three different methods of editing data. The cell edit feature allows the user to edit a single cell the grid at a time, and saves the data when the cell loses focus. The row edit feature works similarly, but does not save the data until the row loses focus. The form edit feature shows a modal dialog instead of editing the values inline. Of these three, the form edit feature is the only one which supports deletes. However, you can still use one of the other "modes" for editing, if you like. Just be aware that when you do a delete, you will be in the form edit code. Also, deletes won’t work if you change the grid’s configuration to disable the form edit feature.

In order to allow the user to delete a row, I need to:

• Add a delete button to the grid’s toolbar. I could, of course, use a custom button or something instead of the grid’s toolbar if I chose to do so.
• Specify a URL for the delete action
• Implement a delete action on the controller

I can accomplish the first two tasks by making a slight change to the JavaScript that configures the grid:

}).navGrid(pager, { edit: false, add: false, del: true, search: false }, {}, {}, {url: "Delete"});

I’ve highlighted the parts I’ve changed in boldface. Changing the "del" property to true turns on the toolbutton. The fifth argument is an object specifying options for the delete action, of which I have only specified the URL. This is important: The only places you can specify the URL for delete are in the arguments to the delGridRow method or in the toolbar’s configuration. There is not a "delete URL" property of the grid itself.

If you don’t specify a delete URL in this way, the grid will presume that you want to send delete requests to the URL you have specified for editing rows. There will be an "oper” value in the submitted form indicating that the request is a delete rather than an edit. But I prefer to use a separate URL for delete, because I think that fits more naturally in the ASP.NET MVC paradigm.

Now let’s write an action to handle the delete:

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult Delete(int id)
{
    var repository = new Repository();
    var deleted = repository.Delete(id);
    if (!deleted)
    {
        Response.StatusCode = 500;
        return Content("Record not found.");
    }
    return Json(true);
}

Again, I’ll note that I’m using a mock repository for this demo so that you can compile and run the solution without needing to set up a database. Since the mock repository is based on IList, the Delete method of the repository returns a Boolean indicating whether or not the requested record was found, since that is how IList behaves. It would probably be more correct to return a partial view in the event of an error rather than specifying the contents directly, but error handling is a subject I hope to cover in more depth later on. For the time being, simply changing the StatusCode will tell the grid when something does not work.

If the delete was successful, I return "true." By default, the grid ignores returned data when the operation is successful. However, you can handle the grid’s afterSubmit event, which will be passed any data you do return. So what you choose to return, and what you do with it, is entirely up to you.

With all that in place, deletes work as expected. I’m not going to update the demo solution just yet, because I’ll be covering edits and inserts real soon now. But first I need to revisit formatting and explain more about LINQ".

Share This | Email this page to a friend

First, let’s fix the date display. The reason the date appeared so oddly is that there is no official support for Date types in JSON, even though JavaScript has a date type. Microsoft, therefore, invented their own way of encoding dates for their JSON serializer, which jqGrid does not understand. There are two possible ways to fix the problem. The first is to keep sending the data to the grid in the Microsoft format, and write a custom formatter for the grid to parse that into a human-readable string. The second is to change the data we send to the grid. Both ways work, but I’m going to demonstrate the second method, mostly because I find C#’s date handling features superior to (and less confusing than) JavaScript’s.

In order to change the date "column" in the results of the query to a formatted string, we simply need to project into an anonymous type. However, this would break the sort, because a text sort on a formatted date will not be in the correct order. Hence, it is necessary to apply the sort before projecting. So I add an OrderBy to the query, and pass null as the orderBy argument to the ToJqGridData method.

public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
{
    var repository = new Repository();
    var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord)
                select new
                {
                    Id = entity.Id,
                    IntProperty = entity.IntProperty,
                    StringProperty = entity.StringProperty,
                    DateProperty = entity.DateProperty.ToString(
                        System.Globalization.CultureInfo.CurrentUICulture)
                };
    return Json(model.ToJqGridData(page, rows, null, search,
        new[] { "IntProperty", "StringProperty", "DateProperty" }));
}

Note that in this revised solution I’ve updated the mock repository to return more sensible dates.

Important update: This code works correctly with LINQ to Objects, as with my demo repository, but will not work with LINQ to SQL. I have addressed this issue in a subsequent post.

Now let’s add a search feature. One of the problems in making a "demo" search is that search is a highly application-specific feature, both in terms of implementation and in terms of user experience. JqGrid’s search feature presumes that you want a search which looks something like this:

Visually, there are lots of variations on this theme available, but in terms of the use case they all presume that the user will specify which fields she would like to search. However, I prefer a "Google-style" interface with just a search box and an implementation that can (hopefully) figure out what the user actually wants. So that’s what I’m going to demonstrate. The needs of your application may be different. The important thing to take away from this demonstration is that the jqGrid search feature can be co-opted to fit your design.

I want the search box to appear above the grid at all times, rather than requiring the user to press a toolbutton to see it. So first I need to add an empty div to the markup in order to indicate where the search box will be positioned, relative to the grid:

    <h2>GridDemo</h2>
    <div id="search"></div>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>

Next, I add a small bit of JavaScript to tell jqGrid to replace this div with a search control:

    search.filterGrid("#" + grid.attr("id"), {
        gridModel: false,
        filterModel: [{
            label: 'Search',
            name: 'search',
            stype: 'text'
        }]
    });

The filterModel tells the grid to show a search control for one column, called "Search." Note that the entities I’m returning do not have a property called "search." That’s OK. When the user types something in the search box and presses enter, the grid will add a query string parameter to its request for the data of the form "search=WhatTheUserTyped". That’s what I want, and the fact that there is no "search" column in the data does not adversely affect the grid in any way.

The first argument to filterGrid is a little odd. I’m required to pass a jQuery selector as a string rather than being able to pass a jQuery object directly. Since this is all inside a method which accepts jQuery objects as arguments (rather than hard-coding the IDs of the DOM objects in the method, for unit-testability), I have to "decode" the jQuery object into a selector string. But note that I did not need to do that when setting up the pager or the grid itself. Like I said before, the grid’s API is consistently inconsistent.

As you can see in the code at the beginning of this post, the GridDemoData action just passes the search query string parameter through to the ToJqGridData method unchanged. Inside that method, I use Dynamic LINQ to alter the IQueryable to implement the search. I’ve cleaned up this method substantially since I first posted the code. Here’s the revised method:

/// <summary>
/// Adds a Where to a Queryable list of entity instances.  In other words, filter the list
/// based on the search parameters passed.
/// </summary>
/// <typeparam name="T">Entity type contained within the list</typeparam>
/// <param name="baseList">Unfiltered list</param>
/// <param name="searchQuery">Whatever the user typed into the search box</param>
/// <param name="searchColumns">List of entity properties which should be included in the
/// search.  If any property in an entity instance begins with the search query, it will
/// be included in the result.</param>
/// <returns>Filtered list.  Note that the query will not actually be executed until the
/// IQueryable is enumerated.</returns>
private static IQueryable<T> ListAddSearchQuery<T>(
    IQueryable<T> baseList,
    string searchQuery,
    IEnumerable<string> searchColumns)
{
    if ((String.IsNullOrEmpty(searchQuery)) | (searchColumns == null)) return baseList;
    const string strpredicateFormat = "{0}.ToString().StartsWith(@0)";
    var searchExpression = new System.Text.StringBuilder();
    string orPart = String.Empty;
    foreach (string column in searchColumns)
    {
        searchExpression.Append(orPart);
        searchExpression.AppendFormat(strpredicateFormat, column, searchQuery);
        orPart = " OR ";
    }
    var filteredList = baseList.Where(searchExpression.ToString(), searchQuery);
    return filteredList;
}

Importantly, this method is simply adding to the expression represented by the IQueryable. It does not actually implement the search. So any search methodology which understands LINQ can deal with its results. At least, theoretically. In practice, LINQ providers vary greatly in terms of which LINQ features they support. So although this method works fine with LINQ to Objects (as used in the rest of the demo solution) I cannot guarantee that it will work with every LINQ provider in the world. You have to try, and revise it to fit the capabilities of the provider you’re using.

Putting all these pieces together, I now have a search feature which does what I want:

Be sure to download the updated solution in order to get the bug fixes and new features.

As a special, no-extra-charge bonus, the updated solution also includes a demonstration of using the grid’s tree view mode with client-side data from a JavaScript function. This has nothing to do with ASP.NET MVC; it’s just an example of how to use a grid feature which I wrote in response to a request in the jqGrid support forums.

That’s all for today. In the next post in the series, I’ll begin to demonstrate the grid’s editing features, and how to use them in an ASP.NET MVC application. But I will also take requests. If there’s a grid feature you’d like to see demonstrated query question regarding using the grid in ASP.NET MVC, please feel free to make a request in comments, and I will answer it as best I can.

Share This | Email this page to a friend

Share This | Email this page to a friend

On The Podcast at Delphi.org, Nick Hodges confirms my earlier speculation about reorganization in the "CodeGear" group at Embarcadero. The whole show is worth listening to. Congratulations are due to Chris Pattinson, who was promoted to Director of Quality for all of Embarcadero. Time to update the subtitle of your blog, Chris! (And maybe even write a post or two…) Sounds like good changes all around. You read it here first, folks!

I wrote a short demo showing how to use jqGrid in tree grid node with client-side data supplied by a JavaScript function.

Share This | Email this page to a friend

Rather than make you set up the Northwind database, I’ve written a quick, mock repository which supplies random data for the application. But the data persists for the lifetime of the app, so I’ll be able to demonstrate sorting, paging, etc. This way you can just run the application, without having to set up SQL Server.

So let’s write some code display a grid.

The data I’m going to display in the grid is a fairly useless type:

    public class SomeEntity
    {
        public int Id { get; set; }
        public int IntProperty { get; set; }
        public string StringProperty { get; set; }
        public DateTime DateProperty { get; set; }
    }

We need two actions. The first will display the page containing the grid. The second supplies data for one page of the grid. Remember, the grid can page through large datasets. It will only request one page of data at a time.

        public ActionResult GridDemo()
        {
            return View();
        }

        public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord)
        {
            var repository = new Repository();
            var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
                new[] { "IntProperty", "StringProperty", "DateProperty" });
            return Json(model);
        }

The first action simply returns a view containing the HTML table which will eventually become the grid. It doesn’t need a model, because the grid will request its data asynchronously. The second action handles the data requested by the grid.

In order to make this work, we need HTML elements for the grid and its pager in the view, along with a reference to the JavaScript which will turn those elements into the grid.

<asp:Content ID="Content2" ContentPlaceHolderID="MainContent" runat="server">
    <h2>GridDemo</h2>
    <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table>
    <div id="pager" class="scroll" style="text-align:center;"></div>
</asp:Content>

<asp:Content ID="Script" ContentPlaceHolderID="ScriptContent" runat="server">
    <script language="javascript" type="text/javascript" src="<%= Url.Content("~/Scripts/Home.GridDemo.js") %>"></script>
</asp:Content>

The JavaScript file is where the grid itself is configured.

$(document).ready(function() {
    GridDemo.Home.GridDemo.setupGrid($("#grid"), $("#pager"));
});

GridDemo.Home.GridDemo = {
    setupGrid: function(grid, pager) {
        grid.jqGrid({
            colNames: ['Int', 'String', 'Date'],
            colModel: [
                        { name: 'IntProperty', index: 'IntProperty' },
                        { name: 'StringProperty', index: 'StringProperty' },
                        { name: 'DateProperty', index: 'DateProperty' },
                      ],
            pager: pager,
            sortname: 'IntProperty',
            rowNum: 10,
            rowList: [10, 20, 50],
            sortorder: "asc",
            url: "GridDemoData"
        }).navGrid(pager, { edit: false, add: false, del: false, search: false });
    }
};

Important to note here are:

• colNames, the grid column captions the user sees.
• colModel,which includes the property name, which is the field name in the returned JSON data which supplies the data for that column, and index, which is the name which will be sent to the GridDemoData action as the sidx (field name to sort on) argument when you click on a grid header. There are a number of other properties you can set here which control editing, with, formatting, etc. See the documentation for full details. Importantly, colNames and colModel must have the same number of entries, and be in the same order.
• url, which tells the grid where to get its data. You’ll notice that the value I’ve passed is the name of the second action.

As I mentioned in the last post, I set a number of the grid options in a script referenced by the Site.Master, in order to prevent having to set them on every grid I create. The configuration idea for the grid on an individual page should be only the things that are unique to that page. So there is a fair bit of configuration for the grid which is not included here. In addition, the Site.Master contains references to the grid’s JavaScript and CSS files.

Finally, it’s worth mentioning that when you add jqGrid to a project you need to edit the jQuery.jqGrid.js file to tell the grid where to find the rest of the JS files it needs. Once you look at this file, it is obvious what you need to change.

At this point, I’m going to make the demo application available for download, so that you can experiment with it yourself. However, I’ll caution that I intend to update it when I write the next post in this series. In other words, the current download is a work in progress. It will change in the next few days.

Download the Solution (587 KB)

So let’s run the solution and see what we get:

OK, the hideous and eyestrain-inducing clash between the default MVC theme and the jQuery UI "lightness" theme included with the grid download notwithstanding, this is actually not bad. Sorting and paging both work, and performance is very good.

But what’s the deal with the data in that date column? It turns out that there is no standard representation of a date in JSON, so Microsoft invented their own. The grid doesn’t know about this special format. We’ll clean that up in the next post in this series and also cover search. Stay tuned!

Share This | Email this page to a friend

Nick Hodges has a new role/job title.

Item 2:

During his presentation to the Columbus Architecture Group a couple of weeks ago, Mike Rozlog handed out business cards which gave his title as "Senior Project Manager, Java Tools." He told us that in a couple of days he would have new business cards, because his role was changing. He didn’t tell us precisely what the new role would be, but I got the feeling that, like Nick, he was happy with the change.

Now, it is of course possible that these two changes happening so close together is merely a coincidence. But my personal speculation is that there are bigger plans afoot, and I’m glad to see that Mike and Nick, two people I trust, seem to be happy with them.

Repeat: This is just my personal speculation. It is based on the evidence above, not inside information. I could be wrong.

Share This | Email this page to a friend

But it turns out that we’re using the grid in a different way, and I think the difference is important. So rather than just referring you to Phil’s post and skipping the rest of the series, I think I still have some important information to contribute. In particular, I’m using the grid in a way which allows me to write an extension method for IQueryable<T> which returns data suitable for the grid, without having to know anything about the type T. If you look closely at the code in Phil’s post, you will see that populating the cell array requires custom code (or use of reflection) every time you want to return data to a grid.

It’s important to me that the method should work without knowing anything about the type of the data in the list because I don’t want to have to write repetitious code in every action method which supplies data to a grid to shape the data according to the grid’s needs, and because I generally use anonymous types for JSON serialization. JSON cannot handle object graphs with cycles (i.e., a "circle" of objects which all reference each other). The .NET JSON serializer will throw an exception if you pass it an object which contains a circular reference, or refers to another object which does. Since this is largely data-dependent and, worse, can be occasionally masked by lazy loading and the like, the safest way to serialize to JSON is to pass it data which provably can never contain a cycle. Anonymous types work really well for this. I’ll probably elaborate on this point in a future post. For the time being, the important thing is that I need to be able to take any IQueryable and turn it into data suitable for the grid, without having to think about what type that list might contain.

In a nutshell, this is what I want to be able to do:

public JsonResult ListGridData(int page, int rows, string search, string sidx, string sord)
{
    var model = repository.SelectAll().ToJqGridData(page, rows, sidx + " " + sord, search,
      new[] { "Column1", "Column2", "Column3" })
    return Json(model);
}

In this example, repository.SelectAll() returns an IQueryable<TSomething>. I’ve written a method, ToJqGridData, which can transform that IQueryable into data suitable for the grid without having to know what TSomething is. (What the heck is that string array? Don’t worry about that just yet; I’ll cover it in a little bit.) The argument names aren’t what I would have picked, but they come from the grid. There is probably a way to change them, but I’ve never bothered to look.

The key to this is a grid property called jsonReader.repeatitems. I’m not sure why it’s called repeatitems, but here’s what it does. When repeatitems is true (the default), data is returned to the grid in a "cell array," like in Phil’s post. That is, an individual row of data in the JSON looks like this:

{id:"1", cell:["cell11", "cell12", "cell13"]}

When repeatitems is false, the data looks like a regular object in JSON format:

{id:"45678",name:"Speakers",note:"note",stock:"false",ship:"4"}

Interestingly, it turns out that this is the format the .NET JSON serializer already uses. So I don’t have to write any code to get this portion of the data into the right shape. I just hand an object to the JSON serializer and let it do its thing. The only things I need to do to make the data work for the grid is to handle paging, sorting, search, and provide certain other bits of data, like the number of rows.

But, as you may have noticed, I’m lazy. Why write code when someone else has already done it? Rob Conery wrote a decent type for paging eons ago, and Troy Goode enhanced it. So I’m going to use their work instead of re-writing paging. It turns out this type already computes most of what the grid needs to know. And Microsoft has this thing called Dynamic LINQ which can deal with ordering and filtering by property names, mostly. All I have to do is tie this stuff together.

I wrote a type to hold the data which the grid needs. You can actually use an anonymous type for this; the JavaScript grid obviously doesn’t care if the data it receives comes from an anonymous type or not, but because I am writing extension methods to IQueryable, it was convenient for me to have a non-anonymous type so that the extension methods could build on each other. This type represents one page of data requested by the grid:

namespace GridDemo.Models
{
    /// <summary>
    /// This type is designed to conform to the structure required by the JqGrid JavaScript component.
    /// It has all of the properties required by the grid. When this type is serialized to JSON, the resulting
    /// JSON will be in the structure expected by the grid when it fetches pages of data via AJAX calls.
    /// </summary>
    [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId="Jq",
        Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
    public class JqGridData
    {
        /// <summary>
        /// The number of pages which should be displayed in the paging controls at the bottom of the grid.
        /// </summary>
        public int Total { get; set; }
        /// <summary>
        /// The current page number which should be highlighted in the paging controls at the bottom of the grid.
        /// </summary>
        public int Page { get; set; }
        /// <summary>
        /// The total number of records in the entire data set, not just the portion returned in Rows.
        /// </summary>
        public int Records { get; set; }
        /// <summary>
        /// The data that will actually be displayed in the grid.
        /// </summary>
        public IEnumerable Rows { get; set; }
        /// <summary>
        /// Arbitrary data to be returned to the grid along with the row data. Leave null if not using. Must be serializable to JSON!
        /// </summary>
        public object UserData { get; set; }
    }
}

Note that I’ve used C# capitalization for the property names. This is not what the grid expects. But I have to tell the grid that I’m using a non-default value for repeatitems anyway, so it’s nice to follow C# conventions.

We have lots of grids in our application, so rather than configuring each grid individually, we set defaults for the grid in one place. In a JavaScript file referenced by the Site.Master, this method is called in the jQuery ready event:

    setDefaults: function() {
        $.jgrid.defaults = $.extend($.jgrid.defaults, {
            datatype: 'json',
            height: 'auto',
            imgpath: '/Content/jqGrid/themes/basic/images',
            jsonReader: {
                root: "Rows",
                page: "Page",
                total: "Total",
                records: "Records",
                repeatitems: false,
                userdata: "UserData",
                id: "Id"
            },
            loadui: "block",
            mtype: 'GET',
            multiboxonly: true,
            rowNum: 20,
            rowList: [10, 20, 50],
            url: "ListGridData.json",
            viewrecords: true
        });
    }

In addition to setting repeatitems, telling the grid to expect the capitalization I mentioned above, this code also sets the grid theme and other options we want to be consistent throughout the application. One important property set in the jsonReader property is id. The value supplied is the name of a column in the returned data which contains a unique ID for the row. You can supply any name you want to, and the type of the property can be just about anything serializable, but it is important that the data you return to the grid contains some unique ID. All of the types in our Entity Framework model contain a unique ID called "Id", so we set this in our defaults for the grid. If you don’t have a single operating name for any data you might supply to the grid, you will have to set this value on the individual grids, when you write the JavaScript for the view.

Now I can write a method which takes a PagedList and transforms it into jqGrid data:

        /// <summary>
        /// Converts a paged list into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="list">The list of records to display in the grid.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data.
        /// Leave null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this PagedList<T> list, object userData)
        {
            return new JqGridData()
            {
                Page = list.PageIndex,
                Records = list.TotalItemCount,
                Rows = from record in list
                       select record,
                Total = list.PageCount,
                UserData = userData
            };
        }

That’s getting somewhere, but it’s not yet enough to let me write the action method I showed above. For that, I need to wrap up the PagedList inside the action so that I can pass the arguments from the grid directly, without having to construct a new PagedList inside of every action method which supplies data to a grid. So I build on the method above:

        /// <summary>
        /// Converts a queryable expression into a format suitable for the JqGrid component, when
        /// serialized to JSON. Use this method when returning data that the JqGrid component will
        /// fetch via AJAX. Take the result of this method, and then serialize to JSON. This method
        /// will also apply paging to the data.
        /// </summary>
        /// <typeparam name="T">The type of the element in baseList. Note that this type should be
        /// an anonymous type or a simple, named type with no possibility of a cycle in the object
        /// graph. The default JSON serializer will throw an exception if the object graph it is
        /// serializing contains cycles.</typeparam>
        /// <param name="baseList">The list of records to display in the grid.</param>
        /// <param name="currentPage">A 1-based index indicating which page the grid is about to display.</param>
        /// <param name="rowsPerPage">The maximum number of rows which the grid can display at the moment.</param>
        /// <param name="orderBy">A string expression containing a column name and an optional ASC or
        /// DESC. You can, separate multiple column names as with SQL.</param>
        /// <param name="searchQuery">The value which the user typed into the search box, if any. Can be
        /// null/empty.</param>
        /// <param name="searchColumns">An array of strings containing the names of properties in the
        /// element type of baseList which should be considered when searching the data for searchQuery.</param>
        /// <param name="userData">Arbitrary data to be returned to the grid along with the row data. Leave
        /// null if not using. Must be serializable to JSON!</param>
        /// <returns>A structure containing all of the fields required by the JqGrid. You can then call
        /// a JSON serializer, passing this result.</returns>
        [SuppressMessage("Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the JavaScript component this type is designed to support.")]
        [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Jq",
            Justification = "JqGrid is the correct name of the grid component we use.")]
        public static JqGridData ToJqGridData<T>(this IQueryable<T> baseList,
            int currentPage,
            int rowsPerPage,
            string orderBy,
            string searchQuery,
            IEnumerable<string> searchColumns,
            object userData)
        {
            var filteredList = ListAddSearchQuery(baseList, searchQuery, searchColumns);
            var pagedModel = new PagedList<T>(filteredList.OrderBy(orderBy), currentPage, rowsPerPage);
            return pagedModel.ToJqGridData(userData);
        }

Now you see that string array (IEnumerable, actually) again. JqGrid supports sorting and searching, and so do I. Like Phil, I suggest using Microsoft Dynamic LINQ in order to take the field names that jqGrid passes and turn them into a sorted, filtered, result set. ListAddSearchQuery is a (longish) method which takes a search string and a list of columns to search and turns them into a Dynamic LINQ expression and applies it to the base list. Unfortunately, search is kind of application-specific. In some applications you might want to search different columns in different ways. Some applications might want "Google-style" searching. It kind of depends upon what you’re doing. So although I will supply the source code for this method when I make the demo project available, I cannot promise that it will be useful for your application. Feel free to substitute your own method here.

It turns out both methods (the one which takes a PagedList and the one which takes an IQueryable) are useful. Generally, I use the method which takes an IQueryable. But every once in a while I want to do something to the data I’m about to supply to the grid before the grid sees it, but after paging, like transform a single page of results in some way. In this case, the action builds up a PagedList, uses that to reduce to one page of data, modifies the results in whatever way is appropriate, and then calls the PagedList extension. I’ve also written additional overloads to both methods which do not have a userData argument, since it is very seldomly used. userData, as the name suggests, is an additional bit of data you can return to the grid with your result set for use in grid event handlers when loading a fresh page of data. Since it’s of type object, you can put as much data in there as you need to, so long as it’s serializable to JSON. In practice, though, we rarely use this feature of the grid.

With these methods, it is easy to write actions as concise as the code I showed at the beginning of this post. Just get a list from somewhere and call ToJqGridData, passing information about which page to display, how to sort, etc.

In a future post, I will tie this all together into a demo application.

Share This | Email this page to a friend

jqGrid provides a nice-looking grid with built-in features such as sorting, editing, search, drill-down, tree lists, and more. It is based upon jQuery and integrates well with ASP.NET MVC. The grid is dual-licensed under both the GPL and MIT licenses (just like jQuery), so it can fit well into both commercial and open-source projects.

If you want to display tabular (grid) data with paging in an ASP.NET MVC application, there are essentially three choices:

1. You can write your own grid components using HTML tables, links, manually-implemented paging, etc. This is actually less work than it might seem at first, and the first release of our ASP.NET MVC software did just this. But our own implementation lacked some of the visual flair of some of the third-party components, and avoided solving some of the harder problems, like in-line editing. Still, it was much easier than I expected, and worked well enough for our initial release.
2. You can use a commercial component designed for ASP.NET. However, many commercial grid components don’t work well with MVC. Telerik has MVC demos available for their grid, and Developer Express has said that they intend to support MVC to some degree in the future. Even when the commercial components do support MVC at all, they don’t tend to make all of their features available in MVC applications, owing to the fundamental difference between the way ASP.NET MVC and "plain" ASP.NET work. If you are building a Web application on ASP.NET MVC, I would never recommend buying an ASP.NET component unless the vendor for that component is fully committed to supporting MVC. You cannot presume that any given ASP.NET component will work at all with MVC unless the vendor explicitly supports this.
3. You can use a grid component which has no knowledge of ASP.NET at all. There are quite a number of these, generally built within the framework of one of the popular JavaScript libraries, and it is, ironically, easier to use one of them within ASP.NET MVC application than it is to use a grid component designed for ASP.NET (but not necessarily MVC).

As I said, we ended up choosing jqGrid. We were strongly influenced by the fact that we already use jQuery extensively in our application. There are several grid plug-ins available for jQuery. jqGrid has a number of advantages over the others I’ve seen, however:

• Supports three different models for editing data within the grid: Single-cell editing, in-line row editing, and editing within a pop-up form.
• Unlike some of the other jQuery grid plug-ins, jqGrid has decent documentation and a large number of demos with source code.
• You can display a simple grid, a "sub grid" (drill down), or a treeview.
• AJAX paging is supported in a number of different formats, including JSON, XML, or JavaScript objects via a custom function.
• The grid is updated very regularly, and a new version, with support for the jQuery UI theming library is under active development right now.
• Tracing through the grid source code is quite easy to do with Firebug, which is useful for figuring out things which aren’t quite covered by the documentation.

The grid also has a few downsides:

• The API is a bit of a mess. Capitalization (significant, in JavaScript) is so varied that it seems almost random. Portions of the code were clearly written by different people with limited collaboration with the original author of the grid. There is no consistent naming convention, and similar functionality is sometimes invoked in wildly different ways. For example, you set the grid height with a custom method, setGridHeight, instead of the jQuery method height. But you reload the grid’s data with a jQuery method, trigger, passing the string "reloadGrid" as an argument. It’s a good thing the grid has decent documentation and includes source code, because you’re going to need them. The API is just not discoverable.
• Although the grid has quite a number of options for editing, it doesn’t necessarily play well with some of the other tools you may want to use in the context of editing, such as jQuery validation. (For starters, the in-line row editing doesn’t actually use an HTML form.) The three different editing modes (single-cell, in-line row editing, and editing in a form) use different source modules, different configuration options, and work differently. Doing editing in a way not explicitly supported by the grid can be challenging.
• Variable names in the source code use a lot of single letter and two letter abbreviations. This is quite common in JavaScript libraries, but it does make the source code a bit challenging to read.
• Although there is both free and paid support available, there is not a commercial company standing behind the grid. So don’t expect round-the-clock support, a toll-free phone number, etc.

After a good deal of experimentation with several grids, we decided to use jqGrid because it has considerably more features than every other grid we examined (when used in MVC application), and because we didn’t find any of its limitations to be show stoppers.

The next post in this series is LINQ Extensions. There, I show an extension method for IQueryable<T> which returns data suitable for the grid, without having to know anything about the type T.

Share This | Email this page to a friend

On the other hand, I have not found in practice that programmers need to be mathematically inclined to become great software developers. Quite the opposite, in fact. This does depend heavily on what kind of code you’re writing, but the vast bulk of code that I’ve seen consists mostly of the "balancing your checkbook" sort of math, nothing remotely like what you’d find in the average college calculus textbook, even.

{
  i = j++ / (x + v);
}

Not exactly the stuff mathletes are made of.

Arithmetic is one kind of math. It is relatively common in elementary math classes, and relatively uncommon in programming. Does Jeff’s example look like the "vast bulk of code you’ve seen?" It certainly doesn’t look like the vast bulk of code I’ve seen.

On the other hand, I do see a good bit of:

• Relational database queries (relational algebra)
• LINQ (functors)
• Composition of lambda expressions (lambda calculus, of course)
• Set operations and logical operators (set theory)
• Etc.

That’s by no means a comprehensive list. For example, every general-purpose programming language I’ve ever used would not exist without the Church-Turing thesis. But I didn’t include that, since most people don’t do programming language design.

All programmers work with mathematical systems day in and day out, whether we recognize them or not. Perhaps the ability to recognize math when we see it will help us better evaluate the importance of understanding it.

Share This | Email this page to a friend

Update: Attendee Bill Melvin wrote a short summary of the presentation.

Share This | Email this page to a friend

1. Decide where you want to go.
2. Aim the car in that direction.
3. Get there.

It’s different when you ride a horse. In that case the process is more like:

1. Decide where you want to go.
2. Communicate your intention to the horse.
3. Arrive at some hybrid of where you want to go and where the horse wants to go.
4. Return to step (1). Repeat.

Now, which of those two would you find more analogous to software architecture? Does that sound like a silly question? The reason that I ask is that I read a paper last night which argues in a persuasive but decidedly unconventional manner that the things you design have, for lack of a better word, intention. In other words, building a cathedral, or a software application is more like riding a horse than driving a car.

The paper is called Designed as Designer, by Richard P. Gabriel. As technical papers go, this one is unusual to say the least. It was written as a response to an address by Fred Brooks at OOPSLA 2007, during his keynote. In the address, Brooks stated:

If we look back then at the 19th century and the things that happened—the cartwright and the textile machinery, Stephenson (the train), Brunel’s bridges and railway, Edison, Ford, the Wright brothers, etc.—these were very largely the designs of single designers or, in the case of the Wright brothers, pairs.

Gabriel disputes this idea, and chooses an unexpected way of making his case. Gabriel is a published author both in the fields of computer science and writing/poetry. The paper draws as much from his experience in the latter field as the former—there is considerably more poetry quoted in the paper than source code. He also manages to weave in, with some relevance, no less, baseball statistics.

The general outline of the paper is as follows: First, there is a running, fictional narrative about a child named Pippo who observes the construction of the cathedral of Firenze. Next, Gabriel addresses Brooks’s point directly. He discusses the relationship between poets T.S. Eliot and Ezra Pound in the evolution of Eliot’s masterpiece The Waste Land. The middle, and longest, section of the paper is an explanation and example of a theory of poetry he has developed based around an almost algorithmic approach to classifying a given text as either poetry or ordinary writing. Contrasting an early version of a published column with a later revision to same, he asserts that the structure of the earlier version leads the poet into the later revision. I should add here that even though I really like poetry, I find his analysis a bit academic for my taste. Returning to the Cathedral of Firenze, Gabriel shows that many people were involved in its design and construction, and that the Cathedral form itself had a bigger impact on the final design than the architect. Then he uses statistical analysis of the career of Roger Maris to show that luck, as much as skill, played a role in Maris’s exceeding Babe Ruth’s home run record.

The final section of the paper includes, almost shockingly at this point, discussion of computer science and one small example of (Lisp) source code. He relates the story of how Gerry Sussman and Guy Steele discovered how two mechanisms they were studying turned out to be exactly the same thing by implementing both of them separately and noticing later that the source code was identical.

If all this sounds a bit, well, crazy, it isn’t. The paper was presented at OOPSLA 2008. I’m not above a bit of schadenfreude, but this is definitely a "laugh with him" rather than "laugh at him" paper. If you have ever had the experience of designing a system, and then uncovering necessary changes in the system when you try to use your design, then you have some idea of what he’s getting at.

I am also reminded, more than a little bit, of Michael Pollan’s book, The Botany of Desire, which calls the evolution of the humans and plants "co-evolutionary," and questions who has domesticated whom. Again, we don’t tend to think of plants as having "intention," but maybe the question should be, "just what is intention, really?" So all in all, it’s a fascinating read.

That said, it seems to me that, to some extent, Brooks and Gabriel may be talking past each other. Surely, Brooks can’t be under the delusion that the Wright brothers invented the airplane in a vacuum? Surely, Gabriel wouldn’t presume that he is? Much of Brooks’s writing focuses on organizational support for creativity. Indeed, Brooks, to me, seems almost to take creativity as a given, more so than examining what it is and what makes it tick. Gabriel seems to be on the other side of this coin, examining the nature of creativity, rather than how to make it work in a large organization.

Both points of view are worth reading, I think.

Share This | Email this page to a friend

http://www.example.com/foo.html

…which is fine, because if you go to such a link, you probably really will get HTML back, to URIs which (often) look something like this:

http://www.example.com/foo.php

…which, as I noted in a previous post, is a lie. Actually, it’s worse than a lie; it’s poor encapsulation. What happens when you come to your senses and stop using PHP? Now you URIs have to change, which means you get to either create redirects for that abomination or break every link to your site. One of the things we’ve learned from the success of Google is that the information encoded in the links between sites is nearly as important as the information on the sites itself.

The web is dynamic. It has to be, because it is too big to compile. And so the same concerns we have when writing code and dynamic languages often apply to creating a web site, whether you use a dynamic language or not. You need to choose very good names, because if you ever need to change them you will either break stuff or have a substantial job on your hands correcting every reference to the name you’ve changed, including references you don’t personally control.

So I’m going to propose a general rule, which states, "Any URI which allows an informed developer to discern the technology used to serve the result is probably broken." The resources you are serving, if they are at all valuable to the user, will probably last longer than the implementation you use to serve them.

Of course, choosing good names is hard. I read a great quote the other day:

There are only two hard things in Computer Science: cache invalidation and naming things. -Phil Karlton

Also, even after you’ve chosen a good URI, you may have to do extra work to make it functional. If routing is not built into the technology stack you’re using, you may have to implement a URL rewriter to make your persistent URIs functional. It’s worth the effort to do so.

Late update: Thank you to commentor Stecki for the link to the following article: Cool URIs don’t change, by Tim Berners-Lee. Well said.

Share This | Email this page to a friend

Mason Wheeler noticed that TObject.InstanceSize returns 8 (bytes). It turns out that this is new in Delphi 2009; in previous releases, TObject.InstanceSize returned 4. But when you look at the definition of TObject in System.pas, you don’t see any fields declared at all.

Four out of the eight bytes are consumed by the VMT; this has been true since the first version of Delphi. You can read more about that in this chapter from Delphi In A Nutshell, which is old, but still accurate insofar as the VMT is concerned.

But what about the remaining four bytes? I didn’t remember the answer, but I enjoy a good puzzle, so I took a closer look at the source code for TObject. While poking around in System.pas, I happened to notice the following constants:

{ Hidden TObject field info }
hfFieldSize          = 4;
hfMonitorOffset      = 0;

OK, that explains the four bytes, but what is this used for? Searching for hfFieldSize yielded the answer:

class function TMonitor.GetFieldAddress(AObject: TObject): PPMonitor;
begin
  Result := PPMonitor(Integer(AObject) + AObject.InstanceSize - hfFieldSize + hfMonitorOffset);
end;

So any object can have a reference to a TMonitor instance, and that reference is as big as a pointer on the target CPU. That reminded me of Allen Bauer’s long series of blog posts last year on the "Delphi Parallel Library," which discuss the TMonitor class in detail. He notes:

The TMonitor class [...] is now tied directly to any TObject instance or derivative (which means any instance of a Delphi class).  [...] For Tiburón [a.k.a. Delphi 2009], you merely need to call System.TMonitor.Enter(<obj>); or System.TMonitor.Exit(<obj>); among the other related methods.

The ability to lock any object is a good feature to have, especially in a multi-CPU, multi-core world. So I’m happy to spend the extra four bytes to get this feature. But I do find the implementation a bit mysterious. Why not just declare an actual field in TObject? I guess one reason would be to reduce the chance of a conflict with existing code. Another possible reason would be to enforce TMonitor’s contract; by obfuscating access to the field, you reduce the chances that an uninformed programmer performs actions on the field which break the contract. But these are just guesses. I don’t know why the Delphi team settled on this implementation. I presume they have some good reason.

Update: In comments, Allen Bauer explains the reasons for this implementation:

The reason for always placing the monitor reference field at the end of the object instance is mainly for backward compatibility. I didn’t want to alter the layout of objects in case there was some code out there that depended upon it. Also, by hiding the implementation the chance of inadvertent mucking with the monitor data was reduced and therefore it increased the overall safety of using it.

If you look closely at what happens on descendant instances, the monitor field is always the last field of the instance. This happens because the compilers (Delphi & C++) ensure that the field is allocated as the last step in the process of laying out the class instance. By doing this, an object laid out in pre-Delphi 2009 will be laid out exactly the same in Delphi 2009 except there is now an extra trailing pointer-sized field.

Share This | Email this page to a friend

Share This | Email this page to a friend

Yes, it is true that ASP.NET MVC does not force you to use one particular ORM. Yes, that means that you’re going to have to expend some thought, and quite possibly even effort, deciding which one is appropriate for you. It will not be sufficient to read a blog post about the Entity Framework from somebody who tried it once and another blog post about NHibernate from somebody who saw a screencast about it and then conclude that there are no workable solutions in the world.

But it’s worth the effort, because you have a big reward in store: Whatever choice you make will work just fine with ASP.NET MVC. Because the framework has no dependencies whatsoever on any ORM, all ORMs work equally well. You get to choose the best tool for your data access needs without worrying that your web framework will reject it.

Share This | Email this page to a friend

Share This | Email this page to a friend

• Stores a list of routes.
• Translates an incoming URI into lists of values and tokens, and passes those lists to the handler specified on the route. In the MVC framework, this feature is used to call the appropriate Controller/Action.
• Translates a list of values into a URI suitable for inclusion within an HTML page. In the MVC framework, this feature is most commonly used in the Html.RouteLink/ActionLink methods you call to generate a link in your Views.

Let’s examine in more detail how MVC uses Routing, and distinguish between the features of Routing and the MVC framework.

Components of a URI

Consider this sample URI:

http://www.example.com:80/SiteRoot/Controller/Action?foo=A&bar=B#anchor

The individual parts of the URI are:

• The scheme (protocol), authority (host), and port ("http://www.example.com:80" in this sample URI). I will mostly ignore this part in the rest of the post, since Routing doesn’t deal with it, except to pass it along when necessary.
• The path ("/SiteRoot/Controller/Action" in this example). Per the URI RFC, this part "serves to identify a resource…" I’ll talk about what that means in a second.
• The query ("?foo=A&bar=B" in this example). My example uses name/value pairs is the query, because that seems to be the most common case. However, the RFC says that the query can be almost anything, and the purpose of the query is to provide a non-hierarchical means of identifying the resource requested by the user.
• The fragment ("#anchor" in this example). I’m going to ignore this, since Routing doesn’t deal with it.
• I’ve omitted some stuff like username/password which is almost never used today.

MapRoute

Now lets examine a simple case of adding a route to the routing system. This is from the default MVC application template:

routes.MapRoute(
    "Default",                                              // Route name
    "{controller}/{action}/{id}",                           // URL with parameters
    new { controller = "Home", action = "Index", id = "" }  // Parameter defaults
);

Note that MapRoute is actually not part of the routing system; it’s an extension method added to RouteCollection by ASP.NET MVC. Calling MapRoute will add a single route to the RouteCollection. Most web applications will have only one RouteCollection, stored in System.Web.Routing.RouteTable.Routes. The most important thing that MapRoute does is to create and pass an MvcRouteHandler instance when adding the route to the RouteCollection. This ensures that when ASP.NET handles an incoming URI which matches this route, control will be passed to the MVC framework for handling the request.

The second argument to MapRoute is named "url", but it’s not really a complete URL. Instead, it is a template for a fragment of a URL containing a portion of the path component of a URI. I say "a portion of the path" because it omits the site root, which I’ve called "/SiteRoot" in the example above. The site root is often empty, but it doesn’t have to be. The URIs produced by the routing system include quite a bit more than the path, however. They also include the scheme/authority (in other words, the protocol and host) and, sometimes, a query.

There are a number of overloads to MapRoute. In addition to the values passed in this simple example, you can also pass constraints (which I’ll discuss in a moment) and namespaces. The namespaces argument is for MVC only (it’s not used by routing) and is useful when you want to put a Controller in a place where MVC wouldn’t otherwise find it.

A route can also contain DataTokens, which are not used by the routing system, but are passed to the route handler. MapRoute doesn’t have an overload which includes these as an argument, and I’m not going to discuss them in this post. ASP.NET MVC uses them only to store the namespaces argument, discussed above.

Parsing a URI

Let’s examine what happens when the routing system is asked to interpret a URI submitted by the user. Importantly, the RouteCollection is ordered. So the routing system will attempt to match the routes in the collection in the order you added them. The first matching route "wins" even if a later route is a "better" match. The incoming URI is compared with the url portion of the route. If it matches, the entire URI, including the query, is tokenized. If there are any constraints on the route, these are considered.

In practice, this means that if you have a lot of routes, then you are almost guaranteed to encounter a bug, sooner or later, where the wrong route is matched to an incoming URI. The only good way to prevent this that I know of is to write unit tests for your routes. Unfortunately, the name of the route is not actually part of the route itself. It is stored in a private, internal dictionary in the RouteCollection. So there is no way to write a unit test which examines the name of the route returned.

Let’s use the sample URI and route from earlier in this post as an example. Out of the entire URI, http://www.example.com:80/SiteRoot/Controller/Action?foo=A&bar=B#anchor, the routing system is only going to examine this bit when processing an incoming URI:

/Controller/Action?foo=A&bar=B#anchor

Does this match the url portion of the route defined above? Recall that it is defined as:

"{controller}/{action}/{id}"

Yes, it matches, because "Controller" can be interpreted as the "{controller}" portion of the url, "Action" for the "{action}" part, and a default value exists for the "{id}" portions (empty string). Note that this means that having default values in a route makes that route much more likely to "match" an incoming URI. If there are defaults for every token in a URL pattern and there are no constraints, then that route will match every incoming URI. The route shown in this post is created by the MVC framework as a "catch all" route for a new application, which contains only that route. That is why it has defaults for all of its values and no constraints; it is intended to match every incoming URI. If, on the other hand, you write a special-purpose route intended only for one particular controller or action, and you provide default values for all of the tokens, then you need to implement constraints to prevent this route for matching other controllers or actions.

Because the url matches, we have five values, so far, to pass as the route data. "controller" will have the value Controller, and "action" will have the value "Action". "id" will have the default value. Also, from the query, "foo" will have the value "A", and "bar" will have the value "B". Now the routing system considers any constraints on the route, but in this case, there aren’t any. Therefore, this list of values is handed to the MvcRouteHandler which was created by MapRoute.

Importantly, note that all of these values are strings. The routing system only deals with URIs, which are always strings. Now, your actions on your controllers can have arguments which are not strings, and the MVC framework will attempt to coerce the strings supplied by the routing system into whatever type is specified in the method signature. But that’s a feature of the MVC framework, not of the routing system. (Yes, the Values property returns type object, but, when matching an incoming URI, the segments are always added as strings.)

Not all URI patterns are as simple as the default route from the standard MVC application template. You can use delimiters other than /, for example. If you have hyphens, parentheses, etc. in your url template, these will be matched against the incoming URI to determine if the route is a match. The following route, for example, would not match the sample URI at the top of the post, even though it contains all of the necessary tokens and has no constraints:

routes.MapRoute(
    "Parenthesized",                                        // Route name
    "{controller}({action})({id})",                         // URL with parameters
    new { id = "" }                                         // Parameter defaults
);

The incoming URI contains no parentheses, so this route can’t match, because the parentheses are needed to to delimit the values for controller and action, and there are no defaults for this values.

Matching a Route With an Action

After a route is matched to an incoming URI, and a list of route values is created, control goes to the MvcRouteHandler. Now the MVC framework can invoke the action on the controller specified by the "action" and "controller" route values. Note that there is nothing special about these names/values to the routing system; they’re just two more values in the list from the routing system’s point of view. But these two names, "action" and "controller" are very important to the MVC framework, for obvious reasons.

MvcRouteHandler (indirectly) calls MvcHandler.ProcessRequest, which constructs the controller using the ControllerFactory, and then calls Controller.Execute, which, in turn, invokes the action using the ControllerActionInvoker. It is at this point that the values in the route are matched against the arguments for the action. In other words, tokenizing the URI and matching those values with action arguments are two entirely separate operations. The tokenizing is done by the routing system, and the matching route values with controller action arguments is done by the MVC framework.

In order to invoke the action, the MVC framework needs to look at the individual arguments to the action, and match them with route values. This is performed by the model binding system.

Note that the name of the argument does not have to correspond to a name in the route values if the model binding system can instantiate an instance of the type of the argument from the values which do exist. For example, if the type of the argument is FormCollection, then this argument can always be passed, no matter what names are present in the route values, and no matter what the name of the argument. The model binding system knows that it can always create a FormCollection by rolling up any values in the form in the incoming POST, if there are any. If there are not, it will simply return an empty FormCollection.

If the action takes an argument which cannot be instantiated based on the data in the route values, that is fine, so long as the argument is nullable. If the action requires a non-nullable argument which is not in the route values, or if the route values contain a value for an argument which cannot be bound into the type of that argument, the MVC framework will throw an exception. Otherwise, the action will be invoked.

Creating a URI With RouteLink

Creating an "a href=" tag with Html.RouteLink is trivial. The routing system looks up the route by name, and populates the URI with values from the RouteCollection you passed. Any values in the RouteCollection not included in the string in the url argument to MapRoute will be added to the resulting link as query string parameters. In my view, you should always prefer RouteLink over ActionLink, whenever there is any ambiguity as to which route will be returned.

Again, let’s consider an example, with the route defined in the MapRoute call earlier in this post:

<%= Html.RouteLink("Link text to display on page", "Default",
    new RouteValueDictionary(new {controller = "Foo", action = "Bar", foo = "A", bar = "B" })) %>

With this example, the order of the routes in the RouteCollection, the constraints, etc., are not needed to find the matching route, because we have specified ("Default") directly. Since there are no tokens called "foo" and "bar" in the url argument we passed to MapRoute, these will be appended to the generated URI as query string parameters. So we end up with the sample URI at the top of this post, less the anchor, provided that the site is configured to live in /SiteRoot.

Creating a URI With ActionLink

ActionLink is more complicated, because the routing system must first find an appropriate route, and then use that route to generate the URI. In a way, the method of finding the route is similar to the method used when parsing a URI, insofar as the first matching route "wins." But in another way, it is very different, because there is no incoming URI to use as a pattern for matching the partial template passed as the url argument to MapRoute. The only thing that the url argument is used for in this scenario is determining which tokens must be included, either directly, or via defaults, in order for a route to match. Let’s re-examine the same scenario considered in the RouteLink section, above, only this time using ActionLink. The call looks like this:

<%= Html.ActionLink("Link text to display on page", "Bar", "Foo",
    new RouteValueDictionary(new {foo = "A", bar = "B" })) %>

Does this match the route defined at the top of this post? Yes it does, and here’s why: That route contains three tokens in its url argument. They are controller, action, and id. I have supplied values for two of these tokens in the call to ActionLink, namely controller and action, via the controller and action arguments in the overload of ActionLink I am using here. The remaining required token, id, is not supplied in the ActionLink call at all. However, there was a default for it (empty string) in the original call to MapRoute, so the route matches even without an explicit value for id. Because foo and bar are present in the RouteValueDictionary, but the url template in the route does not contain {foo} or {bar}, these will be added to the generated URI as query string parameters. Again, the generated URI will be the sample URI at the top of this post, less the anchor, provided that the site is configured to live in /SiteRoot.

Conclusions

I hope this long explanation has taken some of the mystery out of what routing does. Since there’s a lot of detail above, I’m going to summarize some of the information which is important for day-to-day work.

• Routing and the MVC framework are distinct features. MVC knows about Routing, but Routing does not know about MVC. Routing produces a list of values, but does not assign special meaning to them. MVC signs special meaning to the controller and action values.
• Routes include information about a portion of the path in the URI and also the query, since any token not in the path is treated as part of the query.
• The order of routes is very important, both for parsing incoming URIs and for generating URIs with ActionLink.
• Default values in a route will cause the route to match more incoming URIs or sets of values in a RouteValueDictionary. Constraints in a route will cause the route to match fewer incoming URI this or sets of values in a RouteValueDictionary. If there are defaults for every token in a URL pattern and there are no constraints, then that route will match every incoming URI or sets of values in a RouteValueDictionary. For a "catch all" route like the one in the default MVC application, it is fine for all of the tokens to have default values in for there to be no constraints. If you write a special-purpose route for a particular controller or action, however, then you should probably use constraints to prevent it from ever matching any other controller or action.
• If you have more than one route, then you simply must write unit tests to ensure that the correct route is selected for all incoming URIs or values used in your calls to ActionLink.

Share This | Email this page to a friend

Database Workbench supports InterBase and many other DB servers. You can buy the module for one (or more) DB server only if you don’t need all of them.

Share This | Email this page to a friend

I haven’t read this book, but someone in the newsgroups mentioned that it included InterBase. He writes:

"Note that they tested on IB6 — but I have still found most of their advice to be valid."

Share This | Email this page to a friend

When a web application misbehaves, one of the first things you’ll need to do is determine if the problem is in the browser/client (e.g., buggy JavaScript, attempting to use CSS on IE 6, etc.) or in the server application (e.g., returns incorrect HTML, inserts wrong data into database, etc.). In order to make this distinction, it is helpful to use a very simple conception of what a web application does. HTTP is a stateless protocol. Therefore, a web server is expected to always return the same results in response to a certain request, without regard to what has happened beforehand. So we can consider the server as a single function:

The illustration above may appear a bit strange, because of the close proximity of the browser and the database server. In reality, we know that the database server is likely closer to the web server than to the browser. But remember, I am attempting to treat the web server as a pure function here (I’ll explain why in a minute). The state of the database, if there is one, will certainly affect the page returned by the web server. Therefore, it is an input to the function. Similarly, authentication, which can also change the page returned by the web application is most often stored in a cookie. So cookies are also inputs to the server function.

Using this abstraction, it is now very easy to diagnose where the bug exists, by looking at the inputs to the function and its result:

• If the inputs are correct and the result is wrong, then the bug is in the server. But remember that "result" means the HTML returned by the server, not the manner in which the browser renders it. It is possible for correct HTML to be rendered incorrectly, especially if JavaScript or IE 6 are involved.
• If the inputs are incorrect, then the bug is obvious. But remember, inputs can include things like form data which, for some reason, people don’t always tend to examine.
• If the inputs and the result are both correct, but the page still looks wrong in the browser, then the bug is in the browser/client side.

The only circumstance in which this abstraction doesn’t really work is when you’re maintaining state on the client. An example would be when using frames. If you click a link which targets a frame, the result that you see in the browser will be different depending upon what you have done before, because you might have other pages open in other frames. This is an example of why I think that frames are not a good architectural fit with the rest of the web.

Once you’ve determined eactly where the bug lives, you are well on your way to fixing it. You can also describe the bug with more precision. For example, you will say things like "the server returns incorrect HTML (incorrect style on the div with ID ‘foo’) when…" instead of "the panel is on the wrong place in the page when…".

Colophon

It’s not really relevant to the rest of the post, but here is how I made the illustration above. I decided the post would be easier to understand if I included a diagram of the function. So I followed my standard process for using Microsoft Visio, which is something along the lines of the following:

1. Decide that I need a very simple block diagram, and remember that this is something which Visio is supposed to be good at.
2. Attempt to use Visio to produce the diagram.
3. "WTF, I’m not allowed to resize this? I thought that Visio was supposed to be for simple drawings."
4. Decide, "Wow, I’d forgotten how much I really despise Visio."
5. Produce diagram in some other piece of software.
6. Repeat, 6 months later. Some day I’ll learn.

I ended up using Inkscape, which has more than a few rough edges, but which was good enough to produce a simple diagram, which is more than I can personally say for Visio. The drawing includes an MPLed globe image, so I’m releasing the whole drawing under the terms of the MPL.

Share This | Email this page to a friend

Share This | Email this page to a friend

Share This | Email this page to a friend

They told us about the All Access product, which had been officially released that day. One of the most interesting things about this product is that it is installation-free, or so Embarcadero claims. It uses a virtualized registry, etc., so that you can just start the application directly. I’m thinking that this might be useful even outside of the context of All Access. Especially for something like a build machine, where you might want to run it in various different configurations. Sort of like a "lite" VM. It will be interesting to see if Embarcadero uses this technology outside of All Access.

One thing I wonder about right now: Perhaps the biggest differentiator between Delphi Enterprise and Delphi Architect is that with Architect you get Embarcadero ER/Studio. So with the way the SKUs are differentiated right now, All Access is sort of an alternative to Architect. That’s how it seems to me, anyway. I suspect that we will see a different feature mix in Architect in the future.

David expressed a lot of relief at not being part of Borland anymore. He said that since Embarcadero is private, they can talk about upcoming features more freely than when they were part of a publicly held corporation.

David also talked about how Delphi developers who do database work could really benefit from the Embarcadero database tools. He wants to bring the integration much closer. For example, he wants you to be able to step into the Rapid SQL stored procedure debugger directly when debugging through a Delphi application which called a stored procedure in the database. Embarcadero has profilers for Java applications and for database servers; you can see the potential for interaction there, too.

We are looking for a good solution for doing heterogeneous, bidirectional replication between InterBase and SQL Server. We have a working solution already using a SQL Server job and linked servers, but I would prefer a solution that we do not have to develop and maintain. I am aware of CopyTiger, which claims to support both, but doesn’t really seem flexible enough to do what we need. There are a lot of replicators which doesn’t explicitly support InterBase, that support JDBC or ODBC connections, and they could possibly be a solution. David suggested that Embarcadero’s Change Manager product would be a potential solution for us, so we’re going to look into that.

Share This | Email this page to a friend

var emptyInput = $("form input[value='']");

That’s a little hard to read, so note that the quotes around the entire selector are double quotes, but the quotes inside the square brackets are two single quotes. This says, "Give me any input where the value is equal to empty string."

This is broken in jQuery 1.3.1. Try it, and you will see (if you have a debugger attached):

Syntax error, unrecognized expression: value=""]

A fix for the bug has been checked in, so you can get a corrected version in the nightly build. If you prefer to stick with 1.3.1, I’ve found a workaround; do this instead:

var emptyInput = $("form input:not([value])");

This works because empty string is implicitly converted to false in JavaScript.

Update: The bug is fixed in jQuery 1.3.2.

Share This | Email this page to a friend

• HTML, a markup language for displaying rich documents
• HTTP, a stateless protocol for communicating with a web server
• The URI, a short, standardized reference to a network resource

Almost 20 years later, very little has changed, especially from the point of view of the server. To the list above, we have added client-side stuff like JavaScript and CSS, and HTML is now at version 4. But HTTP and the URI are close to unchanged.

And all three of the items in the list above are just text. Not just any text, mind you, but text which is formatted in such a way as to be mostly easy for humans to read.

This means that what web servers do is very simple. They accept text over a text-based protocol and return more text.

Despite this, people occasionally seem to find web servers confusing, in part because we like to lie about what they do. For example, you might see a URI which looks like this:

http://www.example.com/foo.php

This is a lie. If you point a web browser at this URI, you will almost certainly not receive a PHP source file; instead, you will probably get HTML. That the HTML was generated with PHP source code is an implementation detail unsuitable for inclusion in a Uniform Resource Identifier.

Some people might think I’m being pedantic here. How many end users actually care what the extension in the URI is? These are probably the same people who post questions on Stack Overflow wondering why their web application doesn’t work, but not including in the question the only things that the server really cares about, which you can find in the list at the beginning of this post. It doesn’t matter if you generated your HTML using aspx, rhtml, or PHP. It’s still just plain HTML to the browser. When you submit a form, you’re submitting name/value pairs of strings, not "integers" and "dates" and "classes."

That’s why this matters to a programmer: Web applications, as a class, are extremely easy to debug. Tools like Firebug and Fiddler allow you to see the text that is exchanged between the browser and the server, and even pretty it up a little to make it more readable than it already is. But already built-in to your web browser is the single most important tool that you need for debugging a web application; namely, the "View Source" command. Chances are very good that if you look at the HTML, the HTTP, and the URI, you can solve whatever issue you’re having with your web application. But if you can’t, including whichever of these three is relevant to the problem at hand will dramatically increase the chances of getting a solution when you ask someone else to help you out.

Perhaps it’s the fact that we’re used to debugging within an IDE which causes people to look at, for example, aspx source code instead of HTML when a web page doesn’t render the way we’d like, or to debug through model binders instead of looking at a form submission when the input to a controller action is not what we expect. But remember, the browser doesn’t see your aspx or PHP. The browser just sees the HTML you generate.

Perhaps another reason why this tends to confuse people is that certain web frameworks, such as ASP.NET, introduce fictions such as "event handlers" and "postbacks." No such concepts exist within HTTP. Instead, they are (extraordinarily leaky) abstractions built on top of HTTP within a certain framework. But under the hood, there is still just HTTP. If you want to understand the communication between your web application and a browser, look at the HTTP and the HTML.

I’m going to close this post with a recommendation: If you’ve never visited a web site via telnet, go do it right now. It only takes a second, and you’ll learn a lot in the process. This is an easy way to understand at a very fundamental level how web browsers and web servers talk to each other. If you need to debug your web application at a client site and can’t install Fiddler, you can probably use Telnet instead.

This is the first article in a series on the dynamic web.

Share This | Email this page to a friend

public ActionResult UpdateInts(IList<int> ints) {

The MVC framework, in this case, will look at all the inputs on your form, figure out which inputs correspond to which records on the form, find each record individually, and pass the whole list to your action as an IList<T>. Pretty convenient, that.

Note that the format for the name element requires square brackets and a period.

<input type="text" name="products[0].Name" value="Beer" />

There is no way around this required formatting; it is hard-coded into the framework source code. (See DefaultModelBinder.cs.)

Unfortunately, these are special characters in jQuery selectors. As a result, some plug-ins can become unhappy when you use them in an input name. For example, the ClockPick plugin has an option which is useful when you want an image that the user can click on in order to pop up the clock picker instead of clicking on the input itself:

$(".clockpick").clockpick({
    valuefield : 'myfieldname'
});

If the name attribute of your input happens to be myfieldname, this works fine. If, on the other hand, it is products[0].Name, it doesn’t work at all.

The solution is to include an extra set of quotes around the name with the brackets and the dot:

$(".clockpick").clockpick({
    valuefield : "'products[0].Name'"
});

With only one set of quotes, we just have a string containing products[0].Name. With two sets of quotes, the string contains the same thing, except surrounded by quotes. This causes jQuery to not treat the square brackets and the dots as special characters. As the jQuery documentation notes:

Quotes are optional in most cases, but should be used to avoid conflicts when the value contains characters like "]".

Share This | Email this page to a friend

type
  TFoo<TypeParam1; TypeParam2: ISomething> = class

This means that TypeParam2 must support ISomething. In C#, you would use where instead of the colon syntax.

To use (as opposed to declare) a generic type, however, you separate the type parameters with commas, like this:

  Foo := TFoo<integer, TBar>.Create;

Does that seem a little odd? It might to a C# programmer, I guess. But it should look very familiar to a Delphi programmer, because it exactly mirrors the syntax for procedure arguments:

  // declare:
  procedure Foo(ArgA: integer; ArgB: boolean);
  // invoke:
  Foo(A, B);

Indeed, the only reason this is really worth mentioning at all is that if you forget and you use a comma instead of a semicolon when declaring a list of type parameters in a generic type declaration, the message you get from the compiler is not especially helpful.

Share This | Email this page to a friend

Trust me, just leave that little bastard alone.

To understand why, we should review the Windows standards for interacting with radio buttons using the keyboard. If you drop several TRadioButtons onto a group box, then these buttons are treated as a group; although they aren’t quite the same thing as a TRadioGroup, they behave similarly. Namely, if you tab into the group box, then the checked radio button will receive input focus. If you press tab again, you will tab out of the group box. When the checked radio button has focus, pressing the arrow keys will check a different radio button in the group (and un-check all other radio buttons in the group box).

The VCL does not rely on Windows for this behavior; instead, it is implemented in the VCL source code, and you can trace through it.

The reason that this is on my mind is because of a curious bug report I received the other day. A tester reported that tabbing into a radio group on a certain form caused the selection to change (in other words, a different radio button was checked simply by tabbing into the group box). Based upon the explanation above, you can probably see where this is headed, but it took me a long time to figure out. The behavior was easy to reproduce, and I could even see a CN_COMMAND message of BN_CLICKED in the debugger, but it was near the top of the call stack, and I could not figure out where it was coming from.

It seems that if you tab to this particular radio button, and only this particular radio button, and only when you first enter the form, Windows will send BN_CLICKED. But why was I able to tab to a non-selected radio button at all? Shouldn’t focus go to the selected button?

No, not really. Again, Windows doesn’t choose which control is focused when you press tab; the Delphi VCL does. Glossing over a few details, it essentially looks for the next enabled control where TabStop is true. When you click a radio button, in addition to un-checking all of the other radio buttons in the group, the VCL sets their TabStop properties to False.

But only if they weren’t checked in the first place. If a radio button is not already checked, then its TabStop property will not be set false when another radio button in the radio group has its Checked property set true.

So here is what actually happened:

1. A few months ago, a well-meaning developer had done some tab order maintenance on the form in question. He changed the TabStop and TabIndex properties of a number of controls. The changes looked quite innocuous, and even passed code review.
2. Unfortunately, he had set TabStop true on a TRadioButton which was not Checked.
3. Some code in the form configured the radio group by setting the Checked property of a different radio button True. This caused the Checked and TabStop properties of the other radio buttons in the group to be set False, but only if their Checked properties were not true to begin with.
4. Now the tester tabbed into the group box. The first radio button in the tab order was not checked, so a BN_CLICKED message is sent. The tester is surprised that the "wrong" radio button is focused, and, worse, it is checked, as well.

The moral of the story? Just don’t touch the TRadioButton.TabStop property. The VCL is going to manipulate it in code. Leave it at its default value. If you want to change whether or not the user can tab into a group of radio buttons on a TGroupBox, then change the property of the container, instead.

Share This | Email this page to a friend

A first try

So here is a naïve attempt at creating metadata for this sort of data. I’m going to demonstrate this as a database table, but the issues involved are nearly identical when you create similar structures in code.

CREATE TABLE TimeRecords (
  Id INTEGER NOT NULL PRIMARY KEY,
  PersonId INTEGER [...],
  WorkDate DATE,
  StartTime TIME,
  StopTime TIME,
  TotalHours NUMERIC(18, 4));

This might look OK at first glance, but it turns out to be fairly badly broken. Before I discuss why, I should clarify the data types involved.

What isn’t broken

One thing you might notice when looking at this metadata is that I’ve made most of the columns nullable. In my experience, it is very important to allow users to save work in progress, and to only do validations (such as requiring that the Person is filled in) when you actually try to do something important that the time record, such as submit it for payroll. For example, an employee could begin a time record when they "punch in" at the beginning of a shift, and finish the time record when they "punch out" at the end of a shift.

DATE and TIME in this DDL refer to the SQL Standard data types which contain, respectively, a date and a time only. The SQL Standard data type for a date and time in a single column is TIMESTAMP. Not all databases (or programming languages/environments) support such types. In many cases, you will have to do with only TIMESTAMP, or its equivalent, such as TDateTime in Delphi.

Note that TotalHours is NUMERIC, not TIME. It is important not to confuse a time span with a time of day; they are different units. In .NET, there is a TimeSpan type which is appropriate for this.

Problems with the naïve approach

Presuming that the user has filled in all of the required information for a time record, what would we expect to see in a typical row?

If the user is recording only the date and the number of hours worked, it seems pretty obvious: both the WorkDate and the TotalHours fields will be non-NULL, and the StartTime and StopTime columns will be NULL.

What if the user needs to record the start and stop times? Well, the WorkDate, StartTime and StopTime fields almost be non-NULL. But what about TotalHours? We could fill that in, but wouldn’t it be redundant with the information in StartTime and StopTime? Or, worse, could we even be sure that all three fields are kept "in sync?" So we might be tempted to leave that NULL, but then, what if we want to do a SUM in SQL? By filling in TotalHours, we can compute an employee’s total work time, even if some of the records are filled in with start times and stop times, but others were filled in with only the total hours worked.

To compute the total hours on a record that has start and stop times, we might be tempted to subtract the start time from the stop time. But that doesn’t work. An employee could work a shift from 8 PM to 2 AM, yielding a negative result for this calculation.

We could make a rule that we add a day to the stop time before subtracting if it is less than the start time, but this presumes that the employee’s shift was less than 24 hours. Greater than 24 hour shifts may be uncommon, but they do happen; I once worked a 31 hour shift, fresh out of college.

What this means is that recording only the start and stop times without the start and stop dates does not give us sufficient information to compute the total hours worked in all cases, and makes it more difficult in many cases.

Some useful redundancy

Here’s another way to store this data. This is what I generally do. But I’ll forgive you if you’re skeptical when you see it. I’ll discuss some alternatives later.

CREATE TABLE TimeRecords (
  Id INTEGER NOT NULL PRIMARY KEY,
  PersonId INTEGER [...],
  WorkDate DATE,
  StartTime TIMESTAMP,
  StopTime TIMESTAMP,
  TotalHours NUMERIC(18, 4));

First, it is no longer a problem to compute TotalHours from the StopTime and StartTime, if they’re filled in. StopTime minus StartTime will always equal the correct total hours worked, when they are non-NULL. Also, it is no longer a problem to record a greater than 24 hour shift.

But isn’t there some redundancy here? WorkDate is always going to equal the date portion of StartTime, when StartTime is non-NULL. Somebody once suggested to me that it would be possible to just use the date portion of the StartTime TIMESTAMP for a "TotalHours-only"time record (and drop the WorkDate column altogether). The problem with this idea is that a TIMESTAMP always contains time information, whether or not you bother to fill in. If you assign only a date, then the time portion is presumed to be midnight. So there is no way to distinguish between a record where the user intended to fill in only total hours and one where the user intentionally filled in midnight as the StartTime but forgot to fill in the StopTime. That will be a problem when you validate the record prior to submitting it for payroll.

Another way

I found that the approach above works well for me, but those who find the duplication of one date completely untenable might be tempted to do something along the lines of this metadata:

CREATE TABLE TimeRecords (
  Id INTEGER NOT NULL PRIMARY KEY,
  PersonId INTEGER [...],
  WorkDate DATE,
  StartTime TIME,
  StopTime TIMESTAMP,
  TotalHours NUMERIC(18, 4));

There’s no redundancy in this version. You can store both time records with start times and stop times or time records with just elapsed times, and it’s easy to distinguish the two. Unfortunately, certain queries become more difficult. With the version which uses TIMESTAMP for the StartTime column, I can write a query like this to determine who was in the building at a particular point in time:

SELECT
  PersonId
FROM
  TimeRecords
WHERE
  :SomeParameter BETWEEN StartTime and StopTime

Records without start and stop times will be ignored by this query, since they do not contain enough information to answer the question. With a TIME instead of a TIMESTAMP data type for the StartTime column, I would have to use an expression in the where clause to write this query:

SELECT
  PersonId
FROM
  TimeRecords
WHERE
  :SomeParameter BETWEEN (WorkDate + StartTime) and StopTime

That might well mean that my query can no longer be indexed.

Conclusion

Although my preferred schema has a small bit of redundancy, I find it significantly more usable.It is considerably less work to handle copying the date between the WorkDate and StartTime fields, and computing the TotalHours from the StartTime and StopTime, than it is to not have the date in either column.And not having dates in the StartTime and StopTime columns at all is just unusable.

Share This | Email this page to a friend

Because of this, TempData should only be used when you know exactly what the next request is going to be. If you set TempData and your action then returns a ViewResult, then the next request, whatever it happens to be (an AJAX request, another page the user opened in a different tab, etc.), is going to see the TempData value you set, and no other request will see it. I’m guessing that this is never the behavior you want.

Realistically, this means that you should only use TempData immediately before you redirect (in other words, immediately before you return a RedirectResult), because this is the only time when you are sure what the next request from that browser is going to be. I actually think that RedirectData would be a better name for this property. Someone, perhaps Steve McConnell, once noted that you should never include "temp" in a variable name, since variables are, by definition, temporary. "Temp data," to me, is synonymous with "variable." The (functionally similar) "flash" feature in Rails is even more badly named.

If your controller action returns a ViewResult, and you are tempted to put data into TempData, don’t. Use ViewData, instead, in this case.

This may lead to a situation where you might need to have a view look into both ViewData and TempData for a key. One example would be if you have an area in your Site.Master to display error messages on the page. It is likely that one controller action will need to send a message to the view via ViewData (because that action returns a ViewResult), while another action will need to send a message to the view via TempData (because that action returns a RedirectResult, as part of a POST/redirect/GET cycle). In this case, you can use code like this in the Site.Master:

<%= ViewData["Message"] ?? TempData["Message"] %>

Under the hood, TempData works by using server-side Session state, so a distributed session cache like Velocity can be used if your application runs on a server farm. Since the lifetime of data put into TempData is limited to the next request, it is fairly easy to use without worrying about growing the session very large, especially since the most common use for TempData is to store a short error string. But the usual rules on what you can put into a Session apply. If you set the Session state mode to out of process, then the data you put in it must be serializable.

Share This | Email this page to a friend

[DataDirect][ODBC InterBase driver][InterBase]Dynamic SQL Error, SQL error code = -804, SQLDA missing or incorrect version, or incorrect number/type of variables.

I’ve seen this error before when the wrong InterBase client (gds32.dll) version was loaded by the application, or when the wrong number of variables is passed to a stored procedure. But yesterday I found a new way to get the error. It took me quite a while to debug this, so I’m putting the information out in case other people encounter it.

It seems that certain versions of InterBase server don’t respond well to passing a parameter of type ShortInt to a parameterized query that calls a stored procedure with that param as an integer argument. In particular, I found that doing this will fail on InterBase server version 7.5.1.162, but works fine on 7.5.1.80. Changing the type of the parameter to integer fixes the problem. I was doing this via ODBC; I don’t know if it makes a difference or not.

(Why was I using a parameter of type ShortInt in the first place? I wasn’t, exactly. But the value that I assigned to the Param.Value, a variant, was the constant 0. Delphi will treat the constant value 0 as a ShortInt.)

This is a good example of why unit testing can never be sufficient to completely test an application. Since this is a server error, a unit test can’t find it. Since it only happens on certain versions of the server, even basic integration testing might miss it. The only way to have a hope of finding such an error is to do automated testing on multiple environments. In this case, the error prevented a Crystal report from executing at all. We have a testing tool built into our applications that executes all of the reports and saves timing information to a log file. We run this with every QA releasewe do, and compare the timing information with previous releases. Sometimes we catch reports which won’t executed all, as with this case. In other cases, we find that the report has become much faster or much slower, due to a database metadata change.

The report profiler is part of a "debug console" which is built into the applications and provides testing and diagnostic information at runtime. I said before that testing is a valid use case, and this is one example. Another, report-related example is that whenever our applications modified the SQL in a report (to customize the result set for an end user), we have a way to display the final SQL before it is executed by the report. The reason is that a particular customer might make a particular selection that either performs badly or does not execute at all. (End users don’t directly edit SQL; instead they choose the data they want to see graphically, and we build SQL from their choices.) In order to reproduce their problem in a debug environment we need to know precisely what kind of choices they have made, and how the software handle those choices. Displaying the SQL before it is executed answer six questions.

Share This | Email this page to a friend

So let’s pretend that I’d like to extend the default "required" validation in such a way that you are not allowed to enter "foo" in any form field with the "required" class. First, I need to write a function to test this:

function notFoo(value) {
    return value !== "foo";
}

Putting the custom code in a separate function makes it very easy to unit test.

Now, I’d like to replace the default "required" validation with one which first calls my custom method, and then does the regular, default validation, if it passes the custom validation. I can call jQuery.validator.addMethod, and pass "required" as the name argument. Even though this is a standard validator, supplied by the jQuery validation framework, I’m allowed to replace it with my own custom method:

jQuery.validator.addMethod("required", function(value, element, param) {
        return notFoo(value);
        },
    jQuery.validator.messages.required // use default message
    );

So far, I’ve only replaced the standard "required" validator. This is useful by itself; there may be times when you want to write code to completely replace the standard validator. If you put this code in your site master template, you will have accomplished that goal.

But my goal was to extend the default validation, not replace it. So I need to store a reference to the default "required" validator, and then call it, after calling my custom code:

jQuery.validator.methods.oldRequired = jQuery.validator.methods.required;

jQuery.validator.addMethod("required", function(value, element, param) {
        if (!notFoo(value)) {
            return false;
        }
        return jQuery.validator.methods.oldRequired.call(this, value, element, param);
    },
    jQuery.validator.messages.required // use default message
    );

I can’t call the oldRequired method directly, because it uses "this", which will point at the wrong thing if I just invoke it without call.

Before I wrap up this post, I would like to extend thanks to the folks at Stack Overflow who helped me fix a some issues in my first attempt at doing this.

Share This | Email this page to a friend

The problem that I have with a statement like, "Don’t test private methods," is that it is a circular argument. That a method is private implies that you don’t want to test it (or haven’t thought about testing it). So it’s like saying, "Don’t test stuff that you don’t want to test." That’s not very helpful.

Similarly, when people say "test everything," I think they’re being a little vague. I can go along with "(unit) test everything that can practically be tested," but I’m not going to pretend that unit testing will take the place of integration testing, that it is possible to test every condition in a complex software system, etc.

Last year, I asked the question, "Why hide information?" Without needing a definitive answer for that question, let’s agree that, given a certain use case and the code which addresses it, there is some portion which must be public in order to be useful. Internally, however, there is probably some additional code which does not need to be public in order to solve the problem at hand. One popular school of thought says that code which is not needed in the public interface should never be public. (And similarly for protected, etc.)

At this point, I would like to point out that testing code is a legitimate use case. I commonly refactor functioning code to make it more testable. This is code which appears to work just fine, and requires no changes in functionality to the end user. Despite the aphorism, "if it ain’t broke, don’t fix it," I’m willing to make these changes, because I think that the ability to run more tests is a good feature to add to the software.

Some would argue that testing the public interface of a piece of code is sufficient, because any internal bugs which are not reflected in the public behavior are not relevant to how the code is actually used. This is true insofar as it goes, but it turns out that exhaustively testing the public behavior of a complex collection of code is extremely difficult, especially if that code has state.

Let’s imagine a function called Foo which accepts an integer argument and returns some value, but doesn’t use any kind of state. There are, therefore, 2^32 possible test cases for this function. Imagine another function, Bar, with the same characteristics. Again, 2^32 possible test cases. That’s a lot, and perhaps we would not want to test every possible case, but even if we did, it’s nothing that a computer can’t handle. Now imagine a type with a public function called Baz which uses both Foo and Bar internally, passing some internal state of the type. Because Foo and Bar are not needed for the end user’s use case, we decide to make them private. The list of possible test cases for the type now probably exceeds what can be performed with a fast computer in your lifetime. Typically, we try to work around such excessively large numbers of test cases by probing boundary conditions. But if private state is involved, this turns out to be very difficult to do. Given that making a field private implies that the implementation may change, probing the behavior of Baz by trying to make it hit the extremes of Foo and Bar is both difficult and likely to be a maintenance nightmare in the future.

In such a situation, I would argue that the need to test the behavior of Baz is by itself reason enough to make Foo and Bar public. If we are positive that both Foo and Bar work correctly (e.g., because we have exhausted the unit tests), then testing the behavior of Baz is either easy or so trivial as to be unnecessary. Because we don’t want to pollute the namespace of the type containing Baz, and because neither Foo nor Bar use any internal state, it would probably make sense to make them public members of a separate type, probably a static type. The point is that even if they are never used anywhere else in the system, it is worth making them publicly visible for the sole purpose of improving the coverage and efficiency of our testing.

Share This | Email this page to a friend

Imagine that you have been given the task of fixing a bug in the following code. Before you fix the bug, you would like to write a unit test which shows the bug, in order to prevent regressions after you fix it.

procedure TMyDM.qryFooCalcFields(DataSet: TDataSet);
var
  Minimum, PieceRate: Double;
begin
  Minimum := qryFooHOURS.Value / qryFooMINIMUM_WAGE.Value; // oops; should be *
  PieceRate := qryFooPIECE_RATE.Value * qryFooPIECES.Value;
  qryFooEarnings.Value := Max(Minimum, PieceRate);
end;

Now, tests which access a database are, generally speaking, integration tests rather than unit tests. But the code you’ve been asked to fix seems to be pretty wound up in database access. It doesn’t have to be, though. Let’s refactor:

function TMyDM.CalcEarnings(AHours, AMinimumWage, APieceRate, APieces: Double): double;
var
  Minimum, PieceRate: Double;
begin
  Minimum := AHours / AMinimumWage; // oops; should be *
  PieceRate := APieceRate * APieces;
  Result := Max(Minimum, PieceRate);
end;

procedure TMyDM.qryFooCalcFields(DataSet: TDataSet);
begin
  qryFooEarnings.Value := CalcEarnings(qryFooHOURS.Value,
    qryFooMINIMUM_WAGE.Value, qryFooPIECE_RATE.Value, qryFooPIECES.Value);
end;

Resist any temptation to actually fix the bug at the moment; we went to write a unit test which fails before making it pass by fixing the bug. Note that you can make CalcEarnings static, meaning you won’t even need an instance of the data module in order to unit test the function:

procedure TestMyDM.CalcEarnings;
var
  Actual, Delta, Expected, Hours, Minimum, PieceRate, Pieces: Double;
begin
  Hours := 1;
  Minimum := 7;
  PieceRate := 1;
  Pieces := 1;
  Expected := 7;

  Actual := TMyDM.CalcEarnings(Hours, MinimumWage, PieceRate, Pieces);

  Delta := 0.0001;
  CheckEquals(Expected, Actual, Delta);
end;

There are aspects of this code which will be hard to unit test. I could, for example, pass the wrong field value into the refactored function. I can fix that by abstracting the data access layer, but it’s not a complete fix, since I might have the wrong data in the database. The fact is that unit testing can only catch so much, and integration testing will always be necessary, as well. The important point is that I have isolated the bug which I was asked to fix, and future regressions in that code will now be caught.

I wonder if some Delphi users are confused by the term "unit test." The word "unit" means something very different in Delphi than it does in the term "unit test." In "unit testing," the word "unit" refers to a single piece of functionality. In Delphi, a unit (reserved word) is a source code file. When you write unit tests, you do not have to test an entire unit at a time. Unit tests are for specific cases of using a specific function.

Share This | Email this page to a friend

Delphi 2009 includes several new features well suited to a functional style of programming. We’ll explore the use of generics, anonymous methods, and new RTL types while introducing concepts from functional programming.

By the way, comments on this blog should be fixed now; the spam filter had been overly aggressive.

Share This | Email this page to a friend

Last night, the Columbus Architecture Group met to discuss the effects of the economic recession on software vendors, IT departments, and other projects. One area of discussion which I found particularly interesting was the notion of how to build hedges into enterprise or software architecture. In other words, can we examine areas of business risk, and design teams and systems in such a way that they, at least partially, offset this risk?

I’m writing this post more as a question than as a statement of advice, but I will give some examples here in order to clarify what I am asking.

• Last year, my company recognized a need to be in a certain area of the market which we had not previously served. But exactly how to enter this area of the market would depend upon sales needs which were not yet clear. There were two routes we could take, both of which had significant advantages. Taking Fred Brooks’s dictum of "Plan to throw one away; you will anyway" to heart, we started working on both projects, with full knowledge that we would be unlikely to complete both. As anticipated, one of the two projects was eventually left behind. But the other one was better for having undertaken it, and we had mitigated the risk of choosing incorrectly.
• Another participant noted that he had a programmer working on a project which would eventually require to write code to accomplish a certain task. He noticed that another team was working on a similar functional area, although they were not close to completing this work. He asked the first programmer to put off work in this area for another month, which she could easily do, due to the amount of work required in other parts of the project. At the end of the month, they would assess whether the second team had completed enough work to make integration of that team’s code a better idea than writing new code. As it happened, the second team’s project was not far enough along to help the first programmer. However, the cost of rearranging her schedule had been very low, so even though it didn’t provide any actual benefits in the long run, it was still, in hindsight, worth doing.
• Economic conditions which are destructive to one area of a market can be beneficial to another. For example, as companies consolidate due to shrinking markets, there may be an increasing need for software systems integration.

Now I’d like to turn the question over to you: How should architects and developers build hedges into our products and process?

Here’s one response from Jody Dawkins. If you’ve tried to comment in the past couple of days, please forgive the overly aggressive spam filter on this server; it should be fixed now.

Share This | Email this page to a friend

Today I’d like to discuss the "success" of Erlang, how it got there, and what it means.

Is Erlang successful at all? It’s not particularly widely used, there are few books on the subject, and its influence on other languages is debatable. It doesn’t do very well in search-engine-based "metrics," either. On the other hand, one could say that it is "successful" in certain, domain-specific ways. It’s the first example anyone uses of the lightweight-processes-with-message-passing approach to concurrency. It’s proving itself in real-world applications, such as Amazon’s SimpleDB and Ericsson’s phone switches. (If your reaction to that last sentence is, "Of course Ericsson uses it in their phone switches; they invented it," then read the paper, or at least the rest of this post.) The Ericsson AXD301 claims nine nines reliability (99.9999999%). I think that would be a success even if nobody else ever used it.

So the question I’m going to address here is how did Erlang went from a research project to a "successful" tool (in certain domains) with essentially no backing whatsoever, least of all from Ericsson.

Joe Armstrong credits two primary factors for Erlang’s success, neither of which have anything whatsoever to do with Erlang as a language:

• A hardware and software project which Ericsson was developing using C++ collapsed.
• A division at Ericsson banned using Erlang altogether, effectively driving it out of the company.

Although Erlang had proven itself moderately successful in the lab, commercial use of the tool required documentation, libraries, and testing, which, in turn, required more effort to produce than the R&D team could provide. Here’s how Armstrong explains it:

Without the collapse of AXE-N, Erlang would have still remained a Lab experiment and the effort to turn it into a commercial quality product would not have happened. The difference is the many thousands of hours of work that must be done to produce high-quality documentation and to produce and test extensive libraries. AXE-N was a project aimed at developing a new generation of switching products ultimately to replace the AXE10 system. The AXE-N project had developed a new hardware platform and system software that was developed in C++.

Following a series of crisis meetings the project was reorganised and re-started. This time the programming language would be Erlang and hardware from the AXE-N project was salvaged to start the production of a new ATM16 switch, to be called the AXD. This new project was to be the largest-ever Erlang project so far, with over 60 Erlang programmers. At the start of the AXD project, the entire Erlang system was the responsibility of half a dozen people in the Lab. This number was viewed as inadequate to support the needs of a large development project and so plans were immediately enacted to build a product unit, called OTP, to officially support the Erlang system.

This is a really important point, which people have been missing at least since the era when Fred Brooks was writing The Mythical Man Month. Brooks noted that most programmers have had a moment when they write something really cool, and think, "Wow, if I can produce this thing in one week of hard work, why does it take [insert large software company here] two years to produce the functionality I need?" Brooks explained that turning the "really cool" functionality into, say, a piece of an operating system takes many times the effort of writing the functionality in the first place. Adding the library support needed to make the "cool" functionality generally useful (i.e., useful to someone other than the person who wrote it) at least doubles the effort. Sufficiently documenting both the original functionality in the library support you just added doubles it again. Anyone tempted to say, "Just ship the functionality and document later," should consider the reception that greeted the Delphi 2005 and 2006 help systems.

Perhaps you’ve heard about a really cool open source project, and then discovered, when you try to use it, that it was buggy, didn’t have the features required for your application, and had no documentation whatsoever. That’s exactly the same problem at work.

How could an Ericsson division banning Erlang lead to its success? That’s a surprising claim, and the answer is interesting. For some time, the R&D team which developed Erlang had been interested in marketing it to other companies. But they were limited, at first, by the secrecy typical of the era, and later by increasing competition from a more open marketplace:

By 1998, about 40 evaluation systems had been distributed to external users and by now the idea of releasing Erlang subject to an open source license was formed. Recall that at the start of the Erlang era, in 1986, “open source” was unheard of, so in 1986 everything we did was secret. By the end of the era, a significant proportion of the software industry was freely distributing what they would have tried to sell ten years earlier—as was the case with Erlang.

In the end, the ban led to the ability to promote Erlang outside the constraints of Ericsson’s telephone business:

Shortly after the open source release, the majority of the original Erlang development team resigned from Ericsson and started a new company called Bluetail AB with Jane [Walerud] as the chief executive. In retrospect the Erlang ban had the opposite effect and stimulated the long-term growth of Erlang. The ban led indirectly to Open Source Erlang and to the formation of Bluetail. Bluetail led in its turn to the introduction of Erlang into Nortel Networks and to the formation of a small number of Erlang companies in the Stockholm region.

That Armstrong credits these two factors for Erlang’s success is interesting, but can we tie them together into a coherent method for promoting a programming language? Perhaps. Armstrong doesn’t exactly spell it out, but elsewhere in the paper he makes a point which, I think, is the missing link between them:

People are not convinced by theory, only by practice

Share This | Email this page to a friend

I’m going to tell another story which might not seem related at first. But I have a point, I promise.

In the middle of last year, we started a new project based on the ASP.NET MVC framework and the Entity Framework, both of which were essentially beta projects at the time. Despite some (entirely expected) pain from the shifting nature of data frameworks, the project progressed remarkably quickly, and we have begun shipping it to paying customers.

We had planned on using ASP.NET Dynamic Data, but we had to drop that very early in the project. The first thing that you have to do when using Dynamic Data is to call RegisterContext, passing your model, which can be an Entity Framework or LINQ to SQL model. Why only these two frameworks, when both return materialized .NET objects with rich metadata? Good question. What if I don’t want to surface Entity Framework instances in my web application, but prefer to use lightweight data transfer objects instead? What if I don’t want to use the Entity Framework at all?

Worse, the initial release of Dynamic Data cannot handle Entity Framework models containing many-to-many relationships, even though support for these is one of the more significant features that the Entity Framework has over LINQ to SQL. Worse still, Dynamic Data provides no way to scaffold only a subset of your model. I would have been perfectly happy if Dynamic Data just couldn’t provide scaffolding for tables with many-to-many relationships, but could handle other things. Unfortunately, any one thing which Dynamic Data cannot handle in RegisterContext means that you cannot use Dynamic Data at all. So we didn’t.

The non-support for many-to-many relationships will be fixed in a forthcoming version of Dynamic Data, and you can get a preview version now. I expect we’ll give it a try, because the framework has a lot of promise, and portions of it are very nicely done. But part of me wonders which other, entirely legal, things might be lurking in our Entity Framework model which might make Dynamic Data die a speedy death. I don’t see any mention of removing the restriction that Dynamic Data must understand your entire model before it can do anything at all. Even if it does understand our model, I wonder what will happen if I hand it a type which doesn’t happen to be in that model?

This is an area where laziness could be a virtue. Obviously, Dynamic Data needs some sort of metadata model from which to produce its scaffolding. But if it waited to generate this model for types until they were actually needed, it would not die because it didn’t handle something which the application didn’t ever have a need to have scaffolded. Moreover, I think it’s the desire to create a metadata model for every type you might possibly want Dynamic Data support for during the entire lifetime of the application upfront which led to the design decision to support only the Entity Framework and LINQ to SQL. If you create the metadata models on an as-needed basis, using Entity Framework or LINQ to SQL knowledge where possible to enrich the models, then you don’t need a single model in a "known" framework as a starting point.

So the lesson here is that, even within a single-vendor ecosystem such as the standard portions of the Microsoft .NET Framework, dependencies on other projects can be lethal.

Share This | Email this page to a friend

We also try to keep our code relatively small and modular, so a few just install Merb core, which is the core of framework - there are only 6.000 lines of code in there. Just starting up Merb and having it run it’s going automatically takes up less RAM, the critical path will be smaller, we don’t do alias_method_chain, which means that the number of method calls for a regular request is going to be smaller out of Merb.

Now, that’s nice, but it’s even nicer if your preferred ORM doesn’t happen to be ActiveRecord.

I like the ASP.NET MVC framework for the same reason.  It doesn’t supply an ORM because you might not need one, and if you do, there are plenty of existing tools from which to choose.  It doesn’t invent a new templating language.  Though ASP.NET routing was created to support it, routing is no longer part of the ASP.NET MVC framework; it’s now an entirely separate project which other tools can and do use.

By contrast, I have pretty much sworn off / am not even tempted to try certain other frameworks which attempt to do everything required to build an entire application, with strong coupling between the various parts.  Indeed, even though I use the Entity Framework in my ASP.NET MVC applications, these two parts of the application have no knowledge of each other. I’m also incredibly wary of commercial web "components" which don’t play well in ASP.NET MVC, due to dependencies on View State (or worse).  These aren’t web "components;" they are extensions of the WebForms framework, and I’m building web sites, not WebForms sites.

Big frameworks with strong coupling are prisons. Once you go into them, you’re stuck, and it can be incredibly difficult to get out.

Share This | Email this page to a friend

Share This | Email this page to a friend

Facing a state budget crisis in July, California Gov. Arnold Schwarzenegger fired about 10,000 temporary and part-time workers and ordered the 200,000 permanent employees to be paid only the minimum wage of $6.55 an hour until the legislature passed a crisis-solving budget. However, a week later the State Controller John Chiang pointed out that state payroll records could not be changed to accommodate the cut because they were written in the antiquated COBOL computer language, and virtually the only state employees who knew the code were some of the part-timers Schwarzenegger had just fired.

It turns out the whole system is fragile, not just the computers.

Share This | Email this page to a friend

Here’s the trick that I’ll use – and it’s simple to do.
I’ll define a procedure, which I will call Q,
that will use P’s predictions of halting success
to stir up a terrible logical mess.

Do read the whole thing, as it actually does contain a peer-reviewed proof.

Personally, I find rhyme and meter to be underused tools in the fields of mathematics and computer science. I suspect that most programmers don’t read technical journals at all; making the research more entertaining can only help the field. Then again, I’ve been known to cancel spam in verse, so my opinion may be outside of the mainstream.

So anyway, I’m off to adapt Beowulf into a proof in epic verse of the Taniyama–Shimura conjecture.  I’m still dithering over whether to base my adaptation on Seamus Heaney’s translation, or the original, Old English.

Share This | Email this page to a friend

I’ll be speaking at the CodeRage III virtual conference, on the subject of functional programming in Delphi 2009.  Here’s the (rather bland) abstract:

Delphi 2009 includes several new features well-suited to a functional style of programming.  We’ll explore the use of generics, anonymous methods, and new RTL types while introducing concepts from functional programming.

If you’ve been following my blog over the past few months, you might have a fair idea of what to expect.  As you can tell from the somewhat vague abstract, I’m still rounding out the details of the presentation.  If there’s anything you’d like to see me cover, please make your suggestion in comments.

Share This | Email this page to a friend

This speaks to the importance of abstracting data access.  The nice thing about object relational mapping is that once you’ve read the data, your data are just objects, whether or not they’re POCOs. You can and should emphasize this encapsulation by hiding the reading and writing behind a framework such as the Repository pattern.  It is, IMHO, a mistake to ever call something like Context.SaveChanges inside of a controller.

Share This | Email this page to a friend

It is often the case that we use software tools to transform these conceptual models into executable code of one form or another.  For example, a database management tool may know how to transform an ERD into physical database metadata.  A UML design surface might know how to transform a UML model into compilable code.  Most object-relational mappers designed for non-dynamic languages such as C# and Java seem to work by transforming some kind of schema file, usually XML, into compilable code.  And so on…

Inside the application, at runtime, a similar transformation of models occurs.  Let’s examine the pipeline which results in transforming data in a database into HTML which a user can view:

1. The user invokes an action.  This starts a chain of events ultimately resulting in a call to the database server in the form of an SQL statement.  The database server transforms its will on-disk representation of the data into a result set understandable by the data access components.
2. The object-relational mapping transforms the result set into entity instances — instances of classes in a form useful to implementers of business logic.
3. If the application is using the Repository pattern, these entity instances may be passed directly to the controller.  Or they may be transformed into lightweight data access objects.
4. Whatever their form, the entity instances will be wrapped up into a model containing all of the information required to render a view.
5. The view will transform the entities into a different kind of model: HTML.
6. The browser will transform the HTML into a DOM which can be further manipulated via JavaScript.

A similar series of transformations between model types occurs when we accept user input and store it back in the database.

These series of transformations are not so different from the transformations I mentioned above, like transforming a UML model into executable code.  Indeed, compilation itself is just another kind of model transformation, changing a representation which is (hopefully!) readable by a developer into a representation readable by a machine.  If the transformation is lossless, then it can probably be done in both directions.

So one way to look at an application is as a series of transformations from one model type to another, not only in execution, but also in the manner in which the application itself is built. However, the transformations are performed with wildly different implementations:

• A special-purpose executable (e.g., a compiler) can be used to transform the models. Such executables are typically fairly limited; most compilers support only one language, and do not, for example, transform UML models into code.
• We can write code in a high-level language to transform the models, often in a way which is dependent on the specific types or properties involved (e.g, A.Foo = b.Foo), or possibly at a more generic level, such as with a common interface.
• We can create informally-specified principles for model transformation, such as "convention over configuration," which are often sufficient to completely specify the transformation without requiring code.
• We can create systems of rules for model transformation. The formal name for such a system of rules is an algebra, not to be confused with the elementary algebra you learned in high school. This is a set of simple rules designed to be composed into more advanced transformations. One such system is relational algebra, which surfaces (in a heavily mutilated form) in SQL databases.

Now, my purpose in this post is not to claim that one of these methods is superior to the others.  "The best tool for the job" still applies.  The idea of model transformation, however, encompasses more parts of a software system than many programmers commonly suppose.  The notion of treating your source code as a model which can be transformed via an algebra into an equivalent, but faster and provably correct, application is not new, but is perhaps surprising to programmers who have not studied functional programming. So the question is not "How do I create an instance of Bar from an instance of Foo?," but "What kind of transformation is this, and how does it fit into the context of the other transformations I’m performing in this application?"

Such a worldview makes technologies like Haskell’s monads and LINQ, designed as generic solutions for the problem of transforming one type of set into another, even more interesting than they already are.

Share This | Email this page to a friend

Most non-trivial applications have many different models, and for good reason.  If the application uses a relational database, it will have a relational model.  A non-trivial web application may use an application server, containing an entity model with an object-relational mapping, perhaps based on a framework like the Entity Framework or NHibernate.  In some designs, these entity types are not exposed from the application server, but rather represented as lightweight data access objects.  These data access objects are made available to clients of the application server, such as a web application.

Inside the web application, it’s possible that the lightweight data access objects will be made available directly to the View, without being wrapped up inside any other type.  But it’s more likely that additional information will need to be passed to the View, requiring the use of an additional model type, which wraps up the information required by a particular View.  Or, a View might require a different representation of the data access objects, such as XML or JSON.

So let’s count the different types of "Models" I’ve listed so far:

1. Relational data model
2. Entity model, perhaps with an object-relational mapping
3. Lightweight data access objects
4. View-specific models

To be clear, I don’t think the existence of multiple types of models is by itself a problem.  But I do think that the overloading of the term create some confusion about how to follow the MVC application design.

When you create a new ASP.NET MVC web application, Visual Studio creates a Solution containing a single project for the web application itself, with a folder called Models.  Many of the demonstrations of ASP.NET MVC projects that I’ve seen drop a LINQ-to-SQL model inside of this folder.  I understand that people like to keep their demonstrations simple, but as a rule I think that any type of object-relational model should be in a separate assembly from the web application, along with any business logic implementation required.  This second assembly becomes the API for your data which the web application will use.  It can be a web service, or an assembly which exposes methods for accessing the data directly, e.g. with the Repository pattern. However, I keep the Models folder in the web application, and use it for "View models," which handle things like pagination, aggregation of multiple instances required by the view, etc.

Maybe MVC should stand for Models View Controller?

In the second part of this series, I will examine additional types of models, and the general problem of transforming one model into another.

Share This | Email this page to a friend

Share This | Email this page to a friend

So I’d like to note another project which uses the same general concept in a very different manner.  OpenCTF is a free framework which generates DUnit test cases by looking for certain "known" errors in arbitrary components.

Delphi’s relatively limited introspection/RTTI features restrict how far you can go with projects like this.  It’s really only possible to automatically test published properties and published methods with known signatures.  Likewise, most of the Delphi-aware automated testing tools require that you change your application to include special testing interfaces in order to get full functionality from the tool. One of the benefits of enhanced RTTI in the future would be more options for automated testing.

Share This | Email this page to a friend

Memoization is a generic solution to the problem of caching function results. The function Memoize accepts a function as an argument and returns a function which does exactly the same thing, except that it caches the results.  Here’s the prototype:

class function Memoize<A, R>(AFunc: TFunc<A, R>): TFunc<A, R>;

So my version of Memoize will accept any function which takes a single argument of any type, and returns a result of any type. Memoize returns a new function with exactly the same signature. I’ve often written systems to cache the results of database queries in my older Delphi code.  Lately, I’ve been using TClientDataSet to do the same thing, but it still results in several lines of code to accomplish what is essentially a single function call.  Memoize allows me to solve this problem one time and use it for as many situations as I can imagine.

Well, almost any function.  Caching is obviously only useful for functions which are referentially transparent, which is another way of saying that their output is completely determined by their input.  You wouldn’t want a cached version of "Now()" or "Create()", for example. Memoization is most useful when a function performs a lengthy calculation, or queries a database or a web service with a lot of latency.  In order to simulate such a function for the purpose of testing my implementation of Memoize, I’ve written the following test function:

class function TestTMemoize.SlowIncrement(ANum: integer): integer;
begin
  Sleep(100); // simulate web service / db access
  Result := ANum + 1;
end;

This is useless, but it allows me to distribute a testable version of the project without having any dependency on any specific database or Web service.  Feel free to substitute a more useful function if you try this yourself. In order to create a memoized version of this function, I just pass it to Memoize, and call the function that is returned instead of the original one:

procedure DoStuff;
var
  MemoizedFunc: TFunc<integer,integer>;
  iResult: integer;
begin
  MemoizedFunc := TMemoize.Memoize<integer,integer>(SlowIncrement);
  iResult := MemoizedFunc(1);
end;

The Memoize function certainly sounds useful, but how could I write such a thing?  Well, I obviously need generics, or my version of Memoize would be limited to functions which only accepted and returned one specific type.  And I need anonymous methods, because that is the only way that I can return new functions at runtime.  Finally, I need some kind of cache to store pairs of arguments and results, so that successive executions with the same argument can find result in the cache.

Let’s examine the last requirement first.  Delphi 2009’s Generics.Collections unit includes a TDictionary<TKey,TValue> type which, at first glance, it appears to do what I need.  Unfortunately, it isn’t lifetime-managed. So I will just subtype Delphi 2009’s TDictionary<TKey,TValue> and implement a new IDictionary<TKey,TValue> interface I will create.

I will explain why it is important to have a lifetime-managed dictionary in order to implement this function at the end of this post. You can easily understand the rest of the post without grasping this detail, if you’re willing to take my word on the fact that I need a lifetime-managed list. For the time being, let’s just continue with the code. Here’s the full source code for Memoize, which is actually very simple:

class function TMemoize.Memoize<A,R>(AFunc: TFunc<A, R>): TFunc<A, R>;
var
  Map: IDictionary<A, R>;
begin
  Map := TManagedDictionary<A, R>.Create;

  Result := function(arg: A): R
  var
    FuncResult: R;
  begin
    if Map.TryGetValue(arg, FuncResult) then begin
      Exit(FuncResult);
    end;
    FuncResult := AFunc(arg);
    Map.Add(arg, FuncResult);
    Exit(FuncResult);
  end;
end;

In Delphi 2009, you can call Exit with an argument in order to return a certain value from a function immediately, as if you had assigned Result and then called Exit without the argument.  Other than that, I don’t think there is too much to explain about the implementation above.

In order to prove that it works, I wrote a unit test.  The unit test shows that the second execution of a Memoized version of a slow function for the same argument (1) returns the same value and (2) is more than two orders of magnitude faster.  One of the reasons I included this unit test was to demonstrate that unit tests can be used to verify performance goals as well as functional specifications.  In order to profile the function, I’ve "borrowed" Barry Kelly’s useful TBenchmarker class. Here’s the test:

procedure TestTMemoize.TestMemoize;
var
  Cold, Warm: Double;
  ColdResult, WarmResult, AnonResult: integer;
  MemoizedSlowInc: TFunc<integer, integer>;
  BenchmarkProc: TProc;
begin
  MemoizedSlowInc := TMemoize.Memoize<integer,integer>(SlowIncrement);
  BenchmarkProc := procedure()
  begin
    AnonResult := MemoizedSlowInc(1);
  end;

  // first execution should be slow
  Cold := TBenchmarker.Benchmark(BenchmarkProc, 1, 0);
  ColdResult := AnonResult;
  // next time for same value should be fast
  Warm := TBenchmarker.Benchmark(BenchmarkProc, 1, 0);
  WarmResult := AnonResult;

  CheckEquals(ColdResult, WarmResult);
  CheckTrue(Warm < (Cold / 100));
end;

You can download the full source code from CodeCentral.

Finally, here’s the full explanation for why I needed to create a lifetime-managed dictionary:

Anonymous methods, in every implementation I’ve ever seen, can "capture" variables in the declaring method.  This means that you can use variables (or method arguments) declared in the method which defines the anonymous method, without passing them as arguments.  But since we are going to return the anonymous function as the result of the defining function, any object referenced in the anonymous function must not be freed by the defining function.  In Delphi 2009, captured variables are captured by reference (this is similar to C#, but different than JavaScript).  Recall that anonymous methods are reference counted.  When an anonymous method captures a variable in the declaring method, it will be freed when there are no more references to the anonymous method, provided that the captured variable is a lifetime-managed type, such as a string, a record, an interface, etc.

Share This | Email this page to a friend

When you run a search query on CDN, you’ll see just below your search results line of text that looks something like this:

Query processed in 375ms (31.25ms server/network, 343.75ms database)

This text describes exactly how long it took to prepare the query, run it, retrieve all information required for display from the database, check the visibility of every result, sort the results according to the user’s preference, and prepare the set of results in the current page for display.

The times I quoted above came from a run of this query, which searches all indexed sites for C++ source code that contains the word "TClientDataSet". The text is slightly inaccurate: although the total processing time was indeed 375ms, there are other valid methods of counting the other times.

The "server/network" time represents the time it took to send the query to Lucene, get the results from the web service, and convert them to classes and records GetPublished can process. Over 95% of that time is used for transferring the results over the network. The rest is taken by the actual search engine on the Lucene server and the result processing code on the server running GetPublished.

Almost everything else GetPublished does with the results involves the database, so we say the rest of the time belongs to "database". However, for large result sets the Lucene engine processing time - and certainly network transfer time - will be longer. GetPublished actually uses that time to execute additional queries, by running the code in multiple threads.

Here’s the actual processing log from GetPublished for the same query:

Search expression started
Getting list of sites to include in the search
Search includes one or more GetPublished sites
Retrieving external sites
Search includes one or more external sites
Building Lucene query
Lucene query: +appid:(gp blogs blogsteamb cc qc) AND (cpp.source:(+TClientDataSet))
Search sites: 1,5,7,9,10,11,12
Search thread starting
Retrieving language ID
Retrieving site names
Retrieving visible version IDs
Retrieving staged visible version IDs
Retrieving welcome content types for welcome page queries
Welcome content filter:  IN (26,314,322)
Query completed in 31.25ms
The web service returned in 31.25ms
The search engine ran the query in 0ms
Total number of results: 59
Sorting results by AppID and version ID
Adding information for live versions in search results
Adding information for staged versions in search results
Removing versions that are not visible to the user
Updated number of items in search results: 59
Sorting results for welcome page processing
Query ran before retrieving live welcome versions - retrieving now
Live welcome versions:
Language ID: 1
Country ID: 239
Removing welcome pages that are not visible to the user
Final number of items in search results: 59
Parsing dates for sorting
Sorting results
Creating result list
Updating site names for visible results
Creating article headers for display and loading additional information
Search expression processing completed in 375ms

As you can see, GetPublished does a lot more than just run the search query. The reason it does so is that it needs to convert the search results - a list of version records - to a list of articles the user can see. Here are some of the things that affect the visibility of articles and versions:

1. Articles can have multiple versions, but only one version per article can be "live" (that is, visible to users).
2. Articles can be mapped to multiple sites, and have different publishing and expiration dates on each site.
3. GetPublished supports "staging sites", that use a different set of versions.
4. Articles are filtered based on the user’s preferred language.
5. Certain articles may only be visible in specific countries or regions.
6. Articles of a special type, known as a "welcome page", are shown on specific site areas instead of in article lists. On such pages, only one "welcome page" can be visible.

The list of results returned by the search web service is trimmed and expanded based on the search criteria, user preferences, and visibility settings:

1. Versions that are not visible to the user are removed from the list.
2. If an article is mapped to multiple sites that are included in the search, additional result records are created for each site.
3. Welcome pages that are mapped to multiple area are stored once per site, but all valid links are stored in the result record to be displayed later.
4. Of multiple welcome pages mapped to a single area, only the one visible by the user is kept.

Many of these checks can run concurrently. For example, if the search engine takes a while to process the query, GetPublished retrieves all visible articles and welcome pages from the database. If the engine returns quickly, GetPublished only checks the database for versions that are included in the search results. Similarly, GetPublished tries to reduce the working result set as much as possible. For example, when sorting by site name, GetPublished has to set the site name of every result before sorting and paging. When sorting by other fields, GetPublished waits until the results are sorted and only sets the site name for results that are going to be displayed.

Share This | Email this page to a friend

class function TStatistics<T>.Average(
  const AData: TEnumerable<T>;
  AAdder: TBinaryOp<T>;
  ADivider: TBinaryOp<T>;
  AExplicitCast: TFunc<integer, T>): T;
var
  Total, Num: T;
begin
  Total := Sum(AData, AAdder, AExplicitCast);
  Num := Count(AData, AAdder, AExplicitCast);
  Result := ADivider(Total, Num);
end;

class function TStatistics<T>.StandardDeviation(
  const AData: TEnumerable<T>;
  ADeviation, AAdder, ADivider:TBinaryOp<T>;
  ARoot, ASquare: TUnaryOp<T>;
  AExplicitCast: TFunc<integer, T>): T;
var
  Avg, Variance: T;
  SquaredDeviations: TEnumerable<T>;
begin
  Avg := Average(AData, AAdder, ADivider, AExplicitCast);
  SquaredDeviations := Map(
    function(AValue: T): T
    begin
      Result := ASquare(ADeviation(AValue, Avg));
    end,
    AData);
  try
    Variance := Average(SquaredDeviations, AAdder, ADivider, AExplicitCast);
    Result := ARoot(Variance);
  finally
    SquaredDeviations.Free;
  end;
end;

You can download the complete project from CodeCentral.

Even having to use functions like "AAdder" in place of the more conventional "+", I think the algorithm is still fairly easy to read here, at least if you understand the standard deviation.  As Raymond points out in comments to the previous post, however, the syntax may not be familiar to see folks who have spent most of their time in older versions of Delphi.

I’m afraid you’re just going to have to get used to it.  Delphi is well behind the pack in getting support for anonymous methods, and in many environments it is difficult to get much done without them.  JavaScript comes to mind; the asynchronous nature of much of what you have to do in JavaScript makes a continuation-passing style quite practical.  Lambda expressions, which are, very roughly, a more concise version of an anonymous function, are required to do any useful work in LINQ.  In languages like Lisp and Haskell, all "lines" of code are (again, very roughly) functions, which are anonymous by default. Few mainstream languages don’t support this, and many environments leverage it heavily. If you don’t understand such syntax well, you will find it hard to work outside of Delphi, like a COBOL programmer who doesn’t understand OO. You aren’t required to like it, but you have to understand it.

Share This | Email this page to a friend

1. GetPublished indexes the content by calling the search web service.
2. GetPublished stores all data in the database.

The problem is that since we’re updating multiple data stores (the Lucene index and GetPublished’s database), we have no way of controlling the entire process as a single transaction. Writing a distributed transaction manager and extending both GetPublished and Lucene to use it wasn’t really an option, but we needed a way to make sure the database and the Lucene index matched.

When faced with complex design decisions, it help to analyze the risks involved. Here’s my initial risk assessment. It estimates the risk of receiving incorrect search results for index-only and database-only failures (if both operations succeed or fail, the database and index match, and we have no problem).

The risks are based on the following assumptions:

1. Only content in the database is "real" - this is what users see.
2. GetPublished rarely modifies existing content - most edits generate new versions.
3. All search results have to be filtered for visibility by GetPublished.

Based on these assumptions, we can determine that there’s generally little risk in index failures. That risk can be further mitigated by retrying to index later. On the other hand, database-only failures are usually a high risk, so we’d better find a way to avoid them. I therefore based the design on the following requirements:

1. If we were unable to commit changes to the database, we must not update the index.
2. If we were unable to update the index, we should retry later.

The first part is easy: all we have to do is make sure all processing is wrapped in a database transaction. Only if the transaction is successfully committed, can we index the content. By only reading committed data and running the indexing code outside the content processing transaction we can make sure we’re only indexing successfully committed data.

The second part is the real problem. Once data is committed, we have to try to index it. But what if we can’t? We can’t keep trying indefinitely - we need to respond to user requests. So, we need to run the indexing request in another thread, one that can keep trying while we return control to the user.

Background threads are a particular problem in web applications. Web applications usually only live for the duration of a single request. The web server software (IIS, in GetPublished’s case) can - and will - terminate the process, including background thread, at any time. Additional complications arise from GetPublished’s distributed architecture and ASP.NET’s application model. GetPublished is designed to run on multiple servers concurrently (web farm). ASP.NET applications run in application pools, which may be recycled (manually by administrators, or automatically by the server or external monitoring tools), terminating all running threads.

GetPublished solves this problem by implementing a system of background jobs. Instead of arbitrary threads, jobs are special classes that can be tracked and monitored by GetPublished. Since the application runs on multiple servers, and can be restarted at any time, jobs are stored in the database. When GetPublished starts, it creates a job monitoring thread, whose job is to periodically check the database and start job threads as necessary:

Requests that require processing that may take longer than a normal web request can use background jobs. Instead of spawning worker threads, requests can create job records in the database. The job monitoring thread will then pick up the request and spawn a worker thread next time it checks the database. GetPublished even lets administrators monitor, stop, and restart running jobs:

Using background jobs, we can now change the article submission process:

The indexing job is created as part of the same transaction that stores the content. This means the job will only run if the data has been successfully committed. The job calls the web service to index the content, and can safely retry the operation if necessary without blocking the user.

So far, we’ve covered indexing. That’s only half the job of a search engine. Next time, I’ll describe the actual search.

Share This | Email this page to a friend

The previous search engine was based on keywords rather the full-text, and had several limitations:

• The keyword parser did not correctly handle Unicode text, particularly east-Asian languages.
• Lack of a full-text index prevented phrase and proximity searches, and did not allow correct scoring of search results.
• The use of multiple databases by CDN applications prevented unified searches across all web sites.
• It was slow.

Obviously, the engine needed to be replaced. We’ve looked at several search engines in the past, but didn’t find one that matched our needs. We started rewriting our parser, but decided to take another look at the latest version of Lucene - and were pleasantly surprised. Lucene seemed more than capable of handling our needs, and since it is open source, we knew we could tweak it if necessary.

The first problem we needed to solve was how to integrate the Lucene index into our applications. CDN applications are written in several languages (Delphi, Java, C#, and PHP), run on multiple platforms (Native Windows, ASP.NET, and Linux), and use several databases (InterBase, Blackfish SQL, Oracle, and Microsoft SQL Server). In addition, most CDN applications run on multiple load-balanced application servers.

One solution would have been to let Lucene access our databases directly. This would have involved setting up special tables that match the index structure, and maintaining application-specific information on the indexing server. For multiple applications and databases, this can become a serious headache. Instead, we wrapped the search engine in a web service. Each application already controls its content, and notifies the search engine of content changes - indexing new content, re-indexing updated content, and removing deleted content:

Next, we had to decide what constitutes a "document" in the index. A document is a single entry, identified by a unique ID, that can be return as a search result. A single document can have multiple fields, but is included no more than once in search results. Each CDN application has a different concept of a document, which may or may not correspond to what users may consider a single piece of content. Here’s what we came up with:

• For blogs, a single blog post is a document.
• For CodeCentral, both a complete submission and a single source code file within a submission are considered documents. They are indexed using different IDs, so we can search for submissions, source code, or both.
• For GetPublished, a single version of an article is considered a document. GetPublished articles can have multiple versions, and GetPublished uses a complex set of rules to determine actual content visibility on its sites. Instead of indexing all possible visibility combinations, we index each version once, and let GetPublished convert search results into a set of articles visible to the user.
• For QualityCentral, a single report is a document.

GetPublished uses DocAdapter to process submitted files and convert them into HTML articles. Among other things, DocAdapter extracts source code snippets from the submitted document and uses YAPP to automatically syntax-highlight them. Since DocAdapter already knew how to extract source code from an article, it was fairly simple to extend the code to return the original snippets to GetPublished, so they could be indexed by Lucene.

Here is what happens when a user submits an article to GetPublished:

Roughly, the steps are:

1. GetPublished checks the submitted files and form fields and generates the necessary data for the article (for example, a unique article ID for a new article).
2. GetPublished then sends the content to DocAdapter for processing.
3. DocAdapter converts the content to HTML, and extracts all code snippets.
4. Each code snippet is stored, then sent to YAPP for syntax highlighting.
5. DocAdapter merges the syntax-highlighted snippets into the final HMTL.
6. DocAdapter returns the merged HTML, all source code snippets, and any additional required information to GetPublished.
7. GetPublished indexes the content by calling the search web service.
8. GetPublished stores all data in the database.

You’ll notice I painted the line from GetPublished to the web service red. This is because there’s a problem here: the process I just describes only works when there’s no error. If either step 7 (indexing) or step 8 (data storage) fails, we’re left with inconsistent data.

One possible solution would be to switch steps 7 and 8. After we store the data (and commit the transaction) successfully, we can safely call the web service, knowing we’re indexing valid content. This still doesn’t solve the problem of index failure: if the web service call then fails, we’re left with new content that’s not indexed.

Another option is to roll back the entire transaction if step 7 fails. This works in case of an indexing error, but doesn’t help us if indexing succeeded but the database commit failed. Once again, the database and index won’t match.

Since we don’t have distributed transactions with the Lucene index, what we need is a way to tie an indexing request to a successful commit, and the ability to ensure the indexing request succeeds - or at least, notify an administrator if there’s a problem.

In my next post, I’ll describe how GetPublished accomplishes this task.

Share This | Email this page to a friend

This post is the fourth in my series on building a statistics library using Delphi 2009’s new generic types.  If you’re new to the series, you might want to start with the first post.  Today I’m going to discuss the second higher-order function that the library uses internally, Map.

Map is a function which takes a list, and a function which operates on individual items in the list, and returns a new list, formed by executing the function against each entry in the original list.  For example, suppose I had this list of integers:

1, 2, 3, 4, 5

Now let’s say that I call Map, passing this list, and the function Inc(). The result would be this list:

2, 3, 4, 5, 6

Don’t confuse Map with functions like jQuery’s each which merely iterate the list and call a passed function without returning a new list.  That’s a different kind of function.

To calculate the standard deviation of a list, I first need to create a list of the squares of the deviations of each entry in the list. Later, I will perform additional calculations to compute the standard deviation, but first I need this list.  Here’s how I do it, using Map:

  SquaredDeviations := Map(function(AValue: T): T
    begin
      Result := ASquare(ADeviation(AValue, Avg));
    end,
    AData);
    // [...]

The anonymous function I pass as an argument to Map computes the deviation of the individual item it is passed, squares it, and returns a value.  This anonymous function is called once for each item in the original list. And here is the implementation of Map itself:

class function TStatistics<T>.Map(
  AFunc: TUnaryOp<T>;
  const AData: TEnumerable<T>): TEnumerable<T>;
var
  Current: T;
  NewList: TList<T>;
begin
  NewList := TList<T>.Create;
  for Current in AData do begin
    NewList.Add(AFunc(Current));
  end;
  Result := NewList;
end;

Note that I have Map returning a new instance.  It is behaving as a constructor of sorts.  Indeed, it might make sense to have a constructor of this style on a list type. You could also write a version of Map which accepted an entity list to use as the result if it makes your lifetime management easier.

Share This | Email this page to a friend