Columbus Give Camp
If you’re a geek in the Central Ohio area, please consider volunteering for Columbus Give Camp, if your schedule permits.
Share This | Email this page to a friend
Tagged columbus, givecamp, ohioIf you’re a geek in the Central Ohio area, please consider volunteering for Columbus Give Camp, if your schedule permits.
Share This | Email this page to a friend
Tagged columbus, givecamp, ohioCalling IQueryable<T>.Skip(0) seems like it should be "free." In other words, since it will have no effect on the resulting data, there should be little to no performance cost for calling it. But this is demonstrably not true in LINQ to Entities, and it occurs to me that LINQ providers are not required to optimize it away. Therefore, it is probably a good idea to avoid making such a call at all, so that you do not have to concern yourself with whether the provider will generate a slower query if you do it.
Unfortunately, it is not just your own code you need to be concerned with. It turns out that some fairly popular LINQ helpers do this. For example, Rob Connery’s PagedList<T> type does it:
public PagedList(IQueryable<T> source, int index, int pageSize) { this.TotalCount = source.Count(); this.PageSize = pageSize; this.PageIndex = index; this.AddRange(source.Skip((index - 1) * pageSize).Take(pageSize).ToList());
Index, in this code, is the 1-based page you’d like to display. When index is 1, the code calls Skip(0), and you get less efficient SQL. Fixing the problem is quite simple:
this.PageIndex = index; var pageData = index > 1 ? source.Skip((index - 1) * pageSize).Take(pageSize) : source.Take(pageSize); this.AddRange(pageData.ToList());
Naturally, this only optimizes SQL generation for the first page of results, but since this is by far the most common page a user might display, it can net you a substantial win in server performance.
Troy Goode’s updated PagedList type, which I have recommended in the past (and still do, with this tweak!), has the same problem. The solution is almost identical, although Troy’s type is 0-based rather than 1-based. Update: Troy has now posted an update containing this fix, among many other changes.
The root of the problem in the SQL generation is that when you call Skip(0), LINQ to Entities, in at least some cases, generates SQL like this:
) AS [Project3]
WHERE [Project3].[row_number] > 0
ORDER BY [Project3].[TimeRecordDate] DESC
The reference to row_number can have a substantial performance impact. In one case I tried with a SQL Server table containing just over 1 million rows, a simple query ran with sub-second performance without this WHERE clause, and over 4 seconds with the WHERE clause, even though the result set was exactly the same.
Now, is this a bug in LINQ to Entities? Or LINQ itself? It’s hard to say. Both frameworks are doing exactly what you tell them to, even though it doesn’t make a lot of sense. It seems like a missed opportunity for optimization, but I’m open to the possibility that there might be some desirable effect of this that I have simply missed. For my own purposes, I am content to simply not call Skip(0) at all.
Share This | Email this page to a friend
Tagged entity framework, linq, PagedListIn the most recent episode of Hanselminutes, Scott chats with founders of Mustang Software (creators of Wildcat! BBS) Jim Harrer and Scott Hunter about the BBS era. Wildcat! was written in Turbo Pascal, and Harrer and Hunter state that it was one of the first major applications developed in that language. In the first part of the show, they discuss going to California to meet with the Turbo Pascal team, visiting Anders Hejlsberg’s house, and more early history.
It’s worth a listen if you’re interested in the history of the product which started Borland and evolved into Delphi.
Share This | Email this page to a friend
Tagged Anders Hejlsberg, BBS, Hanselminutes, Jim Harrer, Scott Hunter, Turbo PascalI’m going to demonstrate how to use the open-source DayPilot Lite calendar control in an ASP.NET MVC application. I will discuss the capabilities of the control and consider the general problem of how to use controls designed for "plain" ASP.NET in MVC applications. I’m including a demo solution which you can download and run yourself.
Download the demo solution: DayPilot ASP.NET MVC demo
We needed a calendar control for an ASP.NET MVC application. In particular, we needed a control which would show a number of events and when they occur within a particular day, "Outlook-style." I’ll include a screenshot of the demo application here, since it’s the best way to explain the type of control we were looking for:
In my experience, the type of control which works best with ASP.NET MVC is a control which is completely ignorant of ASP.NET, and all the lies such as ViewState and postbacks which accompany it. In particular, I’ve had a good deal of success with controls designed for the jQuery JavaScript framework, which is included, by default, with ASP.NET MVC applications. Unfortunately, I was unable to find a jQuery control which data what we needed. The closest I found was jMonthCalendar which cannot display a single day in the manner illustrated above. I would still prefer a jQuery-based solution if one ever turns up.
I did, however, find a couple of controls designed for "plain" ASP.NET which, from an end-user point of view, it exactly what we needed. One was the DayPilot control illustrated above, the other was the scheduler component included with Telerik’s control suite. Telerik’s control claims some support for MVC, but this is really limited to loading calendar data asynchronously, which was not actually what we wanted to do. Also, Telerik’s control can only be purchased as part of their suite, which is rather expensive if you only need the calendar/scheduler control. Paying for the entire suite would not be a problem for us if it was filled with controls we needed, designed for the environment (MVC) we use. While Telerik seems serious about supporting MVC, they are in the early stages right now, so it is difficult to justify paying for a large suite in order to get a single control, which I would have to use in a mostly-unsupported way. The Telerik control has many more features than the DayPilot control, and, in my opinion, looks a little nicer.
The DayPilot calendar is available in two different editions. There is an open source edition, which is free, and a non-open-source edition, which adds features. The principal difference is that the date version includes editing (via postbacks) and UpdatePanel support, which are not really useful in an ASP.NET MVC application anyway. As with Telerik’s control, I have to adapt the control for use within ASP.NET MVC. So we selected the open source version, even though the paid version is priced reasonably.
In order to use the DayPilot control in ASP.NET MVC, I wanted to use markup which would look familiar to ASP.NET MVC developers:
<%= Html.DayPilot(Model, new DayPilotViewOptions { HourHeight = 30 }) %>
This means that I needed to write an Html helper which will render the control based on the data passed in Model and return the literal HTML markup. In order to make that easier, I created two new types, DayPilotData and DayPilotViewOptions, in order to encapsulate access to the properties of the control. The reason I created two different types was to separate the concerns of data for the events (which will be set up in the controller) and display of the control (which is the concern of the view). The types are trivial, so I’m not going to show them in this post, but you can examine them in the demo solution. There is an overload to this Html helper which does not include the options argument in case you are happy with the defaults.
public static string DayPilot( this HtmlHelper helper, DayPilotData model, DayPilotViewOptions options) { var calendar = new DayPilotCalendar(); if (model != null) { model.CopyTo(calendar); } if (options != null) { options.CopyTo(calendar); } var sb = new System.Text.StringBuilder(); sb.Append("<div class=\"dayPilot\">"); // allows working around td cellpadding bug in css using (var sw = new System.IO.StringWriter(sb)) { using (var tw = new HtmlTextWriter(sw)) { calendar.RenderControl(tw); } } sb.Append("</div>"); return sb.ToString(); }
Note the div. The current version of DayPilot astonishingly, as of this writing, does not put a class anywhere in its rendered markup. Since the stylesheet for the default ASP.NET MVC site conflicts with DayPilot, to some degree, I wrapped the whole control in a div so that I could fix the layout in CSS.
Another way which I could have used the control in ASP.NET MVC would be to register a tag prefix and use code behind/code generation, as with standard ASP.NET. I’ve registered a tag prefix in the demo solution in case you want to try that yourself. Be aware, however, that setting up codebehind and code generation files in an MVC application is somewhat less than simple; doing it right requires, among other things, manual editing of the csproj file. So I do recommend using the control in the way I’ve illustrated.
Finally, here’s the source code for the action which supplies data for the control. Since this is a demo solution, I’m generating random events:
public ActionResult Index() { var r = new Random(); var q = from i in Enumerable.Range(1, 4) let start = DateTime.Today.AddHours(r.Next(47)) select new DayPilotDataItem { Id = i, Start = start, End = start.AddHours(1), Text = string.Format("Event {0}", i) }; var model = new DayPilotData { StartDate = DateTime.Today, Days = 2, Data = q }; return View(model); }
Of course, it’s just as easy to select data from a database.
The current version of DayPilot has a kind of a weird limitation: You cannot make it smaller, in the vertical dimension, than 30 pixels per hour. Based on some inspection of the source code, I think this is fixable if you are comfortable altering the source. But be aware of the limitation.
There are more DayPilot features which I have not covered, but that should be enough to get you on your way. If you have any questions, post them in comments, and I’ll see whdat I can do to help out.
Share This | Email this page to a friend
Tagged asp.net mvc, calendar, DayPilot, scheduler, TelerikIf you spend enough time with the PagedList class that I’ve been using for paging in the method which supplies data to jqGrid, it’s a near-certainty that sooner or later you will see a LINQ error with Count in the call stack. The error may seem confusing, because it has nothing to do with Count. Commentor Graeme has been experimenting with my demo solution, and has run into just this issue:
Finding your blogs very helpful - superbly written - just found out about jqgrid last night and have integrated your code into my project.
One thing that didn’t work for me though was the dates fix above…
adding
.ToString(System.Globalization.CultureInfo.CurrentUICulture)onto the end of my
DateTimevalues makes everything render Ok but when I try doing a search, it fails. Debugging shows the error isMethod ‘
System.String ToString(System.IFormatProvider)’ has no supported translation to SQL.at the
TotalItemCount = source.Count();line of the PagedList.cs file.Any ideas?
In order to understand this error, we need to remember several important things about LINQ:
Where, Take, OrderBy, etc. does not actually execute the query. The query will only be executed when you call a LINQ method which requires execution to produce its results — like Count.Expression representing that method. If the LINQ provider happens to be LINQ to Objects, the method will eventually be invoked when the query is executed. But if it happens to be LINQ to SQL or LINQ to Entities, the method may never be invoked, and instead might be replaced (if possible) with equivalent SQL.ToString method, which is generally supported in LINQ to Entities/SQL, but has a special overload on the DateTime type which is not supported. Worse still, method calls may be supported in some contexts but not others. Graeme reports that LINQ to SQL accepted the ToString overload in the context of a Select, but not in the context of a Where. This makes sense from an implementation point of view, but means programmers have to be extraordinarily careful about what we include in our LINQ queries, especially when they are constructed dynamically.By now it should be obvious what is going on here: The particular overload of ToString which I used in my example is supported in LINQ to Objects but not in LINQ to SQL in a Where expression. My demo solution used a repository based on LINQ to Objects, so the query I showed worked correctly there. But it does not work with LINQ to SQL. Because LINQ is lazy, you will not actually see an error about this until the query is translated into SQL by the LINQ provider, and that does not happen until the call to Count in PagedList.
This means that when you write a LINQ query expression using LINQ to SQL, LINQ to Entities, etc., you are essentially working in a partially dynamic language. Unless you study the documentation very, very carefully, you will not know for certain until runtime if your query will be accepted by the provider.
So how could we work around this problem?
AsEnumerable. Obviously, we would want to apply the paging and ordering first. In this method, we would execute one query (using LINQ to SQL or something) which would retrieve an ordered list of records for the specified page, using AsEnumerable to transform the results into a List. We would then execute a LINQ to Objects query against this list in order to perform the formatting. This will actually work OK, although it can end up being a good bit of code to write for what should be a fairly simple task.Remember, the original query worked just fine. It is only the serialization to JavaScript which caused unexpected results. One way to fix this is to hide the members which are problematic to serialize and replace them with a string version of same:
private class JavaScriptDateFixer { [NonSerialized] internal DateTime _date; public int Id { get; set; } public int IntProperty { get; set; } public string StringProperty { get; set; } public string DateProperty { get { return _date.ToString(System.Globalization.CultureInfo.CurrentUICulture); } } } public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord) { var repository = new Repository(); var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord) select new JavaScriptDateFixer { Id = entity.Id, IntProperty = entity.IntProperty, StringProperty = entity.StringProperty, _date = entity.DateProperty }; return Json(model.ToJqGridData(page, rows, null, search, new[] { "IntProperty", "StringProperty", "DateProperty" })); }
Note that even though this code looks quite different, this does almost exactly the same thing as the code I’ve replaced. The only real difference is that I’ve delayed the call to ToString until after the query is executed. That’s enough to keep LINQ to SQL or LINQ to Entities happy.
So with this fix is everything fine? Maybe. I really like LINQ; I think it’s an extraordinarily powerful tool. But here we have hit upon what I consider to be the single biggest problem with LINQ. The combination of very limited support for all possible query expressions in common LINQ providers like LINQ to SQL and building queries dynamically can be very dangerous. Unless you test all possible query permutations, you cannot be completely confident that any query you build will actually execute successfully. In typical dynamic language programming, you mitigate this problem by writing unit tests. But unit tests are not supposed to connect to a database. Unfortunately, the limitation here is that the point of translation to SQL. So any unit test which mocked the LINQ provider, say, using LINQ to Objects or some other in-memory representation would not actually encounter the limitations we are trying to test.
I have yet to see a really good solution for unit testing LINQ to SQL or LINQ to Entities, one which would combine the desired attributes of not actually connecting to the database while surfacing any potential problems in SQL generation. I suspect that such a beast may never exist for LINQ to SQL, but for LINQ to Entities it seems at least possible. One could write an Entity Framework provider which returns mocked instances of entities. Any LINQ syntax not understood by the Entity Framework would be caught before the mock provider was ever invoked. This is a feature I would like to see in a future version of the Entity Framework.
Share This | Email this page to a friend
Tagged asp.net mvc, entity framework, jqGrid, jQuery, linq, linq to sql, unit testingThe jQuery :nth-child selector has an "index" argument which is 1-based. But the jQuery "index" method, which is arguably the converse of the selector, returns a 0-based result. This is very easy to work around once you’re aware of it, but it’s surprising in a library which is, generally, designed both well and consistently.
Share This | Email this page to a friend
Tagged jQuery indexThis is the fifth post in a series on using jqGrid with ASP.NET MVC. Today, we’re going to begin examining the grid’s editing features by implementing deletes. If you’re new to the series, you might want to start at the beginning.
The delete feature of jqGrid is, oddly, almost entirely undocumented, even though there quite a few examples of different methods of editing, and inserts are documented, to a lesser extent. But it does exist; I just had to read the source code to figure out how it works.
JqGrid has three different methods of editing data. The cell edit feature allows the user to edit a single cell the grid at a time, and saves the data when the cell loses focus. The row edit feature works similarly, but does not save the data until the row loses focus. The form edit feature shows a modal dialog instead of editing the values inline. Of these three, the form edit feature is the only one which supports deletes. However, you can still use one of the other "modes" for editing, if you like. Just be aware that when you do a delete, you will be in the form edit code. Also, deletes won’t work if you change the grid’s configuration to disable the form edit feature.
In order to allow the user to delete a row, I need to:
I can accomplish the first two tasks by making a slight change to the JavaScript that configures the grid:
}).navGrid(pager, { edit: false, add: false, del: true, search: false }, {}, {}, {url: "Delete"});
I’ve highlighted the parts I’ve changed in boldface. Changing the "del" property to true turns on the toolbutton. The fifth argument is an object specifying options for the delete action, of which I have only specified the URL. This is important: The only places you can specify the URL for delete are in the arguments to the delGridRow method or in the toolbar’s configuration. There is not a "delete URL" property of the grid itself.
If you don’t specify a delete URL in this way, the grid will presume that you want to send delete requests to the URL you have specified for editing rows. There will be an "oper” value in the submitted form indicating that the request is a delete rather than an edit. But I prefer to use a separate URL for delete, because I think that fits more naturally in the ASP.NET MVC paradigm.
Now let’s write an action to handle the delete:
[AcceptVerbs(HttpVerbs.Post)] public ActionResult Delete(int id) { var repository = new Repository(); var deleted = repository.Delete(id); if (!deleted) { Response.StatusCode = 500; return Content("Record not found."); } return Json(true); }
Again, I’ll note that I’m using a mock repository for this demo so that you can compile and run the solution without needing to set up a database. Since the mock repository is based on IList, the Delete method of the repository returns a Boolean indicating whether or not the requested record was found, since that is how IList behaves. It would probably be more correct to return a partial view in the event of an error rather than specifying the contents directly, but error handling is a subject I hope to cover in more depth later on. For the time being, simply changing the StatusCode will tell the grid when something does not work.
If the delete was successful, I return "true." By default, the grid ignores returned data when the operation is successful. However, you can handle the grid’s afterSubmit event, which will be passed any data you do return. So what you choose to return, and what you do with it, is entirely up to you.
With all that in place, deletes work as expected. I’m not going to update the demo solution just yet, because I’ll be covering edits and inserts real soon now. But first I need to revisit formatting and explain more about LINQ".
Share This | Email this page to a friend
Tagged asp.net mvc, jqGrid, jQueryThis is the fourth post in a series on using jqGrid with ASP.NET MVC. Today, we’re going to examine search and formatting results. I will implement a search feature and fix the problem with formatting date columns which I observed in the last post. I’ve updated the demo solution with these new features, and also fixed a bug reported by last time. If you’re new to the series, you might want to start at the beginning.
First, let’s fix the date display. The reason the date appeared so oddly is that there is no official support for Date types in JSON, even though JavaScript has a date type. Microsoft, therefore, invented their own way of encoding dates for their JSON serializer, which jqGrid does not understand. There are two possible ways to fix the problem. The first is to keep sending the data to the grid in the Microsoft format, and write a custom formatter for the grid to parse that into a human-readable string. The second is to change the data we send to the grid. Both ways work, but I’m going to demonstrate the second method, mostly because I find C#’s date handling features superior to (and less confusing than) JavaScript’s.
In order to change the date "column" in the results of the query to a formatted string, we simply need to project into an anonymous type. However, this would break the sort, because a text sort on a formatted date will not be in the correct order. Hence, it is necessary to apply the sort before projecting. So I add an OrderBy to the query, and pass null as the orderBy argument to the ToJqGridData method.
public ActionResult GridDemoData(int page, int rows, string search, string sidx, string sord) { var repository = new Repository(); var model = from entity in repository.SelectAll().OrderBy(sidx + " " + sord) select new { Id = entity.Id, IntProperty = entity.IntProperty, StringProperty = entity.StringProperty, DateProperty = entity.DateProperty.ToString( System.Globalization.CultureInfo.CurrentUICulture) }; return Json(model.ToJqGridData(page, rows, null, search, new[] { "IntProperty", "StringProperty", "DateProperty" })); }
Note that in this revised solution I’ve updated the mock repository to return more sensible dates.
Important update: This code works correctly with LINQ to Objects, as with my demo repository, but will not work with LINQ to SQL. I have addressed this issue in a subsequent post.
Now let’s add a search feature. One of the problems in making a "demo" search is that search is a highly application-specific feature, both in terms of implementation and in terms of user experience. JqGrid’s search feature presumes that you want a search which looks something like this:
Visually, there are lots of variations on this theme available, but in terms of the use case they all presume that the user will specify which fields she would like to search. However, I prefer a "Google-style" interface with just a search box and an implementation that can (hopefully) figure out what the user actually wants. So that’s what I’m going to demonstrate. The needs of your application may be different. The important thing to take away from this demonstration is that the jqGrid search feature can be co-opted to fit your design.
I want the search box to appear above the grid at all times, rather than requiring the user to press a toolbutton to see it. So first I need to add an empty div to the markup in order to indicate where the search box will be positioned, relative to the grid:
<h2>GridDemo</h2> <div id="search"></div> <table id="grid" class="scroll" cellpadding="0" cellspacing="0"></table> <div id="pager" class="scroll" style="text-align:center;"></div>
Next, I add a small bit of JavaScript to tell jqGrid to replace this div with a search control:
search.filterGrid("#" + grid.attr("id"), { gridModel: false, filterModel: [{ label: 'Search', name: 'search', stype: 'text' }] });
The filterModel tells the grid to show a search control for one column, called "Search." Note that the entities I’m returning do not have a property called "search." That’s OK. When the user types something in the search box and presses enter, the grid will add a query string parameter to its request for the data of the form "search=WhatTheUserTyped". That’s what I want, and the fact that there is no "search" column in the data does not adversely affect the grid in any way.
The first argument to filterGrid is a little odd. I’m required to pass a jQuery selector as a string rather than being able to pass a jQuery object directly. Since this is all inside a method which accepts jQuery objects as arguments (rather than hard-coding the IDs of the DOM objects in the method, for unit-testability), I have to "decode" the jQuery object into a selector string. But note that I did not need to do that when setting up the pager or the grid itself. Like I said before, the grid’s API is consistently inconsistent.
As you can see in the code at the beginning of this post, the GridDemoData action just passes the search query string parameter through to the ToJqGridData method unchanged. Inside that method, I use Dynamic LINQ to alter the IQueryable to implement the search. I’ve cleaned up this method substantially since I first posted the code. Here’s the revised method:
/// <summary> /// Adds a Where to a Queryable list of entity instances. In other words, filter the list /// based on the search parameters passed. /// </summary> /// <typeparam name="T">Entity type contained within the list</typeparam> /// <param name="baseList">Unfiltered list</param> /// <param name="searchQuery">Whatever the user typed into the search box</param> /// <param name="searchColumns">List of entity properties which should be included in the /// search. If any property in an entity instance begins with the search query, it will /// be included in the result.</param> /// <returns>Filtered list. Note that the query will not actually be executed until the /// IQueryable is enumerated.</returns> private static IQueryable<T> ListAddSearchQuery<T>( IQueryable<T> baseList, string searchQuery, IEnumerable<string> searchColumns) { if ((String.IsNullOrEmpty(searchQuery)) | (searchColumns == null)) return baseList; const string strpredicateFormat = "{0}.ToString().StartsWith(@0)"; var searchExpression = new System.Text.StringBuilder(); string orPart = String.Empty; foreach (string column in searchColumns) { searchExpression.Append(orPart); searchExpression.AppendFormat(strpredicateFormat, column, searchQuery); orPart = " OR "; } var filteredList = baseList.Where(searchExpression.ToString(), searchQuery); return filteredList; }
Importantly, this method is simply adding to the expression represented by the IQueryable. It does not actually implement the search. So any search methodology which understands LINQ can deal with its results. At least, theoretically. In practice, LINQ providers vary greatly in terms of which LINQ features they support. So although this method works fine with LINQ to Objects (as used in the rest of the demo solution) I cannot guarantee that it will work with every LINQ provider in the world. You have to try, and revise it to fit the capabilities of the provider you’re using.
Putting all these pieces together, I now have a search feature which does what I want:

Be sure to download the updated solution in order to get the bug fixes and new features.
As a special, no-extra-charge bonus, the updated solution also includes a demonstration of using the grid’s tree view mode with client-side data from a JavaScript function. This has nothing to do with ASP.NET MVC; it’s just an example of how to use a grid feature which I wrote in response to a request in the jqGrid support forums.
That’s all for today. In the next post in the series, I’ll begin to demonstrate the grid’s editing features, and how to use them in an ASP.NET MVC application. But I will also take requests. If there’s a grid feature you’d like to see demonstrated query question regarding using the grid in ASP.NET MVC, please feel free to make a request in comments, and I will answer it as best I can.
Share This | Email this page to a friend
Tagged .NET, asp.net mvc, C#, javascript, jqGrid, jQueryMy new article, A Crash Course In Failure, has just been published on the architecture site NPlus1. In it, I examine the surprisingly persuasive argument that unplugging entire racks of live, production servers is not only a good idea, but that shutting them down any other way is a mistake, and what this means for how we design software.
Share This | Email this page to a friend
Tagged architecture, error handling, failure, nplus1, system administrationI have a couple of short updates on topics I’ve covered recently.
On The Podcast at Delphi.org, Nick Hodges confirms my earlier speculation about reorganization in the "CodeGear" group at Embarcadero. The whole show is worth listening to. Congratulations are due to Chris Pattinson, who was promoted to Director of Quality for all of Embarcadero. Time to update the subtitle of your blog, Chris! (And maybe even write a post or two…) Sounds like good changes all around. You read it here first, folks!
I wrote a short demo showing how to use jqGrid in tree grid node with client-side data supplied by a JavaScript function.
Share This | Email this page to a friend
Tagged Chris Pattinson, Delphi, jqGrid, Nick HodgesBad Behavior has blocked 1846 access attempts in the last 7 days.