I just finished another post at the Cozi Tech Blog, SerializationException: the constructor was not found

I mentioned three weeks ago that I had just repaved my work dev box and installed the 64-bit version of the Windows 7 RC. Nine or ten years after I first ported parts of IIS to Win64, I am finally running my main desktop on 64-bit Windows. With one exception, it's been painless. Programs have just worked, devices have just worked. There are relatively few native x64 applications, but for the most part it doesn't matter. The cases where it does matter—e.g., shell extensions such as TortoiseSVN—are available as 64-bit binaries.

I briefly flirted with using the 64-bit build of Python, but realized that I would have to recompile several eggs as 64-bit binaries. That was too painful and the 32-bit binary did everything I needed.

Building in Visual Studio 2005 is noticeably faster. I'm not sure how much of it was due to accumulated cruft after 18 months on Vista, but builds there were very slow.

The one exception was a major problem for the first week and a half. Whenever I ran our ASP.NET web application, it would go berserk, eat up all 4GB of my physical RAM, push the working set of IIS's w3wp.exe to 12GB, and max out one of my 4 cores! The only way to maintain any sanity was to run iisreset every 20 minutes to gently kill the process.

WinDbg and Process Explorer showed that the rogue thread was stuck in a loop in mscorjit!LifetimesListInteriorBlocksHelperIterative. I passed a minidump on to my former colleagues in IIS, who sent it to the CLR team. They said:

The only thing I can tell is that it is Regex, and some regex expression compiled down to a method with 456KB of IL. That is huge, and yes 12GB of RAM consumed for something like that is expected.

With that clue, I was able to track down the problem, a particularly foul regex, built from a 10KB string, with 32 alternating expressions, each of which contains dozens of alternated subexpressions. The string is built from many smaller strings, so it's not obvious in the source just how ugly it is. I commented out the new Regex() and my problems went away.

Regardless of how ugly the regex is, this is a major regression in the CLR. This code has been working without blatant problems for two years on the 32-bit flavors of XP, Server 2003, Vista, and Server 2008. I've been meaning to try this code on 32-bit Windows 7, but have been too busy.

(The original, long-gone author was apparently aware that the regex is expensive to create because he runs a background thread to new the regex, which should have told him something. We'll fix the code that uses the regex to do something saner, soon.)

All that aside, I've been happy with the 64-bit version of Windows 7.

In the late 90s when I worked on the classic Active Server Pages dev team, I tried to convince one of the Program Managers that we should make regular updates to browscap.ini, the file that described browser capabilities. He wanted no part of it.

I was pleasantly surprised to learn via Hanselminutes that Microsoft has stepped up to its responsibilities and is now shipping the Mobile Device Browser File on CodePlex. Over 400 mobile devices are defined, with 67 distinct capabilities.

The Hanselminutes podcast is an interesting discussion of the Mobile Web and designing a different experience for mobile browsers. There's more to it than the small screen. You want to think about the scenarios in which the site is likely to be used. The user is probably traveling: give directions. Make phone numbers dialable by using the tel: scheme. Think about server round trips.

Miscellaneous links.

• Do you see the subliminal arrow in the negative space of the FedEx logo at right? Neither did I until I read about it at Edward Tufte's joint. Now I can't stop seeing it.

• One more year to go until the next presidential inauguration on 2009/01/20. Who knows how much more damage Bush can pull off by then? StickerGiant.com has commemorative swag.

• Impeachment is in the air. Watch Rep. Wexler's speech before the U.S. House of Representatives. Read about State Sen. Oemig's hearing in the Washington Legislature.

• Three days ago, I was sent email by DraftBloomberg.com, asking me to sign a petition to draft Mike Bloomberg as an independent candidate for President. I promptly wrote back, refusing on the grounds that (a) I view Bloomberg as a Naderesque spoiler who's likely to take votes from the Democratic nominee, and (b) I find Bloomberg to be an uncompelling candidate who just happens to be rich enough to self-finance. Looking at their site a few minutes ago, I see that they've only managed to scrape up 1,522 signatures, which is pathetic.

• Ron Paul enjoys an improbable level of support on the Internet, raising staggering amounts of money by appealing to the libertarian bloc. But there's compelling evidence that Paul is a Bircherite not a libertarian, with lucrative ties to white supremacists going back more than 20 years.

• Harold Meyerson argues that we are entering a recession and the old remedies won't do, because the US economy is no longer fundamentally sound.

  Wages have been flat-lining for a long time now, the housing bubble isn't going to be reinflated anytime soon, and the upward pressure on oil prices is only going to mount. As in Roosevelt's time, we need a policy that boosts incomes and finds new solutions for our energy needs.

  Scholars & Rogues argue that getting out of Iraq can fund the necessary changes to get us out of a recession.

• Although I'm generally willing to believe the worst of the Bush administration, I've never found the 9/11 conspiracy theories to be plausible. Matt Taibbi debunks 9/11 conspiracy theories to my satisfaction.

• On a positive note, the .NET Source Code is now available. You can debug through the source of the Microsoft libraries, when you need to. Visual Studio 2008 only.

My first project at Cozi is to build a simple REST-style Web Service. Nobody here has done that before.

The first thing that I'm trying to get going is a simple URL rewriter, using an ASP.NET HttpModule.

I'm running Vista as my development desktop for the first time. So far, not bad, but there are lots of new quirks to get used to. I've been a good boy so far and I've left the User Access Control stuff enabled, so that I'm not running with administrative privileges by default.

It's my first exposure to IIS 7. I must say that the IIS UI is much improved (a low bar to surmount).

My first problem was that Skype was squatting on port 80, preventing browser requests going to localhost. This happens to me about once a year on a new dev machine, and I always forget.

To get the HttpModule going, I had to follow Mark Rasmussen's detailed instructions on making URL rewriting on IIS 7 work like IIS 6. The code will be deployed on Windows Server 2003, so IIS 6 compatibility is more important to me than IIS 7 purity.

I was trying to get some debug output appearing in DebugView, but my Trace.WriteLines were not showing up. Some Googling eventually showed me that I had to enable Capture Global Win32, which I never had to do before. Presumably because ASP.NET is executing in a different desktop session.

I had a NameValueCollection embedded inside a larger object. I needed to serialize the larger object into XML and back. Unfortunately, NameValueCollection is not XML serializable. Why I do not know.

A blog comment from Tim Erwin got me started in the right direction. Implement IXmlSerializable and do the work by hand in ReadXml and WriteXml.

Tim's implementation turned out to be overly simple. It didn't handle an empty collection well, nor did it leave the XmlReader in a good state.

I used SGen to examine the deserialization of a List<String> to figure out what else needed to be done.

The following ReadXml seems to work. If I expected to receive XML from untrusted sources, I would make this more robust.

 public void ReadXml(XmlReader reader)
 {
     if (reader.IsEmptyElement)
         return;

     while (reader.Read()
         && reader.NodeType != XmlNodeType.EndElement
         && reader.NodeType != XmlNodeType.None)
     {
         if (reader.NodeType == XmlNodeType.Element && reader.LocalName == "Header")
         {
             reader.MoveToAttribute("name");
             string name = reader.Value;
             reader.MoveToAttribute("value");
             string value = reader.Value;
             Add(name, value);
         }
     }
     reader.ReadEndElement();
 }

 public void WriteXml(XmlWriter writer)
 {
    foreach (string name in nvc.Keys)
    {
        writer.WriteStartElement("Header");
        string value = nvc[name];
        writer.WriteAttributeString("name",  name);
        writer.WriteAttributeString("value", value);
        writer.WriteEndElement();
    }
 }

 public XmlSchema GetSchema( )
 {
    return null;
 }

I also found that I needed to implement custom Equals and GetHashCode, as the NameValueCollection implementations didn't seem to do what I wanted.

 // Have to override GetHashCode() as two apparently identical NameValueCollections
 // will have different hash codes.
 public override int GetHashCode()
 {
     int hash = nvc.Count;

     foreach (string name in nvc)
     {
         hash = 757 * hash  +  101 * nvc[name].GetHashCode()  +  name.GetHashCode();
     }

     return hash;
 }

 public bool Equals(HeadersCollection that)
 {
     if (ReferenceEquals(that, null))
         return false;

     if (ReferenceEquals(this, that))
         return true;

     // Have to explicitly compare the contents of the collections
     // as NameValueCollection.Equals doesn't seem to do what we want.
     // Note: this is independent of order.
     if (nvc.Count != that.nvc.Count)
         return false;

     foreach (string name in nvc)
     {
         if (nvc[name] != that.nvc.Get(name))
             return false;
     }

     return true;
 }

 public static bool Equals(HeadersCollection headersA, HeadersCollection headersB)
 {
     if (headersA == null)
         return (headersB == null);

     if (ReferenceEquals(headersA, headersB))
         return true;

     return headersA.Equals(headersB);
 }

 public override bool Equals(object obj)
 {
     if (obj is HeadersCollection)
         return Equals((HeadersCollection) obj);

     return false;
 }

Over the last few days, I've been adapting an existing native C++ library so that it can be called from managed code. I had written a large number of unit tests with CppUnit and I wanted to be able to call the tests from NUnit.

I suppose that I could have written a new CppUnit TestRunner so that I could call it from NUnit. Instead, I took the cheap-n-dirty route, playing with #define and include paths. It took less time to get working than it did to write this blog post.

Here's the original native CppUnit test code

 //-------------------------------
 // native\FooTest.h
 //-------------------------------

 #include <cppunit/extensions/HelperMacros.h>

 class FooTest : public CppUnit::TestFixture
 {
     CPPUNIT_TEST_SUITE( FooTest );
     CPPUNIT_TEST( testAlpha );
     CPPUNIT_TEST_SUITE_END();
 public:
     void testAlpha();
 };

 //-------------------------------
 // native\FooTest.cpp
 //-------------------------------

 #include "FooTest.h"

 // Registers the fixture into the test 'registry'
 CPPUNIT_TEST_SUITE_REGISTRATION( FooTest );

 void FooTest::testAlpha()
 {
     CPPUNIT_ASSERT( 4 == 2 + 2);
 }

And here's my managed NUnit-based wrapper.

 //-------------------------------
 // managed\FooTest.h
 //-------------------------------

 using namespace NUnit::Framework;

 // Gross hack. Define a completely different NUnit-compatible FooTest
 // test fixture and use #define's to make the CPPUnit-specific
 // stuff build.

 [TestFixture]
 public ref class FooTest
 {
 public:
     [Test] void testAlpha();
 };

 #define CPPUNIT_TEST_SUITE_REGISTRATION(x)
 #define CPPUNIT_ASSERT(x) Assert::IsTrue(x)

I had to make one change to native\FooTest.cpp, to #include <FooTest.h> (angle brackets). This picks up the first FooTest.h in the include path, so that the managed version of FooTest.cpp now picks up managed\FooTest.h, instead of the original.

In last week's tip on using the NVelocity template formatting engine, I described what to set to load a template from an absolute path.

Here's the magic necessary to get NVelocity to load a template from an embedded resource:

 VelocityEngine engine = new VelocityEngine();
 ExtendedProperties properties = new ExtendedProperties();
 properties.AddProperty("resource.loader", "assembly");
 properties.AddProperty("assembly.resource.loader.class",
     "NVelocity.Runtime.Resource.Loader.AssemblyResourceLoader, NVelocity");
 properties.AddProperty("assembly.resource.loader.assembly", "StencilFormatter");
 engine.Init(properties);

We've started using the NVelocity template formatting engine. We were absolutely stymied for an hour, trying to figure out how to get it working with an absolute path to the template file, instead of the relative path shown in the documentation.

The trick is to set file.resource.loader.path. Here's how to load C:\foo\bar\somefile.vm:

 ExtendedProperties props = new ExtendedProperties();
 props.AddProperty("file.resource.loader.path", new ArrayList(new string[]{".", "C:\\"}));
 velocity.Init(props);

 template = velocity.GetTemplate("foo\\bar\\somefile.vm");

I ran into a problem installing some COM+ components today. The installer was using Regsvcs.exe to register each COM+ component. I noticed after a while that the installer wasn't making any progress and that my dual-proc system was stuck at 50% CPU utilization. I attached a debugger to the offending process, regsvcs, and found that it was stuck in the following infinite loop (disassembly courtesy of Reflector):

 internal void System.EnterpriseServices.CatalogSync.Wait()
 {
   if (this._set)
   {
     RegistryKey key1
       = Registry.LocalMachine.OpenSubKey("SOFTWARE\\Classes\\CLSID");
     while (true)
     {
       int num1 = (int) key1.GetValue("CLBVersion", 0);
       if (num1 != this._version)
       {
         break;
       }
       Thread.Sleep(0);
     }
     this._set = false;
   }
 }

There are two severe problems with this code.

1. The loop should time out. There must be some reasonable limit after which you can incontrovertibly say that something must have gone wrong, and throw an exception. There has to be some way to terminate a loop.

2. Never use Sleep(0) in a loop. Sleep(0) yields the processor only if there's a runnable thread. If there isn't, Sleep(0) will return immediately. If the code is sitting in a tight loop, the net effect is that it will maximize the CPU until the thread's quantum is exhausted. There are no other runnable threads, so the scheduler immediately starts this thread again. This code will run until your CPU burns out.

(And, yes, I have committed both of these sins in shipping code. Why do you ask?)

I don't know what the calling code is doing or why CLBVersion isn't being altered by some other thread or process. I had to use RegEdit to modify this value to get the loop to terminate, whereupon RegSvcs immediately did its work and terminated. And then it started all over again, with the next invocation of RegSvcs on another COM+ component. I don't know if the components are really installed properly. I had to leave at that point.

My colleague, Greg, and I spent all day debugging a build break in some unit tests that exercise a webservice interface in legacy .NET 1.1 code. Last night, the tests stopped working on our CruiseControl.NET build server. We couldn't understand it. The tests had been working for months. Now we were getting timeouts in SOAP. The tests essentially mock a SOAP service using the soap.inproc transport and a stub implementation that signaled an event to acknowledge a method being called.

The only thing that had changed in the code tree was that another colleague, Pavel, had discovered that two of our .csproj files somehow shared the same GUID, and had repaired that. But that could hardly have any effect on the WSE2 runtime. Could it?

Turns out that it was the cause of the break. NAnt 0.85 rc2 and rc3 silently failed to build the NUnit assembly because of the duplicated GUIDs. The assembly was not getting propagated to the directory where all the other NUnit assemblies are placed. The CC.NET task that ran the tests never noticed the missing assembly because the test was couched in terms of *.NUnit.dll. And we never noticed that the test hadn't been run in months because we have ~20 such NUnit assemblies, and the NUnit summary output goes on for several screens in CC.NET.

Morals of the story

1. Use NAnt 0.85 rc4, which detects the GUID collision and treats it as a fatal error.

2. Create .csproj files through the IDE, not by taking an existing file and hacking on it. (At least, that's we assume happened.)

3. Assumptions can bite you. We assumed that the code was being run all along, so it took us several hours to draw the connection between Pavel's checkin and the failing NUnit assembly.

4. Don't mock a webservice by implementing a dummy SoapReceiver, hauling in the WSE runtime and a boatload of non-determinism. (Instead, make fun of its dress sense.) For our newer code, we've been taking an approach like this, using partial classes and Rhino Mocks.

5. We have also taken to including our test fixtures in the same assemblies as the code they test. I have mixed feelings about this: it offends my sensibilities to have all this test code compiled into production code. But it would certainly have been hard to miss the build break in production code.

I spent far too much time on Friday trying to make log4net work in a COM+ application.

Someone else had done part of the work necessary, by creating an application.config for the COM+ application and setting a custom Application Root Directory. This was enough to ensure that most of the managed code in the application got their configuration settings; log4net being the

It took some additional work to realize that we needed to add two assembly attributes:

[assembly: log4net.Config.Repository("unique-name")]
[assembly: log4net.Config.XmlConfigurator(ConfigFile="application.config")]

The repository name just needs to be a unique string. We used the name of the assembly.

I needed to add some declarative error checking to some XSLT templates recently. Specifically, I wanted to throw an error if my selects yielded an empty string, indicating that the input XML was wrong.

Unfortunately, there seems to be no easy way of doing this in XSLT, nor in XslTransform. The approved way is to validate against an XSD schema, but for various reasons, I didn't want to go to the hassle of creating one.

I found a partial solution using xsl:message with the terminate="yes" attribute. Under XslTransform.Transform() the following code throws an exception if the XPath expression is empty.

 <xsl:if test="not(/some/xpath/expression)">
     <xsl:message terminate="yes">Missing expression</xsl:message>
 </xsl:if>
 <xsl:value-of select="/some/xpath/expression" />

It doesn't do anything, however, in XMLSpy.

The downside, of course, is that you have to maintain the expression in two places, and the template becomes littered with those annoying tests.

I consider myself to be an expert WinDbg user, when it comes to debugging unmanaged x86 code. I haven't used WinDbg much on managed code, and when I did, I found it quite painful.

Via Scott Guthrie's blog, I discovered Tess Ferrandez's blog. Tess is an escalation engineer in PSS who specializes in ASP.NET and a WinDbg virtuoso. Scott has a list of her best posts. Mind-blowing stuff, but not for the faint of heart.