Friday, September 11, 2009 
Win64

I've been running the 64-bit version of Windows 7 RC since June. It's been quite painless on the whole.

One wrinkle that I ran into was with some batchfiles which launch applications in %ProgramFiles% (normally C:\Program Files). Due to the magic WOW64 redirector, 32-bit applications are actually installed into %ProgramFiles(x86)%—normally C:\Program Files (x86)—instead of %ProgramFiles%. This is transparent to the 32-bit applications, which think they're running in %ProgramFiles% (C:\Program Files).

However, the cmd.exe shell is 64-bit (unless you make a special effort to run the 32-bit cmd.exe in SysWOW64), so batchfiles see the 64-bit %ProgramFiles% which contains 64-bit applications.

Hence, a batchfile that launches an installed 32-bit application on Win64 must use %ProgramFiles(x86)%, not %ProgramFiles%.

It sounds trivial to have a batchfile detect which flavor of %ProgramFiles% it should use, but the parentheses in the environment variable name make it tricky to parse. On earlier versions of Win64, the environment variable was called %ProgramFilesx86%. Presumably they added the strange parentheses into the variable name because the directory name always included them.

Here's a tiny batchfile that will launch the 32-bit DiffMerge correctly on both Win64 and Win32 platforms.

@setlocal
@set _pf=%ProgramFiles%
@if not "[%ProgramFiles(x86)%]"=="[]" set _pf=%ProgramFiles(x86)%
@start "" /b "%_pf%\SourceGear\DiffMerge\DiffMerge.exe" %*

I long ago found that the safest way to test environment variables whose values may include spaces, is to surround them with both double quotes and square brackets.

posted on Saturday, September 12, 2009 6:06:58 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [1]
Friday, July 10, 2009 
Humongous JIT memory leak

I mentioned three weeks ago that I had just repaved my work dev box and installed the 64-bit version of the Windows 7 RC. Nine or ten years after I first ported parts of IIS to Win64, I am finally running my main desktop on 64-bit Windows. With one exception, it's been painless. Programs have just worked, devices have just worked. There are relatively few native x64 applications, but for the most part it doesn't matter. The cases where it does matter—e.g., shell extensions such as TortoiseSVN—are available as 64-bit binaries.

I briefly flirted with using the 64-bit build of Python, but realized that I would have to recompile several eggs as 64-bit binaries. That was too painful and the 32-bit binary did everything I needed.

Building in Visual Studio 2005 is noticeably faster. I'm not sure how much of it was due to accumulated cruft after 18 months on Vista, but builds there were very slow.

The one exception was a major problem for the first week and a half. Whenever I ran our ASP.NET web application, it would go berserk, eat up all 4GB of my physical RAM, push the working set of IIS's w3wp.exe to 12GB, and max out one of my 4 cores! The only way to maintain any sanity was to run iisreset every 20 minutes to gently kill the process.

WinDbg and Process Explorer showed that the rogue thread was stuck in a loop in mscorjit!LifetimesListInteriorBlocksHelperIterative. I passed a minidump on to my former colleagues in IIS, who sent it to the CLR team. They said:

The only thing I can tell is that it is Regex, and some regex expression compiled down to a method with 456KB of IL. That is huge, and yes 12GB of RAM consumed for something like that is expected.

With that clue, I was able to track down the problem, a particularly foul regex, built from a 10KB string, with 32 alternating expressions, each of which contains dozens of alternated subexpressions. The string is built from many smaller strings, so it's not obvious in the source just how ugly it is. I commented out the new Regex() and my problems went away.

Regardless of how ugly the regex is, this is a major regression in the CLR. This code has been working without blatant problems for two years on the 32-bit flavors of XP, Server 2003, Vista, and Server 2008. I've been meaning to try this code on 32-bit Windows 7, but have been too busy.

(The original, long-gone author was apparently aware that the regex is expensive to create because he runs a background thread to new the regex, which should have told him something. We'll fix the code that uses the regex to do something saner, soon.)

All that aside, I've been happy with the 64-bit version of Windows 7.

posted on Saturday, July 11, 2009 6:12:02 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]
Friday, June 19, 2009 
Blown Capacitors

I complained a week ago about my display driver going berserk. I blamed Windows Update, since it happened within hours of a pile of updates being installed. I upgraded to the latest beta NVidia drivers on Monday and it helped for a while, but by Wednesday, it was almost as bad again as it had been last Friday. It was infuriating and I was both entertaining and alarming my neighbors with my cursing.

Today was the last day of a very busy sprint for me and at last I had the time to dig into it. I opened up the case and took a look at both video cards—I have two dual-head cards connected to three monitors—and one of them had partially blown capacitors like those in the picture. I removed the bad card and did some graphics-intensive things for an hour, and the other card behaved flawlessly.

Oddly, until someone mentioned that it might be a hardware problem yesterday, it didn't occur to me, even though a video card blew in this machine last year. I came in one morning to find a black monitor, and when I pulled out that card, I found that some of the capacitors had popped right open with stuffing protruding.

On general principles, I had been meaning to repave this machine for a while. I've had it since December 2007 and it was still running the original installation of Vista. I booted from a DVD, reformatted my C: drive, and installed Windows 7 x64 RC1.

I finally have a 64-bit OS as my primary Windows desktop, so I'll actually be using the Win64 build of Vim that I maintain. My first impressions of Windows 7 on this machine are very favorable, but there's plenty more that I need to install before the machine has everything that I need.

posted on Saturday, June 20, 2009 4:40:37 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]
Thursday, May 21, 2009 
Vim

I updated the Win64 binaries of Vim at vim-win3264 from Vim 7.2.000 to 7.2.182.

I'm amazed that the original binaries were downloaded over 11,000 times since last August.

posted on Friday, May 22, 2009 6:10:38 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]
Thursday, May 07, 2009 
Windows 7 x64 running in Mac VirtualBox 2.2.2

I ported Vim to Win64 but I don't have a convenient Win64 system to test it on.

I decided to install the Win64 build of the Windows 7 RC on VirtualBox, which has supported 64-bit guest operating systems since version 2.0.

It worked without problems on my MacBook Pro. I used VirtualBox's Virtual Media Manager to mount the Windows 7 ISO and installed from that. See also the handy guide. (Why does Windows 7 offer a choice of upgrading from a previous version of Windows on a virgin disk?) After completing the installation of the operating system, I installed the Guest Additions for mouse pointer integration and other goodies.

As always with VirtualBox VMs on my MacBook, I had to fix the Network settings to work over WiFi. When the VM is turned off, go to Settings, choose the Network tab. Change “Attached to” from “NAT” to “Bridged Adapter” and “Name” from “en0: Ethernet” to “en1: AirPort”. Tip: to get a right-click without a mouse, place two fingers on the trackpad and click the trackpad button, or Shift+F10.

I tried installing the Win64 build of Win 7 on my Win32 Vista desktop box at work. The host system bluescreened within seconds of starting the installer! I filed ticket 3963.

I had inadvertently installed the Win32 build first on my work system. That worked fine. It also seemed to have snappy disk I/O. When I unzipped the Win64 Vim binaries (not having realized yet that I had the Win32 Win 7), it was slower than unzipping in the host operating system, but not unreasonably so. On my MacBook, the details pane from the Win 7 zip extractor said that it was running at a mere 260KB per second, which is pitiful. It certainly wasn't that slow when installing the OS onto the virtual disk.

posted on Thursday, May 07, 2009 7:01:14 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]
Saturday, November 29, 2008 
Reach

Perusing Eric S. Raymond's blog recently, I noticed his claim that as a one-time maintainer of GIFLIB, just about every cellphone and browser has some of his software running in it.

That got me thinking about my own reach and where software that I've contributed to can be found.

‘Oh that a man's reach should exceed his grasp, or what's a Heaven for?’

—Robert Browning

I spent seven years on the IIS (Internet Information Services) development team at Microsoft. By any measure, that's a successful product, running one-third of all websites. There are over 100 million registered websites. Many of them are parked and many others see negligible volume, but that's millions, perhaps tens of millions of Windows Server boxes.

Two of those years were spent working on http.sys, the kernel-mode driver that underpins IIS 6. Http.sys was back-ported to Windows XP and released as part of XP SP2 (though IIS 6 never was), and it's an integral part of all later versions of Windows. That's hundreds of millions of Windows boxes.

And then there's Vim. I wrote much of the Win32-specific code–and all the Win64-specific code–but I also made contributions to the core code. Vim has long been the standard implementation of vi in most Linux distributions. And Vim is installed on all recent versions of Mac OS X.

So, I can claim Windows and Linux and Mac–though few cellphones. Not too shabby.

(I'm also one of a small number of people thanked in ESR's Jargon File; in my case, for TeX arcana and painstaking proofreading.)

posted on Saturday, November 29, 2008 9:07:26 AM (Pacific Standard Time, UTC-08:00) 
#    Comments [0]
Sunday, August 17, 2008 
Vim

Fifteen months after the release of Vim 7.1, Bram announced the release of Vim 7.2 last weekend. No major new features, just the consolidation of more than 300 patches. He also included a mention of the new distribution point for Win64 binaries, the vim-win3264 project that I set up at Google Code.

Bram has no way of testing the Win64 version, so I'm providing the official build at vim-win3264. I will no longer provide Win64 binaries for Vim from my own site. The Vim 7.2 sources compile the Win64 binaries cleanly (unlike the 7.1 release). I'll provide occasional intermediate releases up there too, for both Win32 and Win64.

I'm rather surprised to see that the Win64 binary has already been downloaded 248 times in the last week. It's such a pain to look at the logs on my server that I have no idea how many times earlier binaries were downloaded.

posted on Sunday, August 17, 2008 8:17:02 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]
Tuesday, October 23, 2007 

content/binary/virtualization.jpg

Picture this.

An external USB hard drive plugged in to a machine running Win64. The OS has virtualized the underlying transport so that it's essentially indistinguishable from an internal IDE, SCSi, or SATA drive. Call the machine, Boss, and the USB drive, L:.

Boss is running Virtual PC, which is hosting a 32-bit virtual machine on top of Boss's 64-bit OS. Let's call the 32-bit VM, Sidekick.

Sidekick is not only a VM, but a virtual network host. Boss is bridging connections to Sidekick, and Sidekick and Boss both appear on the LAN as separate network hosts.

The USB drive has several ISO images, which Sidekick wants to use. Sidekick connects to \\Master\L$ over the virtual network, and uses a tool like VcdTool to mount the remote ISO on a virtual CD drive.

Amazingly enough, it all just worked for me last night.

I'm trying to set up an environment where I can build Vim with various 32-bit and 64-bit Microsoft compilers and, more importantly, run the Win64 binary. I have a set of VM images with distinct flavors of MSVC, which was necessary to update INSTALLpc.txt and to keep Make_mvc.mak building.

In previous iterations, I got Remote Desktop access to a colleague's Win64 machine, but that was at Atlas, so it's no longer an option. I bought a new AMD64 desktop system a few months ago and over the weekend set it up to dual boot.

posted on Wednesday, October 24, 2007 3:57:53 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]
Monday, February 19, 2007 

http://www.vim.org/images/vim_header.gif

I have updated the Win64 port of Vim. It now includes a working installer, a working "Edit with Vim" shell extension, and the first 195 patches for Vim 7.0. Get it while it's hot!

posted on Tuesday, February 20, 2007 7:49:10 AM (Pacific Standard Time, UTC-08:00) 
#    Comments [1]
Saturday, February 10, 2007 

http://www.georgevreilly.com/blog/content/binary/R6034.png

I have been cleaning up some issues with the Win64 port of Vim, including the Edit with Vim shell extension not working very well. When I built the shell extension with VS 2005 on x86, I would get the following whenever I right-clicked in Explorer:

Microsoft Visual C++ Runtime Library

Runtime Error!

Program: C:\WINDOWS\Explorer.EXE

R6034

An application has made an attempt to load the C runtime library incorrectly. Please contact the application's support team for more information.

There was no mention of which application was at fault, though it was obvious in this case. I have also seen some mention of verclsid in the error dialog, though not when I took this snapshot.

The underlying problem relates to SxS, Fusion, and all that good stuff. By far the simplest fix was for me to statically link with libcmt.lib, instead of msvcrt.lib, rather than figure out the necessary manifest magic.

posted on Sunday, February 11, 2007 3:37:26 AM (Pacific Standard Time, UTC-08:00) 
#    Comments [0]
Sunday, February 04, 2007 

content/binary/laptop-woes.jpg

My laptop scared the crap out of me last night. I came home to find it in a completely unresponsive state: it would not wake up. The hard disk LED was a solid green. I power cycled it and it refused to boot.

It did, however, boot from a Kubuntu Edgy CD, but it did not recognize the hard disk. In desperation, I booted into the BIOS and played with the disk-related menus. That fixed the problem, but I don't know what went wrong, and my faith is shaken in the reliability of this system.

I bought the laptop just over three years ago, shortly before I quit Microsoft, as a replacement for the work laptop that I had been using. It's served me well. I have a reasonably beefy desktop system of the same age, but I almost always use the laptop instead. It's a Compaq Presario X1012QV, with a 1.3GHz Centrino, a WXGA screen, 35GB hard disk, and 1280MB RAM. It had 512MB RAM originally, but I replaced one of the 256MB sticks with a 1GB stick last year, making it more pleasant to use.

For several months now, I've been planning to buy a new Vista-ready laptop this spring, with a Core 2 Duo, ~100GB disk, and 2+GB RAM. I want a 64-bit CPU so that I can occasionally run Win64; e.g., to update the Win64 port of Vim. I'm severely tempted by Apple and expect to end up with some kind of Mac laptop — my first ever Mac. I was hoping to hold out until Mac OSX 10.5 (Leopard) comes out sometime this spring, but the latest rumors say that it's "edging the very limit of the definition of 'Spring' — i.e. mid-June". If the Presario craps out on me again, I'll replace it in short order.

Whether I go with Mac or stick with a PC, I'll continue to run multiple OSes. Kubuntu Linux has been my primary operating system since last June, and I think it's unlikely that Vista will replace it. OSX may well do so.

posted on Sunday, February 04, 2007 9:19:32 AM (Pacific Standard Time, UTC-08:00) 
#    Comments [3]
Wednesday, May 10, 2006 

More than two years in the making, Vim 7.0 is finally out!

(Vim is Vi IMproved, an enormously enhanced version of the classic Unix editor, vi.)

The main features of the 7.0 release are:

  • Spell checking

  • Omni-completion (Intellisense-like)

  • Tabbed pages

  • VimL script language now supports Lists and Dictionaries

I'm going to take credit for some minor features of Vim 7:

WikiPedia summarizes the history of Vim. This enabled me to pinpoint when I first became a contributor to Vim, back in December 1995. I cleaned up the original, rather buggy port of Vim 3.0 to NT, and posted it to the comp.editors newsgroup. Bram invited me to merge my changes into Vim 4.0, which was then under development, and I became the owner of Win32 Vim for the next couple of years.

posted on Thursday, May 11, 2006 6:42:59 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]
Saturday, April 22, 2006 

I've ported Vim to Win64. Native binaries for AMD64 can be found on my Vim page.

In the end, it wasn't all that hard. Last weekend, I fixed approximately 400 warnings that were thrown up by the x86_amd64 cross compiler. Most of them were due to the widening of size_t (especially the value returned from strlen()) and ptrdiff_t to 64 bits. Several years ago, I went through a similar exercise in fixing these warnings for Vim6, but I never finished the port.

This week, I scrounged access to an AMD64 box at work. Today, I turned on the /Wp64 flag, which found several new, subtler problems, where pointers where being truncated to __int32s or conversely __int32s were being widened to pointers. Judicious introduction of (the equivalent of) (INT_PTR) casts fixed most of those.

At that point, I tried running the binary. It refused to start! After a few detours, I had WinDbg installed, and ran gvim under WinDbg. That showed that the error was 14001 (ERROR_SXS_CANT_GEN_ACTCTX, "The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail.") The event log showed nothing.

After more investigation, I found a WinSxS manifest for the Windows Common Controls:

 

processorArchitecture="X86"
version="6.2.0.0"
type="win32"
name="Vim"
/>
Vi Improved - A Text Editor


type="win32"
name="Microsoft.Windows.Common-Controls"
version="6.0.0.0"
publicKeyToken="6595b64144ccf1df"
language="*"
processorArchitecture="X86"
/>



Once the two instances of processorArchitecture="X86" were set to processorArchitecture="AMD64", Vim started working without a hitch. Despite my naïve expectations, none of the other fields in the comctl32 assembly needed to be changed.

posted on Saturday, April 22, 2006 7:55:29 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [1]
Tuesday, March 21, 2006 

I often complain about being busy, no doubt because I have a talent for complicating my life. Things were relatively quiet for a while, but that's not true anymore.

At work, we're close to releasing the first version of our product. Happily, crunch time at Atlas isn't nearly as bad as it was at Microsoft. Instead of working eight-ish hours a day, it's more like nine or maybe ten. The pressure level has risen, of course, but it's far from intolerable.

The real busyness is in my extracurricular life. I'm the president of BiNet Seattle, a bisexual community group, and have been for the last three years. I also do a hell of a lot of the work and I'm burning out. I recently gave notice that I'm stepping down. (It looks like a successor has been found.) Meanwhile, a lot of planning is going on in an effort to revitalize BiNet, as attendance has been dragging.

For the last few years, I've also been heavily involved with The Wild Geese Players of Seattle, as the webmaster and the co-dramaturge. We do readings of Irish literature, particularly that of James Joyce and W.B. Yeats. Every June 16th (Bloomsday), we do a staged reading of a chapter of Ulysses. This year, the longtime director has moved back to Northern Ireland. Currently, I am acting as the director, on top of my other roles, but I don't think I'm the right person for the job, and I'm hoping to find a replacement soon.

I'm a member of Freely Speaking Toastmasters, an LGBT speaking club. I've been working on my CTM for far too long, and I intend to knock off the final three speeches this year.

I resume my woodworking class next week, which is going to tie up ten Tuesday evenings. I haven't decided yet what I'm going to work on this time. In previous years, I built a very nice set of nesting tables and an unsatisfactory pair of bar stools.

In my Copious Spare Time, I'm also making occasional contributions to two open source projects, DasBlog and Vim. I made Vim compile with VC5-VC8, and I promised Bram that I would provide some documentation on debugging Vim with WinDbg and dealing with minidumps. I'd also like to produce a native Win64 version. With DasBlog, I've provided some feedback on the usability of the installation instructions, as well as a fix for dodgy permalinks. I'd also like to make use of my former expertise on IIS performance (see 25+ Tips, 10 Commandments, IIS 5 Tuning, and Professional ASP 3.0) to do some performance tuning of DasBlog.

I'd also like to fit in some time for photography; for reading my way through our enormous backlog of books and magazines; writing the occasional blog post; cooking; bicycle riding; traveling; working out; hanging out with my wife; socializing with my friends; movies; and more. Not to mention all the very dull projects around the house and garden that I've neglected.

posted on Tuesday, March 21, 2006 7:36:59 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [1]
Thursday, June 02, 2005 

Printf Tricks

It may be old-fashioned, but I still find printf (and sprintf and _vsnprintf) incredibly useful, both for printing debug output and for generating formatted strings.

Here are a few lesser-known formats that I use again and again. See MSDN for the full reference.

%04x - 4-digit hex number with leading zeroes

A quick review of some of the basics.

%x prints an int in hexadecimal.

%4x prints a hex int, right-justified to 4 places. If it's less than 4 digits, it's preceded by spaces. If it's more than 4 digits, you get the full number.

%04x prints a hex int, right-justified to 4 places. If it's less than 4 digits, it's preceded by zeroes. If it's more than 4 digits, you get the full number, but no leading zeroes.

Similarly, %d prints a signed int in decimal, and %u prints an unsigned int in decimal.

Not so similarly, %c prints a character and %s prints a string. For wide (Unicode) strings, prefix with l (ell, or w): %lc and %ls.

Note: For the Unicode variants, such as wprintf and friends, %c and %s print wide strings. To force a narrow string, no matter which variant, use the %h size prefix, and to force a wide string, use the %l size prefix; e.g., %hs and %lc.

%p - pointer

The wrong way to print a pointer is to use %x. The right way is to use %p. It's portable to Win64, as well as to all other operating systems.

Everyone should know this one, but many don't.

%I64d, %I64u, %I64x - 64-bit integers

To print 64-bit numbers (__int64), use the I64 size prefix.

%Iu, %Id, %Ix - ULONG_PTR

ULONG_PTR, LONG_PTR, and DWORD_PTR are numeric types that are as wide as a pointer. In other words, they map to ULONG, LONG, and DWORD respectively on Win32, and ULONGLONG, LONGLONG, and ULONGLONG on Win64.

The I size prefix (capital-i, not lowercase-L) is what you need to print *LONG_PTR on Win32 and Win64.

%*d - runtime width specifier

If you want to calculate the width of a field at runtime, you can use %*. This says the next argument is the width, followed by whatever type you want to print.

For example, the following can be used to print a tree:

 void Tree::Print(Node* pNode, int level)
{
if (NULL != pNode)
{
Print(pNode->Left, level+1);
printf("%*d%s\n", 2 * level, pNode->Key);
Print(pNode->Right, level+1);
}
}

%.*s - print a substring

With a variable precision, you can print a substring, or print a non-NUL-terminated string, if you know its length. printf("%.*s\n", sublen, str) prints the first sublen characters of str.

[2005/7/19: fixed a typo in previous sentence (%.s -> %.*s). A little elaboration on the syntax: . in a printf format specification is followed by the precision. For strings, the precision specificies how many characters will be printed. A precision of * indicates that the precision is the next argument on the stack. If the precision is zero, then nothing is printed. If a string has a precision specification, its length is ignored.]

%.0d - print nothing for zero

I've occasionally found it useful to suppress output when a number is zero, and %.0d is the way to do it. (If you attempt to print a non-zero number with this zero-precision specifier, it will be printed.) Similarly, %.0s swallows a string.

%#x - print a leading 0x

If you want printf to automatically generate 0x before hex numbers, use %#x instead of %x.

Other tricks

See the documentation for other useful tricks.

Security

Never use an inputted string as the format argument: printf(str). Instead, use printf("%s", str). The former is a stack smasher waiting to happen.

%n is dangerous and disabled by default in VS2005.

Don't use sprintf. Use the counted version, _snprintf or _vsnprintf instead. Better still, use the StrSafe.h functions, StringCchPrintf and StringCchVPrintf, to guarantee that your strings are NUL-terminated.

[Update: 2008/01/25: See also Printf %n.]

posted on Thursday, June 02, 2005 7:44:26 AM (Pacific Daylight Time, UTC-07:00) 
#    Comments [0]