Beta1 –> Beta2 How Hard Can That Be?

Turned out it was really hard.

We’re using Windows Installer for installing our workstation software. So far using that has been relatively simple and easy. There’s nice support for creating Windows Installer projects in the Visual Studio 2005 and we can build the installers using Visual Studio command-line tools. So our automated building system also produces Windows Installer packages.

Of course there are some problems not yet solved properly – like when the the files to be included in the installation package are moved, removed or added. All these require manually modifying the installer project. Also we’ve not found a way to make the same project produce installation packages with debug binaries (for testing) and release binaries (for release, duh). That could be done by compiling the binaries twice and generating the installer between compilation rounds but so far the pain is not hard enough for anyone make it happen. So we maintain two Windows Installer projects – one including debug binaries and other one including the release binaries.

There have been some plans to explore WiX for easing the automated creation of the installer packages but priorities for this are too low for it to happen.

Also it’s a pity that Visual Studio only supports a subset of Windows Installer features, basic things like service installation, grouping files in features and components and custom installation dialogs are missing.

So we knew there are limitations and inconveniences but as said there wasn’t enough pain for anything to change.

This Monday we started preparing for upgrading our existing beta users to the latest and greatest beta-version of our software. It was supposed to be an easy task as documented so clearly in MSDN. Just increment the product version property and do a reinstallation re-caching the local MSI package.

This Monday the problems started. We had no issues upgrading from previous build to next one. Or from beta1 build to the build following that. But for some odd reason all upgrades from beta1 to the latest builds were failing silently. Everything seemed to go well there were no errors reported doing the upgrade – just no files were updated on the target systems! The local MSI package was cached but that was good for nothing.

It took one and a half days until someone started to suspect that renaming one file in the package might cause the problems. This was found by trial and error: upgrade beta1 with later builds until found the one which is not working. Due to the previously mentioned limitations in MSI package authoring we can’t just double-click an MSI package to try the installation (and it wouldn’t work fora minor upgrade anyway), we need to pass a special, dynamically crafted set of command-line parameters to MsiExec. There’s also some other kind of magic required to stop some components not working upgrade-friendly in the beta1 installations. So testing one upgrade installation might take considerable amount of time.

After the root cause for the problem was found it wasn’t hard to find some answers. As usual, our deares friend Google helped us out in this. Thanks for the supplementary documentation provided:

• Rob Menshing helped in confirming the problem in his article “Doing a small update or minor upgrade in MSI? Use MSIENFORCEUPGRADECOMPONENTRULES“
• InstallShield Help Library had a nice table clarifying the fairly complex MSI strategy about Major Upgrade vs. Minor Upgrade vs. Small Update
• And the final question of WHY THE HELL IS THIS HAPPENING was answered by Heath Stewart in his article about “Why is My Feature Advertised?“

The thing was, Visual Studio puts all files in a single feature (named DefaultFeature) . When we renamed this one file in the installation package, it was basically removed from this feature and a new file was added. Adding files is not bad for upgrades – but removing is. Windows Installer uses black magic, hidden powers and some complex logic to determine when to replace an existing file with one from the upgrade package. And even if it’s being told to reinstall everything (REINSTALL=ALL, repairmode = “force all files to be reinstalled”) it’s still not doing that – since it’s not able to determine the status for this one component which used to be part of this feature. Therefore it skips upgrading anything and still reports no errors (unless MSIENFORCEUPGRADECOMPONENTRULES is set to 1).

To workaround this there are basically three alternatives:

1. Include the old file in the upgrade package. Simplest way unless the old file causes problems when being installed together with the new files. Luckily this wasn’t the case for us.
2. Instead of upgrade do remove feature and install. Problem is, since there’’s only one feature removing it is basically uninstallation of the whole software. Nice way of doing and upgrade.
3. Do major upgrade.

I think we’re going to end up in choosing alternative #1. In the near future we need to seriously re-consider the design decisions we’ve made concerning application deployment, patching, upgrades, uninstallation…

These things are never easy. I personally had high hopes for Windows Installer and still haven’t buried those. These decisions are also often costly yet they might be done hasty.