Release Management and my release tool for full and differential releases

In this post I'll discuss some of the common tasks I perform for release management, and a tool I created to help with it, release.exe. You can find release.exe's source code here on github.

Release Management : Your Mileage May Vary

If you're responsible for software release management, source control is a given--but what else does release management entail? That really depends... it depends on what you hold important, on what constraints come with your target environment(s), and on what customer requirements you have to contend with. Release management might mean nothing more than deploying the latest code from source control to a public cloud; or, it might be a very complex multi-step process involving release packaging, electronic or media transfer to a customer, security scans, patching, approval(s), and network transfers by client IT departments--where some of the process is out of your hands. Whether simple or complex, good release management requires discipline and careful tracking. A well-thought-out procedure, supported with some tools, makes all the difference.

In the release management I regularly perform, common tasks are these:

1. Packaging up a full release to ship to a location, where it will be delivered to the client, go through multiple security processing steps, and eventually end-up on-site, ready for deployment.
2. On-site deployment of an approved release to new or existing servers.

The most interesting new development in all of this has been being able to generate differential releases, where only files that have been changed are delivered. This adds several more common tasks:

3. Packaging up a partial release (just what's changed) to ship to a location, and go through the same processing and approval steps.

4. On-site deployment of an approved partial release to new or existing servers.

Differential releases are massively valuable, especially when your full release might be tens of thousands of files (perhaps spanning multiple DVDs), whereas an update might have only changed a handful of files that take up 1/10th of a DVD. However, getting differential releases to work smoothly and seamlessly requires some careful attention to detail. Most importantly, you need a means to verify what you end up with is a complete, intact release.

To help with release packaging and on-site release verification, I created the release.exe command for Windows. Let's take a look at what it can do.

Hashing: a way to verify that a file has the expected contents

My release.exe command borrows an idea from my Alpha Micro minicomputer days: file hashes and hashed directory files. Back then, our DIR command had a very useful /HASH switch which would give us a hash code for a file, such as 156-078-940-021. Changing even a single byte of a file would yield a dramatically different hash.

When we would ship releases to customers, we would include a directory file of every file with its hash code. On the receiving end, a client could use a verify command which would read the hashed directory file and compare it against the computed hash of each file on the local system--displaying any discrepencies found. This process worked beautifully, and I've always missed having it on Windows. Now I have a version of the same concept in a tool I can use on Windows.

The release command can generate a file hash, with the command release hash:

Command form: release hash file

The hash is a partial MD5 hash. Why partial? Well, the entire hash is really long (20 segments), which is rather onerous if you need to send a hash code to someone or discuss it with someone else. So, I've shortened it to the the first two and last two segements of the full MD5 hash. Since the hash will change dramatically if even one byte changes, this is perfectly adequate for our puposes.

Here's a sample output:

path> release hash readme.txt
05B-8E8-D57-E7C readme.txt

path> release hash release.exe
BB9-AFA-F22-32A release.exe

File hashes will form the basis for packaging up releases with a manifest of files and their hashes; and for verifying those manifests on the receiving side.

Creating A Full Release Manifest

To generate a complete release, we first get the files intended for the release in a folder with the name of the release. For example, if our application's latest changeset in source control was 3105, we might create a 3105_release folder. Within that we copy all of our release files, which will likely include many files and many subfolders.

With the release files copied, we can now use the release create command to create a release manifest:

Command form: release create release-name.txt

3105_release> release create 3105.txt
Creating manifest for c:\3105_release
F7C-2C3-AE1-4BC C:\3105_release\readme.txt
63A-EE0-17F-2D4 C:\3105_release\bin\appmain.dll
9AB-6F4-RE3-007 C:\3105_release\bin\security.dll
3B2-B16-5Ac-007 C:\3105_release\bin\service.dll
47C-08D-A42-FD5 C:\3105_release\bin\en-US\resources.dll
98D-1E1-399-A7A C:\3105_release\Content\css\site.css
652-8A0-52A-ED0 C:\3105_release\Views\Login\login.cshtml
179-488-E60-E22 C:\3105_release\Views\App\main.cshtml
77c-874-963-791 C:\3105_release\Views\App\add.cshtml
6E5-3B0-68C-349 C:\3105_release\Views\Admin\customize.cshtml
E02-C9C-A53-37C C:\3105_release\Views\Admin\settings.cshtml
F01-a37-eed-629 C:\3105_release\Views\Report\monthlysales.cshtml
...

The result of all this is simply to add one file to the release, 3105.txt in this case, which contains every file in the release and its hash. We also add release.exe itself to the release folder. This will give us what we need on the receiving end to verify the release is correct.

Verifying a Release

Once your release has gone through all of the permutations that get it to where it needs to go, and you have deployed it, you'll want to verify that it is complete and intact. Because the release shipped with release.exe and the manifest .txt file, you can easily verify your release by opening a command window, CDing to the root of where the release was deployed to, and using the release verify command.

Command form: release verify release-name.txt

If every file in the manifest is present and has the expected hash, you'll see Verified Release in green.

c:\InetPub\wwwroot> release verify 3105.txt

8713 files checked
Release Verified

If on the other hand there are differences, you will see one or more errrors listed in yellow or red. Yellow indicates a file is present but doesn't have the expected hash. Red indicates a missing file.

c:\InetPub\wwwroot> release verify 3105.txt
FILE NOT FOUND   c:\3105_release\Views\Report\summary.cshtml
A41-BBC-B4B-125  c:\3105_release\Content\css\site.css - ERROR: file is different
782-661-022-411  c:\3105_release\web.config - ERROR: file is different
8713 files checked
3 error(s)

In reviewing the results, note that it may well be normal for a file or two to be different. For example, an ASP.NET web application might have a different web.config file, with settings specific to the target environment.

This simple procedure, which generally takes under a minute even for large releases, is a huge confidence builder that your release is right. If you're in a position where processing steps sometimes lose files, mangle files, or rename files, using release.exe can detect and warn you about all of that.

Creating A Differential Release

At the start of this article I mentioned differential releases, where only changed files are provided. You can generate a differential release (and its manifest .txt file) with the release diff command.

Command form: release diff release-name.txt prior-release-name.txt

Up until now, we have seen variations of the release command that create manifest .txt files or verify them. The release diff command is different: it will not only generate a manifest .txt file, it will also compare it to the prior full release's manfest .txt file--and then delete files from the release folder that have not changed. For this reason, a prominent warning is displayed. The operator must press Y to continue, after confirming they are in the directory they want to be and wish to proceed. Be careful you only run this command from a folder where you intend files to be removed.

Let's say some time has passed since your last full release (3105) and you now wish to issue release 3148--but only a dozen or so files that have changed.

1. You start by creating a 3148_release folder and publishing all of your release files to that folder. So far, this is identical to the process used for full releases.
2. You copy into the folder release.exe and the manifest from the last full release, 3105.txt.
3. Next, you use the release diff command to create a differential release:

3148_release> release diff 3148.txt 3105.txt

Differential release:

    New release manifest file ............ 3148.txt

    Prior release manifest file .......... 3105.txt

    Files common to prior release and this reease will be DELETED from this folder, leaving only new/changed files.

WARNING: This command will DELETE FILES from c:\3148_release\

Are you sure? Type Y to proceed 

3. After confirming this is what you want to do, you press Y and release.exe goes to work.
4. When release.exe is finished, you will see a summary of what it did:

...
Differential release created:
    Release manifest file .................. 3148.txt
    Files in Full Release .................. 8713
    Files in Differential Release .......... 12
    Files removed from this directory ...... 8701

Only 12 files were left in the directory, because the other 8701 files were identical to the last full release--so they don't need to be in the update. Your folder contains only the handful of files that have changed since last release, making for a smaller, simpler release package.

However, the 3148.txt manifest will list every file in the cumulative release and its hash. This is important, because on-site you will be overlaying this partial 3148 release on top of a prior 3105 full release. You want to be able to perform a release verify 3148.txt command which will verify the entire release, not just the changed files.

c:\InetPub\wwwroot> release verify 3148.txt

8713 files checked

Release Verified

Summary: 

The release.exe command has already made my life a lot easier, as someone who has to regularly generate releases--sometimes in a hurry. It is also making deployment a lot less problematic on the customer delivery side: the completeness and correctness of deployments can be immediately ascertained, and if there are problems the specific files are clearly identified.

Download source code