Ghost : A Father-Daughter Project, Part 1

Over the last Thanksgiving break, our oldest daughter Susan came home from college for a few days. She is studying web design and marketing, but has found herself more and more becoming a front-end UX designer. She's been spending a lot of time working on HTML/CSS/JavaScript projects with great success.

While we had a few days together, we decided it would be fun to collaborate on a project that would be useful for her coursework: a program that learns. She suggested implementing the game of GHOST that our family has always played, and that's what we did. The game is here.

Rules of GHOST

In case you're not familiar with it, GHOST is a game well-suited for car trips, sitting around the dinner table, or any other time you have 2 to N people sitting around with nothing to do. It's a spelling game, and if you make a valid word of 3 letters or more, you've lost the round. The first time you lose a round, you're a "G". The next time you lose a round, you're a "GH". Once you get all the way to "GHOST", you're out of the game. Here's a sample sequence or two to give you the idea:

Player 1: K
Player 2: N
Player 3: O
Player 1: C
Player 2: K
Player 3: "That's a word. You're a G."

Player 1: P
Player 2: A
Player 1: C
Player 2: I
Player 1: N
Player 2: A
Player 1: "I challenge you. What was your word?"
Player 2: "Ah, you got me. I didn't want to go out on PACING, so I bluffed. I'm a G-H-O."

Although GHOST is a simple game, there are nuances and strategy to it. Programming a competent player is not as simple to implement programatically as you might think.

Our Game

Our version of Ghost will be a two-player edition, human against computer.

Since Susan has done front-end development but not back-end or database work, I volunteered to put together a starting point program, and write the back-end code to her specifications. Since I spend much of my time with ASP.NET, I created a sample ASP.NET Model-View-Controller (MVC) project, and added web.config and code to connect to a SQL Server database.

Since this game is supposed to learn, it most definitely needs a database so it can grow its word list as it plays. This is just the simplest of databases: one table named Words containing one colum, Word. We initially seeded the word list with a couple dozen entries, but ever since the list has been growing as a result of game play. At the time of this writing, it has over 1400 words. We could of course license a dictionary, but that would defeat the learning purpose of this exercise.

Our first objective was to implement basic game play in order to arrive at a functional game, although not a very smart one. The basic algorithm for game play is this:

Basic Gameplay Flowchart (click to enlarge)

When it's the human's turn, he or she has 3 options:

1. Play: press a letter key to continue the word.
2. Challenge: click a Challenge button.
3. That's a Word: click a That's a Word button.

The Challenge and That's a Word buttons aren't visible unless at least 3 letters have been played.

Initially the human player goes first. In each new round, the game will flip who the starting player is.

Flow for Human Plays a Letter

When the human presses a letter key, the letter is added to the current word in play. Non-letter keys are ignored.

Next, a check is made to see whether the human player has just completed a word: if they have, they have lost the round. This is done with a call to the back end to look up the current word-in-play against the Ghost word list.

SELECT word FROM words WHERE word=@round

If the word is found in the word list, the game informs the player and a loss is scored for them. The player gets the next letter in G-H-O-S-T, and if T has been reached the game is over.

If the word was not found in the word list, the computer needs to make a turn. A query is made of the word list for winning words. Winning words that begin with the current word-in-play and are also the right length (odd or even) such that the computer will win if the word is played out. For example, let's say the current word in play is B A C. That means a winning word must begin with BAC and also be an odd-number of characters. The search for winning words would inlude BACON but not BACK or BACCARAT. If one or more winning words are found, one is randomly selected and the next letter is played.

A good algorithm for selecting a winning word took some thought and experimentation. The example query below shows how a winning word is selected if the human player went first. The first WHERE clause ensures the word selected begins with the word-in-play. The second clause ensures the word selected is longer than the word-in-play. The third clause ensures the selected word, when played out, will result in the human player making a complete word and not the computer. The ORDER BY clause tells SQL Server to select a random order.

SELECT TOP 1 Word from Words 
WHERE word LIKE @round + '%' 
AND LEN(word)>LEN(@round) 
AND ((LEN(word) % 2)=0) 
ORDER BY NEWID()

The above query is actually augmented further, because we don't want to target a winning word only to discover we accidentally made a losing word on the way; for example, planning to play P to pursue the word ZIPPER would be a mistake because ZIP is a losing word. To achieve this, more WHERE clauses are added to the query to ensure the computer does not select any word that could result in a losing position.

If no winning words are found, then the computer must either challenge the user or make a bluff. We came up with this rule: if the word-in-play is 3 letters or more in length and there are no losing words in the word list, a challenge is made. The human player can then admit they were bluffing, or tell the computer their word. If the human player was bluffing, a loss is scored for them. If a word is provided, the game adds the word to its word list and scores a loss for itself.

If not challenging, then the computer must bluff. Initiallly a random letter was selected for bluffing in our game, but that was often too obvious a bluff in game play, with nonsense letter combinations. Susan came up with the idea of scanning the word list for the next letter to play. This results in more credible letter sequences. The bluff letter is played.

Flow for Human Clicks Challenge Button

The human player can click a Challenge button if they don't believe the computer is making a valid word.

The game looks for a word in its word list that begins with the current word-in-play. If found, the user is informed what the word is, and then someone loses the round. Usually this is the human player, except in the case where the computer's word has been fully played: in that case, the computer has made a complete word and loses the round.

If the computer was bluffing (no matching word in the word list), it admits it was bluffing and takes a loss.

Flow for Human Clicks That's a Word Button

The human player can click a That's a Word button to indicate the computer has made a complete word. Since the Ghost game is designed to learn as it plays, it trusts the human player to be truthful (and a good speller), and adds the word to its word list database. Now it knows the new word for future play.

Of course, trusting the human player comes with risks. That's the reason for the next section we'll discuss, Administration.

Administration

Our game has a page for viewing the word list. If you add administrator credentials, this page also allows adding and removing words. This is important because our game trusts the human player in learning new words. If there's a typographic error, or a disallowed word (like a proper name), or profanity, we want to be able to correct it or remove it.

The back end of administrative functions are simple DELETE and INSERT queries to the database.

Summary

Well, that's our game--so far. From a learning / intelligence perspective, Ghost can:

• Learn new words
• Distinguish between potential winning and losing words
• Bluff convincingly

Susan is next going to re-do my placeholder front-end with her own UX design, which I'm sure will be highly creative. We'll cover that in a future Part 2.

I am greatly enjoying teaming up with my daughter on a project--something we haven't done since those middle school science project days that now seem so long ago.

Next: Ghost Game, Part 2: Voice-Enabling for Alexa

Visualizing Workflow Activity with Sankey Diagrams

In this post, I'll demonstrate how something called a Sankey Diagram can be used with charting software to visually show workflow activity.

The Problem of Showing Workflow Activity Effectively

If you've ever worked with business workflows in software, you've likely struggled with the problem of communicating workflow activity to clients: it's important, but it's also difficult. This is especially true with complex workflows. While users are inherently familiar with their own workflow (or a portion of it relating to their position), graphically depiciting activity can be daunting.

It's not difficult to understand why this is difficult problem: just look at how workflows are organized and stored in computer systems. Although you might see workflows depicted with flowcharts or UML diagrams at times, their storage in digital systems tends to be a complex hierarchy of multiple levels, sometimes captured in the form of an XML dialect. Entities involved in the workflow have states and have to be gated through allowable future states depending on the rules of the workflow. Advancing from one state to another can happen in response to a user action; in response to a message from an external system; because a certain amount of time has passed; or can be automatic. On top of all that, some workflow paths may run in parallel.

Most often, you'll see bar/column/pie/donut charts used to show a breakdown of activity within one workflow stage. That's a fine way to show what's going on in a single stage, but it doesn't provide a view of activity across the workflow. That across-the-workflow view can be pretty important: are a significant portion of online orders being returned by customers? You wouldn't get any insight into such connections just looking at workflow activity one stage at a time.

Sankey Diagrams Illustrate Flow Well

It's flow of activity where Sankey diagrams become very helpful. Sankey diagrams are intended to show flow, and they do so with connecting lines or arrows whose size is proportional to the amount of activity.

You can see some simple and complex examples of Sankey Charts on the Google Charts gallery. While there, notice that Google's implementation lets you hover over a particular flow for more information, including count. But even without a visible count, you can tell relative activity by the thickness of the connection between source and destination. In the example below, we can see that the B-X connection has far less activity than the B-Y connection. If you imagine that the start and end states are stages or substages of your workflow, you begin to see the possibilities.

Simple Sankey Diagran

Here's a more complex example from the same Google gallery page that shows flow across a series of states. Even though there's a lot more going on this time, transitions from one state to another are clearly shown and the rate of activity is easy to gauge from the width of the connecting lines. This is what makes Sankey diagrams great for illustrating workflow.

Complex Sankey Diagran

Although the Google library is very good, I'm going to be using the Highcharts chart library for the remainder of this post, simply because that's what I use regularly. Google Charts requires Internet access and the license terms disallow including the library locally in your application; in contrast, Highcharts can be self-contained in the application but the library does need to be purchased. Both libraries render excellent results.

If you want to play around with the idea of Sankey diagrams without doing any coding, check out the SankeyMATIC page, which will let you set up a Sankey diagram without doing any coding. It's a great way to prototype a Sankey diagram before deciding to invest development effort.

In looking at a Sankey diagram, you might get the idea that they must be complex to program but this is not the case at all. Most chart libraries that support Sankey diagrams simply take arrays as data input, wher each array element specifies the name of a source state, the name of a destination state, and a count. The chart library takes it from there, stitching things together for you. Both Google Charts and Highcharts operate this way. We'll see an example of that shortly.

Sample Scenario: An Order Workflow

To show an example, we'll imagine the order workflow for a company that accepts both online orders and phone orders.

• Orders have to be prepaid unless credit is approved. 
• Once an order is prepaid or credit-approved, associates assemble the order by pulling SKUs from a warehouse. The order may also be gift-wrapped. 
• Once assembled, orders are placed on a shipping dock awaiting pick up by the shipping carrier.
• Orders that were not prepaid are billed and followed-up by accounts receivable until the order is paid in full.

Our workflow is implemented as a series of stage-substage-state combinations. Specific actions connect one state to another. For example, submitting an online order that requests credit transitions state from Shopping | Online Order | Order Submitted to Order Configuration | Credit Approval | Credit Review; whereas a prepaid order would transition to Order Configuration | Payment Verification | Applying Payment.

Order Workflow Stages, Substages, and States

Now imagine this system is up and running and users want to see order activity. We could of course do the usual simple charts to show what is happening at each stage of the workflow. That might look something like this:

Single-Stage Views

This is certainly useful information; it lets us look into the breakdown of shopping activity. But it tells us nothing about what's happening across the workflow. So, we're now going to create a Sankey Diagram in Highcharts to provide that added view.

Creating Sankey Diagrams for the Sample Scenario

We can first start out simply, by showing transition from the first stage (Shopping) to the second stage (Order Confirmation). To do that, we'll take a standard JSFiddle for a Highcharts Sankey diagram and simply modify the data array as follows. We're merely supplying array entries, with the source-state, destination-state, and count. Our array includes all of the Shopping stage start states and all of the Order Confirmation stage end states.We know the source and end states from our workflow, and we know the counts by querying our order database.

Below is the resulting Sankey diagram. Even though we're only focusing on the first two stages of the overall workflow, we can see the Sankey diagram yields a very rich view of what is going on. We can hover over the connecting lines for the exact count, but just at-a-glance we can see a great deal. We see that phone orders are miniscule compared to online orders. We see that that most of the orders are in various states of credit approval processing. We are now getting a sense of how activity is flowing, which tells more of a story than just looking at one stage at a time.

Sankey diagram: Shopping Stage to Order Confirmation Stage

Now, let's add the remaining stages. Our data list now looks like this, with more array elements.

And below, the Sankey chart showing activity flow across the entire workflow. Now we really are geting a sense for what's happening across the board (JSFiddle).

I'd like to point out a few useful things about the Highcharts implementation. First off, you can hover over any conection line to get a tooltip showing the end-state name and the count. With some coding, you could also arrange it so that clicking on a section drill down into a detail chart.

Highcharts Sankey Diagram - Detail on Hover

Another useful feature is the menu at top right, which permits the chart to be export as a graphic.

Our sample scenario is a modest workflow: what if your workflow is much more complex, to the point where the diagram is really crammed? Well, you certainly need to keep the informaton understandable and you should strive to avoid overwhelming the user. Here are some strategies to consider:

• Set an appropriate number of colors--too many may reduce understandability. Consider whether it makes sense to color-code stages.
• When showing the full workflow, leave out the most detailed state level and provide that elsewhere.
• Show multi-stage sequences in sections rather than the entire workflow in a single view.
• Allow users to click on a stage to get an expanded view of the detail in that stage.
• Group states of little interest together into a single node or leave them out altogether.

If I were doing this for a client rather than a blog post, I would put extra time on finishing touches: adjusting colors, adjusting font and text effects, and considering whether some of the information is not of interest to its audience. Even without doing so, I hope this introduction to Sankey diagrams provides some insight into how workflow activity can be shown to users in an insightful way.

I only discovered Sankey diagrams recently, but their usefulness was immediately apparent. Not only are they useful, they're also very simple to create using leading chart libraries.If you're facing the challenge of visualizing workflow activity, I encourage you to try them out.

Setting Up Transparent Data Encryption

This post discusses the Transparent Data Encryption (DTE) feature in SQL Server and how to use it.

TDE: What Is Is and Why It Exists

When it comes to database encryption, there are two areas to think about: encryption during transport and encryption at rest.

Encryption during transport means the communication between the database and your client (your application, or SQL Server Management Studio for example) is encrypted. Many developers who use SQL Server are already familiar with specifying Encrypt=True in a connection string. It isn't necessary to create a certificate to use this feature, but in a Production environment you'd want to create a certificate and configure the client to only trust that certificate.

All well and good, but encryption during transport doesn't change the fact that the database data on disk is not encrypted. If you dumped the database file of your Contacts database, you would see visible names and contact information. If someone made off with that file, they'd have access to the data.

This is where encryption at rest comes in: keeping the database data encrypted on disk. That means data is encrypted when inserted or updated, and decrypted when queried. If you consider what would be involved in doing this yourself in your application code, it's pretty daunting: you'd need to be sure encryption and decryption was applied uniformly, and doing so without a performance impact would be a major feat; plus, external applications like report generators would no longer be able to do anything with the database.

Fortunately, the Transparent Data Encryption feature exists and it is extremely well done. Once you turn it on, it just works. Data in the data file is encrypted. Data you work with isn't. Conceptually, you can think of it like the diagram below (and if you want all the specific encryption details, see the Microsoft documentaton link at the top of this post). And as we said earlier, the data can also be encrypted during transport with a connection string option.

In my experience TDE doesn't noticably impact performance. If you're an authorized user who has specified valid credentials, nothing will seem at all different to you. But if you dumped the database files, you would no longer be able to see understandable data.

Although TDE is a very nice feature, it's only available in Enterprise Edition--so it comes at a price. There is one other edition where TDE is available, and that's Developer Edition. This means you can experiment with the feature--or demonstrate it to a client--without having to buy Enterprise Edition up front. Understand, however, that you cannot use Developer Edition in a Production environment.

Enabling TDE

The procedure to enable TDE is not difficult. These are the steps:

1. Install SQL Server Developer Editon or Enterprise Edition.
2. Run SQL Statements to create a key and certificate.
3. Run SQL Statements to enable TDE.
4. Back up the certificate and key file.

1. Install SQL Server Developer Edition or Enterprise Edition

You can download SQL Server Developer Edition from the MSDN web site. For Enteprise Edition, follow the instructions you receive through your purchasing channel to obtain the software.

Create or restore a database, and ensure the database is functional and that you can get to it from SQL Server Management Studio.

2. Run SQL Statements to Create a Certificate

A master key and a certificate are needed for the encryption feature. To create them, run the statements below again the MASTER database.

USE master

GO

CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'my-password';

GO

CREATE CERTIFICATE TDEServerCert WITH SUBJECT = 'My DEK Certificate';

GO

3. Run SQL Statements to Enable TDE

Next, connect to your application database (name App in the example) and run the statements below to enable TDE:

USE App

GO

CREATE DATABASE ENCRYPTION KEY WITH ALGORITHM = AES_128 ENCRYPTION BY SERVER CERTIFICATE TDEServerCert;

GO

ALTER DATABASE App SET ENCRYPTION ON;

GO

4. Back Up the Certificate and Key File

This next step makes a back up of the certificate and key file used for TDE. This step is vital: any backups you make from this point forward cannot be restored unless you have the certificate and key files.

BACKUP CERTIFICATE TDEServerCert TO FILE = 'c:\xfer\TDEServerCert.crt'

    WITH PRIVATE KEY

    (

        FILE = 'c:\xfer\TDEServerCert.pfx',

        ENCRYPTION BY PASSWORD = 'my-password'

    )

Confirming DTE

After enabling DTE, you'll want to confirm your application still works like it always has. 

To confirm to yourself that TDE is really active, or provide evidence to an auditor, you can use this query:

SELECT [Name], is_master_key_encrypted_by_server, is_encrypted from master.sys.databases

This will display a list of databases and whether or not they are encrypted.

name    is_master_key_encrypted_by_server   is_encrypted

master  1                                   0

tempdb  0                                   1

model   0                                   0

msdb    0                                   0

App     0                                   1

If you're still skeptical, you can also dump your database files.

Release Management and my release tool for full and differential releases

In this post I'll discuss some of the common tasks I perform for release management, and a tool I created to help with it, release.exe. You can find release.exe's source code here on github.

Release Management : Your Mileage May Vary

If you're responsible for software release management, source control is a given--but what else does release management entail? That really depends... it depends on what you hold important, on what constraints come with your target environment(s), and on what customer requirements you have to contend with. Release management might mean nothing more than deploying the latest code from source control to a public cloud; or, it might be a very complex multi-step process involving release packaging, electronic or media transfer to a customer, security scans, patching, approval(s), and network transfers by client IT departments--where some of the process is out of your hands. Whether simple or complex, good release management requires discipline and careful tracking. A well-thought-out procedure, supported with some tools, makes all the difference.

In the release management I regularly perform, common tasks are these:

1. Packaging up a full release to ship to a location, where it will be delivered to the client, go through multiple security processing steps, and eventually end-up on-site, ready for deployment.
2. On-site deployment of an approved release to new or existing servers.

The most interesting new development in all of this has been being able to generate differential releases, where only files that have been changed are delivered. This adds several more common tasks:

3. Packaging up a partial release (just what's changed) to ship to a location, and go through the same processing and approval steps.

4. On-site deployment of an approved partial release to new or existing servers.

Differential releases are massively valuable, especially when your full release might be tens of thousands of files (perhaps spanning multiple DVDs), whereas an update might have only changed a handful of files that take up 1/10th of a DVD. However, getting differential releases to work smoothly and seamlessly requires some careful attention to detail. Most importantly, you need a means to verify what you end up with is a complete, intact release.

To help with release packaging and on-site release verification, I created the release.exe command for Windows. Let's take a look at what it can do.

Hashing: a way to verify that a file has the expected contents

My release.exe command borrows an idea from my Alpha Micro minicomputer days: file hashes and hashed directory files. Back then, our DIR command had a very useful /HASH switch which would give us a hash code for a file, such as 156-078-940-021. Changing even a single byte of a file would yield a dramatically different hash.

When we would ship releases to customers, we would include a directory file of every file with its hash code. On the receiving end, a client could use a verify command which would read the hashed directory file and compare it against the computed hash of each file on the local system--displaying any discrepencies found. This process worked beautifully, and I've always missed having it on Windows. Now I have a version of the same concept in a tool I can use on Windows.

The release command can generate a file hash, with the command release hash:

Command form: release hash file

The hash is a partial MD5 hash. Why partial? Well, the entire hash is really long (20 segments), which is rather onerous if you need to send a hash code to someone or discuss it with someone else. So, I've shortened it to the the first two and last two segements of the full MD5 hash. Since the hash will change dramatically if even one byte changes, this is perfectly adequate for our puposes.

Here's a sample output:

path> release hash readme.txt
05B-8E8-D57-E7C readme.txt

path> release hash release.exe
BB9-AFA-F22-32A release.exe

File hashes will form the basis for packaging up releases with a manifest of files and their hashes; and for verifying those manifests on the receiving side.

Creating A Full Release Manifest

To generate a complete release, we first get the files intended for the release in a folder with the name of the release. For example, if our application's latest changeset in source control was 3105, we might create a 3105_release folder. Within that we copy all of our release files, which will likely include many files and many subfolders.

With the release files copied, we can now use the release create command to create a release manifest:

Command form: release create release-name.txt

3105_release> release create 3105.txt
Creating manifest for c:\3105_release
F7C-2C3-AE1-4BC C:\3105_release\readme.txt
63A-EE0-17F-2D4 C:\3105_release\bin\appmain.dll
9AB-6F4-RE3-007 C:\3105_release\bin\security.dll
3B2-B16-5Ac-007 C:\3105_release\bin\service.dll
47C-08D-A42-FD5 C:\3105_release\bin\en-US\resources.dll
98D-1E1-399-A7A C:\3105_release\Content\css\site.css
652-8A0-52A-ED0 C:\3105_release\Views\Login\login.cshtml
179-488-E60-E22 C:\3105_release\Views\App\main.cshtml
77c-874-963-791 C:\3105_release\Views\App\add.cshtml
6E5-3B0-68C-349 C:\3105_release\Views\Admin\customize.cshtml
E02-C9C-A53-37C C:\3105_release\Views\Admin\settings.cshtml
F01-a37-eed-629 C:\3105_release\Views\Report\monthlysales.cshtml
...

The result of all this is simply to add one file to the release, 3105.txt in this case, which contains every file in the release and its hash. We also add release.exe itself to the release folder. This will give us what we need on the receiving end to verify the release is correct.

Verifying a Release

Once your release has gone through all of the permutations that get it to where it needs to go, and you have deployed it, you'll want to verify that it is complete and intact. Because the release shipped with release.exe and the manifest .txt file, you can easily verify your release by opening a command window, CDing to the root of where the release was deployed to, and using the release verify command.

Command form: release verify release-name.txt

If every file in the manifest is present and has the expected hash, you'll see Verified Release in green.

c:\InetPub\wwwroot> release verify 3105.txt

8713 files checked
Release Verified

If on the other hand there are differences, you will see one or more errrors listed in yellow or red. Yellow indicates a file is present but doesn't have the expected hash. Red indicates a missing file.

c:\InetPub\wwwroot> release verify 3105.txt
FILE NOT FOUND   c:\3105_release\Views\Report\summary.cshtml
A41-BBC-B4B-125  c:\3105_release\Content\css\site.css - ERROR: file is different
782-661-022-411  c:\3105_release\web.config - ERROR: file is different
8713 files checked
3 error(s)

In reviewing the results, note that it may well be normal for a file or two to be different. For example, an ASP.NET web application might have a different web.config file, with settings specific to the target environment.

This simple procedure, which generally takes under a minute even for large releases, is a huge confidence builder that your release is right. If you're in a position where processing steps sometimes lose files, mangle files, or rename files, using release.exe can detect and warn you about all of that.

Creating A Differential Release

At the start of this article I mentioned differential releases, where only changed files are provided. You can generate a differential release (and its manifest .txt file) with the release diff command.

Command form: release diff release-name.txt prior-release-name.txt

Up until now, we have seen variations of the release command that create manifest .txt files or verify them. The release diff command is different: it will not only generate a manifest .txt file, it will also compare it to the prior full release's manfest .txt file--and then delete files from the release folder that have not changed. For this reason, a prominent warning is displayed. The operator must press Y to continue, after confirming they are in the directory they want to be and wish to proceed. Be careful you only run this command from a folder where you intend files to be removed.

Let's say some time has passed since your last full release (3105) and you now wish to issue release 3148--but only a dozen or so files that have changed.

1. You start by creating a 3148_release folder and publishing all of your release files to that folder. So far, this is identical to the process used for full releases.
2. You copy into the folder release.exe and the manifest from the last full release, 3105.txt.
3. Next, you use the release diff command to create a differential release:

3148_release> release diff 3148.txt 3105.txt

Differential release:

    New release manifest file ............ 3148.txt

    Prior release manifest file .......... 3105.txt

    Files common to prior release and this reease will be DELETED from this folder, leaving only new/changed files.

WARNING: This command will DELETE FILES from c:\3148_release\

Are you sure? Type Y to proceed 

3. After confirming this is what you want to do, you press Y and release.exe goes to work.
4. When release.exe is finished, you will see a summary of what it did:

...
Differential release created:
    Release manifest file .................. 3148.txt
    Files in Full Release .................. 8713
    Files in Differential Release .......... 12
    Files removed from this directory ...... 8701

Only 12 files were left in the directory, because the other 8701 files were identical to the last full release--so they don't need to be in the update. Your folder contains only the handful of files that have changed since last release, making for a smaller, simpler release package.

However, the 3148.txt manifest will list every file in the cumulative release and its hash. This is important, because on-site you will be overlaying this partial 3148 release on top of a prior 3105 full release. You want to be able to perform a release verify 3148.txt command which will verify the entire release, not just the changed files.

c:\InetPub\wwwroot> release verify 3148.txt

8713 files checked

Release Verified

Summary: 

The release.exe command has already made my life a lot easier, as someone who has to regularly generate releases--sometimes in a hurry. It is also making deployment a lot less problematic on the customer delivery side: the completeness and correctness of deployments can be immediately ascertained, and if there are problems the specific files are clearly identified.

Download source code

My First Chrome Extension: Airgap Scan

Today I wrote my first Chrome Extension, and it was fun. I want to share what the experience was like. The code for this post may be found here: https://github.com/davidpallmann/AirgapScan

Chrome has become a favorite browser to many, and if you do web development at all you have no doubt seen how important Chrome Extensions have become. Some of the ones I use frequently are WhatFont (tells me what font I am looking at when I hover over text in a page) and ng-inspector (AngularJS variable inspector), among many others.

It's always best to learn something new when you have a firm project idea in mind, something that needs to be created. Fortunately, I had a project in mind.

AirgapScan: An Extension That Scans Pages for Disallowed Internet URLs

Today I decided I was in need of a Chrome Extension to help verify whether my web site pages were air-gapped. What is air-gapping? Air-gapping is when your software has to be able to run in a location that allows no Internet access; for various reasons, there are security-minded customers with that requirement. Honoring this requirement is harder than you might think: as a modern developer, we tend to take Internet availability for granted. And, we frequently rely on open source libraries, many of which also take Internet availability for granted.

And so, having made changes to support air-gapping, it's important to test that we haven't missed an Internet reference somewhere. That's why I wanted this Chrome extension: when our testers visit one of our solution's web pages, I want the extension to report if there are airgap violations (that is, Internet access outside of the approved network).

The way I'd like this to work is as follows: you browse to a page in your application. If you want to check your air-gapping, you right-click the AirgapScan icon and select Scan Page. You'll either get a happy green message box telling you all is well, or a red alert box listing the HREF(s) found in the page that are disallowed.

Hello, World

But you have to walk before you can run, so next up was to take a basic tutorial and create a simple "Hello, World!" extension in order to get familiar with the basics. I stumbled across an excellent getting started tutorial by Jake Prins, How to Create and Publish a Chrome Extension in 20 Minutes which walked me through the basics.

I was surprised and pleased to learn just how easy it is to write a Chrome Extension. In a nutshell, here are the basics:

1. Web Technologies. Your extension is written in familiar HTML, CSS, and JavaScript.
2. Developer Mode. While developing, you can easily load and re-load your extension in Chrome as you make changes, making for a great interactive experience as you work. This is done by visiting chrome://extensions and switching on Developer Mode. When you want to load your extension, click LOAD UNPACKED and browse to the folder where your files are. It's a very simple process.
3. Manifest. Your extension starts with a manifest.json file, in which you declare a number of things about your extension--including name, version, icon, permissions needed, and which css / script files it uses.
4. Scripts. You'll write some JavaScript code to do your magic. Depending on what you do, you may have to create more than one based on Google's rules. Again, tutorials and documentation are your friend.
5. You Can Use Your Favorite Libraries. Used to using jQuery? Or one of the many other popular libraries out there? It's fine to include those in your extension--just copy the .js/.css files to your extension folder and declare them in your manifest.
6. Your Extension Can Do A Lot Of Things. In the tutorial I took, I learned I could control the page that is created when a new browser tab is opened. Later on, I learned how to scan the current page's DOM. You can also do things like add context menus to a selection on the page. It's a really powerful platform.

Creating the AirgapScan Extension

1. The Manifest

The first element of any Chrome Extension is the manifest, manifest.json.

{
  "manifest_version": 2,
  "name": "Airgap Scan",
  "author": "David Pallmann",
  "version": "1.0",
  "description": "Scans the page for Internet references. Useful for testing software meant for air-gapped environments (without public Internet access).",
  "background": {
"scripts": [ "background.js" ]
  },
  "icons": {
"128": "icon-128.png"
  },
  "browser_action": {
   "default_icon": "tab-icon.png",
   "default_title": "Airgap Scanner"
 },
  "content_scripts": [
    {
      "matches": [
        ""
      ],
  "css": ["jquery-confirm.min.css"],
      "js": ["jquery-2.2.4.min.js", "jquery-confirm.min.js", "content.js"]
    }
  ],
  "permissions" : [
    "contextMenus"
    ]
}

Key things to note about the manifest:

• It lists required permissions, similar to what you do in a phone app. When the user installs, they'll be asked to confirm they are okay with the extension's required permissions. In my case, I had to list adding context menus as a a permission, since I want to use a context menu item to perform scans on-demand.
• The background property, scripts property specifies one of my script files, background.js.
• The content_scripts object declares some important things. 
• The matches property indicates which URLs the extension is active for (all URLs, in my case). 
• The css property lists CSS files we're including (jquery-confirm.min.css). 
• The js property lists JavaScript files we're including: my own source file content.js, plus libraries jquery.js and jquery-confirm.js.

Why is my JavaScript code in two places (background.js and content.js)? Well, with Chrome extensions there are content scripts which run in the context of page (content.js). But you may also need a global script or page that is running for the lifetime of your extension (background.js).

2. Content.js

Content.js is my page-level JavaScript code. This includes the following important elements:

• A message listener, whose purpose is to listen for the context menu's Scan Page command being clicked. When that happens, the listener invokes the airgapScan function.
• The airgapScan function, which is the heart of the extension. It uses jQuery to find all the HREFs on the page. It discounts some of them, such as mailto: and javascript: links. The rest, it compares against the array of allowableNetworks. If the href partially matches any of the allowable networks, all is well. If not, an error is counted and the URL is added to a list of in-violation URLs. After scanning the HREFs on the page, a green message box (if no errors) or red alert box (displaying the problem URLs) is displayed.

// content.js

// This routes a message from background.js (context menu action selected) to the airgapScan function in this file.

chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
    sendResponse(airgapScan());
});


// airgapScan: scan the page, examine each href. Collect a list of non-allowable hrefs and display an alert.

function airgapScan() {

console.log('--- Airgap Scan ---');

var allowableNetworks = [ '://10.',       // allowed: [http|https]://10.x.x.x 
  '://www.mytestdomain.com'  // allowed: [http|https]://www.mytestdomain.com...
];
var urlCount = 0;
var errorCount = 0;
var url;
var urls = [];
var errList = '';
var listAll = false;

$("a").each(function() {
if (this.href != undefined) {
url = this.href;
if (url!=null && url!='' && url.indexOf('javascript:')==-1 && url.indexOf('mailto:')==-1) {
urlCount++;
urls.push(url);
var error = true;
for (var p = 0; p < allowableNetworks.length; p++) {
if (url.indexOf(allowableNetworks[p])!=-1) {
error = false;
break;
}
}
if (error) {
errorCount++;
console.error('URL outside of network detected: ' + url);
errList = errList + '
' + url;
}

}
}
})
if (listAll && urls.length > 0) {
for (var i = 0; i < urls.length; i++) {
console.log(i.toString() + ': ' + urls[i]);
}
}

console.log('--- end Airgap Scan - URLs: ' + urlCount.toString() + ', Errors: ' + errorCount.toString() + ' ---');

if (errorCount > 0) {
if (errorCount==1) {
$.alert({
//icon: 'fa fa-warning',
type: 'red',
title: 'Airgap Alert',
content: 'Warning: Airgrap scan found 1 url that violates airgap rules:
' + errList,
useBootstrap: false
});
}
else {
$.alert({
//icon: 'fa fa-warning',
type: 'red',
title: 'Airgap Alert',
content: 'Warning: Airgrap scan found ' + errorCount.toString() + ' urls that violate airgap rules:
' + errList,
useBootstrap: false
});
}
}
else {
$.alert({
title: 'Airgap OK',
type: 'green',
content: 'All good: No airgap errors found',
useBootstrap: false
});
}
}

// Default state is that the user initiates a scan from the context menu. Uncomment the line below if you want the scan to automatically run when a page loads. 
//airgapScan();

3. Background.js

Background.js is the lifetime-of-the-extension JavaScript file. It contains

• A call to Chrome.contextMenus.create, which adds a context menu item to the extension, available to the user by right-clicking it's icon.
• A listener to respond to the menu item being clicked. This in turns sends a message to content.js to please invoke the airgapScan method.

// Add "Scan page" action to extension context menu.

chrome.contextMenus.create({

"id": "AG_ScanPage",

    title: "Scan Page",

    contexts: ["browser_action"],

    onclick: function() {

    }

});

// When context menu item is selected, send a message to context.js to run an airgap Scan.

chrome.contextMenus.onClicked.addListener(function(info, tab) {

  if (tab && info.menuItemId=="AG_ScanPage")

    chrome.tabs.sendMessage(tab.id, {args: null }, function(response) {

    });

});

4. Library Files

As mentioned earlier, we are using a few libraries: jquery and jquery-confirm. The .js and .css files for them are included in the folder, and are referenced in the manifest.

5. Icons

Lastly, we have some icons in different sizes. The icon for AirgapScan is shown below.

And that's it. Time from first-time-hello-world-extension to AirgapScan was just a few hours on a Saturday. 

AirgapScan in Action

As I developed AirgapScan, I continually tested in chrome://extensions. When I had an update, I would REMOVE and then LOAD UNPACKED to get the latest changes applied, then visit a fresh page to test it out.

After visiting a page to be tested, the AG icon is visible. Hovering over it shows its name in a tooltip. Right-clicking it shows the Scan Page context menu that the extension code added.

Clicking Scan Page quickly comes back with a message box with the results of the scan. If one or more in-violation HREFs are found, a red alert box itemizes them.

If no violations are found, a green Airgap OK message appears.

You can download this extension here: https://github.com/davidpallmann/AirgapScan

This was a lot of fun, plus I created something that my team needs. Chrome Extensions are surprisingly easy to create and the platform is well thought through which makes it a pleasure to use. I am highly motivated now to create other extensions.