Monday, September 25, 2017

An AngularJS Dashboard, Part 1: Layout, Tiles, and Drag-and-Drop

NOTE: for best results, view the http: version of this page. Else you won't get syntax highlighting.

This is part 1 in a series on creating a business dashboard in AngularJS.

I'm developing a business dashboard, which I'll be using AngularJS for. Since I'm in the midst of learning AngularJS right now, this is a good opportunity to blog my progress as it goes from concept to prototype to refined implementation. In addition to AngularJS, we'll also be using ASP.NET MVC methods for the back end, to handle matters like authentication and data retrieval.

Here's what we're building in Part 1:

Objective

First off, here are my goals:
  • A business dashboard with a variety of tiles (counters, KPIs, tabular data, charts)
  • Tiles support several colors and sizes
  • Tiles can be rearranged by the user, and the user's dashboard layout is remembered
  • Tiles can be clicked on to go to a full detail page
  • Simple configuration, allowing quick and easy add / change / removal of tiles
  • Responsive, able to display well on small phones all the way up to large desktop monitors
Here in Part 1, all we're out to achieve is a dashboard layout, placeholder tiles, and drag-and-drop. We'll build on that foundation in the posts that follow with different tile types, back-end support, and tile actions such as configuration.

Inspiration: Team Foundation Services
Dashboards are a tricky business. I've worked on a number of them, and what often happens is tiles get too ambitious--leading to slow performance, crammed UI, and bloated web pages. In this dashboard, the philisophy is for tiles to show you information, not be workboards in their own right. You can click on a tile to proceed to areas of the software that give you an expanded view with interaction and actions.

My inspiration is to mimic much of the visual design and characteristics of Microsoft's Team Foundation Services dashboard (shown below). While the TFS dashboard is not a business dashboard (it tracks things like bugs, work items, and project progress), there are a number of nice properties of this dashboard that I also want in mine:
  • It renders quickly. 
  • It avoids overly-ambitiious tiles that kill performance and bloat the page. The TFS dashboard designers wisely avoided cramming too much into each tile; when you want more detail and a larger view, you can click through to a page where you perform detailed interaction.
  • It is very customizable, in terms of content, coloring, and size--while keeping configuration fields to a minimal number.
  • It's responsive, and can be used on a variety of devices and screen sizes
In short, it's well done. These are all qualities I'll be striving for in my dashboard as well. Of course I'll also be making my own innovations, but the first milestone is prototyping something similar to this that looks and behaves well.

Microsoft Team Services dashboard - my inspiration

Another goal is to do a proper implementation in AngularJS. As I'm still learning Angular--mostly through tutorials, reading, and creating things--I'll undoubtedly have to refactor and refine as I make progress. I'll do my best, though, to follow Google's guidance and conventions as I go.

Solution Structure

We're beginning with a simple ASP.NET project created in Visual Studio with the structure shown below.

Solution Structure

Key parts of the solution are these:

  • index.html : the default web page, where the dashboard will be rendered
  • dashboard.component.js : my Angular dashboard component
  • dashboard.template.html : the HTML template for the dashboard component
  • dashboard.less : stylesheet

Now let's take a look at these elements to understand what they do.

index.html

Index.html is the HTML page where we'll be rendering our dashboard.

Note: if code views aren't showing with syntax highlighting, change your URL from https: to http:.
<!DOCTYPE html>
<html>
<head>
    <title>Dashboard</title>
    <meta charset="utf-8" />
    <link href="http://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet" />
    <link href="Content/css/dashboard.css" rel="stylesheet" />
    <script src="Scripts/jquery-3.2.1.min.js"></script>
</head>
<body>
    <div ng-app="dashboard">
        <dashboard id="dashboard"></dashboard>
    </div>

    <script src="Scripts/angular.min.js"></script>
    <script src="app/app.module.js"></script>
    <script src="/components/dashboard/dashboard.component.js"></script>

</body>
</html>

The first thing to note here is that there isn't a lot in the HTML page. Things to pay attention to:

1. The page is loading Font Awesome icons, a CSS stylesheet, jQuery, the AngularJS framework, an app.module.js file (which register the Angular app), and a JavaScript file named dashboard.component.js. This last file defines an Angular component named dashboard.
    <link href="http://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet" />
    <link href="Content/css/dashboard.css" rel="stylesheet" />
    <script src="Scripts/jquery-3.2.1.min.js"></script>
    ...
    <script src="Scripts/angular.min.js"></script>
    <script src="app/app.module.js"></script>
    <script src="/components/dashboard/dashboard.component.js"></script>
2. The app.module.js file contains a very small amount of JavaScript code to bootstrap Angular in our application.
(function () {
    'use strict';

    // Define the 'dashboard' module
    angular.module('dashboard', [
      'dashboard'
    ]);

})();
3. The markup declares an Angular app (with the ng-app attribute) named dashboard, and includes a <dashboard> element--that's how you invoke an Angular component, as a custom HTML element. This is one of my favorite features of Angular.
<div ng-app="dashboard">
    <dashboard id="dashboard"></dashboard>
</div>

dashboard.component.js

The dashboard.component.js file defines the dashboard component. Ultimately this will grow quite a bit larger as we add functionality. Today, we're just getting started.
// Register 'dashboard' component, along with its associated controller and template
angular.
  module('dashboard').
  component('dashboard', {
      templateUrl: '/components/dashboard/dashboard.template.html',
      controller: ['$http', function DashboardController($http) {
              ...
          ];

          ...
      }]

  });

Dashboard Component
The angular.module code registers a dashboard module. A component is added to the module, also named dashboard. (If you'd like a detailed understanding of Angular concepts like apps, modules, and components, take the AngularJS PhoneCat Tutorial, which is how I learned it.)
angular.module('dashboard', []); // register module

// Register 'dashboard' component, along with its associated controller and template
angular.
  module('dashboard').
  component('dashboard', {
      templateUrl: '/components/dashboard/dashboard.template.html',
      controller: ['$http', function DashboardController($http) {
          var self = this;

          self.dragTile = function (sourceTileIndex, destTileIndex) {
              try {
                  var sourceTile = self.tiles[sourceTileIndex];

                  self.tiles.splice(sourceTileIndex, 1);  // remove source tile from tile list

                  if (sourceTileIndex < destTileIndex) {
                      destTileIndex--;
                  }

                  self.tiles.splice(destTileIndex, 0, sourceTile); // insert dest tile

                  // Regenerate tile ids.
                  var tile;
                  if (self.tiles != null) {
                      for (var t = 0; t < self.tiles.length; t++) {
                          tile = self.tiles[t];
                          tile.id = 'tile-' + (t + 1).toString(); // tile id (tile_1, tile_2, ...)
                      }
                  }

                  var scope = angular.element('#dashboard').scope();
                  scope.$apply(this);
              }
              catch (e) {
                  console.log('dragTile: EXCEPTION ' + e.toString());
              }
          };

          self.title = 'Sample Dashboard';
          self.tiles = [
              { width: 1, height: 1, color: 'red',    title: 'Tile 01' },
              { width: 1, height: 1, color: 'yellow', title: 'Tile 02' },
              { width: 1, height: 1, color: 'green',  title: 'Tile 03' },
              { width: 2, height: 1, color: 'blue',   title: 'Tile 04' },
              { width: 2, height: 2, color: 'purple', title: 'Tile 05' },
              { width: 1, height: 2, color: 'orange', title: 'Tile 06' },
              { width: 1, height: 1, color: 'gray',   title: 'Tile 07' }
          ];

          // Generate tile properties id, sizeClass, bgClass, and classes.
          var tile;
          if (self.tiles != null) {
              for (var t = 0; t < self.tiles.length; t++) {
                  tile = self.tiles[t];
                  tile.id = 'tile-' + (t + 1).toString(); // tile id (tile_1, tile_2, ...)
                  tile.bgClass = "tile-" + tile.color;    // tile background color class (tile-red, tile-yellow, ...)
                  tile.sizeClass = "tile_" + tile.width.toString() + "_" + tile.height.toString(); // tile size class (tile_1_1, tile_1_2, tile_2_1, tile_2_2)
                  tile.classes = tile.sizeClass + ' ' + tile.bgClass; // full list of classes to customize tile appearance
              }
          }
      }]

  });
The dashboard component contains the following:
  • templateUrl: this is the path to the template file (HTML with Angular directives). When the <dashboard> element in index.html is encountered, the component object is created and renders itself using this template. Our template is named dashboard.template.js.
  • controller: this is the code that has the logic and model for the dashboard. An $http variable is passed into the controller, which we'll use down the road to access back-end services. This is an example of Dependency Injection, which is a central concept in Angular.
Dashboard Controller
Most of the code in the JS file involves the controller, so let's look at some of its noteworthy properties:
  • self : a copy of this, a reference to the controller instance. It's useful to have a self variable for when JavaScript promise functions are invoked. That will come up in later posts.
    var self = this;
  • dragTile: a function for drag-and-drop. Well discuss this later in the post.
  • title: A title for the dashboard. Right now this is hard-coded; eventually, it will be loaded from a stored dashboard definition.
  • tiles: An array of tile definitions. Again, this is hard-coded at present; eventually, the user's tiles, layout, and configuration will be loaded from the application's database. Each tile object has properties that include 
    • title : a title
    • height : logical tile height (1 or 2)
    • width : logical tile width (1 or 2)
    • color: a color name
    In addition to these declared tile properties, several other properties are generated by the controller code:
    • id : unique identifier (tile-1, tile-2, etc.)
    • sizeClass : a CSS class for the tile size 
    • bgClass : a CLS class for the tile's background color.
    • classes : all of the CSS classes to be added to the tile (sizeClass + bgClass)
          self.tiles = [
              { width: 1, height: 1, color: 'red',    title: 'Tile 01' },
              { width: 1, height: 1, color: 'yellow', title: 'Tile 02' },
              { width: 1, height: 1, color: 'green',  title: 'Tile 03' },
              { width: 2, height: 1, color: 'blue',   title: 'Tile 04' },
              { width: 2, height: 2, color: 'purple', title: 'Tile 05' },
              { width: 1, height: 2, color: 'orange', title: 'Tile 06' },
              { width: 1, height: 1, color: 'gray',   title: 'Tile 07' }
          ];
This simple, partly hard-coded structure is enough that we can get started on rendering our dashboard.

dashboard.template.html

The last code artifact we need to look at is the template for the dashboard controller, dashboard.template.html. This is a small fragment of html that contains Angular directives, and this where the magic happens. The logic and model data in the controller are applied to the directives in the template, yielding output to replace the <dashboard> element in the index.html file.
<div class="dashboard-controller">
    <div>
        <div>{{$ctrl.title}}</div>
        <div class="panel">
            <div id="tile-{{$index+1}}" ng-repeat="tile in $ctrl.tiles" class="tile"
                 ng-class="tile.classes"
                 draggable="true" ondragstart="tile_dragstart(event);"
                 ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                {{tile.title}}
            </div>
        </div>
    </div>
Angular Directives
Let's discuss what the Angular directives in the templates do. 
  • On line 5, we see the ng-repeat directive in use in a div. This will execute a loop, one iteration for each item in the controller's tiles array. The variable tile is the current array item, and $index is the zero-based iteration number. 
  • The div markup is generated for each dashboard tile. The tile will have ids of the form tile-1, tile-2, etc. because the id parameter is set to the expression tile-{{$index+1}}.
  • On the same div element is an ng-class directive.  Each tile div starts out with the class tile, but additional CSS classes are needed to control the tile's size and color. ng-class adds additional CSS classes for the tile, based on the classes property of the current tile. Note that we have to use ng-class instead of class so {{tile.classes}}gets evaluated at the right time, when a value is present. 
  • The tile div contains the title for each tile. The expression in double curly braces {{tile.title}} evaluates to the title for that tile.
  • Currently our tiles are simply colored rectangles with a title. There's also an ellipsis (...) that appears in the top right corner if you hover over a tile. That's meant to become a tile action menu.
Tile menu ellipsis appears on hover (future tile menu)

Here's what is actually rendered in the browser by the ng-repeat directive:
<div>
        <div class="ng-binding">Sample Dashboard</div>
        <div class="dashboard-panel">
            <!-- ngRepeat: tile in $ctrl.tiles -->
            <div id="tile-1" ng-repeat="tile in $ctrl.tiles" class="tile ng-binding ng-scope tile_1_1 tile-red" ng-class="tile.classes" draggable="true" ondragstart="tile_dragstart(event);" ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                Tile 01
            </div><!-- end ngRepeat: tile in $ctrl.tiles -->
            <div id="tile-2" ng-repeat="tile in $ctrl.tiles" class="tile ng-binding ng-scope tile_1_1 tile-yellow" ng-class="tile.classes" draggable="true" ondragstart="tile_dragstart(event);" ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                Tile 02
            </div><!-- end ngRepeat: tile in $ctrl.tiles -->
            <div id="tile-3" ng-repeat="tile in $ctrl.tiles" class="tile ng-binding ng-scope tile_1_1 tile-green" ng-class="tile.classes" draggable="true" ondragstart="tile_dragstart(event);" ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                Tile 03
            </div><!-- end ngRepeat: tile in $ctrl.tiles -->
            <div id="tile-4" ng-repeat="tile in $ctrl.tiles" class="tile ng-binding ng-scope tile_2_1 tile-blue" ng-class="tile.classes" draggable="true" ondragstart="tile_dragstart(event);" ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                Tile 04
            </div><!-- end ngRepeat: tile in $ctrl.tiles -->
            <div id="tile-5" ng-repeat="tile in $ctrl.tiles" class="tile ng-binding ng-scope tile_2_2 tile-purple" ng-class="tile.classes" draggable="true" ondragstart="tile_dragstart(event);" ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                Tile 05
            </div><!-- end ngRepeat: tile in $ctrl.tiles -->
            <div id="tile-6" ng-repeat="tile in $ctrl.tiles" class="tile ng-binding ng-scope tile_1_2 tile-orange" ng-class="tile.classes" draggable="true" ondragstart="tile_dragstart(event);" ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                Tile 06
            </div><!-- end ngRepeat: tile in $ctrl.tiles -->
            <div id="tile-7" ng-repeat="tile in $ctrl.tiles" class="tile ng-binding ng-scope tile_1_1 tile-gray" ng-class="tile.classes" draggable="true" ondragstart="tile_dragstart(event);" ondrop="tile_drop(event);" ondragover="tile_dragover(event);">
                <div class="hovermenu"><i class="fa fa-ellipsis-h" aria-hidden="true"></i></div>
                Tile 07
            </div><!-- end ngRepeat: tile in $ctrl.tiles -->
        </div>
    </div>
Drag and Drop
Below the markup, there are JavaScript functions to handle drag and drop.
    <script lang="javascript">
        // Handler for start of tile drag.
        function tile_dragstart(ev) {
            // Add the drag data
            ev.dataTransfer.dropEffect = "move";
            ev.dataTransfer.setData("text/plain", ev.target.id);
        }

        // Handler for tile dragover.
        function tile_dragover(ev) {
            ev.preventDefault();
            // Set the dropEffect to move
            ev.dataTransfer.dropEffect = "move";
        }

        // Handler for tile drop.
        function tile_drop(ev, tiles) {
            ev.preventDefault();

            var sourceTileIndex = parseInt(ev.dataTransfer.getData("text").substring(5)) - 1;
            var destTileIndex = parseInt(ev.target.id.substring(5)) - 1;

            var ctrl = angular.element('#dashboard').scope().$$childHead.$ctrl;
            ctrl.dragTile(sourceTileIndex, destTileIndex);
        }
    </script>
</div>
There are a number of ways to do drag-and-drop, including Angular-specific approaches. Here, I've chosen to use what I'm used to: direct HTML5 drag-and-drop. To make this work, there are events for dragstart (start of drag), dragover (mid-drag), and drop (end of action). The start drag and drop functions are passed the current item, which is a tile It's in the drop function that a function in the controller is invoked to rearrange the items in the tiles array.

  • In tile_dragstart, the id of tile that is being dragged can be identified via event.target.id. The id is saved as data for the drag-and-drop operation, where it will be needed later. The drag cursor is set to a move cursor.
  • In tile_dragover, the drag is in progress. The drag cursor is again set to a move cursor.
  • In tile_drop, we are ready to rearrange the layout. The saved source tile id is retrieved. The drop target--that is, the destination tile's id--is found at event.target.id. Because the tiles have ids like tile-1, tile-2, etc., the tile index in the controller's tiles array can be found by stripping away the first 5 characters of the id and subtracting one.
Lastly, we find the controller object in tile_drop and invoke its dragTile method. The dragTile method uses JavaScript's Array.splice method to remove the source tile and insert it where it was dropped. The $apply method is called to update the Angular scope, which triggers automatic updating of the template view. When the model changes, Angular takes care of updating dependent views automatically. Voila, the dashboard is redrawn in its new layout. Because the elements in the tiles array have been shuffled, the internal ids (tile-1, tile-2, etc.) are re-generated.
          self.dragTile = function (sourceTileIndex, destTileIndex) {
              try {
                  var sourceTile = self.tiles[sourceTileIndex];

                  self.tiles.splice(sourceTileIndex, 1);  // remove source tile from tile list

                  if (sourceTileIndex < destTileIndex) {
                      destTileIndex--;
                  }

                  self.tiles.splice(destTileIndex, 0, sourceTile); // insert dest tile

                  // Regenerate tile ids.
                  var tile;
                  if (self.tiles != null) {
                      for (var t = 0; t < self.tiles.length; t++) {
                          tile = self.tiles[t];
                          tile.id = 'tile-' + (t + 1).toString(); // tile id (tile_1, tile_2, ...)
                      }
                  }

                  var scope = angular.element('#dashboard').scope();
                  scope.$apply(this);
              }
We'll see all of this in action shortly.

dashboard.less

We have one last file to examine, our CSS rules.
/* Dashboard Styles */

    /* panel of tiles */

    .dashboard-panel {
        width: 100%;
        height: 100%;
        padding: 16px;
    }

    /* tile */

    .tile {
        display: inline-block;
        position: relative;
        width: 200px;
        height: 200px;
        top: 0;
        margin-top: 0; 
        margin-right: 16px;
        margin-bottom: 16px;
        padding: 16px;
        background-color: #DDDDDD;
        color: white;
        font-family: roboto, Arial, Helvetica, sans-serif;
        font-size: 18px;
    }

    /* 1x1 tile */

    .tile_1_1 {
        width: 200px !important;
        height: 200px !important;
    }

    /* 2x1 tile*/

    .tile_2_1 {
        width: 448px !important;
        height: 200px !important;
    }

    /* 1x2 tile */

     .tile_1_2 {
        width: 200px !important;
        height: 420px !important;
    }

     /* 2x2 tile */

    .tile_2_2 {
        width: 448px !important; 
        height: 420px !important;
    }

    /* tile colors */

    .tile-gray {
        background-color: #DDDDDD !important;
        color: black !important;
    }

    .tile-red {
        background-color: #E60017 !important;
    }

    .tile-orange {
        background-color: #DB552C !important;
    }

    .tile-blue {
        background-color: #009CCC !important;
    }

    .tile-green {
        background-color: #339947 !important;
    }

    .tile-yellow {
        background-color: #FBFD52 !important;
        color: black !important;
    }

     .tile-purple {
        background-color: #5C197B !important;
    }

     .tile-container {
        display: inline-block;
     }

    .hovermenu {
        position: absolute;
        right: 0;
        top: 0;
        display: block;
        width: 24px;
        height: 24px;
        vertical-align: middle;
        text-align: center;
        color: transparent;
    }

    .hovermenu:hover {
        display: block;
        color: white;
    }

    .bigtext { 
        font-size: 72px;
    }

The CSS rules are very simple right now:

  • dashboard-panel : applied to the div containing the dashboard tiles
  • tile : applied to every tile
  • tile_1_1, tile_1_2, tile_2_1, tile_2_2 : styles that change the size of a tile.
  • tile_red, tile_yellow, tile_blue, etc. : styles that set a tile's color
  • hovermenu : style that makes the tile menu ellipsis (...) apper when hovered over

Running the Code

Now we're ready to try things out. When we run the above code, we get a beginning dashboard rendered.
Dashboard - desktop size window

Let's review what happened when the index.html page was loaded:
  1. The browser loaded the referenced CSS and JavaScript files, including AngularJS and our code for the dashboard component.
  2. The app.module.js code registered the Angular module and dashboard component .
  3. The markup in the page was rendered as a dashboard component by Angular. This included using the dashboard.template.html as a template and executing its directives. The model data in the dashboard controller furnished the dashboard title and tile definitions, which were rendered in a loop by the ng-repeat directive.
True, the tiles are just placeholder rectangles. The dashboard doesn't actually convey any useful information yet. Still, there's a respectable amount of base functionality now in place:
  • A simple JavaScript item collection is rendered as a dashboard
  • Tiles are rendered in the size and color of their definition
  • Whether on a large or small screen, the tiles will responsively arrange themselves to fit
  • Tiles can be dragged and dropped to rearrange the layout
Responsiveness
There's more work to be done to make this a great responsive dashboard, but the tiles across will fill based on available screen size. If we make the browser window smaller to approximate a smaller device such as a tablet, the tile layout changes to fit.
Dashboard - tablet size window
Drag and Drop
Let's test our drag-and-drop functionality. We'll begin dragging tile 4, the wide blue one at the top right, to the left. As we drag, we get a ghost image of the source tile.
Dashboard during Drag
We'll release the drag over tile 2, the yellow tile second from left. When we do, the dashboard rearranges itself. The drop handler invokes the dashboard controller's dragTile function, which removes and inserts the blue tile in the tiles array. When the code applies its changes to the Angular scope, the page is automatically updated with the new tile layout.
Dashboard after Drop
Changing the Dashboard
Thanks to way this is all wired together with Angular, it's quite simple to change the dashboard. If we modify the tile definitions in the controller's tiles array and refresh the page, we'll get a very different dashboard:
          self.title = 'Events Dashboard';
          self.tiles = [
              { width: 1, height: 1, color: 'green',    title: 'Revenue-New York' },
              { width: 1, height: 1, color: 'green', title: 'Revenue-Los Angeles' },
              { width: 1, height: 1, color: 'red',  title: 'Expenses-New York' },
              { width: 1, height: 1, color: 'red',   title: 'Expenses-Los Angeles' },
              { width: 2, height: 2, color: 'blue', title: 'Location Ranking' },
              { width: 1, height: 2, color: 'purple', title: 'Employee Performance' },
              { width: 1, height: 1, color: 'orange',   title: 'Problem Reports' }
          ];
Dashboard with new definitions

Summary

My dashboard is on its way, and doing it in Angular seems to be proceeding well. Angular is giving us a nice separation of concerns. Implementing the dashboard as a component has made it easy to add a dashboard to a page, simply by adding a <dashboard> element.

Much of Angular is elegant, but like any web framework there are areas that are non-obvious, difficult, or contain gotchas. In this lesson, I experienced two of these:
  • It took me a while to find a way to make drag and drop work in concert with the Angular controller, because I wasn't sure how to get to it. What I have now is working, but I'll be looking for a better way to handle that at some point that feels less kludgey.
  • Setting styles for tiles took some time to figure out, until I started using ng-class insted of class.
Next up, we'll be implementing a number of tile types for real. We'll also need to implement a back end for authentication and data operations, and tile actions such as add / configure / delete. See you next time!

Download Source

dashboard_01.zip
https://drive.google.com/open?id=0B4734rvcufGGUTVfT2Fqa0E1dVU

Next: AngularJS Dashboard, Part 2: Counter, Table, Donut Chart, and Column Chart Tiles

Sunday, March 19, 2017

Google Pixel XL Phone Review, Part 2: Google Messes Up Fulfillment

This is Part 2 in a blog series about my new Google Pixel XL phone. In Part 1, I described the ordering process, which was fine except for the long 7-week wait for the phone. I fully expected the next post in this series to be about unpacking and configuring the phone, but unfortunately I need to first post about the fulfillment experience which has not been good.

Original Order

To recap my first post, I ordered my Pixel XL on 2/13/17 and was told to expect it around 4/04/17. I'd ordered with Google Financing over 2 years. I was accepted into Google Financing at the time I placed my order, and received confirming paperwork the following week. All set! All I had to do now was wait until April to get my phone.

Order confirmation, which clearly shows payment method Google Financing

A Problem With Your Order

On March 16, 4 weeks in on my 7-week wait, I received an email from Google Store saying that there was a problem with my order. There was no explanation of what the problem was, but the message did have a Fix It button.

Order on Hold Message

Eager to resolve any issue with getting my phone, I clicked Fix It. This took me to a Google Payments screen which wanted me to update my payment method to an updated credit or debit card. This was odd, because my original order had nothing to do with a credit or debit card: it was made through Google Financing. However, I do have my Google account on a debit card for miscellaneous purchases such as Google Music. After confirming that I was in fact dealing with a real Google web site, I cooperated in updating my Google account payment method to a current debit card and completed the form.

And I guess this was the wrong thing to do, as you'll see in the next section. I want to point out though that this interaction 1) Google provided no other recourse for "correcting the problem with my order" and 2) certainly did not did not indicate that any kind of charge was about to be made.

Unauthorized Debit of Full Price of Phone

At this point I was suspicious. Had I just changed the payment method for my phone order? I went back to call up the status of my original order. It still indicated that payment was through Google Financing, so I relaxed.

Order Check, Confirming that Google Financing was Still Method of Payment

But just a few seconds later, I received an alert from my bank that $968.67 had been debited from my bank account! Wait, what!?!

The unauthorized debit

This was unbelievable. Google had TRICKED ME into paying up front for a phone that had already been ordered and approved for financing--and they did it without telling me what was going to happen.

Your Order Has Shipped

On the next day, 3/17/17, I received an email that my order had shipped (unexpected, because original delivery expectations had been set for April). Delivery of the phone was now promised by 3/23/17. It was nice my phone was coming sooner than originally indicated... but I had been charged nearly $1000 up-front for a phone I had ordered with financing, and that wasn't nice at all.

Interactions with Google Support

Rather upset at this series of events, I proceeded to send a support message to Google. 

My complaint email

The support form promised a reply within 24 hours. 24 hours went by with no reply, but I did receive a reply the following day on 3/18/17.

First response from Google Support - didn't understand or address my issue

Unfortunately, it appears the support person did not even read the entire message. I was assured that my order was not held up any longer (still no explanation has ever been given about what the problem was) and would arrive soon. Nothing about the unauthorized charge or resolving it. I checked with my bank, and the charge had not been reversed.

I responded that I expected this unauthorized charge to be reversed, and wanted to pay for my phone the way it had been ordered, through Google Financing.

My first response, making it clear what action was needed

The Charge Came From Synchrony Bank

The next day (3/19/2017), I received another reply from Google. They said they had looked into the matter, and the bank charge actually came from their financing partner, Synchrony Bank. I would have to take the matter up with Synchrony, because Google had no ability to correct matters directly. They provided basic contact information for Synchrony.

Google Support's second response, saying the charge came from Synchrony

This was disappointing, to say the least. Now I would have to sort out a problem with Google's financing partner? When you are setting up a retail experience, it's fine to have partners but you should take responsibility for providing customers with a seamless experience. That includes managing customer service problems, even if they involve your partners. You wouldn't get this kind of treatment from Amazon: they are famous for stellar customer service.

Sigh. I proceed to the Synchrony bank Contact Us page at https://www.synchronyfinancial.com/contact-us.html, where I see Search or Select a Business. Many companies are listed here. Amazon is listed here. Walmart is listed here. Do you know who isn't listed here? Google.

Synchrony web site contact form - no option for Google

I select Other. Which only gave me an option to call by phone. Up till now, all my interactions has been through email so there would be a record of the communication, but that's not an option here. So, I pick up the phone.

For the next couple of hours, I am on the phone with Synchrony or Google as they in turn transfer me over to the other party.

  • Lavinia at Synchrony doesn't really seem to understand my problem. She assures me any problem must be a Google problem. She transfers me over to Google.


  • Jill at Google does try to help--she spends a lot of time on the phone with me, but ultimately concludes that it is Synchrony who must reverse the charge. I get transferred back to Synchrony.


  • Tina at Synchrony is of no help whatsoever. She doesn't seem to understand anything about the Google-Synchrony relationship or how to resolve issues.

Synchrony's support people are, well, completely useless. They are 100% clueless. Google's support people do seem to be making an effort, but appear to be powerless to actually correct the issue. All of them assure me they want to help me, but each time I end up getting sent back to the other company.

I have now had enough. Apparently no one is able to correct this, and it is extremely frustrating. I decide to accept the situation: my phone is coming, the logistics were screwy, the financing didn't really materialize. It's no longer worth my time. I don't really know who was at fault here but I know it wasn't me. I may just get my next phone from Apple the next time around.

Google Pixel Scorecard

So far, I have to grade my Google Pixel XL experience as follows:


Google gets an F for the order fulfillment experience. This is disappointing to say the least. I like Google. But this has been anything but pleasant. It shows a lot of immaturity when it comes to inventory logistics, reliable order fulfillment processes, and good customer service. Although I received much better support attention from Google than Synchrony, this is ultimately Google's failure because they own the product/process/partnership: it doesn't really matter if the problem was with Google or Synchrony: the responsibility lies with Google and they blew it.

Hopefully my experience with the actual phone (coming next week) will be much better. Stay tuned, I'll let you know.


Friday, March 10, 2017

Google Pixel XL Phone Review, Part 1: The Long Wait

As the new owner of a Google Pixel XL Android phone, I'll be writing a series of posts to review my experiences with Google's new flagship handset. In this first post, I'll... be talking about the ordering experience, because I don't actually have the phone yet. Mostly I'll be complaining about the incredibly looooooong wait for the phone: I ordered it on February 13th, and am expecting to receive it on April 6th. That's a long 7 weeks of waiting. As I write this, I "only" have 4 weeks to go. Talk about agonizing. What I can do in first this post is discuss why I chose this phone (despite the long wait), and what I've learned about it. 
I won't go into the reasons for the delay, since I don't really know them. Google says it's due to high demand. Others speculate it's due to poor planning and management. Since this isn't Google's first phone, it is hard to explain.

Why the Pixel?


Why this phone? Well, my previous two phones have been the Moto X (first generation) and Moto X (second generation), which are also Android phones. I liked the Moto X a great deal, especially the "Moto actions" and other nice touches: I can just wave my hand above the phone to wake it up; I get battery-efficient notification icons; and the phone knows if I am looking at it and won't go to sleep. I also liked that I could design my own customized phone, choosing front (white), back (leather, engraved), and accent (silver). So why not just stay with Moto? Well, I was thrilled when Google bought Motorola Mobility; unfortunately, they've since sold it off to Lenovo. The current Moto phones all seem to have a camera bump, which is something of a deal-breaker for me: I want my phone to lie flat on the desk.

So, I looked around. For a variety of reasons I'm not a big fan of Samsung and LG given past experiences. I'm also quite weary of all the bloatware carriers tend to install on Android phones, often apps that can't be deleted. When Google announced the Pixel it made a lot of sense: no carrier bloatware, a finely-tuned Android experience, high-end hardware, and the best possible integration with Google services. Yes, it's pricey--but it seems like the best choice given what I value (pure Android experience, good hardware, backed by a company I have some respect for). And so, I placed my order and am now waiting (impatiently). In the meantime, I've read dozens of reviews by others so I have a pretty good idea of what to expect.

Having settled on a Pixel, I chose the larger XL model (5.5" screen) - Google's answer to the iPhone Plus. This is a bit of a departure for me, but I'm thinking the larger screen will be easier on the eyes. My wife got an iPhone Plus recently, and it took her about a week to get used to the larger device. I also went with the 128GB storage, because one of my few complaints about the Moto line was that I'm always running low on storage space. As for the color, my first choice would have been silver and after that blue; alas, almost everything was out of stock. When I finally could order something, it was the black. No matter, I'll be putting a skin on it.

The Ordering Experience

The ordering experience was simple enough. Since I'm not going through Verizon (poor coverage where I live, plus refusal to put up with any more carrier bloatware), and since the Pixel isn't available through any other carriers, that means I'm getting it directly from Google. The Google Store ordering experience is what you'd expect from Google: it's simple, and easy to use. Except...

Except that I couldn't actually order one. What's odd about the ordering experience is the backlog, and how that's being handled. When I first visited the Google Store in January, they were sold out, at least in the configurations I was interested in. And, you couldn't pre-order the phone. All you could do is get added to a waitlist (an email would come your way when a phone was available for order). After a few weeks, now in February, I still could not order the phone I wanted. Tired of this, I decided to cave on color, and ordered what was available: a black 128GB Pixel XL.

The Google Store shows you a Pixel and lets you change the base unit (Pixel or Pixel XL), color, and storage. As you change the configuration, you can see the phone from front, side, and back perspectives.


Google Store - Pixel

Here's the unit I ended up ordering: a Black Pixel XL 128GB.


Google Store - Pixel XL

Finally, my phone was ordered--but I was still in for a looong wait. A 7 week wait.

What I'll Like about the Pixel XL

👍 Free Unlimited Cloud Backup of Photos and Videos

The Pixel gives you unlimited storage of images and videos in Google Photos (cloud storage). Android users normally get this feature anyway, but not at full quality. For Pixel owners, they are archived at full quality.

👍 Great Battery Life


The battery life is reportedly very good. There's also fast charging: you're supposed to be able to get 7 hours of battery life from 15 minutes of charging.

👍 Speed

The Pixel's speed is reported as fantastic by reviewers - even after the honeymoon period. This is due to a fast processor plus a really well-tuned edition of Android. This is the whole point of the Pixel line: with Google in control of both the hardware and software, the phone really sings.

👍 Great Android Experience

Reviewers are also in agreement that the Android experience is first-rate. There are all sorts of improvements and refinements to be found, some important and some just to be more Apple-like. Given that the rest of my household is on Apple phones, being more Apple-like won't be a bad thing. 

  • New Pixel Launcher.
  • Round icons. 
  • Long-press to get application shortcuts. 
  • Swipe gestures, such as swiping up to get the app tray. Swipe down on the rear fingerprint sensor to see notifications, swipe back up to dismiss.
Gestures

👍 Camera

Google is advertising the best camera available for a smartphone. Reviews either agree or call it a close second to the iPhone 7. Either way, I'm sure to be happy with this camera.

What I Probably Won't Like About the Pixel XL

👎 Don't Get It Wet

One ding against the Pixel is it isn't waterproof. That's mostly a big deal just because its becoming a standard on high-end phones. For the high price, the Pixel should also be waterproof.

👎 Unique Features that No Longer Are

The features that are supposed to be unique to the Pixel are rapidly become available anywhere. You can install the Pixel launcher on other Android phones. As of this week, the Google Assistant is no longer exclusive to just the Pixel.

👎 Pixel 2 Rumors Already Circulating

Each week, more details are coming to light about the Pixel 2, which will apparently be released near the end of 2017. Meanwhile, I'm still waiting for my first-gen Pixel.

👎 Avoiding Damage

Although the Pixel is reportedly well-made, it also seems prone to picking up scratches, especially on the rear glass panel. I'll admit it, I'm one of those people who prefer to use their phone without a case. This of course has bitten me in the past: my first Moto X cracked when I dropped it just a month in; with my Moto X second gen I was ultra-careful and avoided any mishaps. 

Given the expense of the Pixel, I'm going to give in and get some protection. I've decided to go with a skin from ColorWare, which I recently ordered and will receive in a couple of weeks. It's Techno Blue, and looks like this. Of course, I'm told these skins can also get scratched so I'll still need to be careful.
Pixel XL skin by ColorWare

Well, I think that's all I can say for now. I'll share more when I actually get the thing. In the meantime, I'll be practicing how to be more patient. I hear it's a virtue.

Next: Pixel XL Phone Review, Part 2: Google Messes Up Fulfillment

Friday, March 3, 2017

Searching Blob Documents with the Azure Search Service

One of the core services in the Microsoft Azure cloud platform is the Storage Service, which includes Blobs, Queues, and Table storage. Blobs are great for anything you would use a file system for, such as avatars, data files, XML/JSON files, ...and documents. But until recently, documents in blob storage had one big shortcoming: they weren't searchable. That is no longer the case. In this post, we'll examine how to search documents in blob storage using the Azure Search Service.

Azure Blob Basics

Let's quickly cover the basic of Azure blobs: 

  • Storage Accounts. To work with storage, you need to allocate a storage account in the Azure Management Portal. A storage account can be used for any or all of the following: blob storage, queue storage, table storage. We're focusing on blob storage in this article. To access a storage account you need its name and a key.
  • Containers. In your storage account, you can create one or more named containers. A container is kind of like a file folder--but without subfolders. Fortunately, there's a way to mimic subfolders (you may include slashes in blob names). Containers can be publicly accessible over the Internet, or have restricted access that requires an access key.
  • Blobs. A blob is a piece of data with a name, content, and some properties. For all intents and purposes, you can think of a blob as a file. There are actually several kinds of blobs (block blobs, append blobs, and page blobs). For our purposes here, whenever we mention blob we mean block blob, which is the type that most resembles a sequential file.

Uploading Documents to Azure Blob Storage

Let's say you had the chapters of a book you were writing in Microsoft Word that you save as pdf files--ch01.pdf, ch02.pdf, ... up to ch10.pdf, along with toc.pdf and preface.pdf--which you would like to store in blob storage and be able to search. Here's an example of what a page of this book chapter content looks like:

In your Azure storage account you can create a container (folder) for documents. In my case, I created a container named book-docs to hold my book chapter documents. In the book-docs container, you can upload your documents. If you upload the 12 pdf documents described above, you'll end up with 12 blobs (files) in your container. 

Structure of Azure Storage showing a Container and Blobs

To upload documents and get at your storage account, you'll need a storage explorer tool. You can either use my original Azure Storage Explorer or Microsoft's Azure Storage Explorer. We'll use Microsoft's explorer in this article because it has better support for one of the features we need, custom metadata properties. After downloading and launching the Storage Explorer, and configuring it to know about our storage account, this is what it looks like after creating a container and uploading 12 blobs.

12 pdf documents uploaded as blobs


Setting Document Properties

It would be nice to search these documents not only based on content, but also based on metadata. We can add metadata properties (name-value pairs) to each of these blobs. In the Microsoft Azure Storage Explorer, right-click a blob and select Properties. In the Properties dialog, click Add Metadata to add a property and enter a name and value. We'll later be able to search these properties. In my example, we've added a property named DocType and a property named Title to each document, with values like "pdf" and "Chapter 1: Cloud Computing Explained".

Blob with several metadata properties

Azure Search Basics

The Azure Search Service is able to search a variety of cloud data sources that include SQL Databases, DocumentDB, Table Storage, and Blob Storage (which is what we're interested in here). Azure Search is powered by Lucene, an open-source indexing and search technology. 

Azure Search can index both the content of blob documents and metadata properties of blobs. However, content is only indexable/searchable for supported file types: pdf, Microsoft Office (doc/docx, xls/xlsx, ppt/pttx), msg (Outlook), html, xml, zip, eml, txt, json, and csv. 

To utilize Azure Search, it will be necessary to create three entities: a Data Source, an Index, and an Indexer (don't confuse these last two). These three entities work together to make searches possible.

  • Data Source: defines the data source to be accessed. In our case, a blob container in an Azure storage account.
  • Index: the structure of an index that will be filled by scanning the data source, and queried in order to perform searches.
  • Indexer: a definition for an indexing agent, configured with a data source to scan and an index to populate.

These entities can be created and managed in the Azure Management Portal, or in code using the Azure Search REST API, or in code using the Azure Search .NET API. We'll be showing how to do it in C# code with the .NET API.


Installing the Azure Search API Package

Our code requires the Azure Search package, which is added using nuget. In Visual Studio, right-click your project and select Manage Nuget Packages. Then find and install the Microsoft Azure Search Library.You'll also need the Windows Azure Storage Library, also installed with nuget.

At the top of our code, we'll need using statements for a number of Microsoft.Azure.Search and Microsoft.WindowsAzure namespaces, and some related .NET namespaces:

using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Azure.Search;
using Microsoft.Azure.Search.Models;
using Microsoft.Azure.Search.Serialization;
using Newtonsoft.Json;
using System.ComponentModel.DataAnnotations;
using System.Web;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;

Creating a Search Service

The first step in working with Azure Search is to create a search service using the Azure Management Portal. There are several service tiers you can choose from, including a free service tier which will let you play around with 3 data sources and indexes. To work with your search service in code, you'll need its name and an API admin key, both of which you can get from the management portal. We'll be showing a fake name and key in this article, which you should replace with your actual search service name and key.

Creating a Service Client

To interact with Azure Search in our code, we need to first instantiate a service client, specifying the name and key for our search service:

string searchServiceName = "mysearchservice";
string searchServiceKey = "A65C5028BD889FA0DD2E29D0A8122F46";

SearchServiceClient serviceClient = new SearchServiceClient(searchServiceName, new SearchCredentials(searchServiceKey));

Creating a Data Source

To create a data source, we use the service client to add a new DataSource object to its DataSources collection. You'll need your storage account name and key (note this is a different credential from the search service name and key in the previous section). The following parameters are defined in the code below:

  • Name: name for the data source.
  • Type: the type of data source (AzureBlob).
  • Credentials: storage account connection string.
  • Container: identifies which container in blob storage to access.
  • DataDeletionDetectionPolicy: defines a deletion policy (soft delete), and identifies a property (Deleted) and value (1) which will be recognized as a deletion. Blobs with property Deleted:1 will be removed from the index. We'll explain more about this later.
String datasourceName = "book-docs";
if (!serviceClient.DataSources.Exists(datasourceName))
{
  serviceClient.DataSources.Create(new DataSource()
  {
    Name = datasourceName,
    Type = Microsoft.Azure.Search.Models.DataSourceType.AzureBlob,
    Credentials = new DataSourceCredentials("DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=GL3AAN0Xyy/8nvgBJcVr9lIMgCTtBeIcKuL46o/TTCpEGrReILC5z9k4m4Z/yZyYNfOeEYHEHdqxuQZmPsjoeQ=="),
    Container = new Microsoft.Azure.Search.Models.DataContainer(datasourceName),
DataDeletionDetectionPolicy = new Microsoft.Azure.Search.Models.SoftDeleteColumnDeletionDetectionPolicy() {  SoftDeleteColumnName="Deleted", SoftDeleteMarkerValue="1" }
});
}

With our data source defined, we can move on to creating our index and indexer.

Creating an Index

Next, we need to create the index that Azure Search will maintain for searches. The code below creates an index named book. It populates the Fields collection with the fields we are interested in tracking for searches. This includes:

  • content: the blob's content.
  • native metadata fields that come from accessing blob storage (such as metadata_storage_name, metadata_storage_path, metadata_storage_last_modified,  ...). 
  • custom metadata properties we've decided to add: DocType, Title, and Deleted.
Once the object is set up, it is added to the service client's Indexes collection, which creates the index.

String indexName = "book";
Index index = new Index()
{
    Name = indexName,
    Fields = new List<Field>()
};

index.Fields.Add(new Field() { Name = "content", Type = Microsoft.Azure.Search.Models.DataType.String, IsSearchable = true });
index.Fields.Add(new Field() { Name = "metadata_storage_content_type", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "metadata_storage_size", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "metadata_storage_last_modified", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "metadata_storage_content_md5", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "metadata_storage_name", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "metadata_storage_path", Type = Microsoft.Azure.Search.Models.DataType.String, IsKey = true, IsRetrievable = true , IsSearchable = true});
index.Fields.Add(new Field() { Name = "metadata_author", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "metadata_language", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "metadata_title", Type = Microsoft.Azure.Search.Models.DataType.String });
index.Fields.Add(new Field() { Name = "DocType", Type = Microsoft.Azure.Search.Models.DataType.String, IsSearchable = true });
index.Fields.Add(new Field() { Name = "Title", Type = Microsoft.Azure.Search.Models.DataType.String, IsSearchable = true });

if (serviceClient.Indexers.Exists(indexName))
{
    serviceClient.Indexers.Delete(indexName);
}
serviceClient.Indexes.Create(indexName)

Let's take note of some things about the index we're creating:
  • Some of the fields are built-in from what Azure Search intrinsically knows about blobs. This includes content and all the properties beginning with "metadata_". Especially take note of metadata_storage_path, which is the full URL of the blob. This is marked as the key of the index. This will ensure we do not receive duplicate documents in our search results.
  • Some of the fields are custom properties we've chosen to add to our blobs. This includes DocType and Title.

Creating an Indexer

And now, we can create an indexer (not to be confused with index). The indexer is the entity that will regularly scan the data source and keep the index up to date. The Indexer object identifies the data source to be scanned and the index to be updated. It also contains a schedule. In this case, the indexer will run every 30 minutes. Once the indexer object is set up, it is added to the service client's Indexers collection, which creates the indexer. In the background, the indexer will start running to scan the data source and populate the index. It's progress can be monitored using the Azure Management Portal.

String indexName = "book";
String indexerName = "book-docs";
Indexer indexer = new Indexer()
{
    Name = indexerName,
    DataSourceName = indexerName,
    TargetIndexName = indexName,
    Schedule = new IndexingSchedule()
    {
        Interval = System.TimeSpan.FromMinutes(30)
    }
};
indexer.FieldMappings = new List();
indexer.FieldMappings.Add(new Microsoft.Azure.Search.Models.FieldMapping()
{
    SourceFieldName = "metadata_storage_path",
    MappingFunction = Microsoft.Azure.Search.Models.FieldMappingFunction.Base64Encode()

});

if (serviceClient.Indexers.Exists(indexerName ))
{
    serviceClient.Indexers.Delete(indexerName );
}
serviceClient.Indexers.Create(indexer);

Let's point out some things about the indexer we're creating:

  • The indexer has a schedule, which determines how often it scans blob storage to update the index. The code above sets a schedule of every 30 minutes.
  • There is a field mapping function defined for the metadata_storage_path field, which is the document path and our unique key. Why do we need this? Well, it's possible this path value might contain characters that are invalid for an index column; to avoid failures, it is necessary to Base64-encode the value. We'll have to decode this value whenever we retrieve search results.
Putting this all together, when we run the sample included with this article it takes around half a minute to create the data source, index, and indexer. The index is initially empty, but the indexer is already running in the background and will be ready for searching in about a minute.


Creating data source, index, and indexer

Searching Blob Documents

With all of this set up underway, we're finally ready to do searches.

BlobDocument Class

As we perform searches, we're going to need a class to represent a blob document. This class needs to be aligned with how we defined our index. Our sample uses the BlobDocument class below.

[SerializePropertyNamesAsCamelCase]
public class BlobDocument
{
    [IsSearchable]
    public String content { get; set; }

    [IsSearchable]
    public String metadata_storage_name { get; set; }

    [Key]
    [IsSearchable]
    public String metadata_storage_path { get; set; }

    public String metadata_storage_last_modified { get; set; }

    public String metadata_storage_content_md5 { get; set; }

    public String metadata_author { get; set; }

    public String metadata_content_type { get; set; }

    public String metadata_language { get; set; }

    public String metadata_title { get; set; }

    [IsSearchable]
    public String DocType { get; set; }

    public String Deleted { get; set; } // A value of 1 is a soft delete

    [IsSearchable]
    public String Title { get; set; }
}

Simple Searching

A simple search simply specifies some search text, such as "cloud". 

Up until now we've been using a Service Client to set up search entities. To perform searches, we'll instead use an Index Client, which is created this way:

String indexName = "book";
ISearchIndexClient indexClient = serviceClient.Indexes.GetClient(indexName);

To perform a search, we first define what it is we want to return in our results. We'd like to know the blob document URL, its content, as well as the two custom metadata properties we defined for our blob documents, DocType and Title.

parameters = new SearchParameters()
{
    Select = new[] { "content", "DocType", "Title", "metadata_storage_path" }

};

We call the index client's Documents.Search method to perform a search for "cloud" and return results.

String searchText = "cloud";
DocumentSearchResult<BlobDocument> searchResults = indexClient.Documents.Search(searchText, parameters);

Search Results

The result of our search is a DocumentSearchResult<BlobDocument> object. We can iterate through the results using a loop. When we defined our index earlier, we had to give Azure Search instructions to Base64-encode the metadata storage path field when necessary. As a result, we now need to decode the path.

foreach (SearchResult<BlobDocument> result in searchResults.Results)
{
    Console.WriteLine("---- result ----");
  
    String path = result.Document.metadata_storage_path;
    if (!String.IsNullOrEmpty(path) && !path.Contains("/"))
    {
        path = Base64IUrlDecode(result.Document.metadata_storage_path);
    }
    Console.WriteLine("metadata_storage_path: " + path);
    Console.WriteLine("DocType: " + result.Document.DocType);
    Console.WriteLine("Title: " + result.Document.Title);
}

The path is part of the result that matters the most, because if a user is interested in a particular search result this lets them download/view the document itself. Note this is only true if your blob container is configured to permit public access.

Now that we have enough code to perform a search and view the results, let's try some simple searches. For starters, we search on "pdf". That is not a term that appears in the content of any of the documents, but it is a value in the metadata: specifically, the property Title that we added to each blob earlier. As a result, all 12 documents match:

Search for "pdf" - 12 matches to document metadata

Now, let's try a search term that should match some of the content within these documents. A search for "safe" matches 3 documents:

Search for "safe" - 3 matches to document content


More Complex Searches

We can use some special syntax in our search query to do more complex queries.

To perform an AND between two search terms, use the + operator. The query storage+security will only match documents that contain both "storage" and "security".


AND query

To perform an OR between two search terms, use the | operator. The query dangerous|roi will match documents containing "dangerous" or "ROI".


OR query

In a future post, we'll explore how to perform advanced searches. 

Deleting Documents

Normally, deleting a blob involves nothing more than selecting it in a storage explorer and clicking Delete (or doing the equivalent in code). However, with an Azure Search index it is a little more complicated: if you just summarily delete a blob that was previously in the index, it will remain in the index: the indexer will not realize the blob is now gone. This can lead to search results being returned about documents that no longer exist.

We can get around this unpleasantness by utilizing a soft delete strategy. We will define a property that means "deleted", which we will tell Azure Search about. In our case, we'll call our property Deleted. A "soft delete" will cause the blob to be removed from the index when Deleted:1 is encountered by the indexer--after which it is safe to actually delete the blob. You might consider having an overnight activity scheduled that deletes all blobs marked as deleted.

Summary

With Azure Search, documents in Azure Blob Storage are finally searchable. Using Azure Search gives you the power of Lucene without requiring you to set up and maintain it yourself, and it has the built-in capability to work with blob storage. Although there are a few areas of Azure Search that are cumbersome (notably, Base64 encoding of document URLs and handling of deleted documents), for the most part it is a joy to use: it's fast and powerful. 

You can download the full sample (VS2013 solution) here.