PolyCloud.Storage: A .NET API for Cloud Storage on AWS, Azure, and GCP

Today I'm announcing a new open source API library named PolyCloud.Storage.

PolyCloud.Storage is part of a bigger vision called the PolyCloud-API. The idea is this: as much as cloud platforms are ever striving to competitively differentiate themselves, it's simply a fact that there are many similarities between them. Take cloud storage for example: the SDKs may be different, and the terminology may vary (Azure containers and blobs equate to S3 buckets and objects), but you can basically treat Azure Blob Storage, AWS S3, or Google Cloud Storage like a file system, with folders and files. As it is becoming more and more common to see solutions that can be deployed to more than one cloud platform, the time seems ripe for a common set of API libraries that support the major cloud platforms. That's the idea behind PolyCloud-API.

PolyCloud-Storage is the initial offering in the PolyCloud-API series, and it starts out pretty modestly: with it you can do the following on AWS, Azure, or GCP:

• List folders
• Create a folder
• Delete a folder
• List files
• Upload a file
• Download a file
• Delete a file

Anyone who's played with a cloud storage system knows there's a great deal of other features available, such as versioning, random-access files, and security controls; where is all of that in PolyCloud-Storage? The plan is to introduce these features—where they exist on all of the target cloud platforms—in the days to come. When a feature is novel to just one cloud platform, we probably won't have it in PolyCloud.Storage since we'd like our library methods to work everywhere.

I started with storage because I have a lot of prior experience with it via the original Azure Storage Explorer, which had a whopping 280,000+ downloads in its time. Indeed, I was mulling over the idea of a new version of this tool when the idea for PolyCloud-API struck me. Do expect that you will see some tools coming from me that use PolyCloud.Storage (and other PolyCloud-API libraries as they become available).

This brings us to guiding principles for PolyCloud-API:

• Keep things simple and approachable
• Document the API
• Provide unit tests for the API
• Follow best practices for each cloud platform
• Provide options for error handling by the caller or by the library
• Collaboration = more libraries for more development languages

So, what's PolyCloud-Storage like to use? Here's a unit test for AWS S3:

Console.WriteLine("Test: AWS | CreateFolderAndFile");

// Set your storage account credentials here before running tests

String accessKey = "BKIAIBVVWJCZIPZQM2ZA";
String secretKey = "xQlSUFBSLr5x3FvyccBJy5rcD3e93Ftwzic18gQH";
String endpoint = "us-east-1";

using (Storage storage = Storage.AWS(accessKey, secretKey, endpoint))
{
    storage.HandleErrors = false;
    storage.Open();

    // Create folder

    String folderName = "test-" + Guid.NewGuid().ToString();
    Console.WriteLine("Creating folder " + folderName);
    storage.NewFolder(folderName);

    // Create file in folder

    String fileName = "test.txt";
    String fileName2 = "test2.txt";

    Console.WriteLine("Creating file " + fileName);

    String content = "This is a test.";

    if (File.Exists(fileName)) File.Delete(fileName);
    File.WriteAllText(fileName, content);

    // Upload file

    Console.WriteLine("Uploading file " + fileName);

    storage.UploadFile(folderName, fileName);

    // Retrieve file

    Console.WriteLine("Downloading file " + fileName + " as " + fileName2);

    if (File.Exists(fileName2)) File.Delete(fileName2);
    storage.DownloadFile(folderName, fileName, fileName2);

    // Delete file

    storage.DeleteFile(folderName, fileName);

    // Delete folder

    Console.WriteLine("Deleting test folder");
    storage.DeleteFolder(folderName);

    Console.WriteLine("Verifying content");

    String content2 = File.ReadAllText(fileName2);
    Assert.AreEqual(content, content2, "Downloaded file content did not match uploaded file content");

    storage.Close();
}

Hopefully you find the above code readily understandable; if you've worked with cloud storage SDKs, you may have experienced a mix of approachable and not-so-approachable APIs.

Now, here's the payoff: Let's imagine you wanted to do all of the above, but on Microsoft Azure. The only change you need to make is line 9 (plus different credentials). Instead of Storage.AWS to instantiate an S3 storage client, you use Storage.Azure to instantiate an Azure Blob client. All the other code remains unchanged.

String accountId = "my-blob-storage-account";
String accountKey = "Y7JTgm7B2g53t8MZ1N+GmuQ8d3aHyf0tGo5uglt7OIF3M2roaFiX6jPPI2mWmIOh/6suLEAMVXF5GPXPFl2nOQ==";

using (Storage storage = Storage.Azure(accountId, accountKey))

And as you should now be expecting, the only change needed for Google Cloud Platform is to use Storage.GCP to instantiate.

String jsonPath = @"C:\gcp-storage\helloworld-328219-ca4bb62a59c8.json";
String projectId = "918968012956";

using (Storage storage = Storage.GCP(jsonPath, projectId))

The .NET Framework edition is now up on GitHub at https://github.com/davidpallmann/polycloud-storage and a sister edition for .NET Core is at https://github.com/davidpallmann/polycloud-storage-netcore.

I do think this is a promising idea, and I'm excited about where it can go. I'd like to have some helpful collaborators provide matching libraries for other development languages such as Java. If you're interested, drop me a line.

10 Years In The Cloud: A Retrospective

I am celebrating 10 years of cloud computing work. This post looks back on a decade of cloud activity and where it has led.

2008-2009: Cloud Computing, the New Thing 

In late 2008, working at Microsoft Partner Neudesic, our CTO Tim Marshall and I were invited to a Microsoft feedback session in Redmond about "Project Red Dog". Red Dog, it turns out, was about this new thing called Cloud Computing. Amazon had been doing this for a few years, and Microsoft was going to also enter the market. This "cloud computing" was a new idea and a new way of doing things—but it sounded exciting. A few months later, "Windows Azure" was released. As Neudesic is a consulting company, we started learning it and looking for early prospects.

When Microsoft introduces a new product or service, a lot of work goes into evangelism and education and finding early adopters. As a Microsoft partner, we did a lot of joint work with Microsoft: visits to prospects, proof-of-concept projects, training sessions, code camps.

Tim had his own ideas about developing the market, and one of those was starting Azure user groups in the ten or so locations we had across the United States. I and other colleagues (including Mickey Williams and Chris Rolon) started sponsoring monthly meetings, sometimes held at Microsoft field locations. Since this was all new, meeting attendance could just as easily be 5 or 20 or 50 people, depending. But we kept at it, and we got the word out there, and interest started growing. At meetings we would cover new cloud services that had just become available, or show off things we had built, or discussed useful patterns for applications. It was fun, and there was pizza.

We learned things about the cloud: the infrastructure was really advanced, but the individual hardware components could fail: you had to plan for redundancy and recovery. The economics of the cloud were different: you had to consider lifetime of the data and resources you allocated, else you would "leave the faucet running". Almost everyone who was an early adopter had an Unexpectedly Large Cloud Bill story. Developers giggled with pleasure at the ease of self-deployment; but sometimes you'd hear a horror tale where someone lost important data all because they weren't careful enough when clicking in the management portal. We started reinforcing the importance of separating Production accounts from Development accounts.

2010-2014 : Azure Evangelism and Early Adopters

As Windows Azure was evangelized, prospects started to line up. I participated in a great deal of proof-of-concept project work, sometimes arranged by and paid for by Microsoft. One that stands out was going to Coca Cola headquarters in Atlanta to show how readily existing web sites could be migrated to Windows Azure. The first web site we migrated was in ASP.NET/SQL Server, which was a slam-dunk and just took a handful of days. The second site used Java Server Pages and Oracle—definitely not in my wheelhouse—but in two weeks' time we had migrated it as well.

I wrote The Windows Azure Handbook in 2010, which I believe was the first book out for Azure. The book contained Microsoft messaging from the time: Platform-as-a-Service (PaaS) is better than Infrastructure-as-a-Service (IaaS) and so on. Today Azure is equally well-suited for PaaS and IaaS and the message has changed. We've learned that there are those who value the cloud for innovative new ways of doing things (the PaaS people); but also those who value the ability to leverage existing skills and don't want their world rocked (the IaaS people).

I also released through Neudesic an Azure ROI calculator, long before there was a comprehensive one available from Microsoft. You can see from this screenshot how few cloud services there were in those early years. The number of cloud services available today is vast and ever-expanding.

There were real cloud projects happening too by this time. At first, there had been a lot of interest but prospects seemed hesitant to actually take the plunge. There was for example a great fear of vendor lock-in. Eventually, and with increasing rapidity, adoption started happening. The vast majority of these projects were web site & database migration for established companies; but start-ups had a different mentality, they wanted to do everything in the cloud from Day L.

As head of the Custom App Dev practice at Neudesic, I made sure we had Azure-trained consultants in every region. As new cloud services appeared, this interested our other practices. SQL Azure database and (later on) Power BI interested the SQL / Business Intelligence practice. Service Bus interested the Connected Systems practice.

Badges Awarded to Consulants Who Completed Cloud Training

Microsoft started a Windows Azure category of their Most Valuable Professional program, and I was honored to be a Microsoft MVP from 2010-2014. I met some great MVPs on my visits to Microsoft (and hired one, Michael Collier), along with the Windows Azure product team.

Although activity was intense, Windows Azure wasn't perfect. For three years in a row, Azure went down during the annual MVP summit, usually for reasons like somone having forgotten to renew a security certificate. We MVPs were initially amused, but in later years it meant customers were affected. AWS also seemed to have a hiccup as well once or twice a year. We started educating customers about what dependency on a cloud platform meant for reliability, and fallback plans for when the a region or entire platform was unavailable. Both platforms have improved in reliability since then.

In 2011 Microsoft asked me to teach Azure training sessions in Amsterdam and Germany. This was a fun trip—except for the blistering winter snowstorm—and I met some MVPs including Kris van der Mast and Christian Weyer. This helped me realize that cloud computing was a worldwide phenomenon, and also that different regions had different problems to address: in Europe, for example, there were laws about where clients' data had to be stored, and that didn't always align well with existing data centers.

My Azure class in Munich, Germany

As the years went by, Azure added more and more services and would occasionally drop support for a service (never popular). New data centers were continually added around the world.

Azure Storage Explorer

I created a free storage tool named Azure Storage Explorer and placed it on CodePlex, which turned out to be a hit. Over the next few years, Azure Storage Explorer had over 280,000 downloads! I would do a handful of updates a year to ASE, usually because Microsoft had added a new feature or because the Storage API had changed.

Eventually, there was one breaking API change too many and I stopped maintaining it--but made the source available on CodePlex. A second reason for not working on it is simply how busy I was on cloud projects.

A few years later, Microsoft finally came out with their own tool, with nearly the same name: Microsoft Azure Storage Explorer. You can also now manage storage through the Azure Portal. It's about time!

Recently I've had some thoughts about creating some new, updated cloud tools. See the end of this post for more.

2015-2019: The Maturing Cloud Becomes Essential

Cloud has exploded and is no longer something reserved for brazen early adopters or just a few specialists. At Neudesic, we consult widely on multiple cloud platforms: Microsoft Azure, Amazon Web Services, and now Google Cloud Platform.

New cloud services continue to arrive. There are services for Mobile and APIs and Non-Relational Databases and Distributed Memory Cache and Machine Learning. We now have Serverless Computing (AWS Lambda or Azure Functions), where you don't even have to allocate a server: just upload your function code and the platform takes it from there.

Names were changed. Windows Azure became Microsoft Azure, so the branding wouldn't be focused on one operating system. SQL Azure became SQL Database. Azure Web Sites became Azure App Services. Even Visual Studio Team Services / TFS Online was rebranded as Azure DevOps.

Software-as-a-Service (SaaS)

About 4 years ago I joined a product team to work on creating a Software-as-a-Service offering out of a legacy HR product named HRadvocate. It was a major amount of work to update the architecture and user interface, but eventually we had something deployed to Windows Azure with a reliable SaaS architecture that kept clients' data isolated from each other in separate databases.

SaaS Architecture on Azure

Authentication was initially through Azure Active Directory, with the idea that enterpises could use Microsoft's ADConnect to link their enterprise AD to AAD. It turned out that clients were demanding Active Directory Federation Services (ADFS) integration, so we added support for that. Later we added SAML support so products like PingFederate can be used to authenticate. Now our SaaS product could authenticate each client differently.

An Azure customer required a hybrid architecture, where Azure-hosted HRadvocate needed to integrate with multiple other systems--all of which were local to the enterprise. These systems connected to the former HR system via database integration, a structure that had to be maintained. To fit into this arrangement, I developed SQL Connector, a set of SQL Server functions written in C# that allow enterprise databases to query data in the cloud. This allowed the cloud data to be synced locally. Now, the local systems could continue to use their existing database integration, even though our SaaS was now part of the mix.

Amazon Web Services

I'd obviously been very focused on Microsoft Azure up until now, but that was about to change. Client requirements for HRadvocate led to a decision that we had to be able to run on Amazon Web Services as well as Azure. This led to several years of work on AWS and I am now proficient in it. Getting our solution to work on both Azure and AWS—while keeping a common source code base—was a lot of work but was also very educational. Azure's Cloud Service, SQL Database, Blob Storage, and Redis Cache mapped in a straightforward way to AWS's Elastic Beanstalk/EC2, RDS SQL Server, S3, and ElastiCache. About the only thing we couldn't transition was Azure Active Directory, but that's fine since we offer multiple ways of authentication.

SaaS Architecture on AWS

We also targeted Amazon's Commercial Cloud Services (C2S). To support this we added to the product the ability to run air-gapped (without Internet); this required locating and replacing any code (including from open source libraries) that was taking availability of the web for granted. Chart libraries like Google Charts had to replaced with Highcharts which could be local to the application. We added support for the FIPS 140-2 standard, using only algorithms and code for encryption that been certified to be compliant.

During this time, we continued supporting our product on Azure as well. Being able to run on two cloud platforms provided a lot of insight about what is the same and what is different between leading cloud platforms. There certainly seems to be a lot of copying going on between mainstream cloud platforms: when one provider comes out with a useful cloud service, it's not long before the competition has a very similar service. For example, Amazon has AWS Lambda for serverless-computing while Azure has Azure Functions. For those still worried about vendor lock-in, this keeping-up-with-the-Joneses activity should be comforting. The principles for building a good solution in the cloud transcend any one platform.

The Cloud in 2019

Ten years have gone by, and Cloud has certainly come into the mainstream. Just about all of us now use cloud computing every day, whether we realize it or not. Doing a web search? Streaming a movie? Using a social network? Making an online purchase? Cloud computing is an integral part of that.

Ten years ago, some big tech companies had cloud infrastructure but no one was providing cloud computing services to the public except Amazon. Now, there are clouds by Microsoft, Google, IBM, Oracle, SalesForce, SAP, VMWare, ...the list goes on and on. As for Microsoft, Azure is now also a leading cloud platform: it does PaaS and IaaS; half its VMs are reportedly running Linux; and there are a whopping 54 data centers worldwide. The growth has been phenomenal.

Cloud computing is no longer considered a speculative idea or a novelty for organizations: now, it's a common assumption that you'll be leveraging a cloud in anything new you develop. Ten years ago there was a lot of indecision about whether to go cloud or not; today, going to the cloud is a given, and the discussion is about which platform and which services to use. It's no longer a discussion of IaaS vs. PaaS; the debate now is about whether to leverage the newer cloud native architectures, NoSQL databases, serverless functions, and microservices vs. more traditional architectures. Serverless in particular is a major phenomenon that has opened cloud development to a broader number of people.

Some of my Neudesic colleagues from the early days have gone on to work at Microsoft or Amazon.

Cloud platforms seem to have improved uptime from 10 years ago, but there are still those moments when something goes wrong and a substantial number of clients are affected. You can still be in for a long wait when a cloud platform is recovering from an issue and each customer account has to be restored.

It's been a really interesting decade of cloud work, and there is plenty more to come. The do-it-yourself nature of the cloud is inherently satisfying, as is being able to change your mind and alter your deployment as will. Services that handle the details and let you focus on your application are a joy to use. You still need to know what you're doing architecturally and keep the cloud's different economic model in mind, but things like auto-scale and recovery are increasingly included in new cloud services. New services like Machine Learning are opening up new vistas for developers, and there's never been a more fun time to experiment—for just pennies.

Azure SQL Database Cross Database Joins with Local Databases

Organizations that make use of the cloud sometimes have to integrate that information with local databases. If you have a need to do cross-database joins including an Azure SQL Database, you might be wondering how that can be accomplished. One solution that works is to go to a local database and add a linked server to your Azure database.

Defining a Linked Server with sp_addlinkedserver

To add a linked server, open your local database in SQL Server Management Studio and execute a sp_addlinkedserver statement. You'll need to specify a number of parameters:

• @server : the name you want to use to refer to the database in queries. Use a simple, memorable name. In our example, we'll use abc.
• @datasrc : the data source (Azure database server), such as clients.database.windows.net
• @catalog : the database to connect to. We'll use ABC_PROD in our example.
• @provider : set to SQLNCLI.
• @srvproduct : set to empty string.

sp_addlinkedserver @server='abc', @datasrc=N'clients.database.windows.net', @catalog=N'ABC_PROD', @provider=N'SQLNCLI', @srvproduct=N''

Defining a Linked Server Login

To go with your Azure linked server you'll need to define a login for it. Use a sp_addlinkedsrvlogin statement with these parameters:

• @rmtsrvname : name of the linked server (same value you used for @server in the previous section). Example: abc
• @useself : set to 'false'
• @rmtuser : username. Example: dbrpt
• @rmtpassword : password. Example: s4ssafr4ss!

sp_addlinkedsrvlogin @rmtsrvname='abc', @useself='false', @rmtuser='dbrpt', @rmtpassword='s4ssafr4ss!'

Viewing Linked Server in SSMS Object Explorer

After performing the above steps, refresh your SSMS Object Explorer and you should see your linked server. If you defined everything correctly, you'll also see the database objects available, such as tables and views.

For troubleshooting, there's also a handy Test Connection option if you right-click the linked server name.

Querying the Azure Linked Server

To query your linked database, use the prefix [server][database] before specifying schema/table, where server is the value your used for @server and database is the value you used for @catalog in the first step above. For example,

SELECT * FROM [abc].[ABC_PROD].[dbo].person

Including Azure Linked Server in Cross-Database Queries

Now that you have a working linked server, you can include it in database queries that join to the local database or other linked databases. In the query below, the Azure linked database is joined to a local database table.

There you have it: queries that combine cloud and local data through the use of a Linked Server.

Azure Storage Explorer Preview 3 Now Available: Queue Support, Blob and Table Enhancements

I'm pleased to announce that the next installment of Azure Storage Explorer 6, Preview 3, is now available. Preview 3 adds Queue support along with expanded blob and table features. We've already posted a detailed walk-through of Preview 1 and Preview 2; now let's take a tour of what's new in Preview 3.

Queue Support

Preview 1 provided Blob support and Preview 2 added Table support. Now in Preview 3 you also have Queue support. Similar to how blob and table support work, you have a list of queues in your outline at left. When you select a queue, the top messages in the queue are listed in the main pane at right.

Message List

Actions you can take on queues include creating a queue, deleting the selected queue, or refreshing the queue list. Actions you can take on queue messages include refreshing the message list, creating a new message, creating a copy of an existing message, or popping the top message off of the queue (deleting it). When creating or copying a message, you enter or edit the message text that is to be inserted into the queue.

Creating a New Message

Enhanced Blob Support

We've made several enhancements to blob support. The earlier previews did not include the $logs container, which will exist if you have turned on logging and monitoring for your blob storage. This container is now listed if present.

Earlier previews did not show all blobs in the selected container, such as blobs with virtual directory paths in their name. The blob list is now exhaustive, showing you everything in the selected container.


Content Types

You can now configure a table of content types, which is used to set the Content Type property for a blob that you upload. This matters if blobs are read from public storage by a web browser, as the ContentType header helps the browser figure out how to handle the resource. The Content Type table is a simple table of file extensions and MIME types: if the file you upload ends with an extension listed in your table, its content type is set to the corresponding MIME type. You can view or edit the Content Types list from the top right settings menu ("Edit Content Types").

Editing Content Types for Uploaded Blobs

Enhanced Table Support

Table enhancements include a reordering of columns to put PartitionKey first, and a new upload capability. 

The table support that arrived in Preview 2 provided a means to download entities to CSV, JSON, or XML format - but had no matching upload capability. In Preview 3, you can also upload to tables.

When viewing a table, click the Upload button. On the dialog that appears, select the file containing data to upload and review the format selections and options.

CSV Upload

A CSV upload reads a comma-delimited text file for input. This is a popular simple export format that Excel can be used to edit. Azure Storage Explorer uses the CsvHelper library to parse your data file and insert entities.

When inserting entities, a partition key and row key are required to uniquely identify each entity. In the event your CSV file uses different column names for PartitionKey and RowKey, you can specify on the upload dialog the column names to use.

CSV Upload

Here is an example of valid CSV data that can be uploaded to a table:

"RowKey","PartitionKey","Timestamp","Debut","SecretIdentity"
Batman,DC Comics,8/6/2014 4:07:06 AM +00:00,5/1/1939 12:00:00 AM,Bruce Wayne
Green Lantern,DC Comics,8/6/2014 4:10:52 AM +00:00,7/1/1940 12:00:00 AM,Hal Jordan
Superman,DC Comics,8/6/2014 4:06:52 AM +00:00,4/18/1938 12:00:00 AM,Clark Kent
The Flash,DC Comics,8/6/2014 4:06:58 AM +00:00,1/1/1940 12:00:00 AM,Bart Allen
Iron Man,Marvel Comics,8/6/2014 4:07:58 AM +00:00,3/1/1963 12:00:00 AM,Tony Stark
The Human Torch,Marvel Comics,8/6/2014 4:13:02 AM +00:00,11/1/1961 12:00:00 AM,Johnny Storm
The Thing,Marvel Comics,8/6/2014 4:12:07 AM +00:00,11/1/1961 12:00:00 AM,Ben Grimm
Thor,Marvel Comics,8/6/2014 4:08:58 AM +00:00,8/1/1962 12:00:00 AM,Donald Blake

JSON Upload

A JSON upload reads a JavaScript Object Notation (JSON) format text file for input. JSON is a very popular web transfer format in wide use today. Azure Storage Explorer uses the .NET JavaScriptSerializer to parse your data file and insert entities.

As with CSV uploads, you can specify the field name to be used for RowKey and PartitionKey. In addition, you specify the enclosing object name for the JSON.

JSON Upload

Here is an example of valid JSON data that can be uploaded to a table:

{
    "Entities": [
        {
            "RowKey": "Batman",
            "PartitionKey": "DC Comics",
            "Timestamp": "8/6/2014 4:07:06 AM +00:00",
            "Debut": "5/1/1939 12:00:00 AM",
            "SecretIdentity": "Bruce Wayne"
        },
        {
            "RowKey": "Green Lantern",
            "PartitionKey": "DC Comics",
            "Timestamp": "8/6/2014 4:10:52 AM +00:00",
            "Debut": "7/1/1940 12:00:00 AM",
            "SecretIdentity": "Hal Jordan"
        },
        {
            "RowKey": "Superman",
            "PartitionKey": "DC Comics",
            "Timestamp": "8/6/2014 4:06:52 AM +00:00",
            "Debut": "4/18/1938 12:00:00 AM",
            "SecretIdentity": "Clark Kent"
        }
    ]
}


XML Upload

An XML upload reads an XML file as input, another popular exchange format. Azure Storage Explorer uses the .NET XmlDocument class to parse your data file and insert entities.

As with CSV uploads, you can specify the field name to be used for RowKey and PartitionKey. In addition, you specify the XPath path to locate entity elements in the XML.

XML Upload

Here is an example of valid XML data that can be uploaded to a table:

<?xml version="1.0" ?>
<Entities xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Entity>
    <RowKey>Batman</RowKey>
    <PartitionKey>DC Comics</PartitionKey>
    <Timestamp>8/6/2014 4:07:06 AM +00:00</Timestamp>
    <Debut>5/1/1939 12:00:00 AM</Debut>
    <SecretIdentity>Bruce Wayne</SecretIdentity>
  </Entity>
  <Entity>
    <RowKey>Green Lantern</RowKey>
    <PartitionKey>DC Comics</PartitionKey>
    <Timestamp>8/6/2014 4:10:52 AM +00:00</Timestamp>
    <Debut>7/1/1940 12:00:00 AM</Debut>
    <SecretIdentity>Hal Jordan</SecretIdentity>
  </Entity>
  <Entity>
    <RowKey>Superman</RowKey>
    <PartitionKey>DC Comics</PartitionKey>
    <Timestamp>8/6/2014 4:06:52 AM +00:00</Timestamp>
    <Debut>4/18/1938 12:00:00 AM</Debut>
    <SecretIdentity>Clark Kent</SecretIdentity>
  </Entity>
</Entities>

The default option for error handling is to stop on the first error. The alternative option is to continue past errors and continue uploading.

Summary

Azure Storage Explorer 6 continues to move rapidly toward completion. There are now sufficient features in place that we've made it the default version for download on CodePlex. I hope you find it useful and productive to use.

Azure Storage Explorer 6 Preview 2 Now Available, with Table Support

I'm pleased to announce that the next installment of Azure Storage Explorer, Preview 2, is now available. The big feature in Preview 2 is table support, allowing you to create, delete, and work with with tables and entity records. In addition, we've made some UI improvements and extended our blob support. We've already posted a detailed walk-through of Preview 1, now let's take a tour of what's new in Preview 2.

User Interface Changes

Blob List and Table List Styling
The main pane list view (showing blobs or table entities) now shows alternating rows in contrasting shades of grey. Selections are highlighted in the color associated with the artifact (yellow for blobs, blue for table entities).

Updated Styling of Blob List

Blob List Filters
One thing we've been thinking about is how to limit the display when you have a large number of blobs or table entities in a container; if you have thousands of items, you likely don't want them all listed every time you click on a container or table. In version 5 we implemented results paging, but in version 6 we wanted to do something more versatile. You can limit what is displayed with the new Filter toolbar button. For blobs, this lets  you specify filters such as max records to display, blob type, search text to match, or min and max sizes. You have a similar facility for table entities.

Filter Dialog for Blob List

When filters are active, the Filter toolbar button is highlighted so you're aware you're not seeing all results.

Filtered Blob List

Error Messages
Instead of pop-up dialogs for error reporting, error messages now smoothly stack on the bottom of the main pane, just as background tasks do.

Stacking Error Messages

Table Storage Support
Preview 2 introduces table storage support. When you navigate to a table in the left pane outline, you'll now get a list of entity records and toolbars for acting on the table and its entities.You can click on a column to sort by that property; click a second time to change ascending/descending direction.

Table Entity List

Table Entity Queries
Instead of retrieving all entities from the table, you can specify a query. The Query toolbar button allows to select up to three sets of columns, conditions (equals, not equal to, contains, starts with, ends with), and values. LINQ is then used to retrieve matching entities and the list is refreshed.

Table Query Dialog

Table Entity Filters
You can further restrict your entity list by specifying filters. These include a maximum entity count to display and search text. You can also control which columns should be displayed or hidden in the result list. You can click the check box to save these filter settings as a default.

Table Filter Dialog

Filtered Entity List

Creating, Copying, Editing, and Deleting Entities
You can use the toolbar buttons to create, modify, and remove individual entities.

Creating a New Entity
To enter a new entity, click the New Entity toolbar button and complete the record definition in the dialog that appears. You can add one or more records from here. You can enter binary (01 02 03...), boolean values (True/1/Yes/On, False/0/No/Off), DateTime values, numerics (double, Int32, Int64), GUIDs, or Strings. To not store a field, either set the type to Null or remove the field from the list.

Insert Entity Record Dialog

Updating an Entity
To update an entity, either double-click it or select it and click the View toolbar button. You can modify and update the record.

Copying an Entity
To copy an entity, select it and click the Copy toolbar button. Update the record (minimally changing RowKey/PartitionKey).

Deleting Entities
To delete entities, select one or more entities and click the Delete toolbar button. Confirm the dialog to perform the delete.

Delete Entities Confirmation Dialog

Table Entity Downloads
When viewing a table, you can download selected entities, or all entities, using the Download toolbar button. On the download dialog, select a file format and an output file. You can download to CSV, JSON, or XML format.

Entity Download Dialog

CSV Download
The comma-separated values (CSV) format can be opened in Microsoft Excel.

Result of CSV Download 

JSON Download
The JSON download is one of the more popular web data exchange formats today.

Result of JSON Download

XML Download
The XML download provides another commonly-used data exchange format.

Result of XML Download

Summary
With Preview 2, we've added a significant amount of table functionality to the blob functionality from Preview 1. We hope you find Azure Storage Explorer 6 useful and productive as we continue to move it forward. Do keep in mind the software is still very new and only in preview - please exercise appropriate caution as you use it.

Next: Azure Storage Explorer Preview 3

The Phoenix is Rising: Azure Storage Explorer 6

I'm very excited to announce that, after a long hiatus, work is progressing rapidly on the next version of Azure Storage Explorer and Preview 1 is now available on CodePlex. In this post I want to provide some background context on why the long wait and what you can look forward to in version 6 of Azure Storage Explorer, the "Phoenix" release. In my next post, we'll review Preview 1.

Where We've Been: Azure Storage Explorer versions 1 through 5
I've been involved with Windows Azure from the very beginning, and was honored to be selected as one of the inaugural MVPs by Microsoft. One of my first year contributions was Azure Storage Explorer, a tool for working with Storage Service accounts (blobs/queues/tables). Although there are multiple storage tools available today (both free and commercial), I believe Azure Storage Explorer was the first such tool for Microsoft's cloud platform.

Over the next few years, as the platform grew and extended, we periodically revised or rewrote Azure Storage Explorer, adding support for things like Shared Access Signatures and Logging Policy. By 2012, we were up to a beta of version 5. At that time, the Windows Azure portal had just undergone a major UI change and we styled ASE v5 to match. It looked like this:

Despite some pauses in regular upkeep, Azure Storage Explorer has become pretty popular. If you visit the Azure Storage Explorer CodePlex site, you'll see that there have been, as of this writing, a whopping 184,000+ downloads overall. That's pretty inspiring and humbling. I've been writing software for 35 years, of all varieties, but this has to be the one thing I've done with the broadest adoption.

The Pause
Over the last two years, however, there haven't been any updates to Azure Storage Explorer. If you work in software, then you know time is the enemy of software. The mere passing of time can turn a useful tool into a limited one as platforms change.

So why the pause? Well, I didn't stop working on the Microsoft cloud platform, and I didn't leave Neudesic, and I didn't stop being a software engineer, and I didn't stop valuing community work. But I did have some things going on that made it very difficult to work on Azure Storage Explorer or even to blog.

The pause in upkeep has been due to a combination of medical problems and a very busy project schedule on my part. I've had to work through some challenging orthopedic problems. In 2012 I suffered a tendon tear in my right arm that put me in a lot of pain and made it difficult and painful to do many everyday activities (such as getting dressed!). It took months to get diagnosed and then more months of treatment and therapy to restore use of my right arm. That was bad enough, but I'm now going through the same thing all over again with my left arm! Combine all of that with some ultra-busy projects at work (you know, the kind where you and your team are working around the clock to meet deadlines), and there just hasn't been any bandwidth for Azure Storage Explorer. Or blogging. Or speaking.

But wait a minute, you say. This is the era of open source and community software development. Surely there are others who would like to help work on Azure Storage Explorer. Indeed there are, and a number of talented folks have offered to assist. However, we weren't in a good position for that with version 5 because a lot had changed:

• The Microsoft Storage Client library had changed significantly, now up to version 4, with breaking changes.
• The MSI setup project we used for installation is no longer supported in Visual Studio.
• The way local developer storage is accessed has changed in the platform.
• New data centers in China require tools to use a different set of storage service endpoints.
• Even the name of the platform has changed, now known as Microsoft Azure.
• Our source tree had branched and fragmented a bit too much, and some reconciliation was needed.

All of the above amounted to this: we didn't have a very good starting point that would build with modern tools/frameworks or function well with the current cloud platform. I felt we needed a fresh start with an updated code base if we were going to be able to run this well as an open source project.

Azure Storage Explorer version 6
And so, that brings us to Azure Storage Explorer version 6. I've been able to carve out some time here and there, and we now have a good start on a new code base. Enough to release Preview 1, which is now on CodePlex.

These are some of our objectives for Azure Storage Explorer 6:

• A Visual Studio 2013 WPF solution.
• Uses the latest .NET Storage Client Library, v4.0.
• Uses InstallShield LE (included with Visual Studio)
• The WPF solution uses the Task Parallel Library (TPL) to handle asynchronous tasks
• Has an updated user interface.
• Source code online.
• Set up as an open source project to allow for collaborative contributions.
• Updated for the latest cloud platform.
• Supports configurable storage service endpoints, for example to support data centers in China.
• Has support for newer platform features, such as CORS.

A lot of the improvements in version 6 are subtle and have to do with small improvements that add up to a lot of usability. For example, we've previously had Copy Blob operations, but in version 6 you can copy to any container you want, even in another storage account in another data center. You were previously able to edit block blobs, but now you can also edit page blobs.

With this new code base, we're well-poised to take Azure Storage Explorer places. Once we match the basic feature set past versions have provided, we'd like to add some compelling new capabilities and also take advantage of new Microsoft Azure features (stay tuned for details). Most importantly, we can have multiple contributors involved which means my personal availability won't be a bottleneck in keeping the tool maintained.

The just-released Preview 1 supports blob operations only. Table and queue support will be coming in subsequent updates. Should you upgrade to version 6? If some of the pain points mentioned earlier have been affecting you and you don't mind being an early adopter, yes. If you need table or queue support, or prefer not to be on the bleeding edge with new software, then you might to wait for a later update.

In my next post, we'll take a tour of Preview 1.

Getting Started with Mobility, Part 10: A Back-end in the Cloud with Windows Azure Mobile Services

In this series of posts, we're looking at how to get started as a mobile developer. In Parts 1-9, we examined a variety of mobile platforms and client app development approaches (native, hybrid, web). We've seen a lot so far, but that's only the front-end; typically, a mobile app also needs a back-end. We'll now start looking at various approaches to providing a back-end for your mobile app. Here in Part 10 we'll look at Microsoft's cloud-based Mobile Backend As A Service (MBAAS) offering, Windows Azure Mobile Services.


About Mobile Back-end as a Service (MBaaS) Offerings
To create a back-end for your mobile app(s), you're typically going to care about the following:

• A service layer, where you can put server-side logic
• Persistent data storage
• Authentication
• Push notifications

You could create a back-end for the above using many different platforms and technologies, and you could do so in a traditional data center or in a public cloud. You'd need to write a set of web services, create a database or data store of some kind, provide a security mechanism, and so on.

What's interesting is that today you can make a "build or buy" decision about your mobile back-end: several vendors and open source groups have decided to offer all of the above as a ready-to-use, out-of-box service. Microsoft's Windows Azure Mobile Services is an example of this. Of course, it doesn't do all of your work for you--you're still going to be responsible for supplying a data model and server-side logic. Nevertheless, MBaaS gives you a huge head start. MBaaS is especially valuable if you want to focus your time on your mobile app, not the supporting elements.


Windows Azure Mobile Services
Windows Azure Mobile Services "provides a scalable cloud backend for building Windows Store, Windows Phone, Apple iOS, Android, and HTML/JavaScript applications. Store data in the cloud, authenticate users, and send push notifications to your application within minutes." Specifically, that means you get the following:

• Authentication (to Facebook, Twitter, Microsoft, or Google accounts)
• Scripting for server-side logic
• Push notifications
• Logging
• Data storage
• Diagnostics
• Scalability

The service will also generate for you starter mobile clients for iOS, Android, Windows Phone, Windows 8, or HTML5. You can use these apps as your starting point, or as references for seeing how to hook up your own apps to connect to the service.

Pricing
So what does all this back-end goodness cost? At the time of this writing, there are Free, Standard ($25/month), and Premium ($199/month) tiers of pricing. You can read the pricing details here.


Training
We reference training resources throughout this post. A good place to start, though, is here:

Get Started with Mobile Services
Android    iOS    Windows Phone    Windows 8    HTML5


Provisioning a Mobile Service
The first thing you'll notice about WAMS is the care that's been given to the developer experience, especially your first-time experience. Once you have a Windows Azure account, you'll go to azure.com, sign-in to the management portal, and navigate to the Mobile Services tab. From there, you're only a handful of clicks away from rapid provisioning of a mobile back-end.

1 Kick-off Provisioning
Click New > Mobile Service > Create to begin provisioning a mobile service.

Provisioning a Mobile Service

2 Define a Unique Name and Select a Database
On the first provisioning screen, you'll choose an endpoint name for your service, and either create a database or attach to one you're previously created in the cloud. The service offers a free 20MB SQL database. You'll also indicate which data center to allocate the service in (there are 8 worldwide, 4 in the U.S.)

Provisioning a Mobile Service - Screen 1

Provisioning a Mobile Service - Screen 2

3 Wait for Provisioning to Complete
Click the Checkmark button, and provisioning will commence. It's fast! In less than a minute your service will have been created.

Newly-provisioned Mobile Service Listed in Portal

4 Use the New Mobile Service Wizard
Click on your service to set it up. You'll be greeted with a wizard that walks you through. This is especially helpful if this is your first time using Windows Azure Mobile Services. On the first screen, you'll indicate which mobile platform you are targeting: Windows Store (Windows 8), Windows Phone 8, iOS, Android, or HTML5 (don't worry, you're not restricted to a single mobile platform and can come back and change this setting as often you wish).

Setup Wizard, Screen 1

5 Generate a Mobile App that Uses your Service
Next, you can download an automatically generated app for the platform you've selected, pre-wired up to talk to the service you just provisioned. To do so, click the Create a New App link. This will walk you through 1) installing the SDK you need for your mobile project, 2) creating a database table, and 3) downloading and running your app. The app and database will initially be for a ToDo database, but you can amend the database and app to your liking once you're done with the wizard.

Generating a Mobile Client App for Android

 

In the next section, we'll review how to build and run the app that you generated, and how to view what's happening on the back end.

 



Building and Running a Generated Mobile App
Let's walk through building and running the To Do app the portal auto-generates for you. The mobile client download is a zip file, which you should save locally, Unblock, and extract to a local folder. Next, you can open the project and run it--it's that simple to get started.

Running the App
When you run the app, you'll see a simple ToDo app--one that is live, and uses your back-end in the cloud for data storage. Run the app and kick the tires by adding some tasks. Enter a task by entering it's name and clicking Add. Delete an item by touching its checkbox.

To Do app running on Android phone

Viewing the Data
Now, back in the Windows Azure portal we can inspect the data that has been stored in the database in the cloud. Click on the Data link at the top of the Windows Azure portal for your mobile service, and you'll see what's in the ToDo table. It should match what you just entered using the mobile app.

Database Data in the Cloud

Dynamic Data
One of the great features of Windows Azure Mobile Services is its ability to dynamically adjust its data model. This allows you to change your mobile app's data structure in code, and the back-end database will automatically add new columns if it needs to--all by itself.

Get Started with Data in Mobile Services
Android    iOS    Windows Phone    Windows 8    HTML5


Dynamic data is a great feature, but some people won't want it enabled, perhaps once you're all ready for production use. You can enable or disable the feature in the Configure page of the portal.

Server-side Logic
Windows Azure Mobile Services happens to use node.js, which means server-side logic is something you write in JavaScript.

Mobile Services Server Script Reference

You can have scripts associated with your database table(s), where operations like Insert, Update, Delete, or Read execute script code. You set up these up on the Data page of the portal for your mobile service.

Database Action Scripts

 

Here's an example of a Read script that restricts processing to only return results owned by the current authenticated user:

 

function read(query, user, request) {    

    query.where({        

        owner: user.userId

     });    

    request.execute();

}

 



You can also set up scheduled scripts, which run on a schedule. On the Schedule page of the portal, click Create a Scheduled Job to define a scheduled job.

Creating a Scheduled Job

Once you've define a scheduled job, you can access it in the portal to enter script code and enable or disable the job.

Setting a Job's Script Code

Authentication
You can authenticate against Microsoft accounts, Facebook, Twitter, or Google. This involves registering your app for authentication and configuring Mobile Services; restricting database table permissions to authenticated users; and adding authentication to the app.

Get Started with Authentication
Android    iOS    Windows Phone    Windows 8    HTML5


Push Notifications
Push notifications support is provided for each mobile platform. Tutorials acquaint you with the registration and code steps needed to implement push notifications for  each platform,

Get Started with Push Notifications
Android    iOS    Windows Phone    Windows 8    HTML5


Summary
Windows Azure Mobile Services provides a fast and easy mobile back-end in the cloud. It offers the essential capabilities you need in a back end and supports the common mobile platforms. If you're comfortable expressing your server-side logic in node.js JavaScript, this is a compelling MBaaS to consider.