PolyCloud.Storage: A .NET API for Cloud Storage on AWS, Azure, and GCP

Today I'm announcing a new open source API library named PolyCloud.Storage.

PolyCloud.Storage is part of a bigger vision called the PolyCloud-API. The idea is this: as much as cloud platforms are ever striving to competitively differentiate themselves, it's simply a fact that there are many similarities between them. Take cloud storage for example: the SDKs may be different, and the terminology may vary (Azure containers and blobs equate to S3 buckets and objects), but you can basically treat Azure Blob Storage, AWS S3, or Google Cloud Storage like a file system, with folders and files. As it is becoming more and more common to see solutions that can be deployed to more than one cloud platform, the time seems ripe for a common set of API libraries that support the major cloud platforms. That's the idea behind PolyCloud-API.

PolyCloud-Storage is the initial offering in the PolyCloud-API series, and it starts out pretty modestly: with it you can do the following on AWS, Azure, or GCP:

• List folders
• Create a folder
• Delete a folder
• List files
• Upload a file
• Download a file
• Delete a file

Anyone who's played with a cloud storage system knows there's a great deal of other features available, such as versioning, random-access files, and security controls; where is all of that in PolyCloud-Storage? The plan is to introduce these features—where they exist on all of the target cloud platforms—in the days to come. When a feature is novel to just one cloud platform, we probably won't have it in PolyCloud.Storage since we'd like our library methods to work everywhere.

I started with storage because I have a lot of prior experience with it via the original Azure Storage Explorer, which had a whopping 280,000+ downloads in its time. Indeed, I was mulling over the idea of a new version of this tool when the idea for PolyCloud-API struck me. Do expect that you will see some tools coming from me that use PolyCloud.Storage (and other PolyCloud-API libraries as they become available).

This brings us to guiding principles for PolyCloud-API:

• Keep things simple and approachable
• Document the API
• Provide unit tests for the API
• Follow best practices for each cloud platform
• Provide options for error handling by the caller or by the library
• Collaboration = more libraries for more development languages

So, what's PolyCloud-Storage like to use? Here's a unit test for AWS S3:

Console.WriteLine("Test: AWS | CreateFolderAndFile");

// Set your storage account credentials here before running tests

String accessKey = "BKIAIBVVWJCZIPZQM2ZA";
String secretKey = "xQlSUFBSLr5x3FvyccBJy5rcD3e93Ftwzic18gQH";
String endpoint = "us-east-1";

using (Storage storage = Storage.AWS(accessKey, secretKey, endpoint))
{
    storage.HandleErrors = false;
    storage.Open();

    // Create folder

    String folderName = "test-" + Guid.NewGuid().ToString();
    Console.WriteLine("Creating folder " + folderName);
    storage.NewFolder(folderName);

    // Create file in folder

    String fileName = "test.txt";
    String fileName2 = "test2.txt";

    Console.WriteLine("Creating file " + fileName);

    String content = "This is a test.";

    if (File.Exists(fileName)) File.Delete(fileName);
    File.WriteAllText(fileName, content);

    // Upload file

    Console.WriteLine("Uploading file " + fileName);

    storage.UploadFile(folderName, fileName);

    // Retrieve file

    Console.WriteLine("Downloading file " + fileName + " as " + fileName2);

    if (File.Exists(fileName2)) File.Delete(fileName2);
    storage.DownloadFile(folderName, fileName, fileName2);

    // Delete file

    storage.DeleteFile(folderName, fileName);

    // Delete folder

    Console.WriteLine("Deleting test folder");
    storage.DeleteFolder(folderName);

    Console.WriteLine("Verifying content");

    String content2 = File.ReadAllText(fileName2);
    Assert.AreEqual(content, content2, "Downloaded file content did not match uploaded file content");

    storage.Close();
}

Hopefully you find the above code readily understandable; if you've worked with cloud storage SDKs, you may have experienced a mix of approachable and not-so-approachable APIs.

Now, here's the payoff: Let's imagine you wanted to do all of the above, but on Microsoft Azure. The only change you need to make is line 9 (plus different credentials). Instead of Storage.AWS to instantiate an S3 storage client, you use Storage.Azure to instantiate an Azure Blob client. All the other code remains unchanged.

String accountId = "my-blob-storage-account";
String accountKey = "Y7JTgm7B2g53t8MZ1N+GmuQ8d3aHyf0tGo5uglt7OIF3M2roaFiX6jPPI2mWmIOh/6suLEAMVXF5GPXPFl2nOQ==";

using (Storage storage = Storage.Azure(accountId, accountKey))

And as you should now be expecting, the only change needed for Google Cloud Platform is to use Storage.GCP to instantiate.

String jsonPath = @"C:\gcp-storage\helloworld-328219-ca4bb62a59c8.json";
String projectId = "918968012956";

using (Storage storage = Storage.GCP(jsonPath, projectId))

The .NET Framework edition is now up on GitHub at https://github.com/davidpallmann/polycloud-storage and a sister edition for .NET Core is at https://github.com/davidpallmann/polycloud-storage-netcore.

I do think this is a promising idea, and I'm excited about where it can go. I'd like to have some helpful collaborators provide matching libraries for other development languages such as Java. If you're interested, drop me a line.