Shared Access Signatures for Azure Blob upload

Topics: Writing modules
Jul 7 at 7:18 PM
I need to create a way for clients to upload huge files to a private azure container.
I assume I need to create a service to generate SAS URIs when an authorized user selects a file to upload, but I have no idea how to incorporate this into orchard.

The code to generate the SAS URI is pretty simple and I have it working fine in a console app, but how do I convert it into something usable within Orchard?
Here's the example I'm playing with: https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-shared-access-signature-part-2/#comments

The front-end is based on Gaurav Mantri's post here: http://gauravmantri.com/2013/02/16/uploading-large-files-in-windows-azure-blob-storage-using-shared-access-signature-html-and-javascript/

Thanks for any help or tips!
Developer
Jul 7 at 8:17 PM
Hey, you would probably benefit a lot from looking at the Windows Azure Media Services module that ships with Orchard. It too provides a way to upload large files directly into an Azure container.
Jul 7 at 8:50 PM
Thanks Sipke, it looks like there's a ton more going on in there than I'm looking for. The files aren't for use on the website and the container is private so I'm not sure what I'd be using from the Azure Media module. I'd like to get the following code to return the URI for consumption in a custom form (not dynamic form, custom layout using js and html5 file field), but I don't know what I'm doing ;P
namespace SAS.Generator
{
    class Program
    {
        static void Main(string[] args)
        {
            //Parse the connection string and return a reference to the storage account.
            CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));

            //Create the blob client object.
            CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

            //Get a reference to a container to use for the sample code, and create it if it does not exist.
            CloudBlobContainer container = blobClient.GetContainerReference("sascontainer");
            container.CreateIfNotExists();

            //Insert calls to the methods created below here...
            //Generate a SAS URI for the container, without a stored access policy.
            Console.WriteLine("Container SAS URI: " + GetContainerSasUri(container));
            Console.WriteLine();

            //Require user input before closing the console window.
            Console.ReadLine();
        }

        static string GetContainerSasUri(CloudBlobContainer container)
        {
            //Set the expiry time and permissions for the container.
            //In this case no start time is specified, so the shared access signature becomes valid immediately.
            SharedAccessBlobPolicy sasConstraints = new SharedAccessBlobPolicy();
            sasConstraints.SharedAccessExpiryTime = DateTime.UtcNow.AddHours(2);
            sasConstraints.Permissions = SharedAccessBlobPermissions.Write ;

            //Generate the shared access signature on the container, setting the constraints directly on the signature.
            string sasContainerToken = container.GetSharedAccessSignature(sasConstraints);

            //Return the URI string for the container, including the SAS token.
            return container.Uri + sasContainerToken;
        }
    }
}
Should I be creating a part or a service or a controller with this? I'm just not sure how I should break it into something usable. I've added the connection string to my module's web.config as well as Azure references/NuGet packages

This is also not for backend users, only front end (HTTPS & Authenticated)

I'll keep looking at the media module too.

Cheers,
JV
Jul 7 at 8:53 PM
The front end view will be pretty much this exactly except rather than manually entering the URI, it would be pulled from the previous code and hidden.
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>File Uploader</title>
    <script src="js/jquery-1.7.1.js"></script>
    <link rel="stylesheet" href="css/modern.css"/>
    <script>
        var maxBlockSize = 256 * 1024;//Each file will be split in 256 KB.
        var numberOfBlocks = 1;
        var selectedFile = null;
        var currentFilePointer = 0;
        var totalBytesRemaining = 0;
        var blockIds = new Array();
        var blockIdPrefix = "block-";
        var submitUri = null;
        var bytesUploaded = 0;
         
        $(document).ready(function () {
            $("#output").hide();
            $("#file").bind('change', handleFileSelect);
            if (window.File && window.FileReader && window.FileList && window.Blob) {
                // Great success! All the File APIs are supported.
            } else {
                alert('The File APIs are not fully supported in this browser.');
            }
        });
         
        //Read the file and find out how many blocks we would need to split it.
        function handleFileSelect(e) {
            maxBlockSize = 256 * 1024;
            currentFilePointer = 0;
            totalBytesRemaining = 0;
            var files = e.target.files;
            selectedFile = files[0];
            $("#output").show();
            $("#fileName").text(selectedFile.name);
            $("#fileSize").text(selectedFile.size);
            $("#fileType").text(selectedFile.type);
            var fileSize = selectedFile.size;
            if (fileSize < maxBlockSize) {
                maxBlockSize = fileSize;
                console.log("max block size = " + maxBlockSize);
            }
            totalBytesRemaining = fileSize;
            if (fileSize % maxBlockSize == 0) {
                numberOfBlocks = fileSize / maxBlockSize;
            } else {
                numberOfBlocks = parseInt(fileSize / maxBlockSize, 10) + 1;
            }
            console.log("total blocks = " + numberOfBlocks);
            var baseUrl = $("#sasUrl").val();
            var indexOfQueryStart = baseUrl.indexOf("?");
            submitUri = baseUrl.substring(0, indexOfQueryStart) + '/' + selectedFile.name + baseUrl.substring(indexOfQueryStart);
            console.log(submitUri);
        }
 
        var reader = new FileReader();
 
        reader.onloadend = function (evt) {
            if (evt.target.readyState == FileReader.DONE) { // DONE == 2
                var uri = submitUri + '&comp=block&blockid=' + blockIds[blockIds.length - 1];
                var requestData = new Uint8Array(evt.target.result);
                $.ajax({
                    url: uri,
                    type: "PUT",
                    data: requestData,
                    processData: false,
                    beforeSend: function(xhr) {
                        xhr.setRequestHeader('x-ms-blob-type', 'BlockBlob');
                        xhr.setRequestHeader('Content-Length', requestData.length);
                    },
                    success: function (data, status) {
                        console.log(data);
                        console.log(status);
                        bytesUploaded += requestData.length;
                        var percentComplete = ((parseFloat(bytesUploaded) / parseFloat(selectedFile.size)) * 100).toFixed(2);
                        $("#fileUploadProgress").text(percentComplete + " %");
                        uploadFileInBlocks();
                    },
                    error: function(xhr, desc, err) {
                        console.log(desc);
                        console.log(err);
                    }
                });
            }
        };
 
        function uploadFileInBlocks() {
            if (totalBytesRemaining > 0) {
                console.log("current file pointer = " + currentFilePointer + " bytes read = " + maxBlockSize);
                var fileContent = selectedFile.slice(currentFilePointer, currentFilePointer + maxBlockSize);
                var blockId = blockIdPrefix + pad(blockIds.length, 6);
                console.log("block id = " + blockId);
                blockIds.push(btoa(blockId));
                reader.readAsArrayBuffer(fileContent);
                currentFilePointer += maxBlockSize;
                totalBytesRemaining -= maxBlockSize;
                if (totalBytesRemaining < maxBlockSize) {
                    maxBlockSize = totalBytesRemaining;
                }
            } else {
                commitBlockList();
            }
        }
         
        function commitBlockList() {
            var uri = submitUri + '&comp=blocklist';
            console.log(uri);
            var requestBody = '<?xml version="1.0" encoding="utf-8"?><BlockList>';
            for (var i = 0; i < blockIds.length; i++) {
                requestBody += '<Latest>' + blockIds[i] + '</Latest>';
            }
            requestBody += '</BlockList>';
            console.log(requestBody);
            $.ajax({
                url: uri,
                type: "PUT",
                data: requestBody,
                beforeSend: function (xhr) {
                    xhr.setRequestHeader('x-ms-blob-content-type', selectedFile.type);
                    xhr.setRequestHeader('Content-Length', requestBody.length);
                },
                success: function (data, status) {
                    console.log(data);
                    console.log(status);
                },
                error: function (xhr, desc, err) {
                    console.log(desc);
                    console.log(err);
                }
            });
 
        }
        function pad(number, length) {
            var str = '' + number;
            while (str.length < length) {
                str = '0' + str;
            }
            return str;
        }
    </script>
</head>
<body>
    <form>
        <div style="margin-left: 20px;">
            <h1>File Uploader</h1>
            <p>
                <strong>SAS URI</strong>: 
                <br/>
                <span class="input-control text">
                    <input type="text" id="sasUrl" style="width: 50%"
                           value=""/>
                </span>
            </p>
            <p>
                <strong>File To Upload</strong>: 
                <br/>
                <span class="input-control text">
                    <input type="file" id="file" name="file" style="width: 50%"/>
                </span>
            </p>
            <div id="output">
                 
                <strong>File Properties:</strong>
                <br/>
                <p>
                    Name: <span id="fileName"></span>
                </p>
                <p>
                    File Size: <span id="fileSize"></span> bytes.
                </p>
                <p>
                    File Type: <span id="fileType"></span>
                </p>
                <p>
                    <input type="button" value="Upload File" onclick="uploadFileInBlocks()"/>
                </p>
                <p>
                    <strong>Progress</strong>: <span id="fileUploadProgress">0.00 %</span>
                </p>
            </div>
        </div>
        <div>
        </div>
    </form>
</body>
</html>
Sep 19 at 10:29 PM
I fixed this a while ago, but here's what I added to my part driver
public class StorageManager {
            public static CloudBlobClient BlobClient
            {
                get;
                private set;
            }
            public static CloudQueueClient QueueClient
            {
                get;
                private set;
            }

            static StorageManager() {
                string accountName = "[ACCOUNT NAME]";
                string accountKey = "[ACCOUNT KEY]";
                CloudStorageAccount storage = CloudStorageAccount.Parse(String.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", accountName, accountKey));
                StorageManager.BlobClient = storage.CreateCloudBlobClient();

                EnableCors();

                StorageManager.QueueClient = storage.CreateCloudQueueClient();
            }

            private static void EnableCors() {
                // CORS should be enabled once at service startup
                // Given a BlobClient, download the current Service Properties 
                ServiceProperties blobServiceProperties = BlobClient.GetServiceProperties();

                // Enable and Configure CORS
                EnableCors(blobServiceProperties);

                // Commit the CORS changes into the Service Properties
                BlobClient.SetServiceProperties(blobServiceProperties);
            }

            private static void EnableCors(ServiceProperties serviceProperties) {
                serviceProperties.Cors = new CorsProperties();
                serviceProperties.Cors.CorsRules.Add(new CorsRule() {
                    AllowedHeaders = new List<string>() { "*" },
                    AllowedMethods = CorsHttpMethods.Put | CorsHttpMethods.Get | CorsHttpMethods.Head | CorsHttpMethods.Post,
                    AllowedOrigins = new List<string>() { "*" },
                    ExposedHeaders = new List<string>() { "*" },
                    MaxAgeInSeconds = 900
                });
            }

            public static string CreateSharedAccessPolicy(CloudBlobContainer container) {
                //Create a new stored access policy and define its constraints.
                SharedAccessBlobPolicy sharedPolicy = new SharedAccessBlobPolicy() {
                    SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(120),
                    Permissions = SharedAccessBlobPermissions.Write
                };

                string sasContainerToken = container.GetSharedAccessSignature(sharedPolicy);

                //Return the URI string for the container, including the SAS token.
                return container.Uri + sasContainerToken;
            }
        }