Microsoft Azure .NET Queuing Patterns

Azure Queues are part of the Azure Storage offerings.  Microsoft’s Azure SDK for .NET makes developing with Azure Queues accessible to .NET developers.   SDK documentation is, for the most part, complete.  There are, however, underlying patterns to understand if a developer wants to utilize some more advanced features.  Some examples are the patterns behind message creation/consumption timeouts; and .NET Asynchronous support.  These patterns will be explained and demonstrated in the paragraphs that follow.

    Windows Azure 90 day free trial. Try it Now!    

Getting Started

All code samples are based on the June 2012 Azure CTP and developed in Visual Studio 2010.  A complete setup explanation is beyond the scope of this article.  Instead I’ll provide a synopsis.

First, developers must install the Azure SDK for .NET.  The June 2012 SDK release can be found here http://www.microsoft.com/en-us/download/details.aspx?id=29988.  An Azure Account is required to experiment with Azure Queues.  A quick way to get access is to setup a Trial Account.  The Azure Trial link is everywhere in the Azure documentation.  As of writing this article, there was a trial link here http://www.windowsazure.com/en-us/.  Setting up the Trial version requires an email address, Windows Live Id, a phone number, and a Credit card.  More account setup details can be found here http://www.developer.com/services/step-by-step-guide-to-setting-up-a-windows-azure-free-trial.html

Azure communication is done through email.  The Windows Live Id is for authentication with the Admin portal.  “Keyword” setup information is passed via the phone number or text option to a wireless phone.

After setting up an Azure Account a developer must create a storage account.  An Azure Storage account provisions Azure for code access to Storage mechanisms like: Blobs, Tables, and Queues.  Like all Storage, accessing Queuing from .NET code requires the AccountKey.

Queuing Overview

As stated earlier, what follows is a deeper dive into some important Azure .NET Queuing patterns.  If you’re looking for a basic you’ll find one here http://www.windowsazure.com/en-us/develop/net/how-to-guides/queue-service/

The following code opens access to Azure Queue Storage, sends a message, and then retrieves the message.

var acct = "DefaultEndpointsProtocol=https;AccountName=nametest;AccountKey=??????????";
 
var cloudAcct = CloudStorageAccount.Parse(acct);
var client = cloudAcct.CreateCloudQueueClient();
 
var queue = client.GetQueueReference("testqueue");
queue.CreateIfNotExist();
 
var msg = new CloudQueueMessage("Some data");
 
//2 minutes time to live
queue.AddMessage(msg, new TimeSpan(0, 2, 0));
 
//Unlock if delete not called in 30 seconds.
var recMsg = queue.GetMessage(new TimeSpan(0, 0, 30));
 
Console.WriteLine("Sent and received test message Data == " + recMsg.AsString);
 
queue.DeleteMessage(recMsg);

The CloudStorageAccount class provides a way to validate connect string formatting and create various Azure Storage Objects.  Access to Queuing Storage happens through the CloudQueueClient.  CloudQueueMessage handles the message packaging details.  Developers can send a string or use various .NET Serialization techniques to encode a message payload into bytes.

Queuing Storage requires the Windows.Azure.StorageClient assembly and the Configuration assembly.  Developers will find references to these in the .NET References dialog.  Assemblies are based on .NET 2.0 so they should work with older .NET versions.

Some of the methods in the sample code passed TimeSpan parameters.  The Timespan parameters are part of another Azure Queue Pattern requiring a more detailed explanation.

Lock and Message Timeouts

When deciding how to leverage Azure Queues; there are a number of architectural decisions a developer must make.  Here are sample requirements that have a substantial impact on architecture:

  • Though Message Queuing is Asynchronous; processing a message may require some timeliness. 
  • Overall throughput may be important and Azure storage cost must be considered.  Every message may not be important to process. 
  • Message Queues often control or throttle access to a processing resource like a database. 
  • Queue clients may number in the thousands, but there may be just a handful of processing resources.  Multiple processing resources may be sharing a single Queuing resource. 
  • Resources like Databases may occasionally go offline creating failures during message processing.

Azure Storage charges by usage.  There are containment and accessing costs.  Systems that collect, for example, logging information may be simply transmitting a set of state values on a particular interval.  At times it may not be important to collect every update.  Setting a TimeToLive value ensures that a message will only be consumed if it is meaningful.   TimeToLive is demonstrated in the following code taken from the previous sample.

//2 minutes time to live
queue.AddMessage(msg, new TimeSpan(0, 2, 0));

LockTimeouts are important for message consumption.  If a message causes an Exception in a Queuing client attempting to process the message, a Queuing system should attempt some type of retry.  A retry may not be possible if the Queuing Client has completely crashed.  Azure Queues support LockTimeouts that return a message if it hasn’t been processed within a time interval.  LockTimeouts are demonstrated in the following code.

//Unlock if delete not called in 30 seconds.
var recMsg = queue.GetMessage(new TimeSpan(0, 0, 30));

If client responsiveness is important, one way to achieve responsiveness is to use Asynchronous calls.  Asynchronous methods are part of another Azure Queue pattern.

Asynchronously with Tasks

The .NET Task Parallel Library (TPL) has become a foundation for Concurrency and Asynchronous operations.  TPL is new to .NET 4.0.  Some new .NET 4.5 features are built around the Task Parallel Library.   Understanding how Tasks work is the key to understanding the TPL.  A complete introduction to Tasks is beyond the scope of this article, but you’ll find one here http://www.codeguru.com/columns/experts/article.php/c17197/Understanding-Tasks-in-NET-Framework-40-Task-Parallel-Library.htm .  So, what follows is a Task overview.

Microsoft built TPL to maximize a multi-CPU computer’s potential.  A Task class is a means to package a workload so it can be scheduled on a Thread Pool.  Tasks can be chained together each waiting on the success or failure of the other.  New Tasks can be automatically created contingent on completion of a prior Task; this is called a Continuation.

As mentioned earlier; the Azure SDK is based on an older .NET version so it supports an older Asynchronous operation model called the Asynchronous Programming Model (APM).  In the APM model an operation has a Begin Method that starts the operation and an End Method that concludes the operation.  In between Begin and End a developer must supply a Callback method linking the two operations together.  Luckily TPL includes methods to marry the old model and the new model.  Following is code demonstrating sending and receiving a message asynchronously.

var acct = "DefaultEndpointsProtocol=https;AccountName=nametest;AccountKey=??????????";
var cloudAcct = CloudStorageAccount.Parse(acct);
var client = cloudAcct.CreateCloudQueueClient();
 
var queue = client.GetQueueReference("testqueue");
queue.CreateIfNotExist();
 
var msg = new CloudQueueMessage("Some data");
 
var taskAdd = Task.Factory.FromAsync<CloudQueueMessage>(new 
    Func<CloudQueueMessage,AsyncCallback,object,IAsyncResult>((m,cb,obj)=>
{
    return queue.BeginAddMessage(m,new TimeSpan(0, 2, 0),cb,obj);
}), queue.EndAddMessage,msg,null);
 
taskAdd.Wait();
 
var taskGet = Task.Factory.FromAsync<TimeSpan,CloudQueueMessage>(queue.BeginGetMessage, queue.EndGetMessage,new TimeSpan(0, 0, 30), null);
taskGet.Wait();
 
Console.WriteLine("Sent and received test message Data == " + taskGet.Result.AsString);
 
queue.DeleteMessage(taskGet.Result);
 

Both samples demonstrate Waiting for a Task to complete.  Tasks implement the IAsyncResult interface making them interoperable with the old and the new way.

When the Task completes any returned data can be found in the Result property.  Operations that end with an Exception will have a non-null Exception property.  Asynchronous operations creating an Exception will bubble the exception to the method containing the Wait call.  There are alternatives to calling Task.Wait, however, an Exception must always be inspected. 

Tasks support Waiting on multiple Tasks.  So, for example, the sample could have waited on the Write and the Read rather than waiting on each independently.

Resources at the end of the article include more TPL details.

Conclusion

Azure Queues are part of the Azure Storage offerings.  Microsoft’s Azure SDK for .NET makes developing with Azure Queues accessible to .NET developers.   There is considerable documentation on Azure Queuing basics; so this article did a deep dive into some more advanced Queuing patterns.

Resources

Task Parallel Library resources

Understanding Tasks in .NET Framework 4.0 Task Parallel Library
http://www.codeguru.com/columns/experts/article.php/c17197/Understanding-Tasks-in-NET-Framework-40-Task-Parallel-Library.htm

Microsoft .NET Framework 4.0 Task Parallel Library Continuations
http://www.codeguru.com/csharp/article.php/c18361/Microsoft-NET-Framework-40-Task-Parallel-Library-Continuations.htm

Advanced Task Parallel Library Continuations
http://www.codeguru.com/csharp/article.php/c19209/Advanced-Task-Parallel-Library-Continuations.htm

SpinWait.SpinUntil and the Task Parallel Library
http://www.codeguru.com/csharp/article.php/c19283/SpinWaitSpinUntil-and-the-Task-Parallel-Library.htm

Azure Home site
http://www.windowsazure.com/en-us/

Azure Blog
http://www.windowsazure.com/en-us/community/blog/

June 2012 SDK
http://www.microsoft.com/en-us/download/details.aspx?id=29988



Related Articles