Building the Right Environment to Support AI, Machine Learning and Deep Learning
The cloud is exciting, it is fresh, and it brings wonderful new opportunities and capabilities we haven't had before. There is a but. That but is that not everything CAN or SHOULD be run in the cloud. That's right! Not everything should or can be run in the cloud. Microsoft has worked hard to make the Windows Azure Platform a broad platform that is a la carte in nature. This allows you to consume the parts of the cloud that make sense for you. This also enables a new scenario of building hybrid applications.
Hybrid Applications in Microsoft Azure
Hybrid applications are applications that are broken into pieces, and run in a variety of runtime environments. Each piece is then able to leverage the strengths of its runtime environment to its advantage. For example, we might keep a web application on premises, and move the image storage to the cloud to take advantage of the storage costs and distribution strengths. The Windows Azure Platform lets you take advantage of just the pieces you need to use. Some applications can't run in the cloud because they need to access local hardware (a printer or scanner for example), or you might have data restrictions where you can't possibly move the data to the cloud. A hybrid approach lets you mix and match so you get the best of both worlds.
There are several ways we can get the information to the corporate office. In the past we might use VPNs and have the laptops synchronize a local SQL Express database to the corporate database. This approach can be very high maintenance. Synchronization relationships can need attention, and VPNs are never hands off. Even if they are working well on the corporate office side, it seems they don't always work on the user side. For example, the user might be in a hotel that doesn't allow VPN connections. This approach also requires a synchronous connection, requiring the home server to be online and ready to take connections. This can be an issue when no one is sending messages during the day, and suddenly everyone wants to get their reports in on Friday evening.
Using Microsoft Azure Queues
Windows Azure has four storage options available for use. These include blobs, tables, queues, and SQL Azure. We are going to focus on using Windows Azure Queues. The queue service provides a highly available and scalable way to asynchronously pass messages from the sales reps in the field and the processing server back at the corporate office. Since the queue is stored in the cloud, the sales reps will be able to always send messages to it.
Using the queue also allows us to stop using the VPN, at least for this application. This removes the need to punch holes through firewalls, and leaves the processing server to reach out to the cloud to grab the messages when it wants them.
Format of the Message
Messages in Windows Azure Queues are limited to 8KB in size. This means that when we architect our system we can't exactly stuff a giant document into the message and dump it in the queue. If you need to do this you will want to follow the worker ticket pattern. In this scenario you will upload the large piece of data to be worked on into shared storage (perhaps a blob container or a table) and then submit a work ticket to the queue with a pointer to the document in shared storage.
Our messages will simply contain the doctor Id from our global doctors list, the product sku that was given out, the quantity given, and the id of the sales rep. If we needed to also store the signature of the doctor to prove they received the samples we would store it in a blob container, with a file name that we would know (perhaps the id of the sent message.)
Messages in Azure must be text or serialized binary. In our sample we will simply pass the data as a string, but most teams will choose to use a POCO class to strongly type their messages, and just serialize that class to the message. If you do that you will reduce the interoperability of the queue, since each consumer will have to know how to deserialize that message back to the .NET object you defined.