Working with Google Protocol Buffers and .NET

This month, I was asked to include the topic of the “The Internet of Things,” or IoT if you’re one of these folks who watches all the buzz words as one of my posts. I happen to know that Windows 10 and its support for the “Internet of Things” is quite big news, and lots of developers are chomping at the bit to share their experiences of playing with all this new stuff. So, when I was asked about this, I quietly thought to myself, but WHY should Windows 10 have all the fun? After all, the IoT is not exactly new; we’ve been connecting all manner of things to various networks for well over a decade now!

That got me thinking, what “Things” do I know, that already exist that could be “Useful” to those non-Windows 10 folks who might be playing with regular .NET implementations, but wanting to build and communicate with things that are out there in this brave new world of IoT?

This post, I’m going to show you something called “Google Protocol Buffers,” and then over the next three posts, I’ll pick subjects that broadly fall into the IoT category and will help you build great .NET IoT implementations on any .NET enabled platform.

What Exactly Is “Google Protocol Buffers?”

Google Protocol Buffers (or Protobufs for short) are a data serialization and transmission technology that Google invented quite some time ago now.

As you can well imagine, with a network of machines the size that Google has, they needed a way to perform very high speed data transmission of structured data between nodes in their network.

At the time, like many people, they were using various mechanisms such as XML and Json, and even with relatively lightweight data packets such as those produced by Json, they wanted to shave even more milliseconds off things.

The result of wanting an even faster method led to the birth of Protobufs, and an implementation of the core libraries for just about every platform under the sun, including .NET

Okay, I Get That, but What Makes It So Special, and for That Matter So Ideally Suited for the IoT?

To answer that, I’ll have to deviate just ever so slightly into the world of finance, or more specifically credit and debit card transaction processing.

In the world of finance, ATMs have to communicate with banks, credit card terminals have to communicate with merchants, and all manner of things in between, all very fast and very reliably.

To do this, they employ a transmission method known as “BER-TLV”.

Stripping off the “BER” part where left with “Tag Length Value” which is essentially a very, very compact and efficient binary protocol composed of three parts, as follows:

  1. TAG: This is 1 or 2 bytes holding some kind of message identification number that the communicating terminals understand to mean an account number, or transaction amount or some other “Thing” being transmitted.
  2. LENGTH: This is a 1, 2, or 3 byte value telling the receiving terminal exactly how long the message that follows is, in terms of the number of bytes.
  3. VALUE: This is the actual payload of the message.

Google protobuffers is a very similar way of sending data between communicating entities and, when employed correctly, can allow you to serialise and encapsulate an ENTIRE .NET class into something as small as a few hundred binary bytes, which then can be very efficiently streamed over a UDP or TCP socket connection, or out via a traditional serial port.

If you’re working on an Arduino or a RaspberryPi, you could easily connect something to a Gpio pin and just stream this as a simple digital bit stream. I’ve even used this to send complex structured data between mobile devices using traditional SMS messages over the mobile network.

Put simply, you send ONLY what you need to send, and because of the way it works everything the receiver needs to know is sent in front of the data. This means unpacking it at the other end is very simple and in no way CPU intensive, meaning it’s fast and flexible, even on the most low-power devices.

Enough. Okay… Gimme Some Code ….

Fire up Visual Studio, and start a console program (I should label this as a pre-requisite for my posts ☺).

Once you have a project open, head to NuGet and search for “protobuf-net”, as shown in Figure 1:

Proto1
Figure 1: Protobuf-net in NuGet

You’ll find there are a couple of different implementations, many of which are designed for inclusion in different frameworks, such as “Service Stack”.

I generally use “Marc Garvell’s” implementation. It’s the one I know and the one this post will use.

Once you have the NuGet package installed, head back to your project and add a class called ‘Employee.cs’, as follows:

namespace Protobuffers{
   public class Employee
   {
      public int EmployeeId { get; set; }
      public string FirstName { get; set; }
      public string LastName { get; set; }
      public int Salary { get; set; }
   }
}

Nothing special here, just a normal plain old C# class. We now need to make it aware of Google Protobuf and vice versa.

To do this, we add “attributes” to the class which, if you’re used to using things like entity framework, should feel quite natural. Change your class so it looks as follows:

using ProtoBuf;

namespace Protobuffers
{
   [ProtoContract]
   public class Employee
   {
      [ProtoMember(1)]
      public int EmployeeId { get; set; }

      [ProtoMember(2)]
      public string FirstName { get; set; }

      [ProtoMember(3)]
      public string LastName { get; set; }

      [ProtoMember(4)]
      public int Salary { get; set; }
   }
}

The first thing that most people ask when they see this is “What’s with the numbers?” Protobuf doesn’t serialize the names of members in your classes. Instead, to make the message as small as possible, it uses a numerical index, which it relies on you to tell it.

I could, if I wanted, put all the integers first, followed by the strings if I wanted. In practice, what you normally want to do is just proceed down the class in a linear fashion.

There are a couple of other ways supported by the .NET lib we’re using, including an automatic one that will let Protobuf work this stuff out on its own. Personally, I prefer to tell it, that way I can be 100% sure that I’m getting the exact data in the exact order that I want.

Once you’ve marked up your Protobuf enabled class, you’re then ready to make use of it.

For the purposes of this post, we’ll simply just save the bytes to disk and reload them. In the real world, however, you’d likely send them somewhere else by some different medium.

Make sure your program.cs file looks as follows:

using System;
using System.IO;
using ProtoBuf;

namespace Protobuffers
{
   class Program
   {
      static void Main(string[] args)
      {
         Employee myEmployee = new Employee
         {
            EmployeeId = 1024,
            FirstName = "Peter",
            LastName = "Shaw",
            Salary = 100000
         };

         Console.WriteLine("Saving employee:");
         Console.WriteLine("\tEmployee ID: {0}",
            myEmployee.EmployeeId);
         Console.WriteLine("\tFirst Name: {0}",
            myEmployee.FirstName);
         Console.WriteLine("\tLast Name: {0}",
            myEmployee.LastName);
         Console.WriteLine("\tSalary: {0}",
            myEmployee.Salary);
         Console.WriteLine();

         using(var outputFile = File.Create("outputdata.bin"))
         {
            Serializer.Serialize(outputFile, myEmployee);
         }

         Console.WriteLine("Employee serialized, press return
            to load it back in");
         Console.ReadLine();

         Employee reloadedEmployee;
         using(var inputFile = File.OpenRead("outputdata.bin"))
         {
            reloadedEmployee =
               Serializer.Deserialize<Employee>(inputFile);
         }

         Console.WriteLine("Reloaded employee:");
         Console.WriteLine("\tEmployee ID: {0}",
            reloadedEmployee.EmployeeId);
         Console.WriteLine("\tFirst Name: {0}",
            reloadedEmployee.FirstName);
         Console.WriteLine("\tLast Name: {0}",
            reloadedEmployee.LastName);
         Console.WriteLine("\tSalary: {0}",
            reloadedEmployee.Salary);

      }

   }
}

As you can see, a lot of the code is actually made up of WriteLine statements to output the class data. You can serialize and deserialize the actual class by using the simple calls that the Protobuf library provides.

Before we finish this post, however, there is one small last thing.

YES, you can use this to serialise Lists and Arrays, but I personally would recommend against it.

If you’re using something like Protobuf to send large amounts of data, such as lists of records directly from a database, in my own personal opinion you’re likely not architecting your systems correctly.

Most devices that you would class as being IoT devices will very likely have relatively slow CPUs and quite a small amount of memory. Sending huge amounts of data to these devices will probably make them mis-behave. If you do need to send that level of data, my advice is to get the device to make a request to a JSON endpoint at its own pace and consume what it is able to.

If, however, you want a way to send relatively small, highly efficient structured messages, Protobuf is the perfect choice.

In our earlier example, the binary data generated from our class stands at only 20 bytes long:

Proto2
Figure 2: The output from the preceding code

This is perfect to send via something like SMS to an IoT device with nothing more than a GSM Modem attached.

Found a strange .NET library that makes no sense to you, or want to know if there’s an API for that? Drop me a message in the comments below or come find me on Twitter as @shawty_ds and I’ll see if I can write a post on it for you.

More by Author

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Must Read