Distributed Network Object

The Problem

In a network application, state replication of objects across several or different instances of the application is done by custom application protocols. A typical example is this: You want to create a trainer application where a person is supposed to perform some actions at his computer and you want to replicate the effect at other computers connected to it, simulating the same behaviors. For such a task, sometimes, custom structures or different string formats are used, where the object state is packed into these structures/strings, and pass on to the connected side, which upon receiving it, unpack the structure/string, and retrieve information. Eventually the particular object on to the other side is updated with the information passed on, which in turn would be stored in some collection. You'll have to create a large number of these structures/string formats for each state update.

This model, although used widely, has some issues, especially when it comes to extensibility/flexibility. If one single piece of information that was not earlier being transported has to pass on, the message structures/string has to be updated; this can lead you to make changes in application protocol. Even this minor change can cause a ripple effect and (depending upon the design), one might have to re-engineer several application areas to accommodate the new change—especially, if changes happen frequently, a lot of time, effort, and resources are required to keep things in balance.

The Solution

The "Network Distributed Object Model" offers a simplified way to perform network communication at the object level. This is a network communication framework for applications (client/server) that gives ability to the application programmers to develop a network application without worrying about the underlying network communication details and custom protocols.

The idea is to move network communication at the object level. In a network application, the object at the server side, most likely, will have a corresponding object at the client side. For example, if you are playing a game, and you are controlling a tank, the other players at the network can also see the tank; if you move your tank at your machine, they'll also see the movement at their machines. Or, if you are designing an application for stock markets, if the symbols (like MSFT, YHOO, GOOG, and so forth) at the server side will also have a corresponding object at the client side; if some attribute of these objects (like bid/ask price, print price, and so on) changes at the server side, all clients should be updated with the change occurred at the server side. In this model, the communication is moved down to these objects, rather than the application. Objects, when updated at the server side, communicate with the connected end, and pass on their state to them, which in turn update their own object that corresponds to the server object.

Figure 1: Typical Network Application Communication Model

Figure 2: Distributed Network Object Application Design

A Network Distributed Object is based upon polymorphism; it offers a class named "CNetworkObject", from which an application user can inherit their class and over-ride a method named "NetworkSerialize", which performs serialization. Each network object is assigned a unique ID upon creation; therefore, the order of object creation is important. Two sides that want the information exchange, have to have the same network object creation sequence.

When the objects (inherited from CNetworkObject) are created, they are automatically registered with a network object manager (in other words, CNetworkObjectManager), which maintains a collection of these network objects. In the main application message pump, a method of network object manager (SerializeObjects) has to be called. This acts as a heartbeat to the model which, in one heartbeat, serialize all objects. In the case of the server side, it sends the updated object states, whereas at the client side, it updates the latest object states from the server.

Before you move any further, let me describe the major projects, files, and classes that are part of this model.

The project is split into two sub-projects:

1. Util: This contains utility classes

  1. units.h
  2. SmartPointers.h
  3. globals.h
  4. debug.h
  5. Crc.hpp/.cpp
  6. DataStream.h

units.h has various typedefs, for portability reasons.

#ifdef _MSC_VER
#pragma once

typedef unsigned __int64       uint64_t;
typedef __int64                int64_t;
typedef unsigned long          uint32_t;
typedef long                   int32_t;
typedef short                  uint16_t;
typedef short                  int16_t;
typedef unsigned char          uint8_t;
typedef char                   int8_t;

#if defined (Linux)
#include <stdint.h>

#if defined(freebsd) || defined(tru64_alpha)
#include <inttypes.h>

#if defined(solaris)
#include <sys/int_types.h>

//typedef int8_t int8;
typedef int16_t int16;
typedef int32_t int32;
typedef int64_t int64;

typedef uint8_t uint8;
typedef uint16_t uint16;
typedef uint32_t uint32;
typedefuint64_t uint64;

SmartPointers.h is a stripped-down version, and contains only a template-based Singleton class. Note that it's not a thread safe singleton.

template <class T>
class TSingleton
   typedef T& reference;
   static reference Instance()
      static T obj;
      return obj;
   TSingleton() {}
   ~TSingleton() {}
#define SINGLETON(T) protected:friend class TSingleton< T >;T(){};
#define SINGLETONINSTANCE(T) TSingleton< T >::Instance()

Global.h is another stripped version and has some macros being used in the implementation. FREE_POINTER and FREE_POINTER_ARRAY are just to delete the pointer and pointer arrays, whereas CANONIC is for making a copy constructor and assignment operator of any class as private. They'll be used later in the CNetworkObject class.

#define FREE_POINTER(x)      if(x) { delete (x); (x) = NULL; }

#define FREE_POINTER_ARRAY(x) if(x) { delete[] (x); (x) = NULL; }

#defineCANONIC(x)               \
      private:                  \
      x(const x& obj);          \
      x& operator= (const x& obj);

Debug.h contains some assertions.

Crc.hpp/.cpp contains a trivial class that can calculate a 16 bit CRC.


class CCalcCRC
   static uint16 CalcCRC( const void *pData, const uint32 uSize,
                          uint16 uiCRC =0 );

CDataStream plays a vital role in this model. It manages the byte stream of data and can be used in various situations. In this model, it is used in network object serialization and also in NetComm APIs, where network messages are queued. The internal buffer of CDataStream grows and shrinks itself when needed. Moreover, you can mark certain memory locations, to which you can refer later. Operators for POD types also are overloaded. All these features of this class make us able to implement a distributed network object.

#pragma once

#include "globals.h"
#include <algorithm>

using std::vector;
using std::advance;

class CDataStream
   CDataStream( uint32 uSize = 0 );
   CDataStream( );

   void SetBuffer( char* szBuffer, uint32 uBufSize );
   void SetSize( const uint32 uSize );
   void TrimUnreadSize();
   void Clear( );
   void Rewind( bool bRewindWrite = true );

   uint32 ReserveWriteCell( uint32 uSize );

   void ReleaseWriteCell( uint32 uCellID );
   void WriteToCell( uint32 uCellID, char* szData, uint32 uSize );

   uint32 AddWriteMarker( );
   uint32 AddSizeMarkerCell( );
   uint32 WriteSizeMarkerCell( uint32 uCellID );

   void ReleaseWriteMarker( uint32 uMarkerID );

   uint32 GetWriteSizeSinceMarker( uint32 uMarkerID );

   void SeekBackWritePtr( int32 lByteCount );

   const char * GetMarkedDataPtr( uint32 uMarkerID );
   const char* SeekBackReadPtr( int32 lByteCount );
   const char* SeekForwardReadPtr( int32 lByteCount );
   const char* SeekForwardWritePtr( int32 lByteCount );

   uint32 CheckWriteSize( uint32 uSize );
   uint32 CheckReadSize( uint32 uSize );

   void Write( FILE* phFile, uint32 uDataSize )
   const char* GetDataPtr( int32 lStartIndex = 0) const;
   const char* GetReadPtr( int32 lOffset = 0 ) const;
   uint32 GetDataSize( ) const;
   uint32 GetBufSize( ) const;
   int32 GetReadBytesRemaining( ) const;
   void Write ( const char* pSource, const uint3 uDataSize );
   void WriteString( const char* szSource );

   void Read ( char* pDest, const uint32 uDataSize );
   const char* ReadString( );

   void operator >> ( bool&   rbFlag );
   void operator >> ( uint32& ruInt );
   void operator >> ( int32&  ruInt );
   void operator >> ( int16&  iInt );
   void operator >> ( uint16& uiInt );
   void operator >> ( uint8&  ubVal );
   void operator >> ( float&  rfNum );

   void operator << ( bool   bFlag );
   void operator << ( uint32 uInt );
   void operator << ( int32  uInt );
   void operator << ( int16  iInt );
   void operator << ( uint16 uiInt );
   void operator << ( uint8  ubVal );
   void operator << ( float  fNum );

   void Copy( const CDataStream& cArg );
   CDataStream& operator = (const CDataStream& cArg );

   char*  m_szDataBuf;
   char*  m_szWritePtr;
   char*  m_szReadPtr;
   uint32 m_uBufSize;

   struct SReservedCell
      uint32  uSize;
      uint32  uOff;

   typedef std::vector< SReservedCell > TReservedCell;
   typedef std::vector< uint32 > TMarkerArray;

   TReservedCell  m_cResCellArray;
   TMarkerArray   m_cMarkerArray;

   void CheckSize( uint32 writeDataSize );

Distributed Network Object

2. NetCommLib

This project contains classes directly related to the network distributed object model. It has wrappers over Winsock, in the form of different classes, such as CNetComm, CClient, CHost, and CNetCommMsgQueue; these are further managed in a wrapper, exposing few APIs to application programmers such that they don't have to deal with the underlying network communication mumbo jumbo. They simply can use these APIs (which are hardly few), and can make a fully functional network application. This network communication model (in other words, CNetComm, CClient, CHost, and CNetCommMsgQueue) isn't the best network model on the face of planet earth, and is only for this article, so if you don't like it, you can replace it with your own. All you need is a TCP link. From an application programmer's point of view, you don't have to deal with these classes; you just have to be familiar with the following APIs, which are pretty much self explanatory, defined NetworkWrapper.h, or you can replace the communication library with your own. With the help of these, you can make a client/server application without getting into the hassle of Winsock.

bool NetCommInit( ENetComType eNetComType );
bool NetCommIsMessageAvailable();
bool NetCommIsSystemMessageAvailable();
bool NetCommHost( uint16 uPort );
bool NetCommConnect( const char * pszIP , uint16 uPort );
void NetCommDisconnect();
uint32 NetCommSendMsg( const char * pszData , uint32 uDataSize ,
                       uint32 uType );
uint32 NetCommGetNoOfMessages();
uint32 NetCommGetNoOfSystemMessags();
EMsgType NetCommGetSystemMessage( void*& pData );
SNetworkMsg * NetCommGetMessage();
bool NetCommIsConnected();
uint32 NetCommGenUniqueMsgID();

The other classes that play a main part in the distributed network object model are CNetworkObject and CNetworkObjectManager. CNetworkObject is the base class, from which application programmers have to inherit their classes and over-ride the NetworkSerialize method. This is where the object serialization is performed. An application programmer can read the object state from the stream passed to it as a parameter and can write their state into it. Upon object creation, the objects are maintained in a collection, managed automatically by the other class CNetworkObjectManager. The CNetworkObjectManager class not only manages these objects, but also provides methods to implement network serialization in the main application.

Class interface of CNetoworkObject is as follows

#include "SmartPointers.h"
#include "DataStream.h"
#include <vector>
#include <map>
#include <string>

using std::map;
using std::vector;
using std::string;

struct SNetworkMsg;
class CNetworkObjectManager;

   CANONIC( CNetworkObject );


   typedef CDataStream TDataStream;
   friend CNetworkObjectManager;

   enum EIOType

   virtual ~CNetworkObject();

   virtual bool NetworkSerialize( EIOType eIO,
                                  TDataStream& rStream );

   uint32 GetInstanceID() const { return m_uInstanceID; }
   void SetOwnership(bool bArg) { m_bOwner = bArg; }


   uint16 GetCRC() const { return m_uCRC; }
   void   SetCRC( const uint16 uCRC ) { m_uCRC = uCRC; }

   static uint32 ms_uNetworkObjectInstanceCounter;

   uint16 m_uCRC;

   const uint32   m_uInstanceID;
   uint32         m_uCRCMarkerID;

   bool m_bOwner;

CNetworkObjectManager is a singleton container class that contains a collection of all instances of CNetworkObject. Below is the interface of CNetworkObjectManager. An instnace of CNetworkObject in its constructor registers itself with CNetworkObjectManager and unregisters itself in the destructor.

class CNetworkObjectManager
   SINGLETON( CNetworkObjectManager );

   virtual ~CNetworkObjectManager();

   bool ReceiveNetworkPacket( const SNetworkMsg* pMsg );

   void SerializeObjects();

   void RegisterSerializeObject( CNetworkObject* pSerializeObj);
   void UnRegisterSerializeObject( const CNetworkObject*

   void SetOwnership(bool bOwnership);
   bool IsOwner() const { return m_bOwner; }


   typedef std::map< uint32 , CNetworkObject * >
   typedef std::pair< uint32 , CNetworkObject * >
   TSerializeObjectList  m_SerializeObjectList;

   bool m_bOwner;

Now, let me focus on the design of the Network Distributed Object Model. As was explained earlier, because you're an application programmer, you have to inherit your class(es) from the CNetworkObject class. This class maintains a 16-bit CRC, such that each object of this class will contain a unique 16-bit integer, depending upon the values/states of the object. With the help of this CRC, you'll be able to decide whether or not some state of the object is changed. Being an application programmer, you'll be using the class objects derived from CNetworkObject in a natural way (in other words, by using an assignment operator of course).

CNetworkObjectManager is a Singleton class. Network objects are registered with this class upon creation, and un-registered upon destruction. In the application's main message pump, you have to call the SerializeObjects() method of this class. This is sort of a heartbeat to this model. In each call, the collection will be iterated, NetworkSerialize method will be called, where (if over-ridded by the application programmer correctly), will collect the values of the variables into the stream and a 16-bit CRC checksum will be computed, each object will also contain its own CRC as well, as mentioned earlier; therefore, if at any given time, a newly computed CRC is different than the stored CRC of the object, you can conclude that any state of the object has been changed. In case of no change, of course CRC will remain the same.

At the end of the iteration, you'll have a stream having all the object's state, which has been updated. Note that each object has a unique ID, which should be in the same order at the client end as well. This ID will precede the object state in the stream for each object. Note alsothat your class object, inherited from CNetworkObject, might contain many variables reflecting different states, but you may want only a few to take part in the serialization operation. If that's the case, here is your chance. In the NetworkSerilize method, you have to take care of two aspects:

  1. You only put variables, in the stream or out of stream, which values you want to transfer from one node to another.
  2. The sequence of reading and writing the stream should be same.

At the end of this process, the final stream will be transported to the client through a TCP link (or anything that lets a binary stream flow between two ends).

void CNetworkObjectManager::SerializeObjects()
   // Process outgoing messages...
   static CNetworkObject::TDataStream stream;

   if ( SINGLETONINSTANCE( CNetworkObjectManager ).IsOwner() )
      if ( false == NetCommIsConnected() )

         begin = m_SerializeObjectList.begin() ,
         end = m_SerializeObjectList.end();
      for( ; begin != end; ++begin )
         CNetworkObject * pNetworkObject = begin->second;

         uint32 uMarkerID = stream.AddWriteMarker();

         stream << pNetworkObject->GetInstanceID();

         pNetworkObject->NetworkSerialize( CNetworkObject::ceRead,
                                           stream );

         const char * pDataPtr =
            stream.GetMarkedDataPtr( uMarkerID );
         uint32 uSize = stream.GetWriteSizeSinceMarker( uMarkerID );

         uint16 uCRC = CCalcCRC::CalcCRC( pDataPtr , uSize );
         uint16 uNetObjectCRC = pNetworkObject->GetCRC();

         if ( uNetObjectCRC == uCRC )
            // Rewind stream to the last written point
            uint32 uBytes =
               stream.GetWriteSizeSinceMarker( uMarkerID );
            stream.SeekBackWritePtr( uBytes );

            // Release the market
            stream.ReleaseWriteMarker( uMarkerID );


         pNetworkObject->SetCRC( uCRC );

         // Optimization step to send smaller packets as they are
         // generated so that the network pipe can stay optimally
         // busy.
         if ( stream.GetDataSize() > 4096 )
            NetCommSendMsg( stream.GetDataPtr() ,
                            stream.GetDataSize() ,
                            gs_NetworkObjMsgID );

      if ( stream.GetDataSize() )
         // send serialized buffer...
         NetCommSendMsg( stream.GetDataPtr() ,
                         stream.GetDataSize() ,
                         gs_NetworkObjMsgID );

Distributed Network Object

The counterpart to retrieve the stream and update objects at the client side is ReceiveNetworkPacket. Provided a network structure—in other words, SNetworkMsg—defined in NetCommMessages.h, it extracts the stream out of it, obtains the instance ID of the object, which data is packed ahead, searches the object in the collection, and passes the stream to its NetworkSerialize method, where again, it gives the application programmer a chance to get the data out of the stream and update object variables.

bool CNetworkObjectManager::ReceiveNetworkPacket
   ( const SNetworkMsg* pMsg )
   // Process incoming message...
   if ( pMsg->sHdr.uType == gs_NetworkObjMsgID )
      CNetworkObject::TDataStream& rStream =
         ( CNetworkObject::TDataStream& ) pMsg->cDataStream;

      uint32 uInstanceID = 0;
      rStream >> uInstanceID;

      while ( 1 )
         TSerializeObjectList::iterator iter =
            m_SerializeObjectList.find( uInstanceID );

         if ( iter != m_SerializeObjectList.end() )

            CNetworkObject * pNetworkObject = iter->second;

               NetworkSerialize( CNetworkObject::ceWrite , rStream );

            // Get the next uInstanceID from the stream
            if ( !rStream.CheckReadSize( 0 ) )
               rStream >> uInstanceID;

      return true;

   return false;

It is to note that this model, so far, does not follow Request/Response message orientation. This is an automatically updating mechanism where information exchange is unidirectional. Either the host or client, whoever has ownership, will transmit the object states, and other nodes will be in listening mode. CNetworkObjectManager exposes a method named SetOwnership(bool), which can be used to set the ownership. The node having ownership will be a transmitter and others nodes at the network will be the receiver.

The Sample Application

The sample attached is an MFC application, in which you can either host the application, or connect it to an existing host. An object of class CMyNetworkObject, inherited from CNetworkObject, is created at both the host and client ends, but the host only has the ownership; therefore, after connecting the client with the host, you can click the "Update Object" button; this will increment the object's internal value by a factor of 10, but only the host interval value will be replicated. In the OnTimer() method of the main application, the heartbeat to the distributed network object is pumped, which lets the entire mechanism work. In the sample application, I've created only one object of the class derived from CNetworkObject and updated its value and left the task to create more objects and see how they get updated for you.

In OnTimer(), messages from the underlying network layer are fetched and processed. In the current implementation that I've done, there are two types of messages:

  1. System messages (Host Launch, Host Shutdown, Client Connected, Client Disconnected)
  2. Data message

First, system messages are processed, followed by other messages containing data. You may change it with your favorite network library.

void CNetCommDlg::OnTimer( UINT nIDEvent )
   if ( NetCommIsSystemMessageAvailable() )
      EMsgType eMsgType;
      void* pData = NULL;

      while ( ( eMsgType = NetCommGetSystemMessage( pData ) )
             != ceMsg_None )
         switch ( eMsgType )
            case ceSysMsg_HostLaunced:
               AddMsgToUI( "Host Launched..." );

               CString strMsg;
               strMsg.Format( "Object Value: %d" ,
                              m_oObject.GetData() );
               AddMsgToUI( strMsg );

            case ceSysMsg_HostShutdown:
               AddMsgToUI( "Host Shutdown..." );

            case ceSysMsg_ClientDisconnected:
               AddMsgToUI( "Client Disconnected..." );

            case ceSysMsg_ClientConnected:
               AddMsgToUI( "Client Connected..." );

   if ( SINGLETONINSTANCE( CNetworkObjectManager ).IsOwner() )
      SINGLETONINSTANCE( CNetworkObjectManager ).SerializeObjects();
      while ( NetCommGetNoOfMessages() > 0 )
         SNetworkMsg * pMsg = NetCommGetMessage();
         if ( SINGLETONINSTANCE( CNetworkObjectManager ).
            ReceiveNetworkPacket( pMsg ) )
            static CString strValue;
            strValue.Format( "Object Value: %d" ,
                             m_oObject.GetData() );
            AddMsgToUI( strValue );

When system messages are processed, The SerializeObjects() method of the CNetworkObject class is invoked. In the case of a node, which has the ownership (in this case, the host), it serialize the objects and transmit their states. If the current node isn't a transmitter, it receives network messages and updates the relevant objects by passing the network message structure (containing the stream) to the ReceiveNetworkPacket of CNetworkObjectManager, which in turn, unpack the stream, find the corresponding object, and pass the stream to its polymorphic method (in other words, NetworkSerialize).

You may use it in non-MFC applications as well. You just have to make sure that you'll pump SerializeObjects & ReceiveNetworkPacket adequately.


Situations where changes are very frequent and force you to re-engineer system areas can take benefit of this communication model. In the main application, you can make an assignment in a natural way, and you'll get the relevant updates at the other side without any change elsewhere.

This article was originally published on October 22nd, 2007


Most Popular Programming Stories

More for Developers

RSS Feeds

Thanks for your registration, follow us on our social networks to keep up-to-date