Some Design Suggestions

[scathenC]

Hi All,

I have written an app that continuously processes large amounts of data in a specific way that requires optimum speed and thus all of the computers resources. The app also requires constant data input and gives constant data output. As complex processing is required at all stages I have built three machines for use as follows:

(Machine 1.) - Convert raw realtime data into input data
(Machine 2.) - Process the input data and send output data
(Machine 3.) - Convert the output data to usable information

Now I would like to know what people on these forums opinions are as to the fastest way Machine2 can retrieve and send data between the other machines.

[nuzzle]

Originally Posted by scathenC

Now I would like to know what people on these forums opinions are as to the fastest way Machine2 can retrieve and send data between the other machines.

Are the "machines" physical computers? Then you need fast communication hardware.

I first suggested GPIB here but I realize there are faster ways.

[JohnW@Wessex]

Would a Gigabit Ethernet connection be fast enough,
piping the data through sockets?

You could connect them through dedicated network cards with crossover cables to get the least delay.

[scathenC]

Yeah my first choice at the moment would probably be UDP over Gigabit. Whilst I have been coding for a long time, networking isn't my specialty, and I was eager to see if anyone knew of any better methods. There is no specific requirement for speed, other then simply "as fast as I can do it". There is no need for flow control etc, just something as basic as it can get.

Thanks for the replies guys.

[originalfamousjohnsmith]

Originally Posted by scathenC

Yeah my first choice at the moment would probably be UDP over Gigabit. Whilst I have been coding for a long time, networking isn't my specialty, and I was eager to see if anyone knew of any better methods. There is no specific requirement for speed, other then simply "as fast as I can do it". There is no need for flow control etc, just something as basic as it can get.

Thanks for the replies guys.

Well, I assume you want to know about code not hardware. Obviously for hardware the answer is whatever is rated fastest or whatever you have.

There's no magic involved as far as design goes. You need separate processes on eachmachine running/accepting UDP using asynchronous IO. The hardware handles the work really and queues any network activity so it's not very hard at its core, but lockless multithreaded code can be difficult. It's more a matter of learning how your networking API and OS work and doing it than design, though.

[JohnW@Wessex]

Originally Posted by scathenC

Yeah my first choice at the moment would probably be UDP over Gigabit.

You have to be aware that UDP does not guarantee the order of arrival or even delivery of the packet. This *may* not be a problem on a back-to-back or internal network. TCP would probably be fine and it will catch the occasional glitch in transmission. You can cut down delay times by setting the TCP_NODELAY option so that packets are sent immediately.