Optimization/Theory Questions

[Whitebread]

Hello all,
I am working on a pretty simple app that just transfers data from one machine to another as fast as possible. The app works fine, but I have some theory questions for the gurus:

1. If I am using a TCP Socket does it matter how much I try to send at a time? Since TCP has optimization algorithms built in.

2. If question one does matter then, assuming a closed system where I am the only one talking on the line, what would be the optimal send size? and Why?

3. What about my receive buffer? Should it be the same size as the send buffer? or larger? if so by what factor? but surely not smaller, right?

4. I am running on 100 Ethernet, so what kind of throughput speed should I expect? I am using this simple calculation (# of Bytes transferred)/ (timeFinish-timeStart).

5. Any other hints or tips you have for me would be a great help too.


Sorry for lumping these all together in one post.

Thanks in advance.

[j0nas]

Here are some answers:
1. Probably not, but you may waste CPU time if you call send() with a 1-byte buffer each time.

2. Don't know. I usually have my buffer sizes set to al least the size of 1 MTU (around 1500 bytes).

3. Same as answer 1.

4. Don't know.

5. Keep it simple. Blocking sockets with large buffers. I've seen very strange problems in some nets, so if you want to really do a robust solution, you must test your client/server app on several network speeds and configuration. Some routers/switches lets you set the bandwitdh, as well as full- and half duplex modes.

[Mathew Joy]

I am working on a pretty simple app...
I think it is all the same for simple application. Fine tuning may not be worth because there won't be any noticable difference. Especially if you are working on a LAN. But all depend on how your application behave. But when you are talking of a high end scalable server and transfers are done on remote machine, then it does matter. However, there are lots that come in optimization and it can't be covered in a few posts in little time. Because a lot of factors come into play,like number of connections, amount of data that is transfered, packet size, the communication behavior, the RTT etc, it is hard to give you an advise that helps your application.


1. If I am using a TCP Socket does it matter how much I try to send at a time? Since TCP has optimization algorithms built in. One thing that does matter is how you are doing. The performance increase as you go down the order; Wrappers->apis->kernel->hardware. The best performance will be got obviously in the hardware level. If the wrappers aren't suited for your needs, then you get a poor performance. One of the reason's why IOCP has the highest performance that no other IO model will get is because calls once posted are handled at the kernel level.

How much data to send depends on how you are viewing it. If you are very counsious about the resource you are using, then the send/recv buffer (the application buffer) should a multiples of 4k, because when a call is made and is waiting, then portion of memory is locked in pages. That means if your buffer is only 10b a chunk of 4k will be locked for your 10b buffer.

2. If question one does matter then, assuming a closed system where I am the only one talking on the line, what would be the optimal send size? and Why?As I said previously, it depends on the view you are talking. Comming to the protocol level, each packets are send as an Ethrenet frame. An Ethernet frame's size is 1500b. 20b usually goes for IP header (40b if options are included ) and another 20 for TCP header, the rest for data. If you are sending large amount of data in one go, then this overhead is negligible.
3. What about my receive buffer? Should it be the same size as the send buffer? or larger? if so by what factor? but surely not smaller, right?One thing that you have to do for sure is the receive buffer should never be small. Because if the sender is faster, then you can experience something like the 'silly window syndrome'. The the 'delayed ack' algorithm solves this to an extend, but this is something you should avoid. As said above, multiples of 4k is the best.

4. I am running on 100 Ethernet, so what kind of throughput speed should I expect? I am using this simple calculation (# of Bytes transferred)/ (timeFinish-timeStart).I can't really say that. But obviously you simply can't expect full thruput. For every segment that is sent there should be ack packets comming back. plus retransmission of lost packets, other packet that is floating around etc.

5. Any other hints or tips you have for me would be a great help too.It is really hard to say without knowing quite a few details about your system. My answeres above were without knowing what exactly you are looking for. But those could be used at apropriate places.

There are quite a lot of tips and tricks explained in this forum. Try doing a small amount of searching

[Whitebread]

Thanks for the responses!

Basically my system is just transferring alot of data like 500MB to another machine on the LAN as fast as possible. The critical part of this app is that all the data show up as fast as possible on the other end, that is really all that matters. I have pretty much looked through the whole forum and others, but every optimization post has to do with large servers handling many connections. My problem is a bit different because I only have one connection and want to push it to the max. So, I was looking for some responses from people that have done this before and what they learned or some gurus that could say theoritically what the best solution would be. There has to be some formula for getting max throughput?

Sorry to bug you all with this, but I would just like to know that I have put together an optimal solution for this.

[Tomcat]

The amount of data in each send call should not affect network performance as long as you do try to send faster than 100 Mbps, but obviously it's not optimal for CPU load to call it with 1 byte at a time. As I understand it the TCP stack fills its internal buffer and sends as big packet as possible (MTU) either when that much data is available to send or when a certain time has passed since the first call. You can verify this yourself by running a network analyzer (like the excellent and free Ethereal (http://www.ethereal.com/)).

On the receiving end you simply remove the number of bytes you pass to recv from the buffer, and it's important that you call it often enough to avoid slowing down the connection. I would suggest implementing a test mode that just throws away data to make sure that writing the data to disk doesn't slow things down.

Depending on the type of data you may also want to try using compression of some kind.

[Mathew Joy]

If you are using w2k use overlapped send/recv using IOCP notification. If you are sending a file TransmitFile() gives the best performance. Along with the above suggestions, post multiple sends and the sender and multiple recvs at the reciever side as much as necessary. While using multiple recvs and multithreads make sure the dequeued packets are in order.