I was curious to know the size of the internal winsock read-buffer associated with a telnet socket. I wanted to know how long it would take to fill it on my internet connection, and I wanted to set my recv-buffer the same size so that a single recv would flush it.
Several Microsoft references stated that the default size was 8K. So I called getsockopt with parameters SOL_SOCKET, SO_RCVBUF to confirm this. It returned 1244364 bytes - over 1.2 MB!
I didn't trust either figure. I programmed a loop which did a blocking-recv on a remote telnet server, using a huge 2 MB buffer.. No matter how long I ran it for, the maximum size returned to each recv was 128K. recv would never return more than 128K bytes, even though it was passed a buffer-size of 2M.
I thought that this 128K might be an artefact of my internet-speed. So I had the program sleep for 20 secs after each recv. I thought that this would give the internal buffer plenty of time to fill up to its capacity. But the size returned to recv actually shrank - to exactly 74055 byte.
None of this makes a lot of sense to me. Is the "actual" internal buffer size 128K? Then what is getsockopt returning? And why does Sleep (and SleepEx) have such a bizarre effect? Surely Windows doesn't leave an internal buffer half-full just because the high level process is asleep.
Incidentally, I also tried changing the buffer-size using setsockopt. This function returned without error but had no effect on the buffer-size, regardless of its value, and regardless of whether a connection was open. A subsequent getsockopt always returned 12443644 bytes. I found a reference which stated that this is characteristic of some Windows Sockets implementations. So much for setsockopt.
I am using Microsoft C on XP Home, winsock2.h, wsock32.lib. My internet speed is 512 Kbps.
MikeAThon
July 20th, 2007, 11:30 AM
Try your experiment using the loopback address (127.0.0.1). In other words, run both server and a client on one single machine, and let the client connect to 127.0.0.1 and send to that address.
Transmissions on the loopback address never reach the physical wire, so you will eliminate all issues regarding bandwidth of the network/Internet.
The default buffer size, for both send and recv in newly-created sockets, depends on the OS, but is typically 8K. Are you certain that you are using getsockopt correctly? Is the telnet server your own code? Same for the client?
Maybe it's best if you posted some code.
Mike
fusgerm
July 20th, 2007, 11:19 PM
Thanks Mike, much appreciated. You were right about getsockopt - it was my printf at fault, displaying the buffer address of the int instead of its value (oops!). The actual result was indeed 8K.
Yet this 8K is misleading.
I wrote a server on 127.0.0.1 as you suggested. It sent 1M in fixed-size 256-byte messages. My client looped doing a recv into a huge 512M buffer, sleeping for 10 ms after each recv. The results were:
getsockopt RCVBUF is 8192 bytes
Sleeping for 10 ms after recv
256 bytes received
24832 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
34304 bytes received
32768 bytes received
29104 bytes received
34128 bytes received
24320 bytes received
34304 bytes received
29360 bytes received
25680 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
25856 bytes received
3840 bytes received
Total of 1048576 bytes (1024KB) received
Then I changed the client to send the same amount of data in fixed-size 4K messages. The results were:
i>getsockopt RCVBUF is 8192 bytes
i>Sleeping for 10 ms after recv
i>4096 bytes received
i>23400 bytes received
i>25752 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>40960 bytes received
i>24576 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>28672 bytes received
i>12288 bytes received
i>Total of 1048576 bytes (1024KB) received
Explanation? It seems to me that if there are lower-level buffers full of data at the time recv is called, Windows not only gives recv the contents of RCVBUF but also flushes lower-level buffers at the same time. That would explain why recv gets more data (typically 28672 instead of 25856 bytes) when larger messages are sent - since there is more framing overhead per packet at the lower level.
I tried changing RCVBUF using setsockopt. I increased it from 8K to 32K. The results (still on 4K messages) were:
i>getsockopt RCVBUF is 8192 bytes
i>getsockopt RCVBUF is now 32768 bytes
i>Sleeping for 10 ms after recv
i>7300 bytes received
i>74620 bytes received
i>69296 bytes received
i>66156 bytes received
i>77540 bytes received
i>69632 bytes received
i>69632 bytes received
i>77824 bytes received
i>69580 bytes received
i>77540 bytes received
i>65872 bytes received
i>69632 bytes received
i>77824 bytes received
i>69632 bytes received
i>69632 bytes received
i>36864 bytes received
i>Total of 1048576 bytes (1024KB) received
I would have expected an extra 24K on average per recv, but these sizes are bigger than that. Not sure why.
I repeated the test a few times, and occasionally recv got huge results:
i>2048 bytes received
i>73984 bytes received
i>73216 bytes received
i>69404 bytes received
i>73700 bytes received
i>70608 bytes received
i>216880 bytes received
i>67176 bytes received
i>65688 bytes received
i>64512 bytes received
i>69660 bytes received
i>68172 bytes received
i>74136 bytes received
i>32768 bytes received
i>26624 bytes received
i>Total of 1048576 bytes (1024KB) received
and even:
i>524288 bytes received
i>70144 bytes received
i>68200 bytes received
i>72856 bytes received
i>73984 bytes received
i>72704 bytes received
i>65536 bytes received
i>65280 bytes received
i>32768 bytes received
i>2816 bytes received
i>Total of 1048576 bytes (1024KB) received
If I eliminate the Sleep after the recv, I get results like this:
i>getsockopt RCVBUF is 8192 bytes
i>getsockopt RCVBUF is now 32768 bytes
i>Sleeping for 0 ms after recv
i>4096 bytes received
i>225280 bytes received
i>65536 bytes received
i>12288 bytes received
i>88936 bytes received
i>9368 bytes received
i>3760 bytes received
i>8528 bytes received
i>12288 bytes received
i>12288 bytes received
i>12288 bytes received
i>12288 bytes received
i>65536 bytes received
i>12288 bytes received
i>3760 bytes received
i>8528 bytes received
i>110540 bytes received
i>4148 bytes received
i>32768 bytes received
i>3760 bytes received
i>57008 bytes received
i>94544 bytes received
i>64696 bytes received
i>9368 bytes received
i>3760 bytes received
i>8528 bytes received
i>12288 bytes received
i>9652 bytes received
i>10828 bytes received
i>3760 bytes received
i>336 bytes received
i>12288 bytes received
i>16048 bytes received
i>8528 bytes received
i>4096 bytes received
i>12288 bytes received
i>3760 bytes received
i>8528 bytes received
i>Total of 1048576 bytes (1024KB) received
Hmmm. Perhaps I'm wasting my time trying to figure out how Windows has implemented sockets. I'm inclined to leave setsockopt alone, and just set my recv-buffer to 128K, which seems to be the maximum I ever get from my internet-server.
Mike - to answer your other question, the internet-server is not mine but a third party's. It telnets streaming data to me all day - over 3 ports concurrently. I monitor all 3 ports using select().
My main problem with the service is recovery after a brief internet-outage of 15 secs or more. In that case, one or more sessions are usually lost but my select() is not informed. When select() times out, I reconnect, but then their server doesn't let me tell it where I want to resume from - even though its messages include a sequence-number. Instead, it has to resend everything from start of day - upto 80 MB!
My internet connection is wireless (all I can get), and goes down briefly every few days. They blame my internet connection for being unreliable. I blame their server for a flawed design.
MikeAThon
July 21st, 2007, 02:59 PM
Interesting experiment, but I am not certain that it shows what you thought it might show.
First, please say something that shows that you understand that TCP is a streaming protocol, and not a message-based protocol. You mentioned, for example, that you sent "messages" of 4k, and you might have been referring to the fact that in your code, the length of the transmission was prepended to each transmission. But please say something that shows that you are not surprised when the client receives more than 4k (for example) in one single recv().
Seocnd, with a decreased Sleep() time, from 10 ms to 0 ms, you would not necessarily see significant increases in the amount of data received. This is because of the nature of Sleep(). A call to Sleep(10) will usually not result in a 10 ms sleep. It will usually result in a much longer sleep, something close to a multiple of the CPU hardware clock tick of 10 ms, so it could be 10 ms one time and 20 ms the next time. In addition, the most important effect of Sleep() is a thread-yield to an equal or higher priority thread. So if there is another thread around (and there is, namely the server), that other thread will get a timeslice, which could result in your client sleeping for an extended period.
Likewise, a call to Sleep(0) will yield the thread's time-slice, and will usually result in sleep that's similar to Sleep(10). See this article for actual experimental results on Sleep(): "Quantifying The Accuracy Of Sleep" at http://www.codeproject.com/system/sleepstudy.asp
So, the performance of Sleep() probably explains alot about the relatively small increase in recv() bytes when going from 10 ms down to 0 ms.
As for the actual number of bytes being recv'd, remember that the 8k internal buffer is an initial size only, used in the "advertising" that occurs when a connection is first established, and that the internal TCP window size changes dynamically with time and use. TCP has relatively complex algorithms for estimating round trip time (RTT), so as to adjust the size of the "sliding window" that it reserves for buffering transmissions (until an ACK is received). I am unclear on whether a similar dynamicism occurs in the size of the receive buffer (I don't think it does), but obviously changes in the size of the send buffer will affect the overall efficiency of the throughput (it was designed to this).
For recommendations on the size you should use for your buffer, do some research on the relation of RTT and network bandwidth. Here's a quick overview:
There's no reason to set the buffers as large as you have done (i.e., to 128k). The typical estimate for size is given by this equation:
size = Round-Trip-Time * Bandwidth
where RTT is the round trip time between TCP's sending of data and receipt of its ACK, and Bandwidth is the bandwidth of the network.
For a local network of 100 MBits/sec, a really awful RTT is around 1 millisecond. Using the above equation, you get a buffer size of 100000000 mbps divided by 8 bits per byte times .001 seconds = 12500 bytes.
For Internet connectivity over (say) ADSL, the bandwidth is around 1.5 mbits/sec and the RTT is around 75 milliseconds. So the buffer should be around 1500000/8*.075 = 14062 bytes.
In normal Windows installations, the default size of the buffers is 8K. So in general, yes, it makes sense to increase them. But increase them to 16K, not to 128k like you have done. This is simply wasteful of resources, which might not show up if you are creating only a single socket, but which will quickly manifest itself as more and more sockets are created.
Mike
fusgerm
July 22nd, 2007, 06:15 AM
Thanks Mike, most enlightening.
I was aware that TCP is a streaming protocol and that this accounts for a recv() size that bears no relation to the send() size. What surprised me was that the recv() size can be so huge - upto 128M or 512M, when the SO_RCVBUF is 8K or 32K. The reason, I see now, is that the TCP Window, which recv() drinks from, varies dynamically during reception, and that SO_RCVBUF plays only a minor role during connection establishment.
Microsoft's documentation is somewhat misleading, e.g.:
"An application can specify the maximum receive window size for a connection by using the SO_RCVBUF Windows Sockets option when a connection is initiated." (http://www.microsoft.com/technet/community/columns/cableguy/cg1105.mspx)
or their SDK help: "SO_RCVBUF specifies the total per-socket buffer space reserved for receives. This is unrelated to SO_MAX_MSG_SIZE or the size of a TCP window."
I would have expected that getsockopt on a connected socket would return the dynamic size of the TCP Window, but evidently that is not so.
I wasn't aware that Sleep() does a mandatory thread-switch. That certainly makes my simulation unrealistic for low msecs of Sleep(), since an efficient telnet-client should NOT time out between blocking recv() calls. Nevertheless, even without the Sleep(), my testing shows that recv() may be given as much as 102K locally, or 128K over the internet.
In my case, the calculation RTT * bandwidth works out to under 5K, since my internet speed is 512 Kbps, and the ping-time to the remote server is 70ms, so I can leave SO_RCVBUF alone at 8K.
I still see some benefit in calling recv() with a largish buffer of 32K or so, because my tests show that a recv() size of over 20K (and up to 128K) is often returned during reception from my internet-server, and if my recv() buffer is much smaller than that then I shall be doing unnecessary select() and recv() calls.
Cheers
MikeAThon
July 22nd, 2007, 01:27 PM
I'm glad that this was helpful.
It might also be good to read a few "TCP Overview" types of resources. Here's an arbitrary one, found through a quick Google search: "TCP Overview" at http://www.cse.fau.edu/~sam/course/netp_htm/tcpOverview2.pdf
Mike
codeguru.com
Copyright Internet.com Inc., All Rights Reserved.