Winsock send function...

[dlaugt]

Hello,

I would to understand with a lot of details: how works the send function?

I look in the microsoft msdn that there are options like:
- SO_SNDBUF: Buffer size for sends.
- SO_SNDLOWAT: Sends low watermark.
- SO_SNDTIMEO: Sends time-out.
- SO_MAX_MSG_SIZE: Maximum outbound (send) size of a message.

For send function return, the msdn says:
"If no error occurs, send returns the total number of bytes sent, which can be less than the number indicated by len. Otherwise, a value of SOCKET_ERROR is returned, and a specific error code can be retrieved by calling WSAGetLastError."

I guess that when I call the send function with blocking mode, the winsock API stores data to send in an internal send buffer. After I hope that the API try to send data and return the status of this operation.

But I feel that the send function give the hand back before to be sure that data are correctly sent and data is received by the other.

Somebody can explain me the detailed functional of this function?
How can I be sure that data are correctly sent and data is received by the other?

Thanks.

[Mathew Joy]

Originally posted by dlaugt
- SO_SNDBUF: Buffer size for sends.This is an intermediate buffer between your app and the tcp lever buffer. It is best to leave the buffer as it is. - SO_SNDLOWAT: Sends low watermark. This is a BSD option and not supported in windows - SO_SNDTIMEO: Sends time-out. When you are using blocking sockets send waits forever. This option times out the wait to the specified time in millisecs - SO_MAX_MSG_SIZE: Maximum outbound (send) size of a message.This option is only applicable to message oriented sockets. SOCK_DGRAM
For send function return, the msdn says:
"If no error occurs, send returns the total number of bytes sent, which can be less than the number indicated by len. Otherwise, a value of SOCKET_ERROR is returned, and a specific error code can be retrieved by calling WSAGetLastError."

I guess that when I call the send function with blocking mode, the winsock API stores data to send in an internal send buffer. After I hope that the API try to send data and return the status of this operation.Yea...send() will return only when the process is complete. Technically speaking, the process might not always be like that. The intermediate buffer might be bypassed. But as msdn says you should always check if the data that was submitted was sent successufully by checking the return value.

But I feel that the send function give the hand back before to be sure that data are correctly sent and data is received by the other.

Somebody can explain me the detailed functional of this function?
How can I be sure that data are correctly sent and data is received by the other?
Thanks. If a send call completes successfully, the tcp stack at the other end will get it properly and in order. The rest is left to the client at the other end to retrive it and structure it properly.

[Andreas Masur]

Originally posted by ishtiaque_NSU
If I send a 4K buffer (or less than 4k) using send() function then is it garunteed that the whole buffer will be sent in one shot. That is the send function will always return the same number that I passed in its len parameter provided that the TCP layer send buffer is Empty.
'send()' and 'recv()' do not guarantee that all the data are being sent within one call. Therefore your send and receive functions should look like this (only client send function and server receive function)...

// Client send function
int Send(void *pvBuffer, int iStillToSend)
{
int iRC = 0;
int iSendStatus = 0;
timeval SendTimeout;
char *pBuffer = static_cast<char *>(pvBuffer);

// Set timeout
SendTimeout.tv_sec = 0;
SendTimeout.tv_usec = 250000; // 250 ms

// As long we need to send bytes...
while(iStillToSend > 0)
{
iRC = select(0, NULL, &fds, NULL, &SendTimeout);

// Timeout
if(!iRC)
return -1;

// Error
if(iRC < 0)
return WSAGetLastError();

// Send some bytes
iSendStatus = send(Socket, pBuffer, iStillToSend, 0); // Socket is of type
// 'SOCKET' and is
// the current
// connection socket

// Error
if(iSendStatus < 0)
return WSAGetLastError();
else
{
// Update buffer and counter
iStillToSend -= iSendStatus;
pBuffer += iSendStatus;
}
}

return 0;
}


// Server receive function
int ReceiveNBytes(SOCKET Socket, void *pvBuffer, int iStillToReceive)
{
int iRC = 0;
int iSendStatus = 0;
timeval ReceiveTimeout;
char *pBuffer = static_cast<char *>(pvBuffer);

// Set timeout
ReceiveTimeout.tv_sec = 0;
ReceiveTimeout.tv_usec = 250000; // 250 ms

fd_set fds;

FD_ZERO(&fds);
FD_SET(Socket, &fds);

// As long we need to send bytes...
while(iStillToReceive > 0)
{
iRC = select(0, &fds, NULL, NULL, &ReceiveTimeout);

// Timeout
if(!iRC)
return -1;

// Error
if(iRC < 0)
return WSAGetLastError();

// Receive some bytes
iReceiveStatus = recv(Socket, pBuffer, iStillToReceive, 0);

// Error
if(iReceiveStatus < 0)
return WSAGetLastError();
else
{
// Update buffer and counter
iStillToReceive -= iReceiveStatus;
pBuffer += iReceiveStatus;
}
}

return 0;
}

[dlaugt]

Thanks for yours answers.

Mathew, you said:
If a send call completes successfully, the tcp stack at the other end will get it properly and in order. The rest is left to the client at the other end to retrive it and structure it properly.
Ok now if I have the following situation:
The server call the Send function made by Andreas. Just after that this function is done the connection is interrupted.
When the connection is interrupted, the client call the ReceiveNBytes function made by Andreas.
Does the client receive bytes sent by the server?

My problem is I have a GPRS communication using TCP/IP. This GPRS communication can be not reliable (with maybe a lot of disconnections). I want to be sure to don't loose any byte without to implement an user acknowledge. I would to use, if it possible, the TCP/IP acknowledge. My problem is there are intermediate buffer between my app and the tcp lever buffer. And I would like to be sure to retrieve data from theses buffers when the communication is broken. I try to fix the size of the intermediaire buffers to 0. It's works for GPRS communication. But for local communication, no byte can be transmitted.

Well I have too much questions and I count on gurus to help me.

[Mathew Joy]

Please clarify what do you mean by 'disconnect'. Is it a socket closure, cable pull out (something like that) or what? And what effect do you 'think' your application will get by setting the buffer size 0. As I said, the buffers will be bypassed if you can make the sender/receiver busy. There will be a small decrease in performace (and other issues) at times if the buffer size is set to 0. So it is generaly advised to leave the buffer as it is.

[dlaugt]

I would like to make a soft that is reliable on any situation. 'Disconnect' for me can be a socket closure (by shutdown and closesocket), cable pull out, interference on the air (GPRS) or something like that.

I will make some tests to know how react winsock in theses situations. I will try to put the receive buffer to 0, like this I will be sure that all bytes sent by the sender are received by the receiver. Like this when a disconnection comes, sender go out of the send function with a number of bytes transferred and the receiver go out of the recv function with the same number of bytes received. This solution is not performance because the sender is dependant of the receiver. If the receiver is busy and don't read, the sender can't send.

If you have another solution, I take. One solution is to retrieve data in the internal buffer after a disconnection. But I don't know if it's possible. Somebody knows?

[Mathew Joy]

Originally posted by dlaugt
I would like to make a soft that is reliable on any situation. 'Disconnect' for me can be a socket closure (by shutdown and closesocket), cable pull out, interference on the air (GPRS) or something like that.
Each disconnect means different things and it is meant to be handled differently. If it is a socket closure, and if it is unexpected (during the data transmisssion) then that means some error has happened. The system should be designed so that it can be reconnected and the process can resume. You should use shutdown() so that the send() and recv() is not broken.

In the case of cable pull out, there is no disconnection. You have to use your own timeout schemes to detect it and close the connection if you don't want to wait till the default timeout.

I don't know what is the interference on air, so I can't comment much on it. But I doubt if it will result in a disconnection.

I will try to put the receive buffer to 0, like this I will be sure that all bytes sent by the sender are received by the receiver. Like this when a disconnection comes, sender go out of the send function with a number of bytes transferred and the receiver go out of the recv function with the same number of bytes received. See...when you set the buffer to 0, the incomming data is buffered at the tcp level. Anyway, till disconnection your data will be buffered, first at the tcp level, then in the intermediate level. So I don't see any point in setting it as 0.

[dlaugt]

See...when you set the buffer to 0, the incomming data is buffered at the tcp level. Anyway, till disconnection your data will be buffered, first at the tcp level, then in the intermediate level. So I don't see any point in setting it as 0.If you have buffers at the tcp level, why we need intermediate buffers? What is the role of these intermediate buffers? What is the role of these buffers at the tcp level? But what you said Mathew is true for local network (LAN). I make some tests...

I set the receive buffer to 0 and the send buffer to 0 or 1. It's work for GPRS communication (with a GPRS Wavecom modem). The sender waits that the receiver reads bytes from the socket. But it doesn't work for local communication (local network LAN). I don't know why... If the send buffer is set to 0, the sender can't send any bytes. If the send buffer is set to 1, the sender sends bytes without waiting that the receiver reads bytes from the socket.

I'm surprise that these two communications don't give the same result. Is somebody know more about buffers at the tcp level?

[Mathew Joy]

Frankly speaking, I don't have a first hand idea of what you are trying to achive. Anyway let me give you some technical information, w.r.t the buffer, of what actually happens at the time of send and recv. Maybe this will guide you to make decisions regarding the intermediate buffers.

When you issue a send() call, if there is sufficient space in the buffer, it copies the data to the buffer and the call is completed. Now when the buffer is full, the application send buffer is locked (which means it cannot be paged out). Now after the data in the buffer is processed, the next data is fetched directly from your application buffer. In otherwords, once the app buffer is locked, the data is handed down to the TCP directly for processing, completely bypassing the intermediate buffers. Till the time the data is handed down for processing, your send() will be blocked in the case of blocking sockets or will result in a WSA_IO_PENDING error in the case of non-blocking.

recv() is also simillar. The only difference is that the app buffer is locked when the intermediate buffer is empty, and hence bypassed. Thats why I said in my previous post as long as you can keep the sender/receiver busy, the buffer is bypassed.

Well, a natural question arises. why do you need the intermediate buffers? Isn't the tcp level buffer enough? The answer is resource management. TCP buffer uses Non-paged memory while the intermediate buffer is normal pagable memory. Non-paged memory is very precious resource needed very much by winsock. So if you don't manage it properly, you may end up using up the resource, which may lead to system crash. Thats why I said in my previous post it is generaly best to leave the buffer as it is.

Hope I answered your doubts clearly. The rest is left to you.

:thumb:

[Mick]

Along with what mat said...skip down to the 'who manages the buffers' part

http://msdn.microsoft.com/msdnmag/issues/1000/Winsock/

[hlaalu]

Sorry for resurrecting such an old thread, but this post is the number one hit when you google 'winsock send,' and, while there is lots of good information in this thread, it doesn't seem like dlaugt's original question was answered completely.

I think the core of dlaugt's question is:

"How can I know with absolute certainty what data has been (correctly) received by my peer? Take into consideration that my peer may not be reliably connected to me. E.g. cable pulls, power loss, etc. must be handled properly."

His concern about the 'send' function is that it appears (perhaps because of this intermediary buffer) that 'send' may return a number that does not, in fact, accurately reflect how many bytes his peer was able to receive. It may only be reporting how many bytes were able to be written into the local send buffer.

If the connection to his peer is lost (e.g. his peer's plug gets pulled), and there is still data waiting in the intermediate buffer, he may be in a state were 'send' "reported" that, for example, 100 bytes had been 'sent', but in reality 50 of those bytes are still left in the intermediate buffer -- never to be sent, because his peer has lost its connection to the network and is unable to receive any more data.

Perhaps his question is then, how can I know for certain, in such a situation, what has been sent and received properly, and what hasn't? If, for example, I re-establish my connection to the peer at some later time (say, after the peer has gotten plugged back in), how should I know what data the peer still lacks, so that I can resume transmitting the data, and pick up exactly where the peer left off reading.

From the link that was posted:

http://msdn.microsoft.com/msdnmag/issues/1000/Winsock/

The author claims that a coder might be tempted to disable this intermediate buffer, so that data does not get 'orphaned' in there. But the author goes on to say, that this is not a smart choice to make, because even if you disable the buffer, you still are not guaranteed that your peer has properly acted upon the data you transmitted. This, he claims, is because even though the peer's TCP stack has acknowledged receipt of the data, there is no guarantee that the application lying on top of the TCP stack has, or will ever read the data. E.g., in the event of a power loss, the data may be acknowledged as received by the TCP stack, but lost before the application lying on top of the stack has time to read it out of its receive buffer and act upon it.

The author's scenario implies that disabling the intermediate buffer does in fact, cause 'send' (in blocking mode) to return a number of 'bytes send' that accurately reflects what the peer has acknowledged receipt of. Which is interesting to note, if you have need for such functionality.

I agree with the author of the article. You can use a TCP stack to guarantee that data arrives at your peer uncorrupted, even in the face of adverse network conditions. But, you probably shouldn't rely on TCP to report back to you what has actually been successfully "told" to the peer, in the sense that "telling" the peer something over a network connection implies not only receipt of the raw TCP data stream, but also successful processing of that data by the application on top of the stack.

Perhaps a good example of how to use TCP to "reliably" transmit data even in the face of connection loss, is to look at how FTP resume works. In the event of a connection loss, FTP resumes its work by the peers talking to each other about what data they have. E.g. the file sender asks the file receiver "do you have this file?" If the answer is yes, the file sender asks "oh, and how many bytes of the file do you have?" If the receiver's amount is less than the sender's, the sender says "oh, I have some more of that file for you, go ahead and append this data I'm about to send you ..." That's to say, the guaranteed delivery of information is implemented at the application layer, not the transport layer.

Perhaps some TCP/networking/reliability guru can weigh in on this ...

[MikeAThon]

First off, it's important to understand something that was correct in dlaught's very first post, but which got confused in subsequent posts: dlaught was correct when he speculated that send() returns as soon as your data is accepted by winsock in the internal winsock buffer (whose size is set by SO_SNDBUF). At that point in time, the data has not even been sent onto the wire. It will be sent at a later time. As a consequence, it is impossible to determine through the send() function if the client receives your data successfully.


The article at http://msdn.microsoft.com/msdnmag/issues/1000/Winsock/ is an article about I/O completion ports, which is a very sophisticated architecture relative to the lowly send() function. It's true that IOCP will let you set the SO_SNDBUFF size to zero so that winsock will not waste memory or time in copying your data to an internal winsock buffer. Instead, winsock will simply use the memory that you have allocated, which you guarantee to winsock will remain valid throughout the entire I/O operation. That's part of why the IOCP architecture is "scalable". Let's put this to the side, however, since it's such a large topic, and in any event is not really relevant to the basic question you are asking, which is: how can the sender know if the recipient has received the data that was sent.

To answer this question, you need to recognize that the recipient is broken into layers (remember the ISO/OSI model?). For these purposes, we will focus on the TCP layer and the application layer (i.e., the client/recipient program). The TCP layer uses ACKs and sequence numbers to guarantee arrival of each IP datagram. So, if TCP is working, then it's guaranteed by TCP that all data sent by the sender is received into the TCP layer.

The same cannot be said for the application layer. The application layer might not have received all data, even if the TCP layer is working correctly, for any number of reasons. For example, it might have a logic flaw that prevents the application from calling recv(), in which case the data that was correctly received by the TCP layer into the internal winsock buffer will never be retrieved by the application.

To guarantee arrival at the application layer, therefore, it's necessary for the application layer to implement the same sort of ACK/sequence number handshake that the TCP layer implements. This is a handshake between applications at the application layer, i.e., the sender application on one side and the recipient application on the other.

Note that this implementation has the potential to slow the transfer considerably. It must be implemented thoughtfully to avoid slowing the transfer. Consider a simple (and unusable) algorithm, where the sender sends data and then waits for an ACK from the recipient signifying that the data was safely received before sending the next chunk of data. Such an implementation is disatrous. It requires the entire round trip time (RTT) across the network between send()'s of data, thereby slowing data transfer to a crawl.

Evidentally, you need to work smart with your ACK/sequence numbers, like TCP does. It keeps a sliding window and performs book-keeping on the data that has been sent but not yet ACK'd. so that data can be sent more-or-less continuously, even in the absence of a steady stream of ACKs. The basic idea is to keep as many data packets on-the-fly (i.e., un-ACk'd packets on the wire) as possible.

So, hlaalu, your final observation is a correct one: guaranteed delivery of information is implemented at the application layer, not the transport layer. But it's not easy, and it's not something that can be entered into lightly, without careful implementation of a good algorithm.

Mike