HTTP 1.1 request

[keenlearner]

I am using winsock to build a HTTP 1.1 compliant client, when I made the request with the header
GET / HTTP/1.1[CRLF]
Host : www.google.com[CRLF][CRLF]

After this header is send using function send(), and when I want to receive the html from the server using recv(), it takes sometimes to get the response, as it's because the connection is still alive, the server will only response until the connection timeout, hence I get the response. If I used connection close, I will get the response immediately.

GET / HTTP/1.1[CRLF]
Connection: close[CRLF]
Host : www.google.com[CRLF][CRLF]

So my question is how can I tell the server it's the end of my request without using the Connection: close header field, so that the server will immediately send response without the connection timeout. or what are the conditions that the server will send response besides, connection closed or connection timeout ? Thanks.

[MikeAThon]

The server is probably fine, and has probably sent its response long before the connection is closed.

In fact, your client has probably received the response already. However, your client is probably stuck in a recv() loop, waiting for more information from the server, or for the server to close the connection.

Naturally the server will not send more information since your client hasn't asked for any more. So, your client sits in the recv() loop until the server closes the connection, at which point recv() returns with a value of zero signifying that the connection was closed, and you finally see the original response that was sent from the server long before that.

Debug your code. For example, after the client sends the request to the server, and it's just sitting there before the connection is closed, break execution and see where the client is, and what it's doing. It's probably blocking on a call to recv() or select().

If not, then single step through the client until you see what causes the client to stop responding until after the connection is closed.

Post your recv() code if you can't figure it out.

Mike

[keenlearner]

Thanks, you are very right. :)
So what can I do to know that if I have received all the message ? Is it
By reading the header fields of Content-length and calculate the bytes that has been received from the first two CRLF until specified content-length ?!

[MikeAThon]

Yes, that's basically correct.

But not completely correct. The real answer is that you must follow the HTTP/1.1 protocol exactly, in order to determine when the server's response to a first request is complete. The Content-length header is just one way that the server can indicate the length of his response. In HTTP/1.1 there are other ways too. For example, the server can send a "chunked" response, where each section of the response is preceded by a string of ASCII characters that represents the hexadecimal length of the section that follows.

The precise rules for content length under HTTP/1.1 are complicated. Note that an HTTP/1.1 client MUST support a Transfer-Encoding of "chunked", which is one of the things that adds complications.

The precise rules for content length are specified in RFC 2616 at Section 4.4 (see ftp://ftp.rfc-editor.org/in-notes/rfc2616.txt ):



4.4 Message Length

The transfer-length of a message is the length of the message-body as
it appears in the message; that is, after any transfer-codings have
been applied. When a message-body is included with a message, the
transfer-length of that body is determined by one of the following
(in order of precedence):

1.Any response message which "MUST NOT" include a message-body (such
as the 1xx, 204, and 304 responses and any response to a HEAD
request) is always terminated by the first empty line after the
header fields, regardless of the entity-header fields present in
the message.

2.If a Transfer-Encoding header field (section 14.41) is present and
has any value other than "identity", then the transfer-length is
defined by use of the "chunked" transfer-coding (section 3.6),
unless the message is terminated by closing the connection.

3.If a Content-Length header field (section 14.13) is present, its
decimal value in OCTETs represents both the entity-length and the
transfer-length. The Content-Length header field MUST NOT be sent
if these two lengths are different (i.e., if a Transfer-Encoding
header field is present). If a message is received with both a
Transfer-Encoding header field and a Content-Length header field,
the latter MUST be ignored.

4.If the message uses the media type "multipart/byteranges", and the
ransfer-length is not otherwise specified, then this self-
elimiting media type defines the transfer-length. This media type
UST NOT be used unless the sender knows that the recipient can arse
it; the presence in a request of a Range header with ultiple byte-
range specifiers from a 1.1 client implies that the lient can parse
multipart/byteranges responses.

A range header might be forwarded by a 1.0 proxy that does not
understand multipart/byteranges; in this case the server MUST
delimit the message using methods defined in items 1,3 or 5 of
this section.

5.By the server closing the connection. (Closing the connection
cannot be used to indicate the end of a request body, since that
would leave no possibility for the server to send back a response.)

For compatibility with HTTP/1.0 applications, HTTP/1.1 requests
containing a message-body MUST include a valid Content-Length header
field unless the server is known to be HTTP/1.1 compliant. If a
request contains a message-body and a Content-Length is not given,
the server SHOULD respond with 400 (bad request) if it cannot
determine the length of the message, or with 411 (length required) if
it wishes to insist on receiving a valid Content-Length.

All HTTP/1.1 applications that receive entities MUST accept the
"chunked" transfer-coding (section 3.6), thus allowing this mechanism
to be used for messages when the message length cannot be determined
in advance.

Messages MUST NOT include both a Content-Length header field and a
non-identity transfer-coding. If the message does include a non-
identity transfer-coding, the Content-Length MUST be ignored.

When a Content-Length is given in a message where a message-body is
allowed, its field value MUST exactly match the number of OCTETs in
the message-body. HTTP/1.1 user agents MUST notify the user when an
invalid length is received and detected.


I told you it was complicated <g>. That's why, in other conversations with you, I recommended that you stick with HTTP/1.0, which does not have a need for these rules since there are not persistent connections, and therefore the end of a response is signified unambiguously with a close of the connection.

Mike

[keenlearner]

I have been reading all the HTTP/1.1 protocol which I already printed out the 14 rfc sections. I always found the chunked transfer-encoding never know what does it used for until today I am clearer on how to use it. I like to tried out the persistent connection and pipelining feature of HTTP/1.1, it seems very difficult about HTTP/1.1 but I get more and more familiar with it each of every new day. It's already already 5.20 am at here, got to stop working on the computer. Thanks for always helping. :)