TCP Socket end of data?

[gilad]

I am trying to write a program that will send an HTTP request and get an
HTTP response. However, I am having trouble getting the full payload
(i.e. the web page) returned from the host. I am using a blocking
socket, so I need to test the end of the data somehow, or I get a "hung"
effect while the Socket.Receive() method waits.

I constructed a loop like the following to test for end-of-data, but I
only get part of the data returned for large pages and the loop exits.


Socket IPsocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);

// Socket connection code was here...
// The first read takes place...
// ...then the following is executed:

while ( rBytes > 0)
{

if (!IPsocket.Poll(3000,SelectMode.SelectRead))
{
rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);
mStrm.Write(RecvBytes,0,rBytes);
break;
}


rBytes = IPsocket.Receive(RecvBytes, RecvBytes.Length, 0);

// Write to buffer
mStrm.Write(RecvBytes,0,rBytes);
}




Is there different way I should be testing for the end of the data? I've
tried using Socket.Available(), but found that it returns 0 when there
is in fact more data.

Thanks, JA

 

[David Browne]

I am trying to write a program that will send an HTTP request and get an
HTTP response. However, I am having trouble getting the full payload (i.e.
the web page) returned from the host. I am using a blocking socket, so I
need to test the end of the data somehow, or I get a "hung" effect while
the Socket.Receive() method waits.

I constructed a loop like the following to test for end-of-data, but I
only get part of the data returned for large pages and the loop exits.


Socket IPsocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);

Is there a particular reason you're not using the .NET HTTP classes?

If you want to implement the HTTP protocol using TCP/IP yourself, you'd
better read the spec:

http://www.w3.org/Protocols/rfc2616/rfc2616.html

Pay special attention to the section on the content-length header, the
Connection header, chunked transfer encoding and the 100 continue response
code.


Consider using HTTP 1.0
http://www.w3.org/Protocols/rfc1945/rfc1945

as it's considerably simpler to implement in raw sockets. For instance,
Socket.Recive() in a loop will work in HTTP 1.0 since the server will close
the connection when the response is done.

David

 

[gilad]

Is there a particular reason you're not using the .NET HTTP classes?

I am building a simple proxy server.

If you want to implement the HTTP protocol using TCP/IP yourself, you'd
better read the spec:

http://www.w3.org/Protocols/rfc2616/rfc2616.html

Pay special attention to the section on the content-length header, the
Connection header, chunked transfer encoding and the 100 continue response
code.

I've done this, but the problem that I see is that the Content-Length
and other fields are not always present. For instance, here is what is
returned from "www.wired.com" as the HTTP header.

HTTP/1.1 200 OK
Date: Wed, 31 Aug 2005 18:48:35 GMT
Server: Apache/1.3.31 (Unix) PHP/4.3.9
Cache-Control: max-age=300
Expires: Wed, 31 Aug 2005 18:53:35 GMT
Connection: close
Content-Type: text/html

(From here on there is just HTML content)

I have found that I either loop with Receive until it hangs (no more
data available, and it blocks), or I've got to check the end of the
transmission. As you can see from the above, there is really nothing to
indicate packet length in the HTTP header.

I am reading the packets in buffers with 4096 bytes length.