Socket Class Slower Then HTTP Web Request

[mwieder]

I developed an application that automates dwonloading of a web page
hundreds of times. I first developed it by using http web request, but
then I read up that HTTP Web Request has some overhead that I thought I
could eliminate if I re-wrote it with sockets. However, what I have
found is that sockets actually works significantly slower (I get about
1/2 to 2/3 of the performance I get with HTTPWebRequest).
Now, the web server sends a close command on every request so even if I
post a Keep-Alive request I have to create and open a new socket for
every request, but I assume the HTTPRequest class must do the same
thing under the covers. Below is my Socket Send Method (very basic).
Can anyone explain this strange disparity in performance?

public string Send(string strRequest)
{
m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
ProtocolType.Tcp);

m_sock.Connect(ipe);

string strRetPage = null;

Encoding ASCII = Encoding.ASCII;

Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
Byte[] RecvBytes = new Byte[256];
m_sock.Send(ByteGet, ByteGet.Length, 0);
Int32 bytes;

do
{
bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
strRetPage = strRetPage + ASCII.GetString(RecvBytes, 0, bytes);
} while (bytes > 0);

m_sock.Shutdown(SocketShutdown.Both);
m_sock.Close();

return strRetPage;

}

 

[Feroze [msft]]

The best thing to do is to do a network sniff of the app running with
sockets code and compare it to the app running with HttpWebRequest code.

 

[Joerg Jooss]

I developed an application that automates dwonloading of a web page
hundreds of times. I first developed it by using http web request,
but then I read up that HTTP Web Request has some overhead that I
thought I could eliminate if I re-wrote it with sockets.

<rant>
Isn't it funny how people run off and reinvent the wheel because of
rumors :-/

However,
what I have found is that sockets actually works significantly slower
(I get about 1/2 to 2/3 of the performance I get with HTTPWebRequest).
Now, the web server sends a close command on every request so even if
I post a Keep-Alive request I have to create and open a new socket for
every request, but I assume the HTTPRequest class must do the same
thing under the covers.

Did you use a network sniffer to verify that? Creating a new TCP
connection will create some significant overhead.

Below is my Socket Send Method (very basic).
Can anyone explain this strange disparity in performance?

public string Send(string strRequest)
{
m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
ProtocolType.Tcp);

m_sock.Connect(ipe);

string strRetPage = null;

Encoding ASCII = Encoding.ASCII;

Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
Byte[] RecvBytes = new Byte[256];
m_sock.Send(ByteGet, ByteGet.Length, 0);
Int32 bytes;

do
{
bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
strRetPage = strRetPage + ASCII.GetString(RecvBytes, 0, bytes);
} while (bytes > 0);

m_sock.Shutdown(SocketShutdown.Both);
m_sock.Close();

return strRetPage;

}

You're using a rather small read buffer and decode the incoming bytes
on each iteration (and use ASCIIEncoding, which breaks all non-US-ASCII
content).

You're also using naive string concatenation ("+"), which will create a
lot of temporary objects. Use a MemoryStream or a StringBuilder to
buffer the incoming data.


Cheers,

 

[mwieder]

the netwrok sniff returns the same thing for both. The time difference
is due to the work being done on my machine in between requests. I
also suspect the issue is creating and deleting the socket every time,
but because the server forces closure of the socket (attempts at re-use
confirm this) I must open a new socket to connect. I would think the
HTTPWebRequest class must do teh same thing, I just can't understand
why it is slower...

 

[mwieder]

Hi - thanks for the suggestions - I understood everything you suggested
(see new code below) except the comment on the ASCIIEncoding - please
change th code below to illustarte what you are suggesting there.
thanks!

public string Send(string HttpRequest)
{
m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
ProtocolType.Tcp);

m_sock.Connect(ipe);

Encoding ASCII = Encoding.ASCII;

Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
Byte[] RecvBytes = new Byte[16384];
m_sock.Send(ByteGet, ByteGet.Length, 0);
Int32 bytes;

StringBuilder sbDataReceived = new StringBuilder();

do
{
bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
sbDataReceived.Append(ASCII.GetString(RecvBytes, 0, bytes));

} while (bytes > 0);

m_sock.Shutdown(SocketShutdown.Both);
m_sock.Close();

return sbDataReceived.ToString();

}

 

[Feroze [msft]]

Look at this post, which will show you what the suggested way of downloading
is:

http://blogs.msdn.com/feroze_daud/archive/2004/03/30/104440.aspx

 

[Joerg Jooss]

Hi - thanks for the suggestions - I understood everything you
suggested (see new code below) except the comment on the
ASCIIEncoding - please change th code below to illustarte what you
are suggesting there. thanks!

public string Send(string HttpRequest)
{
m_sock = new Socket(ipe.AddressFamily, SocketType.Stream,
ProtocolType.Tcp);

m_sock.Connect(ipe);

Encoding ASCII = Encoding.ASCII;

Byte[] ByteGet = ASCII.GetBytes(HttpRequest);
Byte[] RecvBytes = new Byte[16384];
m_sock.Send(ByteGet, ByteGet.Length, 0);
Int32 bytes;

StringBuilder sbDataReceived = new StringBuilder();

do
{
bytes = m_sock.Receive(RecvBytes, RecvBytes.Length, 0);
sbDataReceived.Append(ASCII.GetString(RecvBytes, 0, bytes));

} while (bytes > 0);

m_sock.Shutdown(SocketShutdown.Both);
m_sock.Close();

return sbDataReceived.ToString();

}

See Feroze's post for a version that uses a MemoryStream for buffering
and decodes the whole buffer later.

Here's another one, simpler version without characters set detection:
http://tinyurl.com/6aedc

Cheers,

 

[mwieder]

Thanks for your help. Hopefully this optomizes the process as much as
can be done.

 

[Joerg Jooss]

Thanks for your help. Hopefully this optomizes the process as much as
can be done.

No probs. I still recommend sticking to HttpWebRequest, though. There's
more to a proper HTTP implementation that sending a

GET /path/page.html HTTP/1.1
Host: www.foo.org

over a TCP socket.

Cheers,