A
atlaste
Hi,
In an attempt to create a full-blown webcrawler I've found myself
writing a wrapper around the Socket class in an attempt to make it
completely async, supporting timeouts and some scheduling mechanisms.
I use a non-blocking approach for this, using the call to 'poll' to
support the async mechanism - rather than the 'begin' and 'end'
functions. I already found that connecting doesn't set the
"isconnected" variable correctly (SocketException is thrown: non-
blocking has this effect...) - but doesn't appear to be a problem
because poll, read and write work fine.
For measuring the performance of the crawler, I started "perfmon.msc"
and added the "active connections" item from object "TCP". After a
while I found the number of this performance counter to reach over
300K connections (!), enough to start worrying...
My crawler is designed to support around 200 connections simultaneous.
"netstat -an" doesn't support this finding, but does show hundreds of
connections that are in either "CLOSE_WAIT", "FIN_WAIT_2" or another
closing state.
After a host has completed, I try to disconnect the TCP/IP connection.
I've attempted combinations of "shutdown(both)", (async) "disconnect"
and "close(0)" - where no combination appears to have the desired
effect. When the application is shut down, all connections (including
the CLOSE_WAIT connections) are removed. The FIN_WAIT_2 connections
linger forever...
Perhaps someone knows a solution to this problem?
Greetings,
Stefan de Bruijn.
In an attempt to create a full-blown webcrawler I've found myself
writing a wrapper around the Socket class in an attempt to make it
completely async, supporting timeouts and some scheduling mechanisms.
I use a non-blocking approach for this, using the call to 'poll' to
support the async mechanism - rather than the 'begin' and 'end'
functions. I already found that connecting doesn't set the
"isconnected" variable correctly (SocketException is thrown: non-
blocking has this effect...) - but doesn't appear to be a problem
because poll, read and write work fine.
For measuring the performance of the crawler, I started "perfmon.msc"
and added the "active connections" item from object "TCP". After a
while I found the number of this performance counter to reach over
300K connections (!), enough to start worrying...
My crawler is designed to support around 200 connections simultaneous.
"netstat -an" doesn't support this finding, but does show hundreds of
connections that are in either "CLOSE_WAIT", "FIN_WAIT_2" or another
closing state.
After a host has completed, I try to disconnect the TCP/IP connection.
I've attempted combinations of "shutdown(both)", (async) "disconnect"
and "close(0)" - where no combination appears to have the desired
effect. When the application is shut down, all connections (including
the CLOSE_WAIT connections) are removed. The FIN_WAIT_2 connections
linger forever...
Perhaps someone knows a solution to this problem?
Greetings,
Stefan de Bruijn.