Enhancing SpiderInNet

[Tim Gaunt]

Hi,

I'm primarily a web developer and have recently moved onto .NET and am
slowly realising the enormous potential both through ASP.NET but
integrating all the different services.

While teaching myself .NET I decided to start building a spider to get a
load of information off other sites (thanks for the points on here guys)
and on of the spiders that seems to suiting my needs is the Spider In
..NET article on MSDN
(http://msdn.microsoft.com/msdnmag/issues/02/10/SpiderinNET/default.aspx)
but I would now like to expand on what I have altered so far (simply
restricting the spider to search URL's matching certain parameters) to
cope with loss of connection and pausing the process, is there any way
of doing this? I have been reading about pausing processes and needing
other threads and all kinds so have got a little confused and would
apprechiate any advice given.

Many thanks in advance.

Tim

 

[Darious Snell]

You can kick off a "watcher" type thread that looks at connection
status / socket traffic-over-time etc. This thread could suspend/unsuspend
or kill "spidering" threads for you.

 

[Tim Gaunt]

ah great thanks, I'll look into that now, for some reason though the
spider just stops part way through spidering which I'm yet to figure out
thanks again.

Tim