.Net Web Spiders

[nondisclosure007]

et al,

Does anyone have any good links as to where to start in writing a
program like this?

I want to make something custom so that I can pull certain websites,
extract certain data from them, store in a SQL server to search
against.

I'm sure there's lots out there, I just can't seem to find it.

Albeit there might be programs out there to do this, but none that I
have seen to a decent job. And besides, I want to get some expirience
in VB.Net programming.

The first version can be just a simple spider that scans 1 website (I
have a test website w/ 6 pages on it). I figure this is small enough
for a start.
Any links anyone can provide would be greatly appreciated.

thx.

 

[Cor Ligthert]

Hi,

You are probably not the first one, you can go at least two directions

Use the axwebbrowser or use let say HttpWebRequest.

To search a document the best way is in my opinion MSHTML
The axwebbrowser is easy, mshtml is really very hard stuff to go for a
newbie.

HttpWebRequest
http://msdn.microsoft.com/library/d...ml/frlrfsystemnethttpwebrequestclasstopic.asp

webbrowser
http://support.microsoft.com/?kbid=311303

some faqs
http://support.microsoft.com/default.aspx?scid=kb;EN-US;311284

mshtml
http://msdn.microsoft.com/library/default.asp?url=/workshop/browser/hosting/hosting.asp

I hope this helps a little bit?

Cor

 

[nondisclosure007]

Thanks. I'll do some reading and give one, if not all of them, a shot.
thx.