.NET Web Scrapper project

[Radith]

Hi all,

I am about to commence a web scrapper project using vb/c# + sql server. I am
currently a uni. student with no commerical experience (btw I am doing this
as a commercial project). Firstly, the aim of this scrapper is to: Scrape
content off a website and store them in a SQL Server database. The content
is dynamic. However, before the scraping; some HTML parsing must be done.
I.E. I have to iterate through all possible registration numbers (feeding
the reg. #'s via my app.) through most likely the use of a HTML parser.

So, I just want to know if HTML parsing would be the answer for that sort of
iteration that's required and also require some help on the scraping
component. Can someone please point me in the right
direction.

All forms of help appreciated. Thanks in advance for all your help...

Cheers,
Radith

P.S.: I know this questions seems broad but if any of y'all out there has
had any experience at doing such a thing, please refere me to some
resources. Btw, information in re: to this seems to be VERY scarce.

 

[Michael Nemtsev]

Using WebRequest and StreamReader you can get the page contend.
After that you need use either the Regexp or xml (if page is only xtml
compatible) u can iterate through the page to extract necessary values

google to find codesnippets or samples

 

[Peter Bromberg [C# MVP]]

Havea look at Simon Mourier's HtmlAgilityPack, which I believe is now hosted
on codeplex.com.

--Peter
"Inside every large program, there is a small program trying to get out."
http://www.eggheadcafe.com
http://petesbloggerama.blogspot.com
http://www.blogmetafinder.com