Can I automate browsing of this site?

[branden.hughes]

I'm trying to automate this for a mass-processing of GDC number
information (scrapping). I've tried everything I can think of and
every example I've come across to no avail. The root site is:
http://www.dcor.state.ga.us/GDC/OffenderQuery/jsp/OffQryForm.jsp

posting the following:
http://www.dcor.state.ga.us/GDC/OffenderQuery/jsp/OffQryForm.jsp?Institution=vDisclaimer=True

Yields the form I need to fill out to access individual GDC records.
On this page, I need to use option UNO_NO (gdc number) and pass along
an actual GDC number (of which I have thousands to process).

Problem is that the form doesn't post to itself:
<form method="post" action="OffQryRedirector.jsp"

If I try opening a webrequest with:
http://www.dcor.state.ga.us/GDC/Off...noCaseNoRadioButton=UNO_NO&vOffenderId=365172

It just returns me to the main page.

Any ideas whatsoever would be extremely appreciated. I've tried
cookies (as best as I understand them), multiple webrequests, setting
autoredirect to true, etc. This doesn't seem like an absurd request,
so I'm hoping I'm missing some piece of understanding that clears up
the automation of this process. Thanks!

 

[Dave Sexton]

Hi Branden,

If you get this working, what happens if they change the page? Rename a
field? Rename the page? How will your program handle a change in
javascript validation logic that prevents the page from posting some time
after you finally get it working?

A web service might be much better for you. Try contact the site admin and
see if there is a better way. Will they let you connect directly to their
database?

Are you displaying the HTML that is retrieved from the WebRequest in your
application? Could you just use a WebBrowser control instead?

Problem is that the form doesn't post to itself:
<form method="post" action="OffQryRedirector.jsp"

That shouldn't matter if you pass all the data that it expects.
OffQryRedirector.jsp might do something simple like check the HTTP_REFERER
on POST. I don't think that HTTP_REFERER will be accessible to their server
when using a WebRequest object unless it's added by code but I'm not sure of
that.

If I try opening a webrequest with:
http://www.dcor.state.ga.us/GDC/Off...noCaseNoRadioButton=UNO_NO&vOffenderId=365172

It just returns me to the main page.

Does OffQryRedirector.jsp expect just a query string? What about the
required post data?

This doesn't seem like an absurd request,
so I'm hoping I'm missing some piece of understanding that clears up
the automation of this process. Thanks!

Maybe not absurd but maybe not possible as well. Don't rule out the
possibility that it won't work. You may be forced to find another way. Try
my suggestion of contacting the site admin for a web service or data access
ability of some sort. That would probably be your best and only shot.

HTH