WebClient - detect response redirect

[Glen]

I'm writing a console utility to download specific files from web sites
based on the command line options. In most cases, I can trap the 404
error when the file isn't available because the operator mistyped the
URL or it's offline for whatever reason. The problem I'm running into
is with certain sites where the admin has set up a redirect to handle
the 404 condition and redirects the request to another page.

In this case, the redirected page gets downloaded and saved which is not
the desired result. This utility is being used in a scheduling process
to download and process specific data files from public web sites and
the files must exist, or return an error to halt processing.

Is there anyway to tell the WebClient or WebRequest objects to not allow
redirected content? Or, is there a property in either of the objects
that reflects the actual URL of the source when it's redirected? Or, is
there an alternative object that can be used for this purpose to
download files from web sites?

Any help would be greatly appreciated.

- Glen

 

[Adam Clauss]

I don't think there is a way using the WebClient class.

If you are using HttpWebRequest, there is the AllowAutoRedirect property
which may allow you to accomplish what you need.

 

[Cor Ligthert]

Glen,

I know a lot of sites that have an URL only as a start point and than
redirect, how do you handle those when you have your implementation.

Not that I have the answer however maybe is finding it opening a new problem
and is searching for this solution useless..

Just my thought,

Cor

 

[Glen]

Hi Cor,

Since I'm not building a browser and I'm looking to retrieve very specific
data from the web, I expect that any changes in the remote data source will
be documented and handled by the system operators. Again, this is a critical
task, much like getting files via FTP. I don't expect that when I log into
an FTP site, I'm going to get a different file than the one I requested.

In any case, thank you for your input.

- Glen

 

[Glen]

Thanks Adam. I'll give that one a look.

- Glen

I don't think there is a way using the WebClient class.

If you are using HttpWebRequest, there is the AllowAutoRedirect property
which may allow you to accomplish what you need.

 

[Glen]

Unfortunately, this parameter doesn't appear to work in some cases. The
site in question is using IIS 5.0 and is not returning a redirect code or
page not found code in any instance. All redirects are done silently with
an OK status returned.

 

[Adam Clauss]

Hmm, wait, so the website is not telling the client to redirect anywhere?
In this case - there probably is no way to tell that anything was happening.
It sounds like the redirection is actually happening on the server side of
things...

 

[Glen]

Hi Adam,

I checked a few things and even in the browser (IE) when you enter and
incorrect path on this particular server it doesn't change the actual
address when it displays the help text. I guess it's more of a content
substitution than a redirect scenario on the server side to handle 404
errors.

I'm going to have to figure out another way to validate the download and
return the appropriate codes to the application.

Thanks for your help.

- Glen

 

[Adam Clauss]

NP... good luck on that.

Do you know what type of file you are EXPECTING to retrieve? Could you
check the actual content you downloaded to see if it is correct (or some
sort of "error" page?)?

 

[Glen]

Yes, these particular files are regulated (pipe delimited) with constant
headers. I'll probably just build a simple script to open the file and
verify the header in a seperate step when the transfer is completed. At
least this way, I can keep my C# project generic enough to use with other
processes.

- Glen