using HttpWebRequest, HttpWebResponse ?

[Bill Woodruff]

I wrote an application which uses WebRequest to access several thousand small
graphics file on an on-line server, uses GetResponseStream and Image.FromStream
to copy their contents, and then saves the copies to a local hard disk.

It seemed "prudent" to me to insert a wait state after each access to a file on
the server like :

System.Threading.Thread.Sleep(250);

Testing using this scenario on fifty or so of the images went ok, but I am
wondering, before I "turn the application loose," if there are any "best
practices" regarding multiple accesses to a server like this. If I were using an
AxWebBrowser, I could be waiting on the DocumentComplete event, but the
performance of WebRequest seems great, and I like the idea of doing everything
without the overhead of the webbrowser control, and any "visual presentation" of
the process or results other than a progress bar.

Any hints, tips, ideas, pointers on creating robust (and "polite") large-scale
access to on-line image resources will be very appreciated.

thanks !

Bill Woodruff
dotScience
Chiang Mai, Thailand

 

[Joerg Jooss]

I wrote an application which uses WebRequest to access several
thousand small graphics file on an on-line server, uses
GetResponseStream and Image.FromStream to copy their contents, and
then saves the copies to a local hard disk.

It seemed "prudent" to me to insert a wait state after each access to
a file on the server like :

System.Threading.Thread.Sleep(250);

Testing using this scenario on fifty or so of the images went ok, but
I am wondering, before I "turn the application loose," if there are
any "best practices" regarding multiple accesses to a server like
this. If I were using an AxWebBrowser, I could be waiting on the
DocumentComplete event, but the performance of WebRequest seems
great, and I like the idea of doing everything without the overhead
of the webbrowser control, and any "visual presentation" of the
process or results other than a progress bar.

Any hints, tips, ideas, pointers on creating robust (and "polite")
large-scale access to on-line image resources will be very
appreciated.

You didn't say how many requests are sent in parallel, or whether you're
using HTTP 1.1 persistent connections. In case of persistent connections,
the 250 ms delay seems to be a waste of time to me -- there's no risk of
port exhaustion.

The HTTP spec says:

"Clients that use persistent connections SHOULD limit the number of
simultaneous connections that they maintain to a given server. A single-user
client SHOULD NOT maintain more than 2 connections with any server or
proxy."

This is also the default behaviour for WebRequest instances.

Cheers,