System.IO.StringReader ReadToEnd()

[KevinB]

I'm doing screen scraping from a Windows Application and dumping certain data
into a SqlDb, it's one of our parent websites, through a Proxy server using
the the WebClient class. I have used the WebRequest/Response classes but the
only benefit I got from that was the abiliy to control the timeout on the
WebResponse. I can control the timeout on the ProxyServer here and that's
all I need. For the issue I'm having the code below should suffice:

System.Net.WebClient wc = new System.Net.WebClient();
System.Net.WebProxy wp = new System.Net.WebProxy();
wp.UseDefaultCredentials = true;
wc.Headers.Add("User-Agent", "Mozilla/4.0+");
wc.Proxy = wp;
strm = wc.OpenRead(nextSubURL);
System.IO.StreamReader sr = new System.IO.StreamReader(strm);
System.IO.StringReader strReader = new System.IO.StringReader(sr.ReadToEnd());

Here's my problem, eventually I'm getting a timeout on sr.ReadToEnd. I'm
not getting it everytime; could it be that some pages being dumed into the
StreamReader are just to big? I know that I can't control that but if there a
way to stop that from timing out?

Thanks everyone.

 

[Scott M.]

I think is is because occassionally what you are reading takes too long to
read. The only thing you can do is increase your timeout time.

-Scott

 

[KevinB]

Scott, while I certainly appreciate the obvious answer and I'm absolutly a
fan of sarcasm; you're not helping.

Thanks anyway.

Kevin
--
Kevin C. Brown
Developer

I think is is because occassionally what you are reading takes too long to
read. The only thing you can do is increase your timeout time.

-Scott

 

[KevinB]

To answer my own question,

Here is my original code, below it is what I did to fix my problem.

System.Net.WebClient wc = new System.Net.WebClient();
the fix:

sr.DiscardBufferedData();
and of course you should always:
sr.close();

is in a loop grabbing a different nextSubURL from a table in my Db and I was
putting a different HTML page in the StreamReader everytime, clearing it
would be a tremendous help solve all timeout problems. Since the sr object
doesn't have a timeout method that isn't an option.

It's a neat little windows app, works great.

--
Kevin C. Brown
Developer

Scott, while I certainly appreciate the obvious answer and I'm absolutly a
fan of sarcasm; you're not helping.

Thanks anyway.

Kevin

 

[Scott M.]

I wan't trying to be sarcastic at all. I was simply responding your comment
and question:

Yes.

No, you'll need to increase the timeout value.

If you are unhappy with these clear answers to your questions, perhaps you
should ask a better question.

-Scott

Scott, while I certainly appreciate the obvious answer and I'm absolutly a
fan of sarcasm; you're not helping.

Thanks anyway.

Kevin

 

[KevinB]

My appologies.
--
Kevin C. Brown
Developer

I wan't trying to be sarcastic at all. I was simply responding your comment
and question:


No, you'll need to increase the timeout value.

If you are unhappy with these clear answers to your questions, perhaps you
should ask a better question.

-Scott

 

[Jon Skeet [C# MVP]]

Here is my original code, below it is what I did to fix my problem.

System.Net.WebClient wc = new System.Net.WebClient();

the fix:

sr.DiscardBufferedData();

No, that's not going to fix it at all. How could it? If you call it
before ReadToEnd() it's just going to throw away nothing (because you
haven't read anything yet). If you call it *after* ReadToEnd(), it
obviously can't stop

and of course you should always:
sr.close();

Well, you shouldn't do it explicitly - you should use a "using"
statement to wrap it in a try/finally block automatically.

That could actually be the fix - if you weren't closing the
StreamReader, it's possible that it wasn't closing the network
connection in the background, and the next request to the same server
may have been waiting for that connection to be closed.

is in a loop grabbing a different nextSubURL from a table in my Db and I was
putting a different HTML page in the StreamReader everytime, clearing it
would be a tremendous help solve all timeout problems.

No, you weren't putting a different HTML page into the StreamReader -
you were creating a *new* (empty) StreamReader. Big difference.

 

[KevinB]

Jon,

I couldn't figure out how to accomplish my goal of passing in a new URL into
the OpenRead and getting the appropriate response without putting the entire
block of code (see below) in the loop. So the answer to your question is Yes.
If I could instantiate the StreamReader further up and assign the instance
the Stream in the loop I'd do that but it didn't seem to work. Using
sr.DiscardBufferedData(); seems to be working ok now but the fact that I'm
going through our proxy server causes other problems but there isn't much I
can do about that, politricks ya know.

strm = wc.OpenRead(nextSubURL);
System.IO.StreamReader sr = new System.IO.StreamReader(strm);
System.IO.StringReader strReader = new System.IO.StringReader(sr.ReadToEnd());


Thank you!

 

[Jon Skeet [C# MVP]]

I couldn't figure out how to accomplish my goal of passing in a new URL into
the OpenRead and getting the appropriate response without putting the entire
block of code (see below) in the loop.

And what's wrong with that?

So the answer to your question is Yes.

Which question? It's certainly not the answer to "how can it make a
difference?" which is the only question I can see in my post.

If I could instantiate the StreamReader further up and assign the instance
the Stream in the loop I'd do that but it didn't seem to work.

Why would you want to do that in the first place? The fact that you're
reusing even the *Stream* variable seems like a bad idea (in terms of
code cleanliness) let alone the StreamReader.

Using sr.DiscardBufferedData(); seems to be working ok now but the fact that I'm
going through our proxy server causes other problems but there isn't much I
can do about that, politricks ya know.

How can sr.DiscardBufferedData() make any difference when you're
reassigning the variable anyway? Whenever you find a solution which
seems weird, it's probably not the solution you think it is.

No, my bet is on your call to Close() being the solution. I strongly,
strongly suspect that getting rid of sr.DiscardBufferedData() will make
no difference to your code.

I would still suggest a using statement though, so you still close the
StreamReader (and hence the underlying Stream) even if you get an
exception.