Downloading multiple WebPages and saving to text file help needed!



Code below ----

I'm trying to save some specific web pages to disk as text files. I searched the Internet and found a basic example which I changed to fit my needs.
I tested it out first on a single URL and it worked fine. Now when I incorporate an array of URL's, it fails to work. The first "responseFromServer"
properly retrieves and displays "" (or any other valid URL for the first webrequest.create), but when I use the urls from the
array in the for/next structure, I never get a responseFromServer. My guess is that there must be some kind of initialization to the WebRequest,
Stream, or WebResponse.

A new command for me is "reader.ReadToEnd". In one fell swoop it reads the entire webpage. Very cool. Now I also want to write the entire
"responseFromServer" to disk as a text file. I've seen commands like "File.WriteAllText" and it seems to work well for what I need. Should I look
into any other commands to save the "responseFromServer" to disk as a text file?



Imports System
Imports System.IO
Imports System.Net
Imports System.Net.WebClient
Imports System.Windows.Forms
Public Class Form1

Private Sub btnProcess_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnProcess.Click
Dim x As Integer
Dim aryUrls(15) As String
aryUrls(0) = ""
aryUrls(1) = ""
aryUrls(2) = ""
aryUrls(3) = ""

Dim request As WebRequest = WebRequest.Create("")
Dim response As WebResponse = request.GetResponse()
Dim dataStream As Stream = response.GetResponseStream()
Dim reader As New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()

MsgBox(responseFromServer) <=== Shows

For x = 0 To 3
request = WebRequest.Create(aryUrls(x))
dataStream = response.GetResponseStream()
responseFromServer = reader.ReadToEnd()
MsgBox(responseFromServer) <=== Shows nothing

End Sub

Cor Ligthert [MVP]


To check this kind of routines is a hell of a job and yours seems to me
incomplete so even not to debug..
It looks like the sample on our webpages but even than I find that always
painfull to debug.

Be aware however that if there are frames used on a webpage, you get with
this kind of routines only the MainFrame.

That was the first that came up into my mind.


Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question