J
Jarod_24
I have some problems with using Net.WebClient
This function of mine recieves an array with Urls, and Filenames, and is supposed to download all the files (binary files) that is
listed in the array, but in some cases it dosen't work. Normally it would download files (binary) to a folder, but on some occations
it will only download the first file in the array as a binary, and then the rest of the files downloaded will actually only contain
HTML. These are supposed to be video/images so that's very wrong. If i try downloading the files manually usung Internet Explorer it
works like it should (all files are downloaded as binary).
Private Sub downloadFiles(ByVal files() As FileToUri_Structure, ByVal baseUrl As Uri)
Dim i As Integer
Dim web As New Net.WebClient
Dim h As New Net.WebHeaderCollection
h.Add("Referer", baseUrl.AbsoluteUri)
web.Headers = h
For i = files.GetLowerBound(0) To files.GetUpperBound(0)
web.DownloadFile(files(i).AbsoluteUri, files(i).FilePath)
Debug.WriteLine("i= " & i & vbLf & web.ResponseHeaders.ToString)
Next
End Sub
When i looked at the web.ResponseHeaders i saw that the difference was in the "Server" header. The malfunctioning response also
returned "text/html" as the "Content-Type", and an additional "X-Pad" field. "Connection: close" was also something that was not
returned in the other (working) attempts
i=0
Date: Sat, 16 Oct 2004 21:06:58 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.1
Last-Modified: Mon, 04 Oct 2004 23:34:34 GMT
ETag: "1a5f4a-9d761-4161de0a"
Accept-Ranges: bytes
Content-Length: 644961
Connection: close
Content-Type: text/plain
i=1
Date: Sat, 16 Oct 2004 21:07:01 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.1
Last-Modified: Mon, 12 Apr 2004 06:48:24 GMT
ETag: "39196b-274-407a3bb8"
Accept-Ranges: bytes
Content-Length: 628
Connection: close
Content-Type: text/html
X-Pad: avoid browser bug
I googled and found this: "In six cases the HTTP message "X-Pad: avoid browser bug" was sent. This was an unannounced change to
Apache version 1.2b9 which provided a fix for a bug in Netscape versions 2.x, 3.x and 4.0b2."
(http://www.ariadne.ac.uk/issue34/web-watch)
I tried with another webserver and that returned "Server: Apache/1.3.27 (Unix) PHP/4.3.2" and there downloads worked perfectly.
Based on this result im suspecting that this might be a problem between Apache and PHP version 4.3.1 handling the requests
correctly, but since the first file in the array allways downloads sucessfully, i'm not completely convinced.
I tried adding a delay of 2 seconds between the downloads and but that didn't fix the problem. I also tried to declare a "web = new
Net.WebClient" before each new download in case it was something with the "web" object. I also tried adding additional headers like
"User-Agent" and so on to the "web.Header" that is sendt to the webserver. None of this fixed the problem.
Does anybody know how to go around to solve this.
This is how the "web.ReturnHeaders" of a working download looked like
i=0
Date: Sat, 16 Oct 2004 22:01:30 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.2
Last-Modified: Tue, 21 Sep 2004 23:03:38 GMT
ETag: "71d21f-e151f-4150b34a"
Accept-Ranges: bytes
Content-Length: 922911
Content-Type: text/plain
i=1
Date: Sat, 16 Oct 2004 22:01:35 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.2
Last-Modified: Tue, 21 Sep 2004 23:03:17 GMT
ETag: "71d21e-dd69f-4150b335"
Accept-Ranges: bytes
Content-Length: 906911
Content-Type: text/plain
This function of mine recieves an array with Urls, and Filenames, and is supposed to download all the files (binary files) that is
listed in the array, but in some cases it dosen't work. Normally it would download files (binary) to a folder, but on some occations
it will only download the first file in the array as a binary, and then the rest of the files downloaded will actually only contain
HTML. These are supposed to be video/images so that's very wrong. If i try downloading the files manually usung Internet Explorer it
works like it should (all files are downloaded as binary).
Private Sub downloadFiles(ByVal files() As FileToUri_Structure, ByVal baseUrl As Uri)
Dim i As Integer
Dim web As New Net.WebClient
Dim h As New Net.WebHeaderCollection
h.Add("Referer", baseUrl.AbsoluteUri)
web.Headers = h
For i = files.GetLowerBound(0) To files.GetUpperBound(0)
web.DownloadFile(files(i).AbsoluteUri, files(i).FilePath)
Debug.WriteLine("i= " & i & vbLf & web.ResponseHeaders.ToString)
Next
End Sub
When i looked at the web.ResponseHeaders i saw that the difference was in the "Server" header. The malfunctioning response also
returned "text/html" as the "Content-Type", and an additional "X-Pad" field. "Connection: close" was also something that was not
returned in the other (working) attempts
i=0
Date: Sat, 16 Oct 2004 21:06:58 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.1
Last-Modified: Mon, 04 Oct 2004 23:34:34 GMT
ETag: "1a5f4a-9d761-4161de0a"
Accept-Ranges: bytes
Content-Length: 644961
Connection: close
Content-Type: text/plain
i=1
Date: Sat, 16 Oct 2004 21:07:01 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.1
Last-Modified: Mon, 12 Apr 2004 06:48:24 GMT
ETag: "39196b-274-407a3bb8"
Accept-Ranges: bytes
Content-Length: 628
Connection: close
Content-Type: text/html
X-Pad: avoid browser bug
I googled and found this: "In six cases the HTTP message "X-Pad: avoid browser bug" was sent. This was an unannounced change to
Apache version 1.2b9 which provided a fix for a bug in Netscape versions 2.x, 3.x and 4.0b2."
(http://www.ariadne.ac.uk/issue34/web-watch)
I tried with another webserver and that returned "Server: Apache/1.3.27 (Unix) PHP/4.3.2" and there downloads worked perfectly.
Based on this result im suspecting that this might be a problem between Apache and PHP version 4.3.1 handling the requests
correctly, but since the first file in the array allways downloads sucessfully, i'm not completely convinced.
I tried adding a delay of 2 seconds between the downloads and but that didn't fix the problem. I also tried to declare a "web = new
Net.WebClient" before each new download in case it was something with the "web" object. I also tried adding additional headers like
"User-Agent" and so on to the "web.Header" that is sendt to the webserver. None of this fixed the problem.
Does anybody know how to go around to solve this.
This is how the "web.ReturnHeaders" of a working download looked like
i=0
Date: Sat, 16 Oct 2004 22:01:30 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.2
Last-Modified: Tue, 21 Sep 2004 23:03:38 GMT
ETag: "71d21f-e151f-4150b34a"
Accept-Ranges: bytes
Content-Length: 922911
Content-Type: text/plain
i=1
Date: Sat, 16 Oct 2004 22:01:35 GMT
Server: Apache/1.3.27 (Unix) PHP/4.3.2
Last-Modified: Tue, 21 Sep 2004 23:03:17 GMT
ETag: "71d21e-dd69f-4150b335"
Accept-Ranges: bytes
Content-Length: 906911
Content-Type: text/plain