how to determine progress downloading single file?

[Rich P]

Using the following code I am downloading a (large) file. I fill a byte
array and use a filestream to write the contents of the byte array to
the disk. I am also using a backgroundworker thread. How can I monitor
the download progress? I call ReportProgress from the
backgroundWorker2.IsBusy loop. But how do I determine the end portion
of the operation? I know the file size, but when looking at Windows
Explorer - the file doesn't appear until the process is finished. How
can I monitor the amount of data being written to the disk at some given
time so that I can display the progress in a progress bar? Do I
physically monitor the file size on the disk as it grows - with say a
fileInfo object in the backgroundWorker2.IsBusy loop - something that I
can't see until the operation is completed?

---------------------------------------
WebClient request = new WebClient();
request.Credentials = new NetworkCredential(ftpUserID, ftpPassword);
byte[] fileData = request.DownloadData(ftpServer + "/" + fileName);
FileStream file = File.Create(filePath + "\\" + fileName);
file.Write(fileData, 0, fileData.Length);
file.Close();
---------------------------------------

int i = 0;
while (backgroundWorker2.IsBusy)
{
backgroundWorker2.ReportProgress(i, "x");
i++;
Thread.Sleep(100);
Application.DoEvents();
}

 

[Peter Duniho]

Using the following code I am downloading a (large) file. I fill a byte
array and use a filestream to write the contents of the byte array to
the disk. I am also using a backgroundworker thread. How can I monitor
the download progress? I call ReportProgress from the
backgroundWorker2.IsBusy loop. But how do I determine the end portion
of the operation? I know the file size, but when looking at Windows
Explorer - the file doesn't appear until the process is finished. How
can I monitor the amount of data being written to the disk at some given
time so that I can display the progress in a progress bar?

Your code need to inject itself into the process, rather than using a
"one-shot" method like DownloadData().

In this particular example, you should use WebClient.OpenRead() instead of
DownloadData(). Then with the Stream you get back, you can read blocks of
data, updating your progress indicator between each block.

To read the blocks of data, you either put the loop into the DoWork
handler for a BackgroundWorker and call BackgroundWorker.ReportProgress()
after each block, or you can use the Stream.BeginRead() method and update
your progress in each callback.

In any case, this code is decidedly incorrect:

int i = 0;
while (backgroundWorker2.IsBusy)
{
backgroundWorker2.ReportProgress(i, "x");
i++;
Thread.Sleep(100);
Application.DoEvents();
}

Polling the IsBusy property is bad, sleeping for 100ms between calls to
DoEvents() is bad, and calling DoEvents() is bad. Calling the
BackgroundWorker.ReportProgress() method from other than your DoWork event
handler for the BackgroundWorker is really awful. On top of all that, it
doesn't really look like you're reporting progress in a useful way, as the
idea is for the progress value to actually be in some way representative
of the relative proportion of progress made (e.g. a percentage value).
Incrementing your progress indicator by 1 every tenth of a second doesn't
provide the user with useful information.

Fortunately, if you move the i/o operations off the main GUI thread
correctly, none of that dedicedly incorrect code is necessary. You'll
still need to fix things so that you correctly calculate the percentage
progress -- by dividing the number of bytes received so far by the total
number expected -- but you can eliminate all polling/looping in the main
GUI thread, and allow the worker thread to simply update the GUI as
needed. No need for sleeping or calling DoEvents() at all.

Pete

 

[Rich P]

Thanks very much for your reply. Here is a sample I found using
.OpenRead

--------------------------------------------------------
System.Net.WebClient Client = new WebClient();
Stream strm = Client.OpenRead("http://www.csharpfriends.com");
StreamReader sr = new StreamReader(strm);
string line;
do
{
line = sr.ReadLine();
listbox1.Items.Add(line);
}
while (line !=null);
strm.Close();
--------------------------------------------------------

I am supposing I could place this inside the Dowork of backgroundWorker.
How do I write to the disk? Then do I calculate percentage of progess
based on lines in StreamReader?




Rich

 

[Arne Vajhøj]

Using the following code I am downloading a (large) file. I fill a byte
array and use a filestream to write the contents of the byte array to
the disk. I am also using a backgroundworker thread. How can I monitor
the download progress? I call ReportProgress from the
backgroundWorker2.IsBusy loop. But how do I determine the end portion
of the operation? I know the file size, but when looking at Windows
Explorer - the file doesn't appear until the process is finished. How
can I monitor the amount of data being written to the disk at some given
time so that I can display the progress in a progress bar? Do I
physically monitor the file size on the disk as it grows - with say a
fileInfo object in the backgroundWorker2.IsBusy loop - something that I
can't see until the operation is completed?

---------------------------------------
WebClient request = new WebClient();
request.Credentials = new NetworkCredential(ftpUserID, ftpPassword);
byte[] fileData = request.DownloadData(ftpServer + "/" + fileName);
FileStream file = File.Create(filePath + "\\" + fileName);
file.Write(fileData, 0, fileData.Length);
file.Close();
---------------------------------------

int i = 0;
while (backgroundWorker2.IsBusy)
{
backgroundWorker2.ReportProgress(i, "x");
i++;
Thread.Sleep(100);
Application.DoEvents();
}

See long example attached below.

Arne

===============================

using System;
using System.Drawing;
using System.IO;
using System.Net;
using System.Windows.Forms;
using System.Threading;

public class MainForm : Form
{
private ProgressBar bar;
private Button start;
private Button abort;
private Thread t;
public MainForm()
{
bar = new ProgressBar();
start = new Button();
abort = new Button();
SuspendLayout();
bar.Location = new Point(50, 50);
bar.Size = new Size(200, 50);
bar.Name = "Progress Bar";
start.Location = new Point(50,150);
start.Size = new Size(200, 50);
start.Name = "Start Button";
start.Text = "Start";
start.Click += new EventHandler(StartClick);
abort.Location = new Point(50,250);
abort.Size = new Size(200, 50);
abort.Name = "Abort Button";
abort.Text = "Abort";
abort.Click += new EventHandler(AbortClick);
ClientSize = new Size(300, 350);
Controls.Add(bar);
Controls.Add(start);
Controls.Add(abort);
Name = "Main Form";
Text = "Main Form";
ResumeLayout(false);
}
void Reset(int n)
{
bar.Minimum = 0;
bar.Maximum = n;
bar.Value = 0;
}
void Update(int n)
{
bar.Value = n;
}
delegate void ResetHandler(int n);
delegate void UpdateHandler(int n);
void Copy()
{
HttpWebRequest req =
(HttpWebRequest)WebRequest.Create("http://www.tmk.com/ftp/vms-freeware/fileserv/unzip.zip");
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
int tot = (int)resp.ContentLength;
if(bar.InvokeRequired)
{
bar.Invoke(new ResetHandler(Reset), new object[] { tot });
}
else
{
Reset(tot);
}
Stream f1 = resp.GetResponseStream();
Stream f2 = new FileStream(@"C:\unzip.zip", FileMode.CreateNew,
FileAccess.Write);
int sofar = 0;
byte[] b = new byte[1000];
int n;
while((n = f1.Read(b, 0, b.Length)) > 0 )
{
f2.Write(b, 0, n);
sofar += n;
if(bar.InvokeRequired)
{
bar.Invoke(new UpdateHandler(Update), new object[] {
sofar });
}
else
{
Update(sofar);
}
}
f2.Close();
f1.Close();
resp.Close();
}
void StartClick(object sender, EventArgs e)
{
t = new Thread(new ThreadStart(Copy));
t.Start();
}
void AbortClick(object sender, EventArgs e)
{
t.Abort();
}
[STAThread]
public static void Main(string[] args)
{
Application.Run(new MainForm());
Application.Exit();
Environment.Exit(0);
}
}

 

[Peter Duniho]

Thanks very much for your reply. Here is a sample I found using
.OpenRead

--------------------------------------------------------
System.Net.WebClient Client = new WebClient();
Stream strm = Client.OpenRead("http://www.csharpfriends.com");
StreamReader sr = new StreamReader(strm);
string line;
do
{
line = sr.ReadLine();
listbox1.Items.Add(line);
}
while (line !=null);
strm.Close();

Yes, sort of. It sounds like you simply want the raw bytes from the
stream, rather than interpreting them as text. So you'd leave out the
StreamReader stuff in that code example. But the basic idea is the same.

How do I write to the disk?

See the FileStream class. Create one, for each block of bytes you read
from the WebClient stream, write the same block of bytes (taking care to
use the actual length of the read, not necessarily the length of the
byte[] buffer) to the FileStream instance.

Then do I calculate percentage of progess
based on lines in StreamReader?

It's _possible_ that the Stream that is returned from the
WebClient.OpenRead() method will support the Length property (I don't know
for sure...I'm not really all that familiar with this area of .NET). If
so, you can use that. Otherwise, I believe you can get the response
headers from the WebClient instance and look for the Content-Length header
to determine the number of bytes you'll be reading.

Alternatively, see Arne's reply which eschews the WebClient class in favor
of the slightly lower-level WebRequest/Response classes, which provide
more direct access to the elements of the response you need.

Pete

 

[Rich P]

Thanks all for replies. I am a little confused though
void Copy()
{
HttpWebRequest req =
(HttpWebRequest)WebRequest.Create("http://www.tmk.com/ftp/vms-freeware/f
ileserv/unzip.zip");
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
int tot = (int)resp.ContentLength;
if(bar.InvokeRequired)
{
bar.Invoke(new ResetHandler(Reset), new object[] { tot });
}
else
{
Reset(tot);
}
Stream f1 = resp.GetResponseStream();
Stream f2 = new FileStream(@"C:\unzip.zip", FileMode.CreateNew,
FileAccess.Write);
int sofar = 0;
byte[] b = new byte[1000];
int n;
while((n = f1.Read(b, 0, b.Length)) > 0 )
{
f2.Write(b, 0, n);
sofar += n;
if(bar.InvokeRequired)
{
bar.Invoke(new UpdateHandler(Update), new object[] {
sofar });
}
else
{
Update(sofar);
}
}
f2.Close();
f1.Close();
resp.Close();
}
<<

In a previous post it was mentioned that .Invoke.Required... was not
favored. Would this be a case where .Invoke... is the way (or a way) to
go? Or is there an alternative?

Thanks all,

Rich

 

[Peter Duniho]

[...]
In a previous post it was mentioned that .Invoke.Required... was not
favored. Would this be a case where .Invoke... is the way (or a way) to
go? Or is there an alternative?

Arne may not subscribe to my opinion, thus the use of InvokeRequired in
his code. You can do it either way. Just use the code as Arne shows it,
or write the "if" statements differently. For example, instead of:

if(bar.InvokeRequired)
{
bar.Invoke(new ResetHandler(Reset), new object[] { tot });
}
else
{
Reset(tot);
}

Just write:

bar.Invoke((MethodInvoker) delegate
{
Reset(tot);
});

(If you're worried about the overhead of the anonymous method, then just
write out the call to Invoke() as Arne shows it, just without the "if"
statement and the contents of the "else" block...i.e. unconditionally call
Invoke()).

Pete

 

[Arne Vajhøj]

In a previous post it was mentioned that .Invoke.Required... was not
favored.

MS put the method there.

Makes sense to me to use it.

Would this be a case where .Invoke... is the way (or a way) to
go?

Unless you tell win forms not to check and you decide to
take the chance by updating the GUI from another thread,
then Invoke is needed.

Arne

 

[Arne Vajhøj]

[...]
In a previous post it was mentioned that .Invoke.Required... was not
favored. Would this be a case where .Invoke... is the way (or a way) to
go? Or is there an alternative?

Arne may not subscribe to my opinion, thus the use of InvokeRequired in
his code. You can do it either way. Just use the code as Arne shows
it, or write the "if" statements differently. For example, instead of:

if(bar.InvokeRequired)
{
bar.Invoke(new ResetHandler(Reset), new object[] { tot });
}
else
{
Reset(tot);
}

Just write:

bar.Invoke((MethodInvoker) delegate
{
Reset(tot);
});

(If you're worried about the overhead of the anonymous method, then just
write out the call to Invoke() as Arne shows it, just without the "if"
statement and the contents of the "else" block...i.e. unconditionally
call Invoke()).

The code is not using anonymous method, because it is old code.

Arne

 

[Peter Duniho]

MS put the method there.

Makes sense to me to use it.

Hmph.

Then I'll provide the short form of my objection to it: the
Control.Invoke() method will execute the delegate immediately, rather than
queuing it to the message queue for execution, if the call to Invoke() is
already on the correct thread (i.e. InvokeRequired would return "false").
In other words, Control.Invoke() is already making the exact same check.

I suppose if in the unlikely event the common case was to not need to
invoke, you might get a marginal performance improvement by checking it
before calling Invoke(), thus avoiding the call to Invoke() altogether.
But typically, if you're writing code to handle invoking, it's because
that's expected to be the common case (often if it's not the common case,
the logic for invoking is hoisted further up, to the point in the code
where it _is_ the common case).

For the general pattern, I also don't like the way MSDN suggests writing a
method that does two completely different things depending on state (i.e.
either call Invoke() with itself as the target method, or do some real
work), but your code example doesn't really suffer from that (in both code
paths of the "if" you call the same method...you just vary how it's
called).

For a more detailed rant (on my blog, which I really should get around to
writing for again), see:
http://msmvps.com/blogs/duniho/arch...chnique-for-using-control-invoke-is-lame.aspx

Pete

 

[Rich P]

Thanks for this code sample. I just tried it out as is. It works fine.
I will try it next on my ftp files. I will be chewing on this for a
little while.

Thanks again.

Rich

 

[Rich P]

OK. Here is my final product that appears to do what I want --
including the progressbar. Any comments/suggestions are welcome.

----------------------------------------------------
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.IO;
using System.Net;
using System.Threading;

...

private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
try
{
FtpWebRequest sizeRequest =
FtpWebRequest.Create("ftp://myftpsvr/somedir/myfile.mp4") as
FtpWebRequest;
sizeRequest.Method = WebRequestMethods.Ftp.GetFileSize;
sizeRequest.Credentials = new NetworkCredential("myuid", "mypwrd");
sizeRequest.UsePassive = true;
sizeRequest.UseBinary = false;
sizeRequest.KeepAlive = false; //close the connection when done
FtpWebResponse sizeResponse = sizeRequest.GetResponse() as
FtpWebResponse;
long fileSize = sizeResponse.ContentLength;

FtpWebRequest request =
FtpWebRequest.Create("ftp://myftpsvr/somedir/myfile.mpr") as
FtpWebRequest;
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = new NetworkCredential("myuid", "mypwrd");
request.UsePassive = true;
request.UseBinary = false;
request.KeepAlive = false; //close the connection when done

FtpWebResponse response = request.GetResponse() as FtpWebResponse;

Stream reader = response.GetResponseStream();

long fileBytesRead = 0;
int n;
byte[] buffer = new byte[1024];
using (Stream streamFile = File.Create(@"C:\1C\myfile.mp4"))
{
while (true)
{
int bytesRead = reader.Read(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
break;
}
else
{
streamFile.Write(buffer, 0, bytesRead);
}
fileBytesRead += bytesRead;
n = Convert.ToInt32(fileBytesRead * 100 / fileSize);
backgroundWorker1.ReportProgress(n, "in use");
}
}
reader.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message, "Problem downloading file",
MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
}

private void backgroundWorker1_ProgressChanged(object sender,
ProgressChangedEventArgs e)
{
progressBar1.Value = e.ProgressPercentage;
}

private void backgroundWorker1_RunWorkerCompleted(object sender,
RunWorkerCompletedEventArgs e)
{
Console.WriteLine("Download completed!");
}

 

[Arne Vajhøj]

Hmph.

Then I'll provide the short form of my objection to it: the
Control.Invoke() method will execute the delegate immediately, rather
than queuing it to the message queue for execution, if the call to
Invoke() is already on the correct thread (i.e. InvokeRequired would
return "false"). In other words, Control.Invoke() is already making the
exact same check.

Is that documented behavior or just current implementation behavior?

Arne

 

[Peter Duniho]

[...]

In other words, Control.Invoke() is already making the exact same check.

Is that documented behavior or just current implementation behavior?

That's debatable. But, while the documentation doesn't explicitly state
that specific behavior, IMHO that's simply an example (of many) of the
documentation being incomplete. If it _didn't_ do this -- that is, if it
didn't simply invoke the delegate directly and instead always queued it
and waited -- then there would be a serious deadlock risk that would be
VERY important to document; yet, there's no mention of such a risk in the
documentation at all.

In other words, in this particular case, lack of a specific warning is
tantamount to documentation of the desired behavior.

I don't see how this could ever change. Running the risk of causing new
deadlock bugs to appear in .NET code that's been working fine for decades
is just too large a barrier.

Pete

 

[Peter Duniho]

OK. Here is my final product that appears to do what I want --
including the progressbar. Any comments/suggestions are welcome.

Looks basically fine to me. I assume the strings for filenames are just
placeholders for your post, and that you really do get the length of the
same file that you eventually download (in the code you posted, they are
different).

The only things that looks like a real problem per se is that you are
seeing the UseBinary property to "false". Even for text files, I never
used ASCII mode, and for sure a binary file should be using binary mode.

Of course, everyone has a slightly different take on things, so...some
things I would have done differently :

-- Made the buffer larger. 4096 is the same size as a memory page,
and 8192 is roughly the length of a single TCP packet, either of which
should give you somewhat better throughput. But in reality, it probably
doesn't matter too much, especially if this isn't intended to be
high-capacity server code.

-- Idiomatically, I think it's more common to write a "while" loop
where the continue condition is the actual read, rather than using "true"
and included a break later on. So this:

while (true)
{
int bytesRead = reader.Read(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
break;
}
else
{
streamFile.Write(buffer, 0, bytesRead);
}
//...
}

Becomes this:

int bytesRead;
while ((bytesRead = reader.Read(buffer, 0, buffer.Length) > 0)
{
streamFile.Write(buffer, 0, bytesRead);
//...
}

Even in the former construction, the "else" on the "if" is superfluous.
The true condition leads to an exit from the loop, so every line of code
inside the loop but after the "if" statement is basically an "else"
already.

-- I would never call a Stream variable "reader"; too much potential
for confusion with the various "...Reader" types (e.g. StringReader,
StreamReader, XmlReader, etc.). I'd call a Stream a "stream".

-- I would also put the initialization of the "reader" variable (named
differently, of course ) inside a "using" block.

-- I wouldn't use "as" for casting the WebRequest and WebResponse.
Since the cast should always succeed, you should just go ahead and do an
explicit cast. That way, if something _does_ break, you get an invalid
cast exception telling you exactly what went wrong, instead of a null
reference exception.


And some observations not really specific to your code:

-- I wish that FtpWebResponse.ContentLength, or the Stream.Length of
the returned Stream, would indicate the size of the file. It seems kind
of silly you have to query the file size directly before downloading.
But, I suppose the FTP protocol doesn't provide for a reliable alternative
(though, the "SIZE" command isn't mentioned in RFC959 anyway, and I have
no idea how reliable that is itself)

-- One annoying thing about the "SIZE" command is that when you
attempt to execute the "SIZE" command on a directory or link, the FTP
server replies with a 550 (which actually seems fine to me), and when .NET
sees that it panics and closes the connection (that seems very annoying to
me). This wouldn't be so bad, except that there's no reliable way to
avoid sending the "SIZE" because the only detailed directory info FTP
offers isn't standardized (you get whatever the server considers a
"detailed directory listing"). So, either you try to parse the detailed
listing and hope your server is cooperating, or you have to reopen the
connection after each time you send "SIZE" for a name that's not actually
a file.

I did some Googling, and found that there's an "FTPClient" library out
there that is more "FTP-like" in its feature-set. I find the .NET
FtpWebRequest somewhat frustrating because it's so stateless. It works
great if you're dealing with a few files at a time, but the code gets a
bit messy if you need to pretend to the user that he's working in
individual directories, even though everything internally is essentially
absolute filenames.


In case you can't tell, prompted by your questions, I went ahead and did
some playing around with the FtpWebRequest stuff myself. I'd used it a
little before, but not enough to know some of the "ins and outs" that are
relevant to your use. I did learn some new stuff, including the fact that
it turns out my web host where I do most of my FTP-ing can't keep up with
my current broadband connection (it's about a third the speed! ).

And then I thought, well, I might as well play around with various
parameters and see if I can affect the speed where the program itself
might be a bottleneck, so I went to set up an FTP server on my Mac. Only
to discover that for most files (but not all), it rejects the "SIZE"
command (i.e. not just for directories and links, like I was seeing on my
other server).

Fer cryin' out loud.

Anyway, it's late...I'm done playing, for now. Hope you find some of my
comments useful.

Pete

 

[Rich P]

Hi Piete,

Yes, your comments were very useful. I have something that works. Now
I just need to taylor it a little bit more. I will implement your
suggestions about the array size, but I think I will stay with the
double query of the file size because I know that works, and I will
modify my While loop as you suggest and change some of the variable
names - to stay with best practices.

Thanks Pete and Arne for your help and suggestions.

Rich

 

[Peter Duniho]

Hi Piete,

Yes, your comments were very useful. I have something that works. Now
I just need to taylor it a little bit more. I will implement your
suggestions about the array size, but I think I will stay with the
double query of the file size because I know that works,

Sorry if I wasn't clear: I'm not aware of an alternative to querying the
file size before downloading the file. AFAIK, FTP doesn't provide the
file length as part of the RETR command, so unless .NET were adding a SIZE
command before the RETR (and it doesn't appear to be), the only way to get
the data would be to send the SIZE command first.

If you do know of an alternative that you're simply choosing not to use,
I'd love to know about it for my own edification.

Anyway, glad you got it working!

 

[Rich P]

If you do know of an alternative that you're simply

choosing not to use, I'd love to know about it for my own
edification.

I don't have any alternatives at this time. In my test case (where I
hardcode the ftpserver and filename) I first run the query for the
filesize

FtpWebRequest sizeRequest =
(FtpWebRequest)FtpWebRequest.Create("myftpsrverAndFileName")

sizeRequest.Method = WebRequestMethods.Ftp.GetFileSize;<<--

sizeRequest.Credentials = new NetworkCredential("myuid", "mypwrd");
sizeRequest.UsePassive = true;
sizeRequest.UseBinary = false;
long fileSize = sizeResponse.ContentLength;

then I call the .DownloadFile method.

Rich

 

[Peter Duniho]

If you do know of an alternative that you're simply
choosing not to use, I'd love to know about it for my own
edification.

I don't have any alternatives at this time. In my test case (where I
hardcode the ftpserver and filename) I first run the query for the
filesize [...]

Ah, okay. As far as I know, that's your only option. Your previous reply
seemed to suggest that _I_ had suggested an alternative that you weren't
going to use. Guess I was just confused about what you meant.

Thanks!