ftpwebrequest: Upload is slow...

[deostroll]

Hi,

I just tried uploading files with the ftpwebrequest class. I find that
they are really slow. However I am not sure if this is because of
me... I am writing everything to the input stream byte by byte so that
I can calculate the progress. However I only calculate only after 1024
bytes have been uploaded. Is there a better way to do this?

Moreover are there any general property settings for the object so
that ftp uploading is at least faster relatively speaking...

--deostroll

 

[Family Tree Mike]

Hi,

I just tried uploading files with the ftpwebrequest class. I find that
they are really slow. However I am not sure if this is because of
me... I am writing everything to the input stream byte by byte so that
I can calculate the progress. However I only calculate only after 1024
bytes have been uploaded. Is there a better way to do this?

Moreover are there any general property settings for the object so
that ftp uploading is at least faster relatively speaking...

--deostroll

You can still calculate progress percentage if you use large byte blocks,
such as 1024 bytes. Percentage is still bytes sent divided by total bytes.
You just incriment the bytes sent by the block size, rather than by one.
This may not be your problem as to speed, but it may help.

 

[deostroll]

You can still calculate progress percentage if you use large byte blocks,
such as 1024 bytes.  Percentage is still bytes sent divided by total bytes.
You just incriment the bytes sent by the block size, rather than by one.
This may not be your problem as to speed, but it may help.

I notice that the cpu usage goes to 100% while I do uploads! Is this
normal?
--deostroll

 

[Family Tree Mike]

I notice that the cpu usage goes to 100% while I do uploads! Is this
normal?
--deostroll

That doesn't sound right, but it's hard to tell without seeing your code why
that might occur.

 

[deostroll]

That doesn't sound right, but it's hard to tell without seeing your code why
that might occur.

The high CPU usage is probably high because of creating/destroying a
progressbar control. I've written a custom class for a datagridview
progressbar column. Here while "painting" it so happens that I create
a new progressbar control and draw its image in the cell meant to show
the progress updates. Probably this is why it happens...

Is it normally supposed to be slower than using ftp.exe? Why is this
so?

--deostroll

 

[Family Tree Mike]

The high CPU usage is probably high because of creating/destroying a
progressbar control. I've written a custom class for a datagridview
progressbar column. Here while "painting" it so happens that I create
a new progressbar control and draw its image in the cell meant to show
the progress updates. Probably this is why it happens...

This "sounds" like you are creating a new progress bar in each repaint. I
hope that is not what you meant...

Is it normally supposed to be slower than using ftp.exe? Why is this
so?

--deostroll

I ran a few comparisons of ftp.exe versus the async upload code example at
http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx

It's true that the timings are somewhat slower (~40%) using the async
upload. If you are seeing times increase significantly in your code, I would
look at running your upload code in a console app to eliminate any issues
with your UI update logic.

Mike

 

[deostroll]

This "sounds" like you are creating a new progress bar in each repaint.  I
hope that is not what you meant...





I ran a few comparisons of ftp.exe versus the async upload code example athttp://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx

It's true that the timings are somewhat slower (~40%) using the async
upload.  If you are seeing times increase significantly in your code, Iwould
look at running your upload code in a console app to eliminate any issues
with your UI update logic.

Mike

As such the ftp-ing (and other stuff) happen on a separate thread in a
synchronous manner. The UI update is asynchronous. But its not a
direct UI update. I am updating a datatable. My datagridview is bound
to the datatable. However I synchronize the updates to this table.

If I adopted the async solution how will I get the progress updates?

--deostroll

 

[Family Tree Mike]

As such the ftp-ing (and other stuff) happen on a separate thread in a
synchronous manner. The UI update is asynchronous. But its not a
direct UI update. I am updating a datatable. My datagridview is bound
to the datatable. However I synchronize the updates to this table.

If I adopted the async solution how will I get the progress updates?

--deostroll

Sorry, I did not mean to imply trying the async method. I was just trying
to point out an existing example and the speed of that example relative to
ftp.exe.

If your upload times are significantly more than the ftp.exe (like a factor
of two or more), then it is likely beneficial to time your upload code
outside of any UI. If the upload code runs resonably well away from any UI,
then you've narrowed the areas you need to review in your code.

Mike

 

[Family Tree Mike]

Sorry, I did not mean to imply trying the async method. I was just trying
to point out an existing example and the speed of that example relative to
ftp.exe.

If your upload times are significantly more than the ftp.exe (like a factor
of two or more), then it is likely beneficial to time your upload code
outside of any UI. If the upload code runs resonably well away from any UI,
then you've narrowed the areas you need to review in your code.

Mike

I should have added, that if you wanted to use the code I referred to above,
you could just add a PercentComplete property to the FtpState class and
monitor that value from some other part of the code which updates the UI.

Mike

 

[deostroll]

I should have added, that if you wanted to use the code I referred to above,
you could just add a PercentComplete property to the FtpState class and
monitor that value from some other part of the code which updates the UI.

Mike

Hi,

Is this the correct way to upload via batch?

for (i = 0; i <= TotalChunks; i += 1024)
{
//Console.WriteLine(i);
ftpStream.Write(fileContents, i, 1024);
ftpStream.Flush();
}
ftpStream.Write(fileContents, 0, fileContents.Length);
ftpStream.Flush();

 

[Family Tree Mike]

Hi,

Is this the correct way to upload via batch?

for (i = 0; i <= TotalChunks; i += 1024)
{
//Console.WriteLine(i);
ftpStream.Write(fileContents, i, 1024);
ftpStream.Flush();
}
ftpStream.Write(fileContents, 0, fileContents.Length);
ftpStream.Flush();

The code does not appear correct, as you are looping over all "Chunks", but
incrimenting by the apparent chunk size. Step through your loop if the
number of chunks was 3. Your loop is exited after the first loop. After
the loop though, you write all the bytes, which seems wrong...

Regarding your question "Is Flush() known to cause an overhead?". Well,
flushing after every write will be slower than not flushing after every
write. You should flush at the end.

 

[deostroll]

The code does not appear correct, as you are looping over all "Chunks", but
incrimenting by the apparent chunk size.  Step through your loop if the
number of chunks was 3.  Your loop is exited after the first loop.  After
the loop though, you write all the bytes, which seems wrong...

Regarding your question "Is Flush() known to cause an overhead?".  Well,
flushing after every write will be slower than not flushing after every
write.  You should flush at the end.

Thanx for the response. I have realized the mistake. Here is a better
version

for (i = 0; i < TotalWholeChunks; i++)
{

ftpStream.Write(fileContents, i*1024, 1024);
ftpStream.Flush();
}
ftpStream.Write(fileContents, i*1024, LastChunkLength);
ftpStream.Flush();
ftpStream.Close();
-----------------------------

If I don't call Flush() then wouldn't my calculation for the progress
completed be incorrect?

--deostroll

 

[Family Tree Mike]

Thanx for the response. I have realized the mistake. Here is a better
version

for (i = 0; i < TotalWholeChunks; i++)
{

ftpStream.Write(fileContents, i*1024, 1024);
ftpStream.Flush();
}
ftpStream.Write(fileContents, i*1024, LastChunkLength);
ftpStream.Flush();
ftpStream.Close();
-----------------------------

If I don't call Flush() then wouldn't my calculation for the progress
completed be incorrect?

--deostroll

Yes that code looks better.

As to the calculation of progress completed, your percentage completed
values will be resonable even without the inner loop flush. Here is why I
say that. When the data is written to the stream, from the client's point of
view, it is done. If it is flushed, that just means it is pushed through the
wire. There is no guarantee that the file is actually written at all on the
server even with a flush. That is totally dependent on the server side ftp
services. So in essense, the best you can do for progress info is how much
you have pushed off the client.

 

[deostroll]

Yes that code looks better.

As to the calculation of progress completed, your percentage completed
values will be resonable even without the inner loop flush.  Here is why I
say that.  When the data is written to the stream, from the client's point of
view, it is done.  If it is flushed, that just means it is pushed through the
wire.  There is no guarantee that the file is actually written at all on the
server even with a flush.  That is totally dependent on the server sideftp
services.  So in essense, the best you can do for progress info is how much
you have pushed off the client.

Actually we would want to know that the ftp process has completed;
only then we can carry on with the next steps (which is outside the
scope of the application I am developing). I've observed that without
using flush a chunk of data gets written and the iteration proceeds
almost immediately. My guess is, underneath there is some sort of
buffering going on. But my execution flow is blocked at the point
where I close the input stream( ftpStream.Close() ). As per current
logic for computing percentage complete, it will show 100% but the
process is still waiting. I mean, the last line in that code is a
status update that says "completed". Now when it gets blocked it would
show 100 % and still say uploading...

However I did find it is better to upload chunks rather than bytes
(hats off 2 u) . But the strange thing is my cpu usage went high
when I did byte-by-byte uploads, Now my cpu usage is considerably low?
I don't understand why?
--deostroll