C# Batch Process Program Slows Dramatically Over Time

[JBudge]

I've created a program that will create zips and exes of all the
product downloads on our website (mostly clipart and PowerPoint
templates, around 6000 total), but the program slows down dramatically
after the first few minutes. The process never gets above 24meg, all
child processes called are properly closed when they are done. I've
checked the GDI Objects for each process in Task Manager and none of
them get above 84 (that process is the UI for the program).

Simple rundown of the program flow:

1) Program checks user defined directory for a specific file type
(either PNG or POT) and creates an array of file names.

2) Program iterates through array and creates a zip file (using CLI
Winzip) that contains the POT or PNG along with a readme and eula txt
file. After the zip file is created, the program then creates a self
extracting archive (exe) using seabuilder from that zip file.

3) Program moves on to next file in array.


Each process call for WinZip and SEABuilder is terminated when the zip
or exe has been created.

I've noticed the slowdown most after the 40th file is created (usually
within the first minute). When left running long enough (a few hours),
the program can slow to the point where it is only making 5 zips/exes
every minute.

I've tried revamping the code a bit, but nothing seems to make much of
a difference. Can anyone give me a hint as to where to start looking
for leaks and/or performance issues?

Thanks,

Jason

 

[David Browne]

I've created a program that will create zips and exes of all the
product downloads on our website (mostly clipart and PowerPoint
templates, around 6000 total), but the program slows down dramatically
after the first few minutes. The process never gets above 24meg, all
child processes called are properly closed when they are done. I've
checked the GDI Objects for each process in Task Manager and none of
them get above 84 (that process is the UI for the program).

Simple rundown of the program flow:

1) Program checks user defined directory for a specific file type
(either PNG or POT) and creates an array of file names.

2) Program iterates through array and creates a zip file (using CLI
Winzip) that contains the POT or PNG along with a readme and eula txt
file. After the zip file is created, the program then creates a self
extracting archive (exe) using seabuilder from that zip file.

3) Program moves on to next file in array.


Each process call for WinZip and SEABuilder is terminated when the zip
or exe has been created.

I've noticed the slowdown most after the 40th file is created (usually
within the first minute). When left running long enough (a few hours),
the program can slow to the point where it is only making 5 zips/exes
every minute.

I've tried revamping the code a bit, but nothing seems to make much of
a difference. Can anyone give me a hint as to where to start looking
for leaks and/or performance issues?

Since this is really a script, the problem is likely not in C# code.
Instrument the code to take timings for each step in the process and see
where the slowdown occurs. You can start by just outputining the timings to
the console.

David

 

[Marc Gravell]

An interesting problem, but I'm not sure that there's enough there to start
looking for lag... unless you are constantly adding anything to a collection
somewhere? debug info perhaps? (but then I would expect memory to
increase...).

From a very similar project I know it is possible... in a quick test using
of my code (using SharpZipLib [not WinZip], and without the SEABuilder step)
I measured 1066 zip creations in 47s (working on a tree of 1478 files in 267
folders [i.e. some get larger when zipped, so left "as was"], going from
100MB to 39MB).

So I may be able to assist given more information - unless SEABuilder is the
issue... (which sounds unlikely if the processes are being cleanly exited).

Marc