App Performance help

[Guest]

Hi,
I am developing an app that will unzip several zip files and append the
contents of the zip files to one text file. The zip files contains one text
file with a standard format but the size of the file will be an average of
100k to 300k with at least 7000 rows.

Could I use a simple string (Dim s as string)? So first I would unzip a
file, read its contents, store it in the string then unzip the second, append
it's contents to the same string ........after all the files are unzipped I
then write the contents of the string to a text file. Is that the proper way?

Thanks

 

[Nick Malik [Microsoft]]

It would work for a small set of files. Problem is that your plan is
terrible on memory. Strings are immutable in .Net which means that you
create a string for a file, create a string for another file, then throw
them both away as soon as you have a string for the two files together.

If you are simply appending, then use a textreader and textwriter.

Psuedocode
open the source zip
get a list of contained files
open the destination file for append
for each file in the zip
unzip the file (into memory?) I don't know what library you are
using
open the input file stream
while not eof(input file stream)
read a block of text (say, about 20K?)
write the block of text to the destination file
end while
close unzipped file (and clean up, if necessary)
next file
close destination file


This will scale to situations where the zip contains large individual files
as well as when the zip contains large numbers of files.

--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.

 

[Guest]

Each of my zip files has a single file and the file is about 200k with about
7000 rows. I'll have to loop abou 10 zip files. Should I unzip to memory
orunzip the file to a folder then read the file and append each file to the
new file created?

 

[Cor Ligthert [MVP]]

Chris,

For your problem is the stringbuilder, be aware that you have to set a
reference to System.Text to use it.

http://msdn.microsoft.com/library/d...ml/frlrfsystemtextstringbuilderclasstopic.asp

I hope this helps,

Cor

 

[Nick Malik [Microsoft]]

Like I said, I don't know what unzip library you are using. Look into the
code in the library to see which is better at handling memory. In general,
you will get more performance by keeping things in memory, but you may get
better memory management by throwing things to disk.

--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.

 

[Guest]

Hi,
How do I destroy a stringbuilder after the data is sent to a text file.
Should I use

sb=nothing

Thanks

 

[Cor Ligthert [MVP]]

Chris,

There are only very few cases in managed languages where you have to set an
object to nothing or whatever like that.

Most of those is where you have placed objects globaly, otherwise they will
go out of scoped and automaticly be released.

I hope this helps,

Cor