Big file generator?

  • Thread starter Thread starter Mark Carter
  • Start date Start date
REM said:
Very good idea!
Click to expand...

However such a large file may simply be created by marking the
appropriate entries in the hd cluster tables (...desperately tries to
remember back to his old dos days ... and fails ...) and not by actually
writing data into a huge file.
It sounds like you want to do the latter, actually write 20GB of data.
 
REM said:
Pretty cool! If, for some reason it doesn't work I have a utility that
can easily be can be altered to do this. It currently writes 1, 5, 10,
50 and 100 meg files and a million character file with no line breaks.
Click to expand...

The one above does not execute in W98SE.

REM:
If your existing util is a small standalone program, could you post a URL for it?
I would use it to generate 1,5,10 MB text files for testing.

Thanks,

Mike Sa
 
The one above does not execute in W98SE.

REM:
If your existing util is a small standalone program, could you post a URL for it?
I would use it to generate 1,5,10 MB text files for testing.
Click to expand...

I got interested in this, and I've put together a small utility which
creates a text file in 10 million byte (decimal) increments. The user
selects the number of 10 M units to write. My utility measures the
time it takes to write each 10 M unit, and writes that info plus the
cumulative time to the screen as it writes. So my util has more
purposes that just creating large files. I'm interested also in
checking writing speed under various conditions.

It's a 16 bit DOS program that runs fine in a DOS window, and should
run on any version of Windows or in plain DOS. I need to put in a ASM
module that checks free drive space. The utility will refuse to write
if insufficient free drive space is available. I'll probably have that
done today and put the util up at my web site later today or tomorrow.

I'm seeing approximately one second to write each 10 M unit (10
million bytes per second) on one test machine in a DOS window. Thus
each billion bytes takes about 100 seconds to write on that PC. Your
mileage will vary :)


http://home.epix.net/~artnpeg
 
Art said:
I got interested in this, and I've put together a small utility which
creates a text file in 10 million byte (decimal) increments. The user
selects the number of 10 M units to write. My utility measures the
time it takes to write each 10 M unit, and writes that info plus the
cumulative time to the screen as it writes. So my util has more
purposes that just creating large files. I'm interested also in
checking writing speed under various conditions.

It's a 16 bit DOS program that runs fine in a DOS window, and should
run on any version of Windows or in plain DOS. I need to put in a ASM
module that checks free drive space. The utility will refuse to write
if insufficient free drive space is available. I'll probably have that
done today and put the util up at my web site later today or tomorrow.

I'm seeing approximately one second to write each 10 M unit (10
million bytes per second) on one test machine in a DOS window. Thus
each billion bytes takes about 100 seconds to write on that PC. Your
mileage will vary :)


http://home.epix.net/~artnpeg
Click to expand...
Thanks, Art. But I'm interested in smaller increments, 1,2,4, to at most 10 MB
generated flles. I could never duplicate REM's results before. Editpad for him
worked on large files, mine choked over about 800 KB. My registry keeps changing,
so I want to run tests later.

Mike Sa
 
ms <[email protected]> wrote:
Click to expand...
Thanks, Art. But I'm interested in smaller increments, 1,2,4, to at most 10 MB
generated flles. I could never duplicate REM's results before. Editpad for him
worked on large files, mine choked over about 800 KB. My registry keeps changing,
so I want to run tests later.
Click to expand...

Mike: Note the utility I listed creates all files at the same time.
There are 1, 2, 5, 10, 50, 100 meg files and one with a million
characters without any line breaks.

You can delete whatever you want. Just make sure you have the drive
space available.

I had 256 megs of ram on that test machine. Any quality program
'should' open the 100 meg file if you have that much free ram. If not,
the swap file is used and that's a pretty slow operation.
 
I want to test out a hard disk by generating a large file (let's say
20Gb) to it. I don't care what its contents actually are, it could be
random as far as I can.

Is there such a utility to do this?
Click to expand...
DupBlock is a text duplication utility that generates blocks of text
that are alike except for one or more sequenced variables. You supply
the default text section to be repeated and specify variable names in
the positions where a variable is to be used. You then define the
variables, which can be counter or list variables. Specify the number of
iterations to perform and then generate the duplication. Depending on
your needs this may be faster than copy/pasting text manually in a text
editor and then doing tedious manual edits.

DubBlock is for Windows 95/98/ME NT/2000/XP.

http://www.funduc.com/dupblock.htm

Don't let the scripting scare you! I haven't used the scripting yet, but
I'm sure at some point I will. The program meets my requirements which
are: creates files which contain the data I want and the size I that
want and it's free.

As a test I created a 1024 byte file with all ones. I used the character
"ÿ" (which is "FF" in hex) and pasted it into "text to duplicate" then
set Repeat Block to 1024. I then copied and pasted the contents of the
1024 byte file into "text to duplicate" and set "Repeat Block" to
1048576 which created a 1GB file with all ones in it. Using a "Repeat
Block" count of 10485760 will create a 10GB file or using a count of
20971520 will create a 20GB file.

This program may do more than you want or need and be larger then you'd
like, but when I was searching for a program to create files with the
data I wanted and the size that I wanted them to be, this one turned out
to be a real find for me.
 
Mark Carter skrev:
I want to test out a hard disk by generating a large file (let's say
20Gb) to it. I don't care what its contents actually are, it could be
random as far as I can.

Is there such a utility to do this?
Click to expand...

Sure it is. It should be in your base install.

dd if=/dev/zero of=/var/tmp/big.file bs=1024 count=20000000

For a 20Gb file

or

dd if=/dev/zero of=/var/tmp/big.file bs=1024 count=2000000000

For a 2Tb file
 
REM said:
Mike: Note the utility I listed creates all files at the same time.
There are 1, 2, 5, 10, 50, 100 meg files and one with a million
characters without any line breaks.

You can delete whatever you want. Just make sure you have the drive
space available.

I had 256 megs of ram on that test machine. Any quality program
'should' open the 100 meg file if you have that much free ram. If not,
the swap file is used and that's a pretty slow operation.
Click to expand...
Thanks, REM. Mine has 96 MB RAM. Dunno if that's a reason, but Edxor is the only
editor I can use on files of 1.5 MB or so. My old favorite, Editpad Classic,
started to choke over 900 KB.

Mike Sa
 
I had 256 megs of ram on that test machine. Any quality program
'should' open the 100 meg file if you have that much free ram. If not,
the swap file is used and that's a pretty slow operation.
Click to expand...

Editors don't necessarily require much ram since they can work with
file pointers and short (maybe only 50k or 100k) sections of a file at
a time. Editing (writing) is done with multiple filing operations.
Thus, large files will take noticeable editing time, not because of
the use of the swap file, but simply because the large file is being
re-pieced together when additions or deletions are made.

File viewers can view any portion of really huge files seemingly
"instantly" because of the use of file pointers. They requre
insignifcant ram.

A truly "quality" editor would handle small and medium sized files up
to several tens of megs in size in extended memory up to the point
when the file has to be re-written (when done editing it). It would
be smart enough to handle huge files using pointers and filing methods
so it could handle files of any size.

But there's no way around having to re-write a file that's been edited
<smile>, and that takes noticeable time with huge files.


http://home.epix.net/~artnpeg
 
Editors don't necessarily require much ram since they can work with
file pointers and short (maybe only 50k or 100k) sections of a file at
a time. Editing (writing) is done with multiple filing operations.
Thus, large files will take noticeable editing time, not because of
the use of the swap file, but simply because the large file is being
re-pieced together when additions or deletions are made.
Click to expand...
File viewers can view any portion of really huge files seemingly
"instantly" because of the use of file pointers. They requre
insignifcant ram.
Click to expand...
A truly "quality" editor would handle small and medium sized files up
to several tens of megs in size in extended memory up to the point
when the file has to be re-written (when done editing it). It would
be smart enough to handle huge files using pointers and filing methods
so it could handle files of any size.
Click to expand...

In the test, I loaded up to 100 meg files and looked for "The End,"
which ends each of the test data files. There were very few that could
find the end, even when scrolling. The best, EditPad include, would do
so.
But there's no way around having to re-write a file that's been edited
<smile>, and that takes noticeable time with huge files.
Click to expand...

True. I think some of the utilities to cut columns and such that were
suggested are valuable assets. Anything to trim the working data down
some helps.

I haven't tried, but I don't think any editor I tried could
successfully open, edit, and save a text file in the gig arena on the
machine I was using. I've got a new machine. I'll try some larger
files and see how it does.
 
Back
Top