What's the fastest HD setup?

[J. Franchino]

I have a scientific application which is gathering data in real time
and the way things work out, I need to write 513k files to the hard
drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
second). What is the most robust, reliable way to assure this kind of
speed?

We already have a small RAID array (2 HDs, RAID0), but it maxes out at
60MB/sec. That's just under what I need. But more to the point, it
can't consistently give me 60MB/sec. Should I just add more HDs to my
RAID array? If so, what kind should I get? What kind of performance
can I expect then? Thanks in advance. james

 

[Al Dykes]

I have a scientific application which is gathering data in real time
and the way things work out, I need to write 513k files to the hard
drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
second). What is the most robust, reliable way to assure this kind of
speed?

We already have a small RAID array (2 HDs, RAID0), but it maxes out at
60MB/sec. That's just under what I need. But more to the point, it
can't consistently give me 60MB/sec. Should I just add more HDs to my
RAID array? If so, what kind should I get? What kind of performance
can I expect then? Thanks in advance. james

Do you need to do this continuously ? For how long ? How big is your
budget ?

A couple Gig of memory as a RAMdisk could buffer about an hour of data
that could be dumped to disk as time allows.

It may be that a couple digital videotape recorders may be your best
bet if you need more that that.

 

[J. Clarke]

I have a scientific application which is gathering data in real time
and the way things work out, I need to write 513k files to the hard
drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
second). What is the most robust, reliable way to assure this kind of
speed?

We already have a small RAID array (2 HDs, RAID0), but it maxes out at
60MB/sec. That's just under what I need. But more to the point, it
can't consistently give me 60MB/sec. Should I just add more HDs to my
RAID array? If so, what kind should I get? What kind of performance
can I expect then? Thanks in advance. james

Do you for some reason need to have the data committed to the physical
surface of the disk at the end of the acquisition cycle? If so, and if I'm
reading what you posted correctly, then you need to rethink your strategy
because a commercial disk can buffer up to 4 seconds of your data in its
internal RAM, which would defeat such a purpose.

If that's not the case, then you should take a look at a buffering
strategy--store the data in RAM as it is captured and write it to the disk
in the interval before the next acquisition cycle starts. Any current
production disk should be fast enough to support this if I've read the
numbers you provided above correctly.

 

[CWatters]

I have a scientific application which is gathering data in real time
and the way things work out, I need to write 513k files to the hard
drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
second). What is the most robust, reliable way to assure this kind of
speed?

Ok there are two parts to this problem. The average data rate and the peak
data rate....

1) Average data rate

Say we aim for 10 cycles per second. That works out at 513 x 1024 x 10 =
5.2MBytes/sec which doesn't sound too hard.

2) Peak data rate

513 x 1024 /8^-3 = 65MBytes/S

So it's clear the problem is the peak data rate not the average data rate.
Why 8mS?

If you can't give the CPU more than 8mS to do this what you need is a buffer
to reduce the peak data rate. Something like a RAM disk.

Perhaps even something as simple as writing the data to a shared drive on
another PC over a fast LAN might work. Far from the most efficient route
though.

 

[J. Franchino]

I need to do it in 8ms or less because that's all the clock time I
have. I'm grabbing at 30 fps, then doing some online data reduction to
have a final output of 513k. But since I'm doing a lot of online
computations, I only have about 8ms left in my cycle to have it written
to disk. The rest of the time, the CPU is busy.

I'd like to have the process be able to run indefinately at least up to
30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
seconds/minute = 6.9 GB. Far to big to fit in RAM.

I'm already running a multithreaded process, and I'll try some more
code tricks, but the simpler solution (though more expensive) is to
reduce the amount of time it takes to write to disk. Hence my
question. Thanks. I appreciate all of your responses so far.

 

[Eric Gisin]

If you are creating 7.5 files/sec you are doing almost 100 IO/sec on NTFS. You
can probably do that on SCSI 10K or a Raptor. FAT32 should be faster.

You can write 1/2MB in 8ms, but not create a file. You could pre-create all the
files ahead of time to eliminate file-create overhead.

 

[Peter]

I need to do it in 8ms or less because that's all the clock time I

have. I'm grabbing at 30 fps, then doing some online data reduction to
have a final output of 513k. But since I'm doing a lot of online
computations, I only have about 8ms left in my cycle to have it written
to disk. The rest of the time, the CPU is busy.

I'd like to have the process be able to run indefinately at least up to
30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
seconds/minute = 6.9 GB. Far to big to fit in RAM.

I'm already running a multithreaded process, and I'll try some more
code tricks, but the simpler solution (though more expensive) is to
reduce the amount of time it takes to write to disk. Hence my
question. Thanks. I appreciate all of your responses so far.

First find your bottleneck. I don't think it is a disk, more likely it is a
CPU.
Do you have AntiVirus software turned on?
Turn it off and try this test on a quiet system:
@echo off
for /L %%a IN (1,1,4000) DO copy source target%%a >nul

My two systems (AV off) produced the following results:

PIII-933 WD200BB (20GB)
4000 files 566KB each (26 files per second)
152.8s, ~270IOps, 16.0MB/s, CPU 36%

2xPIII-1.3G and 2xMaxtor 9 (80GB) - RAID0
5000 files 566KB each (43 files per second)
115.6s, ~800IOps, 24.5MB/s, CPU0 20%, CPU1 27%

With AV turned on, CPU was much higher, IOps less than a half.
So it depends what your CPU is doing at the time of file aceess...
You may try at least with the hyperthreading turned on and a decent CPU.

 

[Odie Ferrous]

I need to do it in 8ms or less because that's all the clock time I
have. I'm grabbing at 30 fps, then doing some online data reduction to
have a final output of 513k. But since I'm doing a lot of online
computations, I only have about 8ms left in my cycle to have it written
to disk. The rest of the time, the CPU is busy.

I'd like to have the process be able to run indefinately at least up to
30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
seconds/minute = 6.9 GB. Far to big to fit in RAM.

I'm already running a multithreaded process, and I'll try some more
code tricks, but the simpler solution (though more expensive) is to
reduce the amount of time it takes to write to disk. Hence my
question. Thanks. I appreciate all of your responses so far.

Run it from a server machine with a RAM drive of 8GB?

I don't know offhand what the largest RAM drive you can have will be - I
used to use something called Ramdisk Pro and I had the option of
formatting the ram disk as NTFS or FAT32 - so I assume you will be able
to use 8GB formatted as NTFSs.


Odie

 

[J. Clarke]

I need to do it in 8ms or less because that's all the clock time I
have. I'm grabbing at 30 fps, then doing some online data reduction to
have a final output of 513k. But since I'm doing a lot of online
computations, I only have about 8ms left in my cycle to have it written
to disk. The rest of the time, the CPU is busy.

I'd like to have the process be able to run indefinately at least up to
30 minutes. That implies 0.513MB x 7.5 files/second x 30 minutes x 60
seconds/minute = 6.9 GB. Far to big to fit in RAM.

I'm already running a multithreaded process, and I'll try some more
code tricks, but the simpler solution (though more expensive) is to
reduce the amount of time it takes to write to disk. Hence my
question. Thanks. I appreciate all of your responses so far.

You may be hitting the real world limit of the PCI bus here. Typically in
network servers the PCI bus becomes a bottleneck somewhere between 250 and
400 Mb/sec--400 works out to 50 MB/sec with no overhead. That means that
no matter how fast your disks are if you're connecting the host adapter via
the PCI bus you're not likely to get above 50 MB/sec. In your application
in principle you might be able to but it's not the way to bet.

So to start out, I'd try staying with my existing disks and going to a
faster bus and host adapter. While you're about it it you might want to
consider going to an AMD-64 system--some of them can handle 32 gig or more
of RAM--with a PCI-X bus and a PCI-X host adapter.

Beyond that, you should be able to burst 513K to any disk on the market in
8ms if the bus will handle it--they all have 2 meg buffers or larger, some
up to 16, and interfaces that can handle 100+ MB/sec into that buffer. One
strategy, once you've got a fast enough bus in place, would be to alternate
your writes to several different disks--write to one, then the next cycle
write to the next, then the next cycle write to the next and so on in turn
until the first has had enough time to complete its write.

 

[Arno Wagner]

I have a scientific application which is gathering data in real time
and the way things work out, I need to write 513k files to the hard
drive in under 8ms each acquisition cycle (7.5 acquisition cycles per
second). What is the most robust, reliable way to assure this kind of
speed?

We already have a small RAID array (2 HDs, RAID0), but it maxes out at
60MB/sec. That's just under what I need. But more to the point, it
can't consistently give me 60MB/sec. Should I just add more HDs to my
RAID array? If so, what kind should I get? What kind of performance
can I expect then? Thanks in advance. james

The only choice to get sustained, untinterrupted high bandwidth
access is RAM. Whether this is main memory or a solid state disk
depends on your specific needs and how much money you have.

I suspect memeoy is cheapest, solid state disk is most reliable.

Arno

 

[CWatters]

I need to do it in 8ms or less because that's all the clock time I
have. I'm grabbing at 30 fps, then doing some online data reduction to
have a final output of 513k. But since I'm doing a lot of online
computations, I only have about 8ms left in my cycle to have it written
to disk. The rest of the time, the CPU is busy.

30fps = 33mS...

You need to set up some buffers. Could you set up a double buffer? Compute
on one frame while writing the results of the previous frame to disc. That
gives 33mS for writing the data bringing the bandwidth required down to
about 15MBytes/Sec.

 

[CWatters]

You may be hitting the real world limit of the PCI bus here. Typically in
network servers the PCI bus becomes a bottleneck somewhere between 250 and
400 Mb/sec--400 works out to 50 MB/sec with no overhead.

He only hits >50MB/sec because he's trying to write all the data in 8mS. A
simple double buffer would allow upto 33mS (30fps) and reduce the rate to
about 15MBytes/Sec.

 

[J. Clarke]

He only hits >50MB/sec because he's trying to write all the data in 8mS.

If you read my earlier post you would be aware that I had already addressed
this point.

A simple double buffer would allow upto 33mS (30fps) and reduce the rate
to about 15MBytes/Sec.

If his circumstances are such that that is a feasible solution. He seems
adamant about his 8 ms write window.

 

[Arno Wagner]

CWatters wrote:

If you read my earlier post you would be aware that I had already addressed
this point.

If his circumstances are such that that is a feasible solution. He seems
adamant about his 8 ms write window.

As I (and others) said, RAM and solid-state disks are the only solution
with a filesystem, since all other will have larger access times,
at least occasionally.

Maybe a single 15000RPM SCSI disk used in sequential mode (i.e. no
filesystem, sequential sector writes only) with no other devices
on the SCSI bus will work also.

In both cases 66MHz and or 64 bit PCI, alternatively PCI-E,
would likely need to be used as bus.

Arno

 

[Peter]

What OS do you run?

 

[J. Clarke]

As I (and others) said, RAM and solid-state disks are the only solution
with a filesystem, since all other will have larger access times,
at least occasionally.

Not if he's using the disk exclusively for his data and if necessary writing
to several drives. Not unless the OS is generating some unnecessary
overhead. All he has to do is burst his 533k (or whatever the exact
number he gave was) into the buffer and then move on and let the drive
digest that. If the drive hasn't digested that by the next cycle then use
two drives and alternate. If that doesn't allow long enough for the drive
to digest the burst, then three, or four.

Nothing elaborate like RAID, just several drives and alternating writes.

Maybe a single 15000RPM SCSI disk used in sequential mode (i.e. no
filesystem, sequential sector writes only) with no other devices
on the SCSI bus will work also.

Seagate says that the minimum sustained sequential transfer rate on a 36 gig
X15 is 80 MB/sec, which should be adequate.

In both cases 66MHz and or 64 bit PCI, alternatively PCI-E,
would likely need to be used as bus.

PCI-X, which is both 64 bit and 66 MHz or faster. PCI-E is limiting--there
are very few host adapters out for it as yet, one SATA that I know of and
no SCSI.

 

[Al Dykes]

Not if he's using the disk exclusively for his data and if necessary writing
to several drives. Not unless the OS is generating some unnecessary
overhead. All he has to do is burst his 533k (or whatever the exact
number he gave was) into the buffer and then move on and let the drive
digest that. If the drive hasn't digested that by the next cycle then use
two drives and alternate. If that doesn't allow long enough for the drive
to digest the burst, then three, or four.

Nothing elaborate like RAID, just several drives and alternating writes.


Seagate says that the minimum sustained sequential transfer rate on a 36 gig
X15 is 80 MB/sec, which should be adequate.


PCI-X, which is both 64 bit and 66 MHz or faster. PCI-E is limiting--there
are very few host adapters out for it as yet, one SATA that I know of and
no SCSI.

The OP requirement, as stated, (500kb block of data every 8MS) makes
this a fairly tight real-time requirement and getting generic hardware
and consumer OS, even with a RAMdisk, to never drop a frame is
unlikely, IMO. If the OP can ocassionally loose a frame it gets a lot
easier.


Google for real-time linux and see what you find.

 

[CWatters]

If you read my earlier post you would be aware that I had already addressed
this point.

Sorry JC. Been busy.

Colin

 

[CWatters]

The OP requirement, as stated, (500kb block of data every 8MS)

No not "every" he said "in" 8mS. He implies he only has 8mS left in a 30fps
(eg 33mS) cycle. He's capturing data every 33mS, then appears to be
computing it for 25mS. That produces 513K of data that (he says) needs to be
written to disk before the next lot of computation has to start 8mS later.
The long term average is around 15MBytes/S but the way he's doing it the
peak is >65MBytes/Sec which as JC points out is probably hitting the PCI bus
limit.

 

[Peter]

No not "every" he said "in" 8mS. He implies he only has 8mS left in a
30fps

(eg 33mS) cycle. He's capturing data every 33mS, then appears to be
computing it for 25mS. That produces 513K of data that (he says) needs to be
written to disk before the next lot of computation has to start 8mS later.
The long term average is around 15MBytes/S but the way he's doing it the
peak is >65MBytes/Sec which as JC points out is probably hitting the PCI bus
limit.

Probably not. These are the maximum rates:
32bit PCI, 33MHz - 133MB/s
64bit PCI, 33MHz - 266MB/s
64bit PCI, 66MHz - 532MB/s
64bit PCI-X, 100MHz - 800MB/s
64bit PCI-X, 133MHz - 1GB/s