Software vs. Hardware RAID - several questions

[Carlos Moreno]

Hi,

I'm doing some tests -- and getting a few surprises -- with
software RAID on the following setup:

Asus A7N8X Motherboard (Presumably Ultra ATA 133; PCI 2.2)
with an Athlon 2800 and 1GB of RAM. (with the latest BIOS
upgrade -- rev. 1010-B)

I have a PCI IDE controller card (the ones that come with
the Maxtor ATA133 drives -- it's really a Promise card, re-
branded to Maxtor).

Three Hard drives Western Digital 200GB ATA100.

Windows 2000, with SP4 installed.

When I measure raw performance on each of the drives (using
the DskSpeed utility), I get an impressive 57MB/sec. However,
when I configure them on RAID-0 (software RAID, or stripped
volume, as Windows 2000 calls it), it only goes up to 78MB/sec.

Even more surprising is that when putting a tripple RAID-0
volume (a volume stripped across three drives), I only get
81MB/sec.

I was expecting that the overhead given by RAID-0 would be
basically negligible, and thus a software RAID solution
would duplicate or triplicate the speed.

Is this normal? What bottleneck am I experiencing? (I
think the bandwidth for the PCI bus -- PCI 2.2, according
to Asus' manual -- is 266MB/sec, correct? So, there should
be no bottleneck once the data reaches the controllers)

I am, of course, putting the three hard drives on three
separate channels (MB IDE-1: one crappy hard drive with the
system -- just for testing purposes; MB IDE-2: WD200 hard
drive + CD-ROM -- the CD-ROM is inactive most of the time,
and the raw performance of this HD is 60MB/sec anyway;
Promise Controller IDE-1: WD200 HD; Promise IDE-2: WD200 HD)

Can you see any obvious red flag? Or are the numbers I'm
getting what I should be expecting?

Thanks for any comments!

Carlos
--

 

[Egil Solberg]

Is this normal? What bottleneck am I experiencing? (I
think the bandwidth for the PCI bus -- PCI 2.2, according
to Asus' manual -- is 266MB/sec, correct?

I thought it was 133 MB/s.

32 bits x 33 MHz= 1,056Gbit/s

 

[Carlos Moreno]

I thought it was 133 MB/s.

32 bits x 33 MHz= 1,056Gbit/s

???

AFAIR, PCI 2.2 supports 66MHz and 64-bit. The Promise card's
datasheet describes it as "Ultra ATA/133 Controller for 66MHz
PCI" -- and they do say (I quote directly from the datasheet),
"Ultra133 also supports 66MHz PCI motherboards that offer a
broader bandwidth of up to 266MB/sec ..."

Since the Asus manual for the motherboard talks about PCI2.2,
I assumed this 266MB/s would be the useable bandwith.

????

Carlos
--

 

[Peter]

???

AFAIR, PCI 2.2 supports 66MHz and 64-bit. The Promise card's
datasheet describes it as "Ultra ATA/133 Controller for 66MHz
PCI" -- and they do say (I quote directly from the datasheet),
"Ultra133 also supports 66MHz PCI motherboards that offer a
broader bandwidth of up to 266MB/sec ..."

Since the Asus manual for the motherboard talks about PCI2.2,
I assumed this 266MB/s would be the useable bandwith.

As far as I know A7N8X has 5 32bit 33MHz PCI slots.
Thus 133MB/s applies.

 

[J. Clarke]

???

AFAIR, PCI 2.2 supports 66MHz and 64-bit. The Promise card's
datasheet describes it as "Ultra ATA/133 Controller for 66MHz
PCI" -- and they do say (I quote directly from the datasheet),
"Ultra133 also supports 66MHz PCI motherboards that offer a
broader bandwidth of up to 266MB/sec ..."

Since the Asus manual for the motherboard talks about PCI2.2,
I assumed this 266MB/s would be the useable bandwith.

While PCI 2.2 may _allow_ for 66 MHz and 64-bit that doesn't mean that
either is _required_. Unless the docs specifically state 66 MHz or 64 bit
or both or you identify 64-bit slots, assume 32bit 33 MHz.

 

[Benjamin Gawert]

I'm doing some tests -- and getting a few surprises -- with
software RAID on the following setup:

Asus A7N8X Motherboard (Presumably Ultra ATA 133; PCI 2.2)
with an Athlon 2800 and 1GB of RAM. (with the latest BIOS
upgrade -- rev. 1010-B)

I have a PCI IDE controller card (the ones that come with
the Maxtor ATA133 drives -- it's really a Promise card, re-
branded to Maxtor).

Well, the low-end Promise cards aren't really fast...

Three Hard drives Western Digital 200GB ATA100.

Windows 2000, with SP4 installed.

When I measure raw performance on each of the drives (using
the DskSpeed utility), I get an impressive 57MB/sec. However,
when I configure them on RAID-0 (software RAID, or stripped
volume, as Windows 2000 calls it), it only goes up to 78MB/sec.

Even more surprising is that when putting a tripple RAID-0
volume (a volume stripped across three drives), I only get
81MB/sec.

I was expecting that the overhead given by RAID-0 would be
basically negligible, and thus a software RAID solution
would duplicate or triplicate the speed.

No way! RAID0 increases performance, but not that much that it
doubles/triples data rates with two/three drives..

Is this normal? What bottleneck am I experiencing? (I
think the bandwidth for the PCI bus -- PCI 2.2, according
to Asus' manual -- is 266MB/sec, correct? So, there should
be no bottleneck once the data reaches the controllers)

Wrong. Your A7N8X only has a single 32bit 33MHz PCI bus which do a
theoretical(!) max of 133MB/s. The real life performance usually is probably
around 100MB/s. Then add the fact that the PCI controller has to share this
bandwidth with all other PCI devices You have, and You're where You are...

Can you see any obvious red flag? Or are the numbers I'm
getting what I should be expecting?

You get around the best You can expect. But don't worry, I don't get that
much more with a Promise S150SX4 SATA RAID Controller and really fast
Seagate 250GB SATA drives on a 266MB/s separate PCI bus...

Benjamin

 

[Carlos Moreno]

Is this normal? What bottleneck am I experiencing? (I
think the bandwidth for the PCI bus -- PCI 2.2, according
to Asus' manual -- is 266MB/sec, correct?

I thought it was 133 MB/s.

32 bits x 33 MHz= 1,056Gbit/s

???

AFAIR, PCI 2.2 supports 66MHz and 64-bit. [...]

While PCI 2.2 may _allow_ for 66 MHz and 64-bit that doesn't mean that
either is _required_. Unless the docs specifically state 66 MHz or 64 bit
or both or you identify 64-bit slots, assume 32bit 33 MHz.

You are absolutely correct. A little additional digging
through the Asus manual revealed that the expansion PCI
bus is 32-bits at 33MHz.

Ironically, they do say that the on-board PCI peripherals
do benefit of an 800MB/sec bandwidth (that surprised me;
is this also covered by PCI 2.2? 100MHz / 64-bit?), and
they specifically say that both IDE channels use parallel
data paths to maximize throughput, etc etc.

I say ironically, because even with the latest BIOS upgrade,
the drives connected through those two channels are only
recognized as 137GB drives :-(

Thanks,

Carlos
--

 

[Carlos Moreno]

Hi Benjamin,

Thanks for all the info and tips!

One comment, though:

You get around the best You can expect. But don't worry, I don't get that
much more with a Promise S150SX4 SATA RAID Controller and really fast
Seagate 250GB SATA drives on a 266MB/s separate PCI bus...

This quite surprises me. I know that "in theory, there is
no difference between theory and practice; in practice, there
is" and all... But normally practical results can be
reconciled with a theoretical analysis. I can't see how
this applies in here.

Assuming no transfer limitations other than the hard drives
themselves, why wouldn't RAID-0 with four drives get very
close to multiplying times four the speed? What happens
in the electronics realm (buses, memory, CPU, etc.) is still
a lot faster than the transfer rates we're talking about;
and the software processing to simply split blocks of data
into 2, or 3, or 4, should be sufficiently low... (hmmm,
now that I think about it, perhaps when we talk about fast
drives, and trying to quadruple the speed, then what I just
said is no longer true? I mean, perhaps then the overhead
of the processing -- even if done on a hardware-RAID card --
is significant when we're trying to get 200+ MB/sec effective
rate?)

Thanks,

Carlos
--

 

[Egil Solberg]

Carlos Moreno wrote:

Ironically, they do say that the on-board PCI peripherals
do benefit of an 800MB/sec bandwidth (that surprised me;
is this also covered by PCI 2.2? 100MHz / 64-bit?), and
they specifically say that both IDE channels use parallel
data paths to maximize throughput, etc etc.

Some time ago I gave that some thoughts as well. Onboard devices enumerate
as PCI devices, and so I thought they had the same bandwidth limitations as
regular PCIdevices that go inte the PCIslots.
Using diagnostic software like SisoftSandra, I get the information that my
computer (A7N8X-Deluxe) has 2 PCIbusses (0and 1), one for onboard devices
and one for the PCIslots and other controllers mounted to the PCB (Silicon
image SATA controller).
I have come to the conclusion after reading around that onboard devices are
not limited by old PCIspecs. The interconnect between northbridge and
southbridge is Hypertransport (800MB/s) and is now the current limitation,
hence your confusion.

I say ironically, because even with the latest BIOS upgrade,
the drives connected through those two channels are only
recognized as 137GB drives :-(

It does not help upgrading system BIOS, when you use a PCI IDEcontroller.
You will need to flash the BIOS on the PCIcard.

 

[Carlos Moreno]

It does not help upgrading system BIOS, when you use a PCI IDEcontroller.
You will need to flash the BIOS on the PCIcard.

No, wait!

The drives that I connect through the PCI Promise controller
card are recognized as 200GB; but then, they're held hostage
of the 33MHz PCI bus offering 133MB/sec bandwidth.

The drives that I connect through the on-board IDE controller
(which is the one going through the special/ultrafast PCI bus)
are seen as 137GB; a BIOS upgrade should address that, from
what I've read -- however, in this case it doesn't :-(

Carlos
--

 

[J. Clarke]

Hi Benjamin,

Thanks for all the info and tips!

One comment, though:


This quite surprises me. I know that "in theory, there is
no difference between theory and practice; in practice, there
is" and all... But normally practical results can be
reconciled with a theoretical analysis. I can't see how
this applies in here.

Assuming no transfer limitations other than the hard drives
themselves, why wouldn't RAID-0 with four drives get very
close to multiplying times four the speed? What happens
in the electronics realm (buses, memory, CPU, etc.) is still
a lot faster than the transfer rates we're talking about;
and the software processing to simply split blocks of data
into 2, or 3, or 4, should be sufficiently low... (hmmm,
now that I think about it, perhaps when we talk about fast
drives, and trying to quadruple the speed, then what I just
said is no longer true? I mean, perhaps then the overhead
of the processing -- even if done on a hardware-RAID card --
is significant when we're trying to get 200+ MB/sec effective
rate?)

The disk scheduling problem is typically the subject of a full semester
course in a computer science curriculum, and that just scratches the
surface. It's not as simple as it would seem. One can still get a
doctoral dissertation out of it.

Read up on the actual functioning of RAID-0 and then try to apply that to
some practical situations (multiple small files, single large files,
multiple large files, files being written while others are being read, etc)
and you'll see that the range of circumstances under which it gives a major
performance boost are limited.

For example you talk about splitting blocks of data. How large is a block?
If it's smaller than the stripe size it doesn't get split.

 

[Nick]

No, wait!

The drives that I connect through the PCI Promise controller
card are recognized as 200GB; but then, they're held hostage
of the 33MHz PCI bus offering 133MB/sec bandwidth.

The drives that I connect through the on-board IDE controller
(which is the one going through the special/ultrafast PCI bus)
are seen as 137GB; a BIOS upgrade should address that, from
what I've read -- however, in this case it doesn't :-(

Carlos

That's not a big deal if you use a proper software to access these
drives.

Nick

 

[Peter]

Asus A7N8X Motherboard (Presumably Ultra ATA 133; PCI 2.2)
with an Athlon 2800 and 1GB of RAM. (with the latest BIOS
upgrade -- rev. 1010-B)

Which motherboard PCB revision do you have?
2.0
1.06
1.04

"At this point in time, ASUS A7N8X deluxe and the basic version A7N8X have
several PCB revisions. There are 1.04, 1.06 and 2.0. Please note that
besides the differences between deluxe version and non-deluxe version, which
means users cannot put the BIOS for deluxe version on non-deluxe version
A7N8X. The new BIOS for PCB rev.2.0 is not backward compatible to rev. 1.0X
PCB as well. If users flash the BIOS for PCB rev.2.0 onto PCB rev.1.0X,
users may experience unstable systems or even crashed systems."

 

[Carlos Moreno]

That's not a big deal if you use a proper software to access these
drives.

Are you referring to device-driver-level software? If the OS
doesn't see it, how would this software access it?

In case someone else is reading this and having similar trouble;
I discovered one additional detail (from some Windows newsgroups);
if I connect the 200GB drive through the PCI card (on which it
is properly recognized as 200GB) and I create all the partitions
filling up the 200GB (or I guess at least exceeding the 137GB
threshold), then when I connect it to the on-board IDE controller
it is seen as a 200GB.

Curiously enough, as soon as I delete the partitions, the drive
automatically turns into 137GB (well, reported by Win2K's Disk
Manager as 128.0GB). This still does not allow me to create an
arbitrary RAID partition that includes this drive, because I
would have to create it while connected to the PCI card, and
then as soon as I move it, all the "special" partitions become
invalid :-(

I can't quite visualize where and what exactly the bug is that
would cause this behaviour...

Cheers,

Carlos
--

 

[Peter]

Are you referring to device-driver-level software? If the OS

doesn't see it, how would this software access it?

In case someone else is reading this and having similar trouble;
I discovered one additional detail (from some Windows newsgroups);
if I connect the 200GB drive through the PCI card (on which it
is properly recognized as 200GB) and I create all the partitions
filling up the 200GB (or I guess at least exceeding the 137GB
threshold), then when I connect it to the on-board IDE controller
it is seen as a 200GB.

Curiously enough, as soon as I delete the partitions, the drive
automatically turns into 137GB (well, reported by Win2K's Disk
Manager as 128.0GB). This still does not allow me to create an
arbitrary RAID partition that includes this drive, because I
would have to create it while connected to the PCI card, and
then as soon as I move it, all the "special" partitions become
invalid :-(

I can't quite visualize where and what exactly the bug is that
would cause this behaviour...

Win2K? Maybe that would help you to understand the 137GB issue:
http://support.microsoft.com/default.aspx?scid=kb;en-us;305098

 

[Eric Gisin]

That is a serious bug in Win2K. I don't think Microsoft acknowledges it.

If there are no parts, then Disk Manager reports the size from drive
identification. This is limited to 137GB if the BIOS reports that.

If partitions extend beyond the above size, Windows uses it and corrupts the
disk, often overwritting the beginning (MBR, boot, FAT).

 

[Carlos Moreno]

Which motherboard PCB revision do you have?
2.0
1.06
1.04

2.0 (1010-B is listed as the latest for PCB rev. 2.0)

Carlos
--

 

[Carlos Moreno]

Win2K?

Way ahead of you!

Just to clarify semantics... Win2K can not be the bug...
We all know Windows is more like a NEST and the warmest of
all homes for any amount and breed of bugs... Right?

Maybe that would help you to understand the 137GB issue:
http://support.microsoft.com/default.aspx?scid=kb;en-us;305098

I did understand the 137GB issue (I think ) -- 2^28 possible
sectors, half a K each, gives an eigth of a terabyte... (you
need both the BIOS and the Operating System to properly handle
the issue by sticking to the newer 48-bit sector addressing
length)

But the thing is that I do have the latest SP for Win2K (SP4);
and it does see 200GB drives, as long as I connect the drives
through the PCI card. That led me to the conclusion that it
would be the MB's BIOS the one that wasn't able to handle the
137GB issue. But then, as I mentioned, if the drive has been
partitioned while connected through the PCI card, then when
later connected through the MB's IDE connector, it is seen as
a full 200GB drive...

That last part is what I find strange. I guess the bug in
Win2K (including SP4) may be that for the different pieces of
software related to this issue, some of them have the bug
fixed and some of them don't? (so, if the drive has to be
partitioned, perhaps that portion of the OS functionality
still has the 28-bit limitation, while the parts that read
the partition table and use it don't have the limitation?)

Incidentally, my registry settings did not have the
EnableBigLba value (which they mention in the link you
kindly posted). I just added it; perhaps it'll fix the
whole issue (I guess I'll know after rebooting)

Thanks,

Carlos
--

 

[Carlos Moreno]

That is a serious bug in Win2K. I don't think Microsoft acknowledges it.

If there are no parts, then Disk Manager reports the size from drive
identification. This is limited to 137GB if the BIOS reports that.

If partitions extend beyond the above size, Windows uses it and corrupts the
disk, often overwritting the beginning (MBR, boot, FAT).

Scary enough... It's perhaps worse than what you describe
in the second paragraph -- it makes sense that if the BIOS
reports 137GB the OS would report it that way. But in my
case, the BIOS is reporting it correctly -- I have the
latest BIOS, which according to whatI read, does handle
large drives.

So, the Win2K bug is particularly curious -- with SP4 and
with a BIOS reporting the correct drive size and geometry,
if the drive has no partitions, Win2K reports it as 128GB
(corresponding to 137 billion bytes). If there are partitions,
then Win2K reports it as 186GB...

BTW, all this got fixed as soon as I entered the Registry
parameter, EnableBigLba, following Microsoft's KB report;
after rebooting my machine, the drive, with no partitions
whatsoever, got correctly reported as 186GB (thanks again
Peter, for the pointer!)

Cheers,

Carlos
--

 

[Peter]

Scary enough... It's perhaps worse than what you describe
in the second paragraph -- it makes sense that if the BIOS
reports 137GB the OS would report it that way. But in my
case, the BIOS is reporting it correctly -- I have the
latest BIOS, which according to whatI read, does handle
large drives.

So, the Win2K bug is particularly curious -- with SP4 and
with a BIOS reporting the correct drive size and geometry,
if the drive has no partitions, Win2K reports it as 128GB
(corresponding to 137 billion bytes). If there are partitions,
then Win2K reports it as 186GB...

BTW, all this got fixed as soon as I entered the Registry
parameter, EnableBigLba, following Microsoft's KB report;
after rebooting my machine, the drive, with no partitions
whatsoever, got correctly reported as 186GB (thanks again
Peter, for the pointer!)

So, why do you use Win2K?
I believe XPSP1/2003 does not have that (particular) problem.