At some point in time, some of the folks here doubted that DDR2 would
present a capacity problem. I would like to instruct those individuals
to read the following PDF, in particular, page 37.

http://www-5.ibm.com/ch/events/facht...erformance.pdf

Just so you don't miss the highlights:

DDR2-400/533 supports 8 DIMMs across two channels
DDR2-667 supports 4 DIMMs across two channels
DDR2-800 supports 2 DIMMs across two channels

Perhaps now it should become abundantly clear why FBD is in fact
necessary for high bandwidth. There's an illustration of this on page
42 as well.

DK

On 6 Jul 2006 19:07:16 -0700, "David Kanter" <(E-Mail Removed)> wrote:

>At some point in time, some of the folks here doubted that DDR2 would
>present a capacity problem.

What "folks" would that be?

> I would like to instruct those individuals
>to read the following PDF, in particular, page 37.
>
>http://www-5.ibm.com/ch/events/facht...erformance.pdf

Doesn't work and appears to require registation... in German.:-(

>Just so you don't miss the highlights:
>
>DDR2-400/533 supports 8 DIMMs across two channels
>DDR2-667 supports 4 DIMMs across two channels
>DDR2-800 supports 2 DIMMs across two channels
>
>Perhaps now it should become abundantly clear why FBD is in fact
>necessary for high bandwidth. There's an illustration of this on page
>42 as well.

--
Rgds, George Macdonald

George Macdonald wrote:
> On 6 Jul 2006 19:07:16 -0700, "David Kanter" <(E-Mail Removed)> wrote:
>
> >At some point in time, some of the folks here doubted that DDR2 would
> >present a capacity problem.
>
> What "folks" would that be?

I wonder...

> > I would like to instruct those individuals
> >to read the following PDF, in particular, page 37.
> >
> >http://www-5.ibm.com/ch/events/facht...erformance.pdf
>
> Doesn't work and appears to require registation... in German.:-(

Doh, sorry about that. Perhaps the presentation was IBM internal. I
managed to DL a copy of it yesterday, but it could be that they just
changed it to require registration...

If you want a copy, send me an email, and I'll shoot it over. It has
some interesting comparisons of K8 to Dempsey, and boy, Dempsey doesn't
look attractive. Woodcrest looks pretty nice though.

[snip]

DK

On 6 Jul 2006 19:07:16 -0700, "David Kanter" <(E-Mail Removed)>
wrote:

>At some point in time, some of the folks here doubted that DDR2 would
>present a capacity problem. I would like to instruct those individuals
>to read the following PDF, in particular, page 37.
>
>http://www-5.ibm.com/ch/events/facht...erformance.pdf
>
>Just so you don't miss the highlights:
>
>DDR2-400/533 supports 8 DIMMs across two channels
>DDR2-667 supports 4 DIMMs across two channels
>DDR2-800 supports 2 DIMMs across two channels
>
>Perhaps now it should become abundantly clear why FBD is in fact
>necessary for high bandwidth. There's an illustration of this on page
>42 as well.
>
>DK

I'm no memory expert so take this with a shovel of salt, but as I
understand it.

Current FB Dimm is DDR2 memory with a different module interface. The
interface is point to point fast serial allowing more modules to be
attached for a comparable pin count. The interface does not have a
shared bus between modules. It is the extra bus loading which
mandates the use of slower DDR2 with current bus architectures when
more modules are added.

The capacity issue is the number of pins on the controller required
for a memory channel.

The extra bandwidth is due in part to being able to use higher speed
modules when larger numbers are used, and being able to use higher
numbers of modules.

This isn't DDR2 Vs FB Dimm - this is serial Vs parallel, and
everythings going serial 'cos in general it tends to be faster.

Ryan

Ryan Godridge wrote:
> On 6 Jul 2006 19:07:16 -0700, "David Kanter" <(E-Mail Removed)>
> wrote:
>
> >At some point in time, some of the folks here doubted that DDR2 would
> >present a capacity problem. I would like to instruct those individuals
> >to read the following PDF, in particular, page 37.
> >
> >http://www-5.ibm.com/ch/events/facht...erformance.pdf
> >
> >Just so you don't miss the highlights:
> >
> >DDR2-400/533 supports 8 DIMMs across two channels
> >DDR2-667 supports 4 DIMMs across two channels
> >DDR2-800 supports 2 DIMMs across two channels
> >
> >Perhaps now it should become abundantly clear why FBD is in fact
> >necessary for high bandwidth. There's an illustration of this on page
> >42 as well.
> >
> >DK
>
> I'm no memory expert so take this with a shovel of salt, but as I
> understand it.

Did you get a copy of the PDF? George mentioned he couldn't access
it...let me know and I'll email it.

> Current FB Dimm is DDR2 memory with a different module interface. The
> interface is point to point fast serial allowing more modules to be
> attached for a comparable pin count.

So the issue isn't modules/pin, it's bandwidth/pin. Each pin costs
quite a bit of time/effort/money, so the lower the pin count, the
better. FBD is 70 pins/channel, DDR2 is 240 pins/channel.

In turn, fewer pins means more channels. More DIMMs/channel is a
result of going serial, as you noted below.

> The interface does not have a
> shared bus between modules.

Right, the address and command lanes are bypass routed to the next DIMM
in the channel. That's why the unloaded latency is higher, because you
have several hops to get to a DIMM.

> It is the extra bus loading which
> mandates the use of slower DDR2 with current bus architectures when
> more modules are added.

That's right. In general for DDR, capacity decreases as bandwidth
increases.

> The capacity issue is the number of pins on the controller required
> for a memory channel.

Sort of. The capacity issue is two fold:

1. # of channels, which is what you identified above - FBD can have
more channels with a given number of pins --> higher total bandwidth

2. Number of DIMMs/channel. In FBD, you can have up to 8
DIMMs/channel for any speed grade, whereas in DDR2 there is a hard
limit that decreases as the bandwidth goes up.

To be fair, the latency does get a bit worse with each DIMM you add,
but performance will actually increase for the 2nd DIMM and possibly
the 3rd, since the memory controller can play more interleaving
tricks...however, losing a bit of latency is not a huge deal compared
to letting your data set spill to disk (or having to buy the super high
capacity 4 and 8GB DIMMs, which cost an arm, leg and your first born
child).

> The extra bandwidth is due in part to being able to use higher speed
> modules when larger numbers are used, and being able to use higher
> numbers of modules.

Yes and no. Using more modules does not increase the theoretical peak
bandwidth, but will increase the average bandwidth. The real gain in
bandwidth is because Intel uses 4 channels of FBD where AMD uses 2
channels of DDR2. Now, in a 2S system, AMD will have a total of 4
channels of DDR2; so things will be somewhat more equal. However, the
capacity is definitely going to be a problem. For DDR2-800, a 2S
system would have 4 DIMMs, which means you cannot get nearly enough
memory.

I think the bottom line is that it means instead of trading capacity
for bandwidth, you trade capacity for latency (and some extra heat).

> This isn't DDR2 Vs FB Dimm - this is serial Vs parallel, and
> everythings going serial 'cos in general it tends to be faster.

Yup, that's right. It really is a serial versus || kind of thing. The
last thing to mention, is that since the routing is easier for FBD, the
boards should be a bit cheaper and easier to make. I couldn't really
quantify that though...

DK

On 7 Jul 2006 17:52:11 -0700, "David Kanter" <(E-Mail Removed)>
wrote:

>Ryan Godridge wrote:
>> On 6 Jul 2006 19:07:16 -0700, "David Kanter" <(E-Mail Removed)>
>> wrote:
>>
>> >At some point in time, some of the folks here doubted that DDR2 would
>> >present a capacity problem. I would like to instruct those individuals
>> >to read the following PDF, in particular, page 37.
>> >
>> >http://www-5.ibm.com/ch/events/facht...erformance.pdf
>> >
>> >Just so you don't miss the highlights:
>> >
>> >DDR2-400/533 supports 8 DIMMs across two channels
>> >DDR2-667 supports 4 DIMMs across two channels
>> >DDR2-800 supports 2 DIMMs across two channels
>> >
>> >Perhaps now it should become abundantly clear why FBD is in fact
>> >necessary for high bandwidth. There's an illustration of this on page
>> >42 as well.
>> >
>> >DK
>>
>> I'm no memory expert so take this with a shovel of salt, but as I
>> understand it.
>
>Did you get a copy of the PDF? George mentioned he couldn't access
>it...let me know and I'll email it.
>
I managed to get it on the third attempt - dunno what's going on.

Nice FB Dimm for dummies type video linked from here -

http://www.futureplus.com/products/f..._overview.html

>> Current FB Dimm is DDR2 memory with a different module interface. The
>> interface is point to point fast serial allowing more modules to be
>> attached for a comparable pin count.
>
>So the issue isn't modules/pin, it's bandwidth/pin. Each pin costs
>quite a bit of time/effort/money, so the lower the pin count, the
>better. FBD is 70 pins/channel, DDR2 is 240 pins/channel.
>
>In turn, fewer pins means more channels. More DIMMs/channel is a
>result of going serial, as you noted below.
>
>> The interface does not have a
>> shared bus between modules.
>
>Right, the address and command lanes are bypass routed to the next DIMM
>in the channel. That's why the unloaded latency is higher, because you
>have several hops to get to a DIMM.
>

"Unloaded latency" ?

>> It is the extra bus loading which
>> mandates the use of slower DDR2 with current bus architectures when
>> more modules are added.
>
>That's right. In general for DDR, capacity decreases as bandwidth
>increases.
>
>> The capacity issue is the number of pins on the controller required
>> for a memory channel.
>
>Sort of. The capacity issue is two fold:
>
>1. # of channels, which is what you identified above - FBD can have
>more channels with a given number of pins --> higher total bandwidth
>
>2. Number of DIMMs/channel. In FBD, you can have up to 8
>DIMMs/channel for any speed grade, whereas in DDR2 there is a hard
>limit that decreases as the bandwidth goes up.
>
Is that a question of the current controller implementations or
intrinsically in the i/o standard?

>To be fair, the latency does get a bit worse with each DIMM you add,
>but performance will actually increase for the 2nd DIMM and possibly
>the 3rd, since the memory controller can play more interleaving
>tricks...however, losing a bit of latency is not a huge deal compared
>to letting your data set spill to disk (or having to buy the super high
>capacity 4 and 8GB DIMMs, which cost an arm, leg and your first born
>child).
>

It's swings and roundabouts again, but I think Intel are right with
this one. Parallel interfaces have got nowhere to go.

>> The extra bandwidth is due in part to being able to use higher speed
>> modules when larger numbers are used, and being able to use higher
>> numbers of modules.
>
>Yes and no. Using more modules does not increase the theoretical peak
>bandwidth, but will increase the average bandwidth. The real gain in
>bandwidth is because Intel uses 4 channels of FBD where AMD uses 2
>channels of DDR2. Now, in a 2S system, AMD will have a total of 4
>channels of DDR2; so things will be somewhat more equal. However, the
>capacity is definitely going to be a problem. For DDR2-800, a 2S
>system would have 4 DIMMs, which means you cannot get nearly enough
>memory.
>
I'm confused when you say more modules does not increase theoretical
bandwidth - is that a width / depth issue. I.e. if the interface was
wide enough adding extra modules would add bandwidth.

>I think the bottom line is that it means instead of trading capacity
>for bandwidth, you trade capacity for latency (and some extra heat).
>
>> This isn't DDR2 Vs FB Dimm - this is serial Vs parallel, and
>> everythings going serial 'cos in general it tends to be faster.
>
>Yup, that's right. It really is a serial versus || kind of thing. The
>last thing to mention, is that since the routing is easier for FBD, the
>boards should be a bit cheaper and easier to make. I couldn't really
>quantify that though...
>
>DK

I'd like to think you're right there, but I bet we won't see the
difference

[snip]
> I managed to get it on the third attempt - dunno what's going on.
>
> Nice FB Dimm for dummies type video linked from here -
>
> http://www.futureplus.com/products/f..._overview.html
>
> >> Current FB Dimm is DDR2 memory with a different module interface. The
> >> interface is point to point fast serial allowing more modules to be
> >> attached for a comparable pin count.
> >
> >So the issue isn't modules/pin, it's bandwidth/pin. Each pin costs
> >quite a bit of time/effort/money, so the lower the pin count, the
> >better. FBD is 70 pins/channel, DDR2 is 240 pins/channel.
> >
> >In turn, fewer pins means more channels. More DIMMs/channel is a
> >result of going serial, as you noted below.
> >
> >> The interface does not have a
> >> shared bus between modules.
> >
> >Right, the address and command lanes are bypass routed to the next DIMM
> >in the channel. That's why the unloaded latency is higher, because you
> >have several hops to get to a DIMM.
> >
>
> "Unloaded latency" ?

So unloaded latency would be the time it takes to access data in memory
when the system is otherwise idle. So you remove any queuing delay.
When someone says that an MPU's memory latency is X nanoseconds,
usually they mean unloaded. Loaded latencies can be substantially
higher, depending on the load. Just as an example, with DDR2 you have
bus turn around (on top of queuing), which is where a bubble is
inserted between R and W activities. FBD supports simultaneous R and W
transactions.

[snip]

> >Sort of. The capacity issue is two fold:
> >
> >1. # of channels, which is what you identified above - FBD can have
> >more channels with a given number of pins --> higher total bandwidth
> >
> >2. Number of DIMMs/channel. In FBD, you can have up to 8
> >DIMMs/channel for any speed grade, whereas in DDR2 there is a hard
> >limit that decreases as the bandwidth goes up.
> >
> Is that a question of the current controller implementations or
> intrinsically in the i/o standard?

I'm not sure which you are referring to, so I'll answer for both. 8
DIMMs/channel is a hard limit for FBD. DDR2 the limit is in the I/O
standard. Fundamentally the issue is the number of DRAM devices that
each channel can support, which has dropped off rather substantially.
Going from SDR-->DDR halved that, going to DDR2-400 halved that again,
and DDR2-667 and DDR2-800 will only make it worse.

> >To be fair, the latency does get a bit worse with each DIMM you add,
> >but performance will actually increase for the 2nd DIMM and possibly
> >the 3rd, since the memory controller can play more interleaving
> >tricks...however, losing a bit of latency is not a huge deal compared
> >to letting your data set spill to disk (or having to buy the super high
> >capacity 4 and 8GB DIMMs, which cost an arm, leg and your first born
> >child).
> >
>
> It's swings and roundabouts again, but I think Intel are right with
> this one. Parallel interfaces have got nowhere to go.

Yup I agree. The one thing that AMD is right about is that the heat
can be an issue. The initial implementation of AMB's are not as cool
as everyone would have liked.

> >> The extra bandwidth is due in part to being able to use higher speed
> >> modules when larger numbers are used, and being able to use higher
> >> numbers of modules.
> >
> >Yes and no. Using more modules does not increase the theoretical peak
> >bandwidth, but will increase the average bandwidth. The real gain in
> >bandwidth is because Intel uses 4 channels of FBD where AMD uses 2
> >channels of DDR2. Now, in a 2S system, AMD will have a total of 4
> >channels of DDR2; so things will be somewhat more equal. However, the
> >capacity is definitely going to be a problem. For DDR2-800, a 2S
> >system would have 4 DIMMs, which means you cannot get nearly enough
> >memory.
> >
> I'm confused when you say more modules does not increase theoretical
> bandwidth - is that a width / depth issue. I.e. if the interface was
> wide enough adding extra modules would add bandwidth.

Yes. Going wider (i.e. more channels) adds bandwidth. Going deeper
(more DIMMs/channel) increases effective bandwidth.

Time for ASCII diagrams

------|--|--|
CPU-|
------|--|--|

Will have better effective bandwidth than:

------|
CPU-|
------|

Both will have better peak bandwidth (and better real bandwidth) than:

------|--|--|
CPU-|

> >I think the bottom line is that it means instead of trading capacity
> >for bandwidth, you trade capacity for latency (and some extra heat).
> >
> >> This isn't DDR2 Vs FB Dimm - this is serial Vs parallel, and
> >> everythings going serial 'cos in general it tends to be faster.
> >
> >Yup, that's right. It really is a serial versus || kind of thing. The
> >last thing to mention, is that since the routing is easier for FBD, the
> >boards should be a bit cheaper and easier to make. I couldn't really
> >quantify that though...
> >
> >DK
>
> I'd like to think you're right there, but I bet we won't see the
> difference

Probably not. A lot of times 'cheaper' really seems to mean "we won't
jack up the price and make you bleed", rather than "it will be less
expensive".

DK

On Fri, 07 Jul 2006 15:33:08 -0400, George Macdonald
<fammacd=!SPAM^(E-Mail Removed)> wrote:

>On 6 Jul 2006 19:07:16 -0700, "David Kanter" <(E-Mail Removed)> wrote:
>>http://www-5.ibm.com/ch/events/facht...erformance.pdf
>
>Doesn't work and appears to require registation... in German.:-(

IBM doesn't like you, George It downloads ok without registration
for me.

--
A Lost Angel, fallen from heaven
Lost in dreams, Lost in aspirations,
Lost to the world, Lost to myself

On 7 Jul 2006 17:52:11 -0700, "David Kanter" <(E-Mail Removed)> wrote:

>Ryan Godridge wrote:
>> On 6 Jul 2006 19:07:16 -0700, "David Kanter" <(E-Mail Removed)>
>> wrote:
>>
>> >At some point in time, some of the folks here doubted that DDR2 would
>> >present a capacity problem. I would like to instruct those individuals
>> >to read the following PDF, in particular, page 37.
>> >
>> >http://www-5.ibm.com/ch/events/facht...erformance.pdf
>> >
>> >Just so you don't miss the highlights:
>> >
>> >DDR2-400/533 supports 8 DIMMs across two channels
>> >DDR2-667 supports 4 DIMMs across two channels
>> >DDR2-800 supports 2 DIMMs across two channels
>> >
>> >Perhaps now it should become abundantly clear why FBD is in fact
>> >necessary for high bandwidth. There's an illustration of this on page
>> >42 as well.
>> >
>> >DK
>>
>> I'm no memory expert so take this with a shovel of salt, but as I
>> understand it.
>
>Did you get a copy of the PDF? George mentioned he couldn't access
>it...let me know and I'll email it.
>
>> Current FB Dimm is DDR2 memory with a different module interface. The
>> interface is point to point fast serial allowing more modules to be
>> attached for a comparable pin count.
>
>So the issue isn't modules/pin, it's bandwidth/pin. Each pin costs
>quite a bit of time/effort/money, so the lower the pin count, the
>better. FBD is 70 pins/channel, DDR2 is 240 pins/channel.
[snipped]

If you are counting signal pins, it's more like 150 pins per channel for
DDR2....

/daytripper

> >So the issue isn't modules/pin, it's bandwidth/pin. Each pin costs
> >quite a bit of time/effort/money, so the lower the pin count, the
> >better. FBD is 70 pins/channel, DDR2 is 240 pins/channel.
> [snipped]
>
> If you are counting signal pins, it's more like 150 pins per channel for
> DDR2....

No, I'm counting command, data, power, ground, everything. After all,
8 channels of DDR2 without any power or ground makes for an extremely
low performance system...

DK