PCI-Express over Cat6

[Patrick Geoffray]

I dunno. I know even less about expansion bus protocols than I do about
most other things. Is there anything you can do with any available
out-of-the box interconnect that you can't do with lower latency using
PCI-Express? Limited bandwidth and distance, to be sure, but how could
you beat the latency?

Most of the end-to-end latency these days comes from the PCI and the
link (SerDes + distance). So PCI-Express-only would not save much. As
Del noted, Intel is working on a switching extensions to PCI-Express,
but the PCI-Express protocol is not really designed for that: flow
control is very tight, as you would expect on a very short
point-to-point connection. If PCI-Express wants to go outside the box,
it will have to deal with some tough problems with flow control.

Patrick

 

[daytripper]

Most of the end-to-end latency these days comes from the PCI and the
link (SerDes + distance). So PCI-Express-only would not save much. As
Del noted, Intel is working on a switching extensions to PCI-Express,
but the PCI-Express protocol is not really designed for that: flow
control is very tight, as you would expect on a very short
point-to-point connection. If PCI-Express wants to go outside the box,
it will have to deal with some tough problems with flow control.

Respectfully, I disagree with that last sentence. Unless the mission is
redefined, PCI Express can certainly go outside the crate.

/daytripper (the question remains why one would do that...)

 

[Sander Vesik]

IB and PCI-Express should be pretty comparable. (PCI express isn't out of
the box yet)
Ethernet with RDMA and hardware offload is in the same ballpark.
Rapid I/O, Fibre Channel, are contenders depending on task.

Is latency a big deal writing to a disk or graphics card?

It can easily be for a graphics card.

 

[Andrew Reilly]

It can easily be for a graphics card.

Why? Aren't they write-only devices? Surely any latency
limitation is the 10ms or so of human perception, and anything in the
circuitry is neither here nor there at that scale.

 

[Rob Warnock]

+---------------
| Most of the end-to-end latency these days comes from the PCI and the
| link (SerDes + distance). So PCI-Express-only would not save much. As
| Del noted, Intel is working on a switching extensions to PCI-Express,
| but the PCI-Express protocol is not really designed for that: flow
| control is very tight, as you would expect on a very short
| point-to-point connection. If PCI-Express wants to go outside the box,
| it will have to deal with some tough problems with flow control.
+---------------

Indeed. Quite a bit of the difference between "GSN" (a.k.a. the ANSI
HIPPI-6400 standard) and the SGI "XIO" (switched-fabric I/O to multiple
PCI busses) it was based on was the need to increase the low-level
retransmisson buffers and sequence space to allow a potential 1km
distance at full bandwidth[1], compared to the ~10m permitted by XIO.
This added considerably to the die area of the PHY/PMD part.

Also note that at "only" 10m range, XIO devices *already* needed
rather large retransmisson buffers and sequence space...


-Rob

[1] Though note that un-repeatered GSN can still only go 30m in copper.
This is for electrical reasons, not flow-control window size.

 

[Rupert Pigott]

Why? Aren't they write-only devices? Surely any latency

Off the top of my head, at least two requirements exist, namely
screenshots and flyback sychronisation...

Cheers,
Rupert

 

[Andrew Reilly]

Off the top of my head, at least two requirements exist, namely
screenshots and flyback sychronisation...

Both of which appear, on the surface, to be frame-rate type events: i.e.,
in the ballpark of the 10ms event time that I mentioned in the part that
you snipped. Not a latency issue on the order of memory access or
processor cycle times...

[Don't graphics cards generate interrupts for flyback synchronization?]

 

[Rupert Pigott]

Both of which appear, on the surface, to be frame-rate type events: i.e.,
in the ballpark of the 10ms event time that I mentioned in the part that
you snipped. Not a latency issue on the order of memory access or

Hmmm, how about querying the state of an OpenGL rendering pipeline
that happens to be sitting on the graphics card ? I don't think that
it's ever been true to say GFX cards are write only, and I'm not sure
I'd ever want that.

Cheers,
Rupert

 

[KR Williams]

Hmmm, how about querying the state of an OpenGL rendering pipeline
that happens to be sitting on the graphics card ? I don't think that
it's ever been true to say GFX cards are write only, and I'm not sure
I'd ever want that.

Why wouldn't things be rendered in memory and then DMA'd to the
graphics card? Why would the processor *ever* care what's been
sent to the graphics subsystem? I'm from (close enough to)
Missouri, and you're going to have to show us, Rupert.

 

[Robin KAY]

Why wouldn't things be rendered in memory and then DMA'd to the
graphics card? Why would the processor *ever* care what's been sent
to the graphics subsystem?

Why would you have your main processor(s) render a scene when you have a
dedicated graphics processor to do it? In the case of the graphics cores
I've used, reading from the framebuffer is needed to i) make sure the
FIFO has enough spaces for the register writes you're about to issue,
and ii) to synchronize the graphics core and host's usage of video
memory (e.g. so you can reuse a memory buffer once the graphics
operation that was using it has completed). These wouldn't be too
difficult to overcome if interconnect latency become a problem, but as
graphics cards become increasingly flexible there's more useful
information they can provide to the host. Hardware accelerated collision
detection for example.

 

[David Schwartz]

Why wouldn't things be rendered in memory and then DMA'd to the
graphics card?

Because then the rendering process would be eating system memory
bandwidth.

Why would the processor *ever* care what's been
sent to the graphics subsystem?

Because it may have to make decisions based upon that information. I
don't know enough about modern graphics hardware to know if it actually does
this, but it has been at times logical to use the graphics hardware to help
you make decisions about other issues. For example, a game may want to
display some information about an object if and only if that object is
visible to you. That may be the graphics card's decision, since it has to
decide that anyway.

DS

 

[Rupert Pigott]

Why wouldn't things be rendered in memory and then DMA'd to the
graphics card? Why would the processor *ever* care what's been
sent to the graphics subsystem? I'm from (close enough to)
Missouri, and you're going to have to show us, Rupert.

Try starting here :

http://www.opengl.org

Take a look at the spec. There are numerous papers on OpenGL
acceleration hardware too. FWIW I have been quite impressed by
the OpenGL spec, seems to give a lot of freedom to both the
application and the hardware.

For a more generic non-OpenGL biased look at 3D hardware you
might want to check out the following :

"Computer Graphics Principles and Practice",
2nd Edition by Foley/van Dam/Feiner/Hughes,
published by Addison Wesley.

Specifically chapter 18 "Advanced Raster Graphics Architecture"
for a discussion on various (rather nifty) 3D graphics hardware
and chapter 16 "Illumination and Shading" for a heavy hint as to
why it's necessary.

I can also recommend Jim Blinn's articles in IEEE CG&A, last
time I read them was 1995. The articles I read by Blinn were
focussed on software rendering using approximations that were
"good enough" but still allowed him to get his rendering done
before hell froze over. IIRC Blinn had access to machinary that
would *still* eat a modern PC for breakfast in terms of FP and
memory throughput in those apps.

However I think times are a-changing. Life might well get a lot
more interesting when CPU designers start looking for new things
to do because they can't get any decent speed ups on single
thread execution speed.

Cheers,
Rupert

 

[Robert Redelmeier]

Why would you have your main processor(s) render a scene
when you have a dedicated graphics processor to do it?

I think you're talking 3-D while Keith is talking 2-D.

In 3-D there's simply too much drudge work (shading,
perspective) and not enough interaction back to the control
program to need or want the CPU. 2-D is much simpler and often
requires considerable CPU interactivity (CAD) with the display.

-- Robert

 

[KR Williams]

I think you're talking 3-D while Keith is talking 2-D.

Nope. 3-D is no different. AGP wuz supposed to make the
graphics channel two-way so the graphics card could access main
memory. DO you know anyone that actually does this? PLease!
With 32MB (or 128MB) on the graphics card, who cares?

In 3-D there's simply too much drudge work (shading,
perspective) and not enough interaction back to the control
program to need or want the CPU. 2-D is much simpler and often
requires considerable CPU interactivity (CAD) with the display.

Sure, so why does the 3-D card want to go back to main memory,
again? The graphics pipe is amazingly one-directional. ...and
thus not sensitive to latency, any more than in human terms.

 

[KR Williams]

Because then the rendering process would be eating system memory
bandwidth.

Nope. YOu're thinking so AGP (no one uses it, or ever has).

Because it may have to make decisions based upon that information. I
don't know enough about modern graphics hardware to know if it actually does
this, but it has been at times logical to use the graphics hardware to help
you make decisions about other issues. For example, a game may want to
display some information about an object if and only if that object is
visible to you. That may be the graphics card's decision, since it has to
decide that anyway.

Nope. Any feedback is certainly within human response. Low
latency (in CPU terms) is irrelevant to graphics subsystems.

 

[KR Williams]

Try starting here :

http://www.opengl.org

Take a look at the spec. There are numerous papers on OpenGL
acceleration hardware too. FWIW I have been quite impressed by
the OpenGL spec, seems to give a lot of freedom to both the
application and the hardware.

For a more generic non-OpenGL biased look at 3D hardware you
might want to check out the following :

"Computer Graphics Principles and Practice",
2nd Edition by Foley/van Dam/Feiner/Hughes,
published by Addison Wesley.

Specifically chapter 18 "Advanced Raster Graphics Architecture"
for a discussion on various (rather nifty) 3D graphics hardware
and chapter 16 "Illumination and Shading" for a heavy hint as to
why it's necessary.

WHy don't you tell us why it's necessary, rather than spewing
some irrelevant web sites. THe fact is that the graphics channel
is amazingly unidirectional. THe processor sends the commands to
the graphics card and it does it's thing in its own memory. AGP
was a wunnerful idea, ten years before it was available.

I can also recommend Jim Blinn's articles in IEEE CG&A, last
time I read them was 1995. The articles I read by Blinn were
focussed on software rendering using approximations that were
"good enough" but still allowed him to get his rendering done
before hell froze over. IIRC Blinn had access to machinary that
would *still* eat a modern PC for breakfast in terms of FP and
memory throughput in those apps.

Oh, my. 1995, there's a recent article. I don't remember. Did
graphics cars have 128MB of their own then? Did systems even
have 128MB. Come on! Get with the times.

However I think times are a-changing. Life might well get a lot
more interesting when CPU designers start looking for new things
to do because they can't get any decent speed ups on single
thread execution speed.

"They" are. ;-) Though you're still wrong about the graphics
pipe. It really isn't latency sensitive, any more than humans
are.

 

[David Schwartz]

Nope. YOu're thinking so AGP (no one uses it, or ever has).

Okay, then you tell me why things aren't rendered in memory and then
DMA'd to the graphics card.

DS

 

[Robert Redelmeier]

Nope. 3-D is no different. AGP wuz supposed to make the
graphics channel two-way so the graphics card could access main
memory. DO you know anyone that actually does this? PLease!
With 32MB (or 128MB) on the graphics card, who cares?

I'm not at all sure what point you're trying to make here.
Forgive me if I flounder around a bit. The graphics card
_does_ access main memory. AFAIK, for both 2D & 3D after
rendering in system RAM the CPU programs the GPU to do BM
DMA to load the framebuffer vram.

No-one in their right mind tries to get the CPU to read
the framebuffer. It is dead slow because vram is very busy
being read to satisfy the refresh rate. It is hard enough for
the GPU to access synchonously and this is what the multiple
planes and the MBs of vram are used for.

Sure, so why does the 3-D card want to go back to main memory,
again? The graphics pipe is amazingly one-directional. ...and
thus not sensitive to latency, any more than in human terms.

My understanding is that in 3-D the advanced functions in
the GPU (perspective & shading) can handle quite a number of
intermediate frames before requiring a reload from system ram.
But it does require a reload. How's the graphics card gonna
know what's behind Door Number Three?

-- Robert

 

[Rupert Pigott]

Sure, so why does the 3-D card want to go back to main memory,
again? The graphics pipe is amazingly one-directional. ...and

I think you might be getting caught out by the wheel of
reincarnation. You can put lots of stuff into hardware or software,
the fashion has changed over the years, but usually the high-end
stuff has gone hardware. These days even the low-end stuff is
going hardware - silicon is cheap.

Now for a why you might want a two way pipe to a GFX card.

In interactive applications you'll want to do stuff like collision
detection which can make use of a lot of data that the 3D pipe
munges. It makes sense to use that data rather than replicate the
work done by that very fast silicon dedicated to the job, right ?

In comp.arch the question of GFX hardware doing double precision
floating point keeps popping up. There are people who want that
hardware to be doing stuff other than blatting pixels around.

My advice is to read the first few chapters of the OpenGL spec,
then try and dig up some papers about 3D hardware. The opengl.org
site has a lot of useful and above all accessible info, it has a
fair bit of example code too.


Cheers,
Rupert

 

[Rupert Pigott]

KR Williams wrote:

[SNIP]

WHy don't you tell us why it's necessary, rather than spewing
some irrelevant web sites. THe fact is that the graphics channel

OpenGL.org is hardly irrelevent with respect to 3D apps and
hardware. :/

is amazingly unidirectional. THe processor sends the commands to
the graphics card and it does it's thing in its own memory. AGP

No, the fact is : It isn't. I've given you some broad reasons
and I've given you some hints on where to start finding some
specifics.

[SNIP]

"They" are. ;-) Though you're still wrong about the graphics
pipe. It really isn't latency sensitive, any more than humans
are.

As long as you consider sites like opengl.org to be irrelevant
you will continue to think that way regardless of what the
reality is.

Cheers,
Rupert