possible divergence in x86 virtualization technologies?

[YKhan]

Intel's Vanderpool and AMD's Pacifica are supposed to bring hardware
support for virtualization to x86 processors. Basically they'll be like
a super Protected Mode which have more privilege than operating
systems. There is speculation that the two implementations will not be
compatible with each other. That might present a fork in x86
instruction sets, although it's not clear if it really matters at this
level since the hypervisor runs so far above the operating system and
applications that they won't even notice it. But it's interesting
architecturally, nonetheless.

Yousuf Khan

 

[Ed]

Intel's Vanderpool and AMD's Pacifica are supposed to bring hardware
support for virtualization to x86 processors. Basically they'll be like
a super Protected Mode which have more privilege than operating
systems. There is speculation that the two implementations will not be
compatible with each other. That might present a fork in x86
instruction sets, although it's not clear if it really matters at this
level since the hypervisor runs so far above the operating system and
applications that they won't even notice it. But it's interesting
architecturally, nonetheless.

Yousuf Khan

Is AMD releasing this tech to desktops or just servers?


http://news.com.com/AMD+jockeys+with+Intel+in+multi-OS+race/2100-1012_3-5600552.html?tag=nefd.lede
"It's hard to imagine that AMD would be so blindingly stupid as to
forward a truly Vanderpool-incompatible virtualization mechanism," a
move that could cause heartburn for many software makers, said
Illuminata analyst Jonathan Eunice. "It's hard to understand why, if
they do intend to do the right thing here, that they're so obstinately
set against just saying so."


Ed

 

[Joe Seigh]

Intel's Vanderpool and AMD's Pacifica are supposed to bring hardware
support for virtualization to x86 processors. Basically they'll be like
a super Protected Mode which have more privilege than operating
systems. There is speculation that the two implementations will not be
compatible with each other. That might present a fork in x86
instruction sets, although it's not clear if it really matters at this
level since the hypervisor runs so far above the operating system and
applications that they won't even notice it. But it's interesting
architecturally, nonetheless.

As long as they're functionally close to each other, it shouldn't be too
much of a problem. The instruction to enter virtual execution probably
is only used in one place.

The bigger problem is the VMCALL instruction to make hypervisor calls.
You need to know what hypervisor you are running under so you know what and how
to make the calls. That means some sort of coordination among the
different vm vendors.

I looked at the Vanderpool preliminary architecture and it allows the hypervisor
to trap the MWAIT instruction but not the PAUSE instruction. Presumably it's
not ok to waste a time slice in MWAIT but is ok to waste it in a PAUSE spin loop
(assuming the guest uses pause in their spin wait loops). That's strange
considering the current MWAIT implementation is pretty suboptimal and probably
no one uses it.

In IBM's operating systems running under VM, the spin loops issue a hypervisor
call to yield the processor, sort of like sched_yield() for Unix. For Intel
guest OSes, they'd have to determine which VM they're running under and use
the corresponding VMCALL, if it exists, to yield the processor.

 

[Eric Smith]

Intel's Vanderpool and AMD's Pacifica are supposed to bring hardware
support for virtualization to x86 processors. Basically they'll be like
a super Protected Mode which have more privilege than operating
systems. There is speculation that the two implementations will not be
compatible with each other.

It's not speculation. AMD has stated it as fact.

Eric

 

[Yousuf Khan]

Is AMD releasing this tech to desktops or just servers?


http://news.com.com/AMD+jockeys+with+Intel+in+multi-OS+race/2100-1012_3-5600552.html?tag=nefd.lede
"It's hard to imagine that AMD would be so blindingly stupid as to
forward a truly Vanderpool-incompatible virtualization mechanism," a
move that could cause heartburn for many software makers, said
Illuminata analyst Jonathan Eunice. "It's hard to understand why, if
they do intend to do the right thing here, that they're so obstinately
set against just saying so."


Ed

Thanks for posting the URL, I just noticed I'd forgotten to do it in my
original posting.

Anyways, there's been no announcement yet, whether they're going to
release it to desktop/server/mobile etc. But my assumption is that it's
going to go everywhere since it doesn't really cost much money to put it
everything. I can't even seeing them bothering to disable it on some
chips but not others.

Yousuf Khan

 

[Yousuf Khan]

It's not speculation. AMD has stated it as fact.

Yeah? Where? So far, AMD is just denying all speculations.

Yousuf Khan

 

[Yousuf Khan]

As long as they're functionally close to each other, it shouldn't be too
much of a problem. The instruction to enter virtual execution probably
is only used in one place.

The bigger problem is the VMCALL instruction to make hypervisor calls.
You need to know what hypervisor you are running under so you know what
and how
to make the calls. That means some sort of coordination among the
different vm vendors.

I looked at the Vanderpool preliminary architecture and it allows the
hypervisor
to trap the MWAIT instruction but not the PAUSE instruction. Presumably
it's
not ok to waste a time slice in MWAIT but is ok to waste it in a PAUSE
spin loop
(assuming the guest uses pause in their spin wait loops). That's strange
considering the current MWAIT implementation is pretty suboptimal and
probably
no one uses it.

MWAIT is one of the new SSE3 instructions linking with Hyperthreading.
PAUSE was one of the older Hyperthreading instructions. Might be an
attempt by Intel to get more people using SSE3, even if they don't need
to use any of its math features.

Yousuf Khan

 

[George Macdonald]

Is AMD releasing this tech to desktops or just servers?


http://news.com.com/AMD+jockeys+with+Intel+in+multi-OS+race/2100-1012_3-5600552.html?tag=nefd.lede
"It's hard to imagine that AMD would be so blindingly stupid as to
forward a truly Vanderpool-incompatible virtualization mechanism," a
move that could cause heartburn for many software makers, said
Illuminata analyst Jonathan Eunice. "It's hard to understand why, if
they do intend to do the right thing here, that they're so obstinately
set against just saying so."

Uh-huh the same company that was so "blindingly stupid" as to do a 64-bit
x86. I'm not sure what is meant by "incompatible" here but I don't see how
it causes "heartburn" for application software developers. Isn't the point
of virtualization that apps are not supposed to notice they don't have a
system all to themselves? If the mechanism is so tangibly application
dependent, what's the point?

 

[Ed]

Uh-huh the same company that was so "blindingly stupid" as to do a 64-bit
x86. I'm not sure what is meant by "incompatible" here but I don't see how
it causes "heartburn" for application software developers. Isn't the point
of virtualization that apps are not supposed to notice they don't have a
system all to themselves? If the mechanism is so tangibly application
dependent, what's the point?

If AMD's solution out performs Intel's at a better price developers may
look at Intel's solution as the one that's incompatible. ;p

http://theinquirer.net/?article=21607
AMD did say that its version would be better because it would support
all the things that make the K8 line unique. HT, embedded memory
controllers, and all the bells and whistles should make for a whopping
good time for all the VMM writers out there. With any luck, Pacifica
will make it easier.

Ed

 

[Joe Seigh]

MWAIT is one of the new SSE3 instructions linking with Hyperthreading.
PAUSE was one of the older Hyperthreading instructions. Might be an
attempt by Intel to get more people using SSE3, even if they don't need
to use any of its math features.

According to here and the Intel manuals
http://softwareforums.intel.com/ids/board/message?board.id=42&message.id=417
MWAIT was meant to be an alternative to processors issuing a HLT while waiting
for work since you could dispence with signaling a processor to wake it out of HLT
wait state. It's quite slow, so it's not something you want to use for spin
locks.

It looks like ADM doesn't have PAUSE so there doesn't appear to be any standard
way to do a spin wait. This makes virtual SMP rather problematic unless you
restricted it and/or implemented gang scheduling.

 

[Alex Johnson]

"It's hard to imagine that AMD would be so blindingly stupid as to
forward a truly Vanderpool-incompatible virtualization mechanism,"

Interesting. Like they were so blindingly stupid as to forward a truly
SSE-incompatible vector processing extension (3D-Now!)?

The things that will need to understand virtualization are the firmware
and the virtualized operating systems. AMD can control the firmware if
it wants. The OS vendors are not likely to support two approaches to
virtualization because it takes a lot of work to virtualize a major OS.

MWAIT probably has to trap because it changes the system state of the
machine that might interfere with virtualization or could present a hole
to violate the integrity of the virtual environment by waiting on a
memory location which is later changed by someone in a separate virtual
partition. MWAIT should only wake the (virtualized) processor if the
correct virtualized system wakes it.

PAUSE does not need to trap because, as the SDM states: "This
instruction does not change the architectural state of the processor
(that is, it performs essentially a delaying noop operation)."

Alex

 

[Joe Seigh]

MWAIT probably has to trap because it changes the system state of the
machine that might interfere with virtualization or could present a hole
to violate the integrity of the virtual environment by waiting on a
memory location which is later changed by someone in a separate virtual
partition. MWAIT should only wake the (virtualized) processor if the
correct virtualized system wakes it.

That's not really a problem. Load reserved/store conditional has a similar
issue. The way you deal with that is to drop the reserve on preemption by
doing a throwaway store conditional. On resumption of execution, the
subsequent store conditional will fail. For MWAIT, on handling preemption
from an interrupt (i/o or timer) the hypervisor would do a MONITOR and
store into a the monitored region to drop the monitoring. Resumption
of the viritual processor would be after the MWAIT, so it would appear
to be a spurious wakeup. So it is not required to trap MWAIT to ensure
integrity of virtual machine state.

AMD doesn't have MWAIT so it's a moot issue for them. Though I suppose
you could simulate the instruction if the opcodes for system instructions
used by AMD and by Intel didn't conflict.

 

[YKhan]

AMD doesn't have MWAIT so it's a moot issue for them. Though I suppose
you could simulate the instruction if the opcodes for system instructions
used by AMD and by Intel didn't conflict.

Most of those instructions are Hyperthreading-related, AMD doesn't
right now have Hyperthreading, so those instructions don't exist for
it. However, once dual-cores arrive, AMD has stated that it will use
the Hyperthreading interface to indentify its cores and likely all of
the Hyperthreading instructions will be added at that point too.

Yousuf Khan

 

[Eric Smith]

Yeah? Where? So far, AMD is just denying all speculations.

It was in an interview on one of the zillion PC review sites; I have no
idea now which one it was. An AMD spokesman said that their
virtualization technology was homegrown rather than being a copy of
anything Intel was working on.

 

[Yousuf Khan]

It was in an interview on one of the zillion PC review sites; I have no
idea now which one it was. An AMD spokesman said that their
virtualization technology was homegrown rather than being a copy of
anything Intel was working on.

Well then, this is a case where AMD can actually get away with it.
Virtualization can't really be considered a part of the x86 software
stack, with a huge legacy behind it. There's maybe five products out
there that do it, and none of them are so complex that they can't
accomodate an Intel method and an AMD method of virtualization.

Yousuf Khan

 

[Wes Felter]

The bigger problem is the VMCALL instruction to make hypervisor calls.
You need to know what hypervisor you are running under so you know what and how
to make the calls. That means some sort of coordination among the
different vm vendors.

I suspect most hypervisors will use VT/Pacifica to do full
virtualization, so that the guest OSes don't even know they're being
virtualized. In that situation the guest OSes do not make any
hypercalls. Maybe there will be optional special drivers (like under
VMware), but these will be specific to the hypervisor so there's no
problem.

 

[Douglas Siebert]

Intel's Vanderpool and AMD's Pacifica are supposed to bring hardware
support for virtualization to x86 processors. Basically they'll be like
a super Protected Mode which have more privilege than operating
systems. There is speculation that the two implementations will not be
compatible with each other. That might present a fork in x86
instruction sets, although it's not clear if it really matters at this
level since the hypervisor runs so far above the operating system and
applications that they won't even notice it. But it's interesting
architecturally, nonetheless.

Kind of a moronic position for those speculators to take since the whole
point of the virtualization tech from both Intel and AMD is that it will
be completely invisible to the OS and user mode code. It only has to be
coded in the hypervisor and there aren't exactly thousands of different
implementations of that kind of software, and the critical sections where
these differences mattered would be quite small. Who cares if there are
two code paths for those sections? If game authors can write completely
different rendering paths for ATI and NVidia cards, I'm sure EMC and MS
can handle making their software handle both Intel and AMD virtualization.
Since Xen is open source it will certainly be done for both, and probably
done by Intel and AMD engineers, at that. Maybe even with support ready
before you can even buy the CPUs themselves.

Anyway there's already a few forks in the x86 world, since Intel doesn't
support 3Dnow!, AMD doesn't support hyperthreading, AMD didn't support
MONITOR and MWAIT in their SSE3 implementation. Oh yeah, for a while
there Intel didn't support AMD64, but they recently fixed that.

Which points out another alternative. That AMD just does what Intel did
when they implemented AMD64. Wait until they come out with the docs about
the tech (which Intel just did) and take a month to copy them with some
minor modifications to suit your CPU's implementation a bit better (like
if you had stuff like glueless SMP support the other guy doesn't) Then
announce it as shipping the following year. If AMD followed that path,
they'd release their info in a month, then announce a ship date of 2006.
Oh wait...

 

[Sander Vesik]

Interesting. Like they were so blindingly stupid as to forward a truly
SSE-incompatible vector processing extension (3D-Now!)?

The things that will need to understand virtualization are the firmware
and the virtualized operating systems. AMD can control the firmware if
it wants. The OS vendors are not likely to support two approaches to
virtualization because it takes a lot of work to virtualize a major OS.

No. It depends on whetever one can build a abstraction layer easily
that would cover both - nobody (well, possibly with one exception) is
going to be fool enough as to do it with sprinkling a lot of assmebler
through all source files.

MWAIT probably has to trap because it changes the system state of the
machine that might interfere with virtualization or could present a hole
to violate the integrity of the virtual environment by waiting on a
memory location which is later changed by someone in a separate virtual
partition. MWAIT should only wake the (virtualized) processor if the
correct virtualized system wakes it.

MWAIT does not need to trap - there needs to be a way to know that
something was mwaiting and continue in that state after a switch back.

 

[YKhan]

Kind of a moronic position for those speculators to take since the whole
point of the virtualization tech from both Intel and AMD is that it will
be completely invisible to the OS and user mode code. It only has to be
coded in the hypervisor and there aren't exactly thousands of different
implementations of that kind of software, and the critical sections where
these differences mattered would be quite small. Who cares if there are
two code paths for those sections? If game authors can write completely
different rendering paths for ATI and NVidia cards, I'm sure EMC and MS
can handle making their software handle both Intel and AMD virtualization.
Since Xen is open source it will certainly be done for both, and probably
done by Intel and AMD engineers, at that. Maybe even with support ready
before you can even buy the CPUs themselves.

That reminds me, how are they adding these extra instructions into the
x86 instruction set? It's been years since I looked into x86 assembly
(since the 386 and 486 days), and previously they used to just add new
instructions into the second bank of instructions by adding a specific
instruction prefix (I believe it was called the "ESC" prefix). However
they couldn't just add those extra instructions anywhere they liked
(well, actually initially they could, since it was just Intel
controlling it all back then), but as that second map got filled up,
they had do it strategically, they had to add them in a
compatible-fashion. Have they opened up a new instruction path
nowadays?

Anyway there's already a few forks in the x86 world, since Intel doesn't
support 3Dnow!, AMD doesn't support hyperthreading, AMD didn't support
MONITOR and MWAIT in their SSE3 implementation. Oh yeah, for a while
there Intel didn't support AMD64, but they recently fixed that.

Also there was a brief period where Intel's version didn't support the
NX-bit in the paging tables.

Which points out another alternative. That AMD just does what Intel did
when they implemented AMD64. Wait until they come out with the docs about
the tech (which Intel just did) and take a month to copy them with some
minor modifications to suit your CPU's implementation a bit better (like
if you had stuff like glueless SMP support the other guy doesn't) Then
announce it as shipping the following year. If AMD followed that path,
they'd release their info in a month, then announce a ship date of 2006.
Oh wait...

Well, it took considerably longer for Intel to copy AMD64 than a month.
I think the AMD spec docs were out a full year before AMD actually
released any products. And then it took Intel another 1.5 years to
release their own version. So altogether it was 2.5 years later for
Intel. Mind you I would guess that the AMD64 instruction set would be
much more complicated to copy than a virtual-machine instruction set.

There was an earlier example for AMD. During the introduction of the
MMX instruction set, AMD had their version released on the same day as
Intel's. There was talk that AMD was planning on doing its own
proprietary instruction set previously, but then it decided to license
MMX from Intel and it simply remapped all of its proprietary
instructions to the MMX instruction mappings instead. AMD was able to
do that virtually overnight. I can imagine if compatibility in virtual
machines is a requirement, then it shouldn't be too difficult to remap
Pacifica to Vanderpool or vice-versa. It must relatively
straightforward in the same way MMX was straightforward.

But it may not be necessary, virtual machines aren't that common that
they need compatibility.

Yousuf Khan