Cross vendor migration ideas

Cross vendor migration ideas

[Alexander Graf]

Hi,

I was thinking a bit about cross vendor migration recently and since  
we're doing open source development, I figured it might be a good idea  
to talk to everyone about this.

So why are we having a problem?

In normal operation we don't. If we're running a 32-bit kernel, we can  
use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit  
kernel with 64-bit userspace, every CPU supports SYSCALL. At least  
Linux is being smart on this and does use exactly these two  
capabilities in these two cases.
But if we're running in compat mode (64-bit kernel with 32-bit  
userspace), things differ. Intel supports only SYSENTER here, while  
AMD only supports SYSCALL. Both can still use int80.

Operating systems detect usage of SYSCALL or SYSENTER pretty early on  
(Linux does this on vdso). So when we boot up on an Intel machine,  
Linux assumes that using SYSENTER in compat mode is fine. Migrating  
that machine to an AMD machine breaks this assumption though, since  
SYSENTER can't be used in compat mode.
On LInux, this detection is based on the CPU vendor string. If Linux  
finds a "GenuineIntel", SYSENTER is used in compat mode, if it's  
"AuthenticAMD", SYSCALL is used and if none of these two is found,  
int80 is used.

I tried modifying the vendor string, removed the "overwrite the vendor  
string with the native string" hack and things look like they work  
just fine with Linux.

Unfortunately right now I don't have a 64-bit Windows installation  
around to check if that approach works there too, but if it does and  
no known OS breaks due to the invalid vendor string, we can just  
create our own virtual CPU string, no?

I'd love to hear comments and suggestions on this and hope we'll end  
up in a fruitful discussion on how to improve the current situation.

Alex

Re: Cross vendor migration ideas

[Anthony Liguori]

Alexander Graf wrote:
> Hi,
>
> I was thinking a bit about cross vendor migration recently and since 
> we're doing open source development, I figured it might be a good idea 
> to talk to everyone about this.
>
> So why are we having a problem?
>
> In normal operation we don't. If we're running a 32-bit kernel, we can 
> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit 
> kernel with 64-bit userspace, every CPU supports SYSCALL. At least 
> Linux is being smart on this and does use exactly these two 
> capabilities in these two cases.
> But if we're running in compat mode (64-bit kernel with 32-bit 
> userspace), things differ. Intel supports only SYSENTER here, while 
> AMD only supports SYSCALL. Both can still use int80.

Obviously we can trap-and-emulate but that would be slow in a relatively 
fast past.

I wonder if patching is an option?

Regards,

Anthony Liguori

Re: Cross vendor migration ideas

[Alexander Graf]

On 12.11.2008, at 16:45, Anthony Liguori wrote:

> Alexander Graf wrote:
>> Hi,
>>
>> I was thinking a bit about cross vendor migration recently and  
>> since we're doing open source development, I figured it might be a  
>> good idea to talk to everyone about this.
>>
>> So why are we having a problem?
>>
>> In normal operation we don't. If we're running a 32-bit kernel, we  
>> can use SYSENTER to jump from kernel<->userspace. If we're on a 64- 
>> bit kernel with 64-bit userspace, every CPU supports SYSCALL. At  
>> least Linux is being smart on this and does use exactly these two  
>> capabilities in these two cases.
>> But if we're running in compat mode (64-bit kernel with 32-bit  
>> userspace), things differ. Intel supports only SYSENTER here, while  
>> AMD only supports SYSCALL. Both can still use int80.
>
> Obviously we can trap-and-emulate but that would be slow in a  
> relatively fast past.

If we can do it without emulation, I'd greatly prefer it, as syscall/ 
sysenter emulation in the hypervisor most probably isn't exactly  
fast ;-). And you don't really want to degrade performance just  
because you're migrating (think flashplayer here). I guess Windows 64- 
bit contains even more 32-bit parts.

> I wonder if patching is an option?

Windows does have background daemons that check code in runtime and  
compares that to checksums. So binary patching might break Windows  
pretty easily. I'm really wondering why the CR8 patching still works -  
maybe even that'll break with Windows 7.

Alex

Re: Cross vendor migration ideas

[Anthony Liguori]

Alexander Graf wrote:
>
> On 12.11.2008, at 16:45, Anthony Liguori wrote:
>
>> Alexander Graf wrote:
>>> Hi,
>>>
>>> I was thinking a bit about cross vendor migration recently and since 
>>> we're doing open source development, I figured it might be a good 
>>> idea to talk to everyone about this.
>>>
>>> So why are we having a problem?
>>>
>>> In normal operation we don't. If we're running a 32-bit kernel, we 
>>> can use SYSENTER to jump from kernel<->userspace. If we're on a 
>>> 64-bit kernel with 64-bit userspace, every CPU supports SYSCALL. At 
>>> least Linux is being smart on this and does use exactly these two 
>>> capabilities in these two cases.
>>> But if we're running in compat mode (64-bit kernel with 32-bit 
>>> userspace), things differ. Intel supports only SYSENTER here, while 
>>> AMD only supports SYSCALL. Both can still use int80.
>>
>> Obviously we can trap-and-emulate but that would be slow in a 
>> relatively fast past.
>
> If we can do it without emulation, I'd greatly prefer it, as 
> syscall/sysenter emulation in the hypervisor most probably isn't 
> exactly fast ;-). And you don't really want to degrade performance 
> just because you're migrating (think flashplayer here). I guess 
> Windows 64-bit contains even more 32-bit parts.
>
>> I wonder if patching is an option?
>
> Windows does have background daemons that check code in runtime and 
> compares that to checksums. So binary patching might break Windows 
> pretty easily. I'm really wondering why the CR8 patching still works - 
> maybe even that'll break with Windows 7.

The Hyper-V spec claims that the accesses to the TPR will always be in 5 
byte instructions.  Presumably this is to allow binary patching to be 
reliable.   I'm pretty sure newer versions of Windows don't access the 
TPR so frequently or at least access it through CR8.

With VT on modern processors, TPR patching is unnecessary so as long as 
they use CR8 on AMD processors, all should be okay.

Although here is obviously another spot where cross vendor migration 
could be problematic :-)

Regards,

Anthony Liguori

> Alex

RE: Cross vendor migration ideas

[Skywing]

Hi,

-----Original Message-----
From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On Behalf Of Alexander Graf
Sent: Wednesday, November 12, 2008 10:50 AM
To: Anthony Liguori
Cc: kvm@vger.kernel.org list; Amit Shah; Avi Kivity; Elsie Wahlig; Serebrin, Benjamin (Calendar); Nakajima, Jun

> Windows does have background daemons that check code in runtime and  
> compares that to checksums. So binary patching might break Windows  
> pretty easily. I'm really wondering why the CR8 patching still works -  
> maybe even that'll break with Windows 7.

> Alex

If you're thinking of PatchGuard, that's only there on amd64 Windows, not 32-bit x86.  However, I think that patching is likely to cause other issues in general and should be avoided.

BTW, 32-bit [x86] programs on amd64 Windows will all eventually get thunked through the 64-bit system call interface, which is always (hardcoded) syscall for amd64 Windows.  It is only if you are running a 32-bit (x86) Windows kernel that you may optionally use one of [sysenter, syscall, int 2e].  In other words, if you are running a 64-bit Windows kernel, even the 32-bit parts will always pick syscall regardless of the reported processor feature set.  The 32-bit binaries that make the system call transitions to kernel mode are different if you are running a 64-bit Windows kernel versus a 32-bit kernel; the code to differentiate which mechanism to use is removed.

(In fact, to be specific, it is always 64-bit code that issues the actual syscall instruction, as the call path in the Wow64 [32-bit Windows program running under an amd64 Windows kernel] case is 32-bit program code -> 32-bit ntdll.dll -> 64-bit wow64.dll -> 64-bit ntdll.dll, where 64-bit ntdll.dll is the system-supplied binary that makes the actual system service calls.  The switch from 32-bit mode to 64-bit mode in amd64 Windows is done in user mode.  All system call transitions are thus done from 64-bit mode in the 64-bit Windows case.  This sequence is independent of CPU vendor.)

You could probably get away with not claiming support for sysenter/syscall for 32-bit guests, as Windows will degrade gracefully to int 2e in that case for guests running a 32-bit kernel.  Performance implications for that still apply of course.

- S

RE: Cross vendor migration ideas

[Serebrin, Benjamin (Calendar)]

This seems like conveniently good news.

Combining all the ideas: 

Hide CPUID.SYSCALL to force 32-bit OSes to use 32-bit legacy SYSENTER,
which is available on AMD and Intel.

64-bit OSes will use SYSCALL for 64-bit code.  CPUID name string should
be "FakeCPUFakeCPU" or some such so a 64-bit OS will avoid SYSCALL and
SYSENTER for 32-bit code.  A 64-bit Linux will use int80 for its 32-bit
code, by Alex's observation.  A 64-bit Windows will use a thunk that
gets to long mode before using SYSCALL, by Skywing's observation.

Intercept #UD and emulate any of the SYSENTER, SYSCALL, SYSRET, SYSEXIT
instructions that fall through the cracks.  I think this addresses
Alex's observations about MacOS/Darwin's use of SYSENTER.  

This leaves you with a hypervisor that doesn't have to know a priori
what the guest is going to do and can do nearly all its tricks through
CPUID control.  The only unsatisfying piece is anything high-frequency
that the hypervisor ends up having to emulate.  Maybe in-guest patching
works on Darwin if it doesn't have any intrusion detection going on?  

Ben

-----Original Message-----
From: Skywing [mailto:Skywing@valhallalegends.com] 
Sent: Wednesday, November 12, 2008 4:02 PM
To: Alexander Graf; Anthony Liguori
Cc: kvm@vger.kernel.org list; Amit Shah; Avi Kivity; Wahlig, Elsie;
Serebrin, Benjamin (Calendar); Nakajima, Jun
Subject: RE: Cross vendor migration ideas

Hi,

-----Original Message-----
From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
Behalf Of Alexander Graf
Sent: Wednesday, November 12, 2008 10:50 AM
To: Anthony Liguori
Cc: kvm@vger.kernel.org list; Amit Shah; Avi Kivity; Elsie Wahlig;
Serebrin, Benjamin (Calendar); Nakajima, Jun

> Windows does have background daemons that check code in runtime and  
> compares that to checksums. So binary patching might break Windows  
> pretty easily. I'm really wondering why the CR8 patching still works -

> maybe even that'll break with Windows 7.

> Alex

If you're thinking of PatchGuard, that's only there on amd64 Windows,
not 32-bit x86.  However, I think that patching is likely to cause other
issues in general and should be avoided.

BTW, 32-bit [x86] programs on amd64 Windows will all eventually get
thunked through the 64-bit system call interface, which is always
(hardcoded) syscall for amd64 Windows.  It is only if you are running a
32-bit (x86) Windows kernel that you may optionally use one of
[sysenter, syscall, int 2e].  In other words, if you are running a
64-bit Windows kernel, even the 32-bit parts will always pick syscall
regardless of the reported processor feature set.  The 32-bit binaries
that make the system call transitions to kernel mode are different if
you are running a 64-bit Windows kernel versus a 32-bit kernel; the code
to differentiate which mechanism to use is removed.

(In fact, to be specific, it is always 64-bit code that issues the
actual syscall instruction, as the call path in the Wow64 [32-bit
Windows program running under an amd64 Windows kernel] case is 32-bit
program code -> 32-bit ntdll.dll -> 64-bit wow64.dll -> 64-bit
ntdll.dll, where 64-bit ntdll.dll is the system-supplied binary that
makes the actual system service calls.  The switch from 32-bit mode to
64-bit mode in amd64 Windows is done in user mode.  All system call
transitions are thus done from 64-bit mode in the 64-bit Windows case.
This sequence is independent of CPU vendor.)

You could probably get away with not claiming support for
sysenter/syscall for 32-bit guests, as Windows will degrade gracefully
to int 2e in that case for guests running a 32-bit kernel.  Performance
implications for that still apply of course.

- S

Re: Cross vendor migration ideas

[Andi Kleen]

"Serebrin, Benjamin (Calendar)" <Benjamin.Serebrin@exchange.amd.com> writes:
>
> 64-bit OSes will use SYSCALL for 64-bit code.  CPUID name string should
> be "FakeCPUFakeCPU"

One problem is that you'll disable a lot of vendor specific setup 
code if you do that. Might have all kinds of unexpected side effects.

Also it might break user space, unless you key the fake vendor CPUID
intercept on ring 0 vs ring 3 (but even if that might not be enough
because some kernel modules can call CPUID on their own)

I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't
be that much slower than int 0x80 hopefully.

-Andi

Re: Cross vendor migration ideas

[Alexander Graf]

On 15.11.2008, at 14:03, Andi Kleen wrote:

> "Serebrin, Benjamin (Calendar)" <Benjamin.Serebrin@exchange.amd.com>  
> writes:
>>
>> 64-bit OSes will use SYSCALL for 64-bit code.  CPUID name string  
>> should
>> be "FakeCPUFakeCPU"
>
> One problem is that you'll disable a lot of vendor specific setup
> code if you do that. Might have all kinds of unexpected side effects.

Yes, but since we're emulating a CPU anyways we don't want vendor  
specific setup, since we might live migrate. And quirks should already  
be handled by the host. We'd only nop them out usually.

> Also it might break user space, unless you key the fake vendor CPUID
> intercept on ring 0 vs ring 3 (but even if that might not be enough
> because some kernel modules can call CPUID on their own)

Is there userspace code that relies on GenuineIntel? What would a new  
vendor do? Can't we just behave the same and have our self-invented  
vendor string added to applications that direly need to know which  
vendor they're running on?

> I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't
> be that much slower than int 0x80 hopefully.

Well emulating them means you're leaving the VM on every user<->kernel  
transition. That's a _huge_ performance hit. I don't have the numbers,  
but IIRC a roundtrip is ~3000 cycles.

So yes, it definitely is safer. But it'll hurt performance a lot -  
which is exactly what you'd want to avoid when migrating between  
vendors.

Alex

Re: Cross vendor migration ideas

[Andi Kleen]

Alexander Graf wrote:
> 
> On 15.11.2008, at 14:03, Andi Kleen wrote:
> 
>> "Serebrin, Benjamin (Calendar)" <Benjamin.Serebrin@exchange.amd.com>
>> writes:
>>>
>>> 64-bit OSes will use SYSCALL for 64-bit code.  CPUID name string should
>>> be "FakeCPUFakeCPU"
>>
>> One problem is that you'll disable a lot of vendor specific setup
>> code if you do that. Might have all kinds of unexpected side effects.
> 
> Yes, but since we're emulating a CPU anyways we don't want vendor
> specific setup, since we might live migrate.

I was mainly thinking of tuning.

For example Linux selects the best suitable page copy
function based on vendor information. Using the wrong page copy
can have a large impact on your performance.

> And quirks should already
> be handled by the host. We'd only nop them out usually.
> 
>> Also it might break user space, unless you key the fake vendor CPUID
>> intercept on ring 0 vs ring 3 (but even if that might not be enough
>> because some kernel modules can call CPUID on their own)
> 
> Is there userspace code that relies on GenuineIntel? 

Yes. Undoubtedly also the same with others.

>> I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't
>> be that much slower than int 0x80 hopefully.
> 
> Well emulating them means you're leaving the VM on every user<->kernel
> transition. That's a _huge_ performance hit. I don't have the numbers,
> but IIRC a roundtrip is ~3000 cycles.

Depends on the CPU. Newer CPUs are faster.

Also you would select the default based on which CPU you're booting
on. So initially and as long as you stay on the same
vendor you'll have full performance.  Only if you switch
vendors later you'll also eat the emulation hit (and get it
back again when you switch back to a system with the original vendor)
But everything will still work correctly and that is the
important part.

Doesn't sound so bad to me.

-Andi

Re: Cross vendor migration ideas

[Avi Kivity]

Andi Kleen wrote:
>> Yes, but since we're emulating a CPU anyways we don't want vendor
>> specific setup, since we might live migrate.
>>     
>
> I was mainly thinking of tuning.
>
> For example Linux selects the best suitable page copy
> function based on vendor information. Using the wrong page copy
> can have a large impact on your performance.
>   

We can tell Linux to retune using a custom ACPI (blech) event.

>>> Also it might break user space, unless you key the fake vendor CPUID
>>> intercept on ring 0 vs ring 3 (but even if that might not be enough
>>> because some kernel modules can call CPUID on their own)
>>>       
>> Is there userspace code that relies on GenuineIntel? 
>>     
>
> Yes. Undoubtedly also the same with others.
>   

This will not work of course for userspace.

>>> I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't
>>> be that much slower than int 0x80 hopefully.
>>>       
>> Well emulating them means you're leaving the VM on every user<->kernel
>> transition. That's a _huge_ performance hit. I don't have the numbers,
>> but IIRC a roundtrip is ~3000 cycles.
>>     
>
> Depends on the CPU. Newer CPUs are faster.
>   

I think emulation will be slower than 3000 cycles, even on newer cpus.  
In addition to the #vmexit, you have to save all registers, walk the 
guest page tables to fetch the instruction, execute the emulator code 
including all the checks, go back to the guest.  Then do the same again 
on sysexit/sysret -- the overhead doubles.

> Also you would select the default based on which CPU you're booting
> on. So initially and as long as you stay on the same
> vendor you'll have full performance.  Only if you switch
> vendors later you'll also eat the emulation hit (and get it
> back again when you switch back to a system with the original vendor)
> But everything will still work correctly and that is the
> important part.
>
> Doesn't sound so bad to me.
>   

Long running guests on a large farm shouldn't expect to stay on the same 
host they were booted on.  But at least for Linux, if we notify the 
guest telling it to retune (which can include the vsyscall path) we can 
avoid the cost entirely.

-- 
error compiling committee.c: too many arguments to function

Re: Cross vendor migration ideas

[Andi Kleen]

> Long running guests on a large farm shouldn't expect to stay on the same 
> host they were booted on.  

If they change frequently and the vendors are even matched in the farm
their chance of running "fast" again will be quite good. The only
worst case would be to start on one vendor, run for a short time
there and then spend all the remaining time on the other. Most other
cases would be better.

Also again I wouldn't expect the impact to be that dramatic. While
there are some workloads which are syscall latency sensitive (and not rely
on the vsyscalls) a lot of others are not. 

> But at least for Linux, if we notify the 
> guest telling it to retune (which can include the vsyscall path) we can 
> avoid the cost entirely.

Hmm, we discussed something like this some time ago anyways (add
a way to rescan CPUID features) because there are microcode updates
around that change some (relatively obscure) CPUID features.

But doing it fully general would be likely quite intrusive. You 
would need to rewrite the CPU initialization code and some other
code in a callback like matter like the PCI code does for hotplug.

I doubt doing that would be worth the effort in real performance gains.

-Andi

-- 
ak@linux.intel.com

Re: Cross vendor migration ideas

[Avi Kivity]

Andi Kleen wrote:
>> But at least for Linux, if we notify the 
>> guest telling it to retune (which can include the vsyscall path) we can 
>> avoid the cost entirely.
>>     
>
> Hmm, we discussed something like this some time ago anyways (add
> a way to rescan CPUID features) because there are microcode updates
> around that change some (relatively obscure) CPUID features.
>
> But doing it fully general would be likely quite intrusive. You 
> would need to rewrite the CPU initialization code and some other
> code in a callback like matter like the PCI code does for hotplug.
>
>   

The primary consumer would be compat mode syscall/sysenter.  Page copy 
would be a secondary consumer.  All the rest are likely in the noise.

Hmm, maybe also the raid5+ algorithms.

-- 
error compiling committee.c: too many arguments to function

Re: Cross vendor migration ideas

[Glauber Costa]

On Sat, Nov 15, 2008 at 2:39 PM, Alexander Graf <agraf@suse.de> wrote:
>
> On 15.11.2008, at 14:03, Andi Kleen wrote:
>
>> "Serebrin, Benjamin (Calendar)" <Benjamin.Serebrin@exchange.amd.com>
>> writes:
>>>
>>> 64-bit OSes will use SYSCALL for 64-bit code.  CPUID name string should
>>> be "FakeCPUFakeCPU"
>>
>> One problem is that you'll disable a lot of vendor specific setup
>> code if you do that. Might have all kinds of unexpected side effects.
>
> Yes, but since we're emulating a CPU anyways we don't want vendor specific
> setup, since we might live migrate. And quirks should already be handled by
> the host. We'd only nop them out usually.
>
>> Also it might break user space, unless you key the fake vendor CPUID
>> intercept on ring 0 vs ring 3 (but even if that might not be enough
>> because some kernel modules can call CPUID on their own)
>
> Is there userspace code that relies on GenuineIntel? What would a new vendor
> do? Can't we just behave the same and have our self-invented vendor string
> added to applications that direly need to know which vendor they're running
> on?
>
>> I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't
>> be that much slower than int 0x80 hopefully.
>
> Well emulating them means you're leaving the VM on every user<->kernel
> transition. That's a _huge_ performance hit. I don't have the numbers, but
> IIRC a roundtrip is ~3000 cycles.
>
> So yes, it definitely is safer. But it'll hurt performance a lot - which is
> exactly what you'd want to avoid when migrating between vendors.

We can possibly do it a little bit better by emulating in the guest directly.
So the deal would be emulation both in the host and the guest. If the
guest have it,
fine. Otherwise, we use host emulation, that is slower, but works.

> Alex
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Glauber  Costa.
"Free as in Freedom"
http://glommer.net

"The less confident you are, the more serious you have to act."

Re: Cross vendor migration ideas

[Avi Kivity]

Glauber Costa wrote:
> We can possibly do it a little bit better by emulating in the guest directly.
> So the deal would be emulation both in the host and the guest. If the
> guest have it,
> fine. Otherwise, we use host emulation, that is slower, but works.
>   

If the guest is modifiable, this is easy.  Include both syscall and 
sysenter paths in the vsyscall page, with a jmp to select between them.  
If we trap a #UD pointing at a syscall or sysenter, emulate it, and 
patch the jmp instruction to point at the other path.  This way we have 
a self-adjusting syscall/sysenter entry point.  We also need to tell kvm 
not to emulate the instruction, so that the #UD actually reaches the kernel.

If the guest is not modifiable, this means patching.  This is much 
harder than tpr patching since we don't have any space to hide our code 
in.  Maybe we can reverse engineer the vsyscall page and hack it, but 
this may take a lot of work.

-- 
error compiling committee.c: too many arguments to function

Re: Cross vendor migration ideas

[Avi Kivity]

Alexander Graf wrote:
>> I wonder if patching is an option?
>
> Windows does have background daemons that check code in runtime and 
> compares that to checksums. So binary patching might break Windows 
> pretty easily. I'm really wondering why the CR8 patching still works - 
> maybe even that'll break with Windows 7.

kvm doesn't patch Vista since the bios isn't mapped.  Vista doesn't bang 
heavily on the TPR like XP does so patching is not needed anyway.

-- 
error compiling committee.c: too many arguments to function

Re: Cross vendor migration ideas

[Andi Kleen]

Anthony Liguori <anthony@codemonkey.ws> writes:
>
> I wonder if patching is an option?

Not by patching instructions because SYSENTER and SYSCALL have
completely different semantics (e.g. regarding register use). That is
why the entry paths are different on both sides.

In theory you could switch the entry paths around, but that would
be highly kernel version specific.

One easy way is to just clear the SEP and SYSCALL32 flags
in CPUID, then int 0x80 will be always used. Of course it'll
cost you in performance significantly. I wouldn't recommend
to make it default.

-Andi

-- 
ak@linux.intel.com

Re: Cross vendor migration ideas

[Alexander Graf]

On 12.11.2008, at 21:06, Andi Kleen wrote:

> Anthony Liguori <anthony@codemonkey.ws> writes:
>>
>> I wonder if patching is an option?
>
> Not by patching instructions because SYSENTER and SYSCALL have
> completely different semantics (e.g. regarding register use). That is
> why the entry paths are different on both sides.
>
> In theory you could switch the entry paths around, but that would
> be highly kernel version specific.
>
> One easy way is to just clear the SEP and SYSCALL32 flags
> in CPUID, then int 0x80 will be always used. Of course it'll
> cost you in performance significantly. I wouldn't recommend
> to make it default.

SEP and SYSCALL CPUIDs are only respected on X86_32. X86_64 just  
ignores them and sets X86_FEATURE_SYS{CALL,ENTER}32 in intel.c / amd.c.

That's why I'm suggesting the vendor hack, because Linux doesn't  
respect the CPUID flags on x86_64.

Alex

Re: Cross vendor migration ideas

[Avi Kivity]

Anthony Liguori wrote:
>
> I wonder if patching is an option?

You would need to patch the syscall/sysenter and sysret/sysexit sites, 
as well as the kernel entry points, since the calling conventions are 
dissimilar.  Finding a place to put the stubs would be difficult, at best.

-- 
error compiling committee.c: too many arguments to function

Re: Cross vendor migration ideas

[Amit Shah]

Hi Alex,

* On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote:
> Hi,
>
> I was thinking a bit about cross vendor migration recently and since
> we're doing open source development, I figured it might be a good idea
> to talk to everyone about this.
>
> So why are we having a problem?
>
> In normal operation we don't. If we're running a 32-bit kernel, we can
> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit
> kernel with 64-bit userspace, every CPU supports SYSCALL. At least
> Linux is being smart on this and does use exactly these two
> capabilities in these two cases.
> But if we're running in compat mode (64-bit kernel with 32-bit
> userspace), things differ. Intel supports only SYSENTER here, while
> AMD only supports SYSCALL. Both can still use int80.
>
> Operating systems detect usage of SYSCALL or SYSENTER pretty early on
> (Linux does this on vdso). So when we boot up on an Intel machine,
> Linux assumes that using SYSENTER in compat mode is fine. Migrating
> that machine to an AMD machine breaks this assumption though, since
> SYSENTER can't be used in compat mode.
> On LInux, this detection is based on the CPU vendor string. If Linux
> finds a "GenuineIntel", SYSENTER is used in compat mode, if it's
> "AuthenticAMD", SYSCALL is used and if none of these two is found,
> int80 is used.
>
> I tried modifying the vendor string, removed the "overwrite the vendor
> string with the native string" hack and things look like they work
> just fine with Linux.
>
> Unfortunately right now I don't have a 64-bit Windows installation
> around to check if that approach works there too, but if it does and
> no known OS breaks due to the invalid vendor string, we can just
> create our own virtual CPU string, no?

qemu has an option for that, -cpu qemu64 IIRC. As long as we expose 
practically correct cpuids and MSRs, this should be fine. I've not tested 
qemu64 with winxp x64 though. Also, last I knew, winxp x64 installation 
didn't succeed with --no-kvm. qemu by default exposes an AMD CPU type.

There are pros and cons to expose a custom vendor ID:

pros:
- We don't need to have all the cpuid features exposed which are expected of a 
physically available CPU in the market, for example, badly-coded applications 
might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or not, not 
exposing what's actually available on every C2D out there is bad. 

cons:
- To expose the "correct" set of feature bits for a known processor, we also 
need to check the family/model/stepping to support the exact same feature 
bits that were present in the CPU.
- We might not get some optimizations that OSes might have based on CPU type, 
even if the host CPU qualifies for such optimizations
- Standard programs like benchmarking tools, etc., might fail if they depend 
on the vendor string for their functionality

For 32-bit guests, I think exposing a pentium4 or Athlon CPU type should be 
fine. For 64-bit guests, the newer the better.

> I'd love to hear comments and suggestions on this and hope we'll end
> up in a fruitful discussion on how to improve the current situation.

I have a patch ready for emulating sysenter/sysexit on AMD systems (needs 
testing). Patching the guest was an option that was discouraged; I had a hack 
ready but it was quickly shelved (again, untested).

Amit

Re: Cross vendor migration ideas

[Alexander Graf]

On 12.11.2008, at 17:52, Amit Shah wrote:

> Hi Alex,
>
> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote:
>> Hi,
>>
>> I was thinking a bit about cross vendor migration recently and since
>> we're doing open source development, I figured it might be a good  
>> idea
>> to talk to everyone about this.
>>
>> So why are we having a problem?
>>
>> In normal operation we don't. If we're running a 32-bit kernel, we  
>> can
>> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit
>> kernel with 64-bit userspace, every CPU supports SYSCALL. At least
>> Linux is being smart on this and does use exactly these two
>> capabilities in these two cases.
>> But if we're running in compat mode (64-bit kernel with 32-bit
>> userspace), things differ. Intel supports only SYSENTER here, while
>> AMD only supports SYSCALL. Both can still use int80.
>>
>> Operating systems detect usage of SYSCALL or SYSENTER pretty early on
>> (Linux does this on vdso). So when we boot up on an Intel machine,
>> Linux assumes that using SYSENTER in compat mode is fine. Migrating
>> that machine to an AMD machine breaks this assumption though, since
>> SYSENTER can't be used in compat mode.
>> On LInux, this detection is based on the CPU vendor string. If Linux
>> finds a "GenuineIntel", SYSENTER is used in compat mode, if it's
>> "AuthenticAMD", SYSCALL is used and if none of these two is found,
>> int80 is used.
>>
>> I tried modifying the vendor string, removed the "overwrite the  
>> vendor
>> string with the native string" hack and things look like they work
>> just fine with Linux.
>>
>> Unfortunately right now I don't have a 64-bit Windows installation
>> around to check if that approach works there too, but if it does and
>> no known OS breaks due to the invalid vendor string, we can just
>> create our own virtual CPU string, no?
>
> qemu has an option for that, -cpu qemu64 IIRC. As long as we expose
> practically correct cpuids and MSRs, this should be fine. I've not  
> tested
> qemu64 with winxp x64 though. Also, last I knew, winxp x64  
> installation
> didn't succeed with --no-kvm. qemu by default exposes an AMD CPU type.

I wasn't talking about CPUID features, but the vendor string. Qemu64  
provides the AuthenticAMD string, so we don't run into any issues I'm  
presuming.

> There are pros and cons to expose a custom vendor ID:
>
> pros:
> - We don't need to have all the cpuid features exposed which are  
> expected of a
> physically available CPU in the market, for example, badly-coded  
> applications
> might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or  
> not, not
> exposing what's actually available on every C2D out there is bad.
>
> cons:
> - To expose the "correct" set of feature bits for a known processor,  
> we also
> need to check the family/model/stepping to support the exact same  
> feature
> bits that were present in the CPU.
> - We might not get some optimizations that OSes might have based on  
> CPU type,
> even if the host CPU qualifies for such optimizations
> - Standard programs like benchmarking tools, etc., might fail if  
> they depend
> on the vendor string for their functionality
>
> For 32-bit guests, I think exposing a pentium4 or Athlon CPU type  
> should be
> fine. For 64-bit guests, the newer the better.

Well, we could create different CPU definitions:

- migration safe (do what is safe for migration)
- CPU specific (like a Core2Duo, necessary to run Mac OS X)
- host (fastest possible, but no migration)

I don't think we could find one definition that fits all, so the user  
would have to define what the usage pattern will be.

>
>
>> I'd love to hear comments and suggestions on this and hope we'll end
>> up in a fruitful discussion on how to improve the current situation.
>
> I have a patch ready for emulating sysenter/sysexit on AMD systems  
> (needs
> testing). Patching the guest was an option that was discouraged; I  
> had a hack
> ready but it was quickly shelved (again, untested).

That sounds useful for misbehaving guests or cases I haven't thought  
of yet. Are you sure you're intercepting the SYSENTER MSRs on AMD, so  
you don't end up only getting 32 bits?

Alex

Re: Cross vendor migration ideas

[Amit Shah]

* On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote:
> On 12.11.2008, at 17:52, Amit Shah wrote:
> > Hi Alex,
> >
> > * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote:
> >> Hi,
> >>
> >> I was thinking a bit about cross vendor migration recently and since
> >> we're doing open source development, I figured it might be a good
> >> idea
> >> to talk to everyone about this.
> >>
> >> So why are we having a problem?
> >>
> >> In normal operation we don't. If we're running a 32-bit kernel, we
> >> can
> >> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit
> >> kernel with 64-bit userspace, every CPU supports SYSCALL. At least
> >> Linux is being smart on this and does use exactly these two
> >> capabilities in these two cases.
> >> But if we're running in compat mode (64-bit kernel with 32-bit
> >> userspace), things differ. Intel supports only SYSENTER here, while
> >> AMD only supports SYSCALL. Both can still use int80.
> >>
> >> Operating systems detect usage of SYSCALL or SYSENTER pretty early on
> >> (Linux does this on vdso). So when we boot up on an Intel machine,
> >> Linux assumes that using SYSENTER in compat mode is fine. Migrating
> >> that machine to an AMD machine breaks this assumption though, since
> >> SYSENTER can't be used in compat mode.
> >> On LInux, this detection is based on the CPU vendor string. If Linux
> >> finds a "GenuineIntel", SYSENTER is used in compat mode, if it's
> >> "AuthenticAMD", SYSCALL is used and if none of these two is found,
> >> int80 is used.
> >>
> >> I tried modifying the vendor string, removed the "overwrite the
> >> vendor
> >> string with the native string" hack and things look like they work
> >> just fine with Linux.
> >>
> >> Unfortunately right now I don't have a 64-bit Windows installation
> >> around to check if that approach works there too, but if it does and
> >> no known OS breaks due to the invalid vendor string, we can just
> >> create our own virtual CPU string, no?
> >
> > qemu has an option for that, -cpu qemu64 IIRC. As long as we expose
> > practically correct cpuids and MSRs, this should be fine. I've not
> > tested
> > qemu64 with winxp x64 though. Also, last I knew, winxp x64
> > installation
> > didn't succeed with --no-kvm. qemu by default exposes an AMD CPU type.
>
> I wasn't talking about CPUID features, but the vendor string. Qemu64
> provides the AuthenticAMD string, so we don't run into any issues I'm
> presuming.

Right -- the thing is, with the default AuthenticAMD string, winp x64 
installation fails. That has to be because of some missing cpuids. That's one 
of the drawbacks of exposing a well-known CPU type. I was suggesting we 
should try out the -cpu qemu64 CPU type since it exposes a non-standard CPU 
to see if guests and most userspace programs work fine without any further 
tweaking -- see the 'cons' below for why this might be a problem.

> > There are pros and cons to expose a custom vendor ID:
> >
> > pros:
> > - We don't need to have all the cpuid features exposed which are
> > expected of a
> > physically available CPU in the market, for example, badly-coded
> > applications
> > might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or
> > not, not
> > exposing what's actually available on every C2D out there is bad.
> >
> > cons:
> > - To expose the "correct" set of feature bits for a known processor,
> > we also
> > need to check the family/model/stepping to support the exact same
> > feature
> > bits that were present in the CPU.
> > - We might not get some optimizations that OSes might have based on
> > CPU type,
> > even if the host CPU qualifies for such optimizations
> > - Standard programs like benchmarking tools, etc., might fail if
> > they depend
> > on the vendor string for their functionality
> >
> > For 32-bit guests, I think exposing a pentium4 or Athlon CPU type
> > should be
> > fine. For 64-bit guests, the newer the better.
>
> Well, we could create different CPU definitions:
>
> - migration safe (do what is safe for migration)

There are multiple ways of approaching this: peg to a least-known good CPU 
type, all of whose instructions will work on processors from both the major 
vendors. However, you never know how the server pools change and you'd want 
to upgrade the CPU type once you know the CPUs that are installed in servers. 
This has to be dynamic and the management application has to take care of 
exposing a CPU that's of a "safe" type for the particular server pool. We 
have to provide ways to mask off CPUID bits as requested by the management 
application. (Each server sends its cpuid to the management application, 
which calculates the safest bits and then conveys this to each server before 
starting a VM.)

> - CPU specific (like a Core2Duo, necessary to run Mac OS X)

This doesn't need any more work -- we already have the ability to select CPU 
types. If the management application has knowledge of the kind of OS being 
installed in a VM (which these days is true), exposing a Core2Duo for a 
Mac-based OS isn't difficult.

> - host (fastest possible, but no migration)

This should be the default.

> I don't think we could find one definition that fits all, so the user
> would have to define what the usage pattern will be.
>
> >> I'd love to hear comments and suggestions on this and hope we'll end
> >> up in a fruitful discussion on how to improve the current situation.
> >
> > I have a patch ready for emulating sysenter/sysexit on AMD systems
> > (needs
> > testing). Patching the guest was an option that was discouraged; I
> > had a hack
> > ready but it was quickly shelved (again, untested).
>
> That sounds useful for misbehaving guests or cases I haven't thought
> of yet. Are you sure you're intercepting the SYSENTER MSRs on AMD, so
> you don't end up only getting 32 bits?

Can you elaborate?

Re: Cross vendor migration ideas

[Alexander Graf]

On 13.11.2008, at 05:35, Amit Shah wrote:

> * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote:
>> On 12.11.2008, at 17:52, Amit Shah wrote:
>>> Hi Alex,
>>>
>>> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote:
>>>> Hi,
>>>>
>>>> I was thinking a bit about cross vendor migration recently and  
>>>> since
>>>> we're doing open source development, I figured it might be a good
>>>> idea
>>>> to talk to everyone about this.
>>>>
>>>> So why are we having a problem?
>>>>
>>>> In normal operation we don't. If we're running a 32-bit kernel, we
>>>> can
>>>> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit
>>>> kernel with 64-bit userspace, every CPU supports SYSCALL. At least
>>>> Linux is being smart on this and does use exactly these two
>>>> capabilities in these two cases.
>>>> But if we're running in compat mode (64-bit kernel with 32-bit
>>>> userspace), things differ. Intel supports only SYSENTER here, while
>>>> AMD only supports SYSCALL. Both can still use int80.
>>>>
>>>> Operating systems detect usage of SYSCALL or SYSENTER pretty  
>>>> early on
>>>> (Linux does this on vdso). So when we boot up on an Intel machine,
>>>> Linux assumes that using SYSENTER in compat mode is fine. Migrating
>>>> that machine to an AMD machine breaks this assumption though, since
>>>> SYSENTER can't be used in compat mode.
>>>> On LInux, this detection is based on the CPU vendor string. If  
>>>> Linux
>>>> finds a "GenuineIntel", SYSENTER is used in compat mode, if it's
>>>> "AuthenticAMD", SYSCALL is used and if none of these two is found,
>>>> int80 is used.
>>>>
>>>> I tried modifying the vendor string, removed the "overwrite the
>>>> vendor
>>>> string with the native string" hack and things look like they work
>>>> just fine with Linux.
>>>>
>>>> Unfortunately right now I don't have a 64-bit Windows installation
>>>> around to check if that approach works there too, but if it does  
>>>> and
>>>> no known OS breaks due to the invalid vendor string, we can just
>>>> create our own virtual CPU string, no?
>>>
>>> qemu has an option for that, -cpu qemu64 IIRC. As long as we expose
>>> practically correct cpuids and MSRs, this should be fine. I've not
>>> tested
>>> qemu64 with winxp x64 though. Also, last I knew, winxp x64
>>> installation
>>> didn't succeed with --no-kvm. qemu by default exposes an AMD CPU  
>>> type.
>>
>> I wasn't talking about CPUID features, but the vendor string. Qemu64
>> provides the AuthenticAMD string, so we don't run into any issues I'm
>> presuming.
>
> Right -- the thing is, with the default AuthenticAMD string, winp x64
> installation fails. That has to be because of some missing cpuids.  
> That's one
> of the drawbacks of exposing a well-known CPU type. I was suggesting  
> we
> should try out the -cpu qemu64 CPU type since it exposes a non- 
> standard CPU
> to see if guests and most userspace programs work fine without any  
> further
> tweaking -- see the 'cons' below for why this might be a problem.

I still don't really understand what you're trying to say - qemu64 is  
the default in KVM right now. You mean winxp64 installation doesn't  
work as is and we should fix it? This has nothing to do with the  
migration problems, right?

>
>
>>> There are pros and cons to expose a custom vendor ID:
>>>
>>> pros:
>>> - We don't need to have all the cpuid features exposed which are
>>> expected of a
>>> physically available CPU in the market, for example, badly-coded
>>> applications
>>> might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or
>>> not, not
>>> exposing what's actually available on every C2D out there is bad.
>>>
>>> cons:
>>> - To expose the "correct" set of feature bits for a known processor,
>>> we also
>>> need to check the family/model/stepping to support the exact same
>>> feature
>>> bits that were present in the CPU.
>>> - We might not get some optimizations that OSes might have based on
>>> CPU type,
>>> even if the host CPU qualifies for such optimizations
>>> - Standard programs like benchmarking tools, etc., might fail if
>>> they depend
>>> on the vendor string for their functionality
>>>
>>> For 32-bit guests, I think exposing a pentium4 or Athlon CPU type
>>> should be
>>> fine. For 64-bit guests, the newer the better.
>>
>> Well, we could create different CPU definitions:
>>
>> - migration safe (do what is safe for migration)
>
> There are multiple ways of approaching this: peg to a least-known  
> good CPU
> type, all of whose instructions will work on processors from both  
> the major
> vendors. However, you never know how the server pools change and  
> you'd want
> to upgrade the CPU type once you know the CPUs that are installed in  
> servers.
> This has to be dynamic and the management application has to take  
> care of
> exposing a CPU that's of a "safe" type for the particular server  
> pool. We
> have to provide ways to mask off CPUID bits as requested by the  
> management
> application. (Each server sends its cpuid to the management  
> application,
> which calculates the safest bits and then conveys this to each  
> server before
> starting a VM.)

IMHO we shouldn't really start to be smart here. There's only so much  
benefit in using the least common dominator between all CPUs in the  
datacenter vs. using the least common dominator between all possible  
CPUs. You'll basically end up enabling some newer SSE instructions.

So I don't think we need to go through the hassle of making this  
dynamic. If you want to migrate your machines - use the migrate  
preset. That won't give you the 150% speed boost on video encoding,  
but should not really be any slower on normal workloads. It does make  
things a lot more transparent to us and the admin of a network though,  
because you know what you'll end up with "-cpu migration".

>> - CPU specific (like a Core2Duo, necessary to run Mac OS X)
>
> This doesn't need any more work -- we already have the ability to  
> select CPU
> types. If the management application has knowledge of the kind of OS  
> being
> installed in a VM (which these days is true), exposing a Core2Duo  
> for a
> Mac-based OS isn't difficult.

There is no sysenter emulation for IA-32e on AMD yet, right? That's  
the only issue I see here and your emulation patch should address that.

>> - host (fastest possible, but no migration)
>
> This should be the default.

I'm not sure. Either host or migration should be the default. This  
actually depends on the workload you have on KVM. For servers you'll  
probably want to have migration be the default. For desktop usage it's  
host. I can't think of a way we can be smart about that on the KVM  
level.

>
>
>> I don't think we could find one definition that fits all, so the user
>> would have to define what the usage pattern will be.
>>
>>>> I'd love to hear comments and suggestions on this and hope we'll  
>>>> end
>>>> up in a fruitful discussion on how to improve the current  
>>>> situation.
>>>
>>> I have a patch ready for emulating sysenter/sysexit on AMD systems
>>> (needs
>>> testing). Patching the guest was an option that was discouraged; I
>>> had a hack
>>> ready but it was quickly shelved (again, untested).
>>
>> That sounds useful for misbehaving guests or cases I haven't thought
>> of yet. Are you sure you're intercepting the SYSENTER MSRs on AMD, so
>> you don't end up only getting 32 bits?
>
> Can you elaborate?

When you write to MSR_IA32_SYSENTER_EIP on AMD, that MSR will be  
directly passed through to the hardware (search for that MSR in  
svm.c). This is because SVM automatically writes the SYSENTER MSRs to  
the SYSENTER fields in the VMCB.

Now, AMD only implements 32 bits here (actually Intel simply extended  
it to 64 bit without coordinating with AMD). So if you use SYSENTER on  
an AMD VM now, only 32 bits of that IP will be stored in the actual  
hardware's MSR, thus in the VMCB, which means that you won't be able  
to boot 64-bit Linux system with "GenuineIntel" vendor specification  
on SVM.

I just wanted to hint you on that, so you can implement it in the  
emulation that you'll hopefully post to the list even though it's  
preliminary, so everyone can contribute to and comment on it.

Alex

Re: Cross vendor migration ideas

[Amit Shah]

* On Thursday 13 Nov 2008 19:08:14 Alexander Graf wrote:
> On 13.11.2008, at 05:35, Amit Shah wrote:
> > * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote:
> >> On 12.11.2008, at 17:52, Amit Shah wrote:
> >>> Hi Alex,
> >>>
> >>> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote:
> >>>> Hi,
> >>>>
> >>>> I was thinking a bit about cross vendor migration recently and
> >>>> since
> >>>> we're doing open source development, I figured it might be a good
> >>>> idea
> >>>> to talk to everyone about this.
> >>>>
> >>>> So why are we having a problem?
> >>>>
> >>>> In normal operation we don't. If we're running a 32-bit kernel, we
> >>>> can
> >>>> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit
> >>>> kernel with 64-bit userspace, every CPU supports SYSCALL. At least
> >>>> Linux is being smart on this and does use exactly these two
> >>>> capabilities in these two cases.
> >>>> But if we're running in compat mode (64-bit kernel with 32-bit
> >>>> userspace), things differ. Intel supports only SYSENTER here, while
> >>>> AMD only supports SYSCALL. Both can still use int80.
> >>>>
> >>>> Operating systems detect usage of SYSCALL or SYSENTER pretty
> >>>> early on
> >>>> (Linux does this on vdso). So when we boot up on an Intel machine,
> >>>> Linux assumes that using SYSENTER in compat mode is fine. Migrating
> >>>> that machine to an AMD machine breaks this assumption though, since
> >>>> SYSENTER can't be used in compat mode.
> >>>> On LInux, this detection is based on the CPU vendor string. If
> >>>> Linux
> >>>> finds a "GenuineIntel", SYSENTER is used in compat mode, if it's
> >>>> "AuthenticAMD", SYSCALL is used and if none of these two is found,
> >>>> int80 is used.
> >>>>
> >>>> I tried modifying the vendor string, removed the "overwrite the
> >>>> vendor
> >>>> string with the native string" hack and things look like they work
> >>>> just fine with Linux.
> >>>>
> >>>> Unfortunately right now I don't have a 64-bit Windows installation
> >>>> around to check if that approach works there too, but if it does
> >>>> and
> >>>> no known OS breaks due to the invalid vendor string, we can just
> >>>> create our own virtual CPU string, no?
> >>>
> >>> qemu has an option for that, -cpu qemu64 IIRC. As long as we expose
> >>> practically correct cpuids and MSRs, this should be fine. I've not
> >>> tested
> >>> qemu64 with winxp x64 though. Also, last I knew, winxp x64
> >>> installation
> >>> didn't succeed with --no-kvm. qemu by default exposes an AMD CPU
> >>> type.
> >>
> >> I wasn't talking about CPUID features, but the vendor string. Qemu64
> >> provides the AuthenticAMD string, so we don't run into any issues I'm
> >> presuming.
> >
> > Right -- the thing is, with the default AuthenticAMD string, winp x64
> > installation fails. That has to be because of some missing cpuids.
> > That's one
> > of the drawbacks of exposing a well-known CPU type. I was suggesting
> > we
> > should try out the -cpu qemu64 CPU type since it exposes a non-
> > standard CPU
> > to see if guests and most userspace programs work fine without any
> > further
> > tweaking -- see the 'cons' below for why this might be a problem.
>
> I still don't really understand what you're trying to say - qemu64 is
> the default in KVM right now. You mean winxp64 installation doesn't

No, the default for KVM is the host CPU type.

> work as is and we should fix it? This has nothing to do with the
> migration problems, right?

Solutions shouldn't involve adding known regressions. If our default cpu type 
changes to one that renders some of the OSes we support right now to become 
nonfunctional, such changes won't be accepted. Of course, we can improve the 
qemu64 cpu type to ensure the popular OS types work properly at the least.

> >>> There are pros and cons to expose a custom vendor ID:
> >>>
> >>> pros:
> >>> - We don't need to have all the cpuid features exposed which are
> >>> expected of a
> >>> physically available CPU in the market, for example, badly-coded
> >>> applications
> >>> might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or
> >>> not, not
> >>> exposing what's actually available on every C2D out there is bad.
> >>>
> >>> cons:
> >>> - To expose the "correct" set of feature bits for a known processor,
> >>> we also
> >>> need to check the family/model/stepping to support the exact same
> >>> feature
> >>> bits that were present in the CPU.
> >>> - We might not get some optimizations that OSes might have based on
> >>> CPU type,
> >>> even if the host CPU qualifies for such optimizations
> >>> - Standard programs like benchmarking tools, etc., might fail if
> >>> they depend
> >>> on the vendor string for their functionality
> >>>
> >>> For 32-bit guests, I think exposing a pentium4 or Athlon CPU type
> >>> should be
> >>> fine. For 64-bit guests, the newer the better.
> >>
> >> Well, we could create different CPU definitions:
> >>
> >> - migration safe (do what is safe for migration)
> >
> > There are multiple ways of approaching this: peg to a least-known
> > good CPU
> > type, all of whose instructions will work on processors from both
> > the major
> > vendors. However, you never know how the server pools change and
> > you'd want
> > to upgrade the CPU type once you know the CPUs that are installed in
> > servers.
> > This has to be dynamic and the management application has to take
> > care of
> > exposing a CPU that's of a "safe" type for the particular server
> > pool. We
> > have to provide ways to mask off CPUID bits as requested by the
> > management
> > application. (Each server sends its cpuid to the management
> > application,
> > which calculates the safest bits and then conveys this to each
> > server before
> > starting a VM.)
>
> IMHO we shouldn't really start to be smart here. There's only so much
> benefit in using the least common dominator between all CPUs in the
> datacenter vs. using the least common dominator between all possible
> CPUs. You'll basically end up enabling some newer SSE instructions.

I'm just saying the management application will do it. So it'll be local to 
the server pool the management app caters to. Not a common denominator for 
all deployments.

> So I don't think we need to go through the hassle of making this
> dynamic. If you want to migrate your machines - use the migrate
> preset. That won't give you the 150% speed boost on video encoding,
> but should not really be any slower on normal workloads. It does make
> things a lot more transparent to us and the admin of a network though,
> because you know what you'll end up with "-cpu migration".

We hardly know what uses KVM will be put to. Server virtualisation, desktop 
virtualisation, combination, what not. If we provide with the flexibility to 
the admin to tune as necessary, it's not a bad option at all. All the 
userspace needs is one tool that can calculate the max. features supported by 
the current CPU and send it over to the management app when asked for. The 
management app does the rest. KVM is not involved at all.

> >> - CPU specific (like a Core2Duo, necessary to run Mac OS X)
> >
> > This doesn't need any more work -- we already have the ability to
> > select CPU
> > types. If the management application has knowledge of the kind of OS
> > being
> > installed in a VM (which these days is true), exposing a Core2Duo
> > for a
> > Mac-based OS isn't difficult.
>
> There is no sysenter emulation for IA-32e on AMD yet, right? That's
> the only issue I see here and your emulation patch should address that.

As I've mentioned before, I've not yet been able to test my patch because I've 
not found the sysenter/sysexit calls being used at all. It's included at the 
end of this mail for review; hopefully someone finds a use-case and we can 
take it forward.

> >> - host (fastest possible, but no migration)
> >
> > This should be the default.
>
> I'm not sure. Either host or migration should be the default. This

I'm suggesting that 'host' should be default. Where do we disagree?

> actually depends on the workload you have on KVM. For servers you'll
> probably want to have migration be the default. For desktop usage it's
> host. I can't think of a way we can be smart about that on the KVM
> level.

For individual runs from the command line, I'd prefer the host to be the 
default. For a wider deployment, the management app will set the defaults as 
necessary (admin-chosen).

> >> I don't think we could find one definition that fits all, so the user
> >> would have to define what the usage pattern will be.
> >>
> >>>> I'd love to hear comments and suggestions on this and hope we'll
> >>>> end
> >>>> up in a fruitful discussion on how to improve the current
> >>>> situation.
> >>>
> >>> I have a patch ready for emulating sysenter/sysexit on AMD systems
> >>> (needs
> >>> testing). Patching the guest was an option that was discouraged; I
> >>> had a hack
> >>> ready but it was quickly shelved (again, untested).
> >>
> >> That sounds useful for misbehaving guests or cases I haven't thought
> >> of yet. Are you sure you're intercepting the SYSENTER MSRs on AMD, so
> >> you don't end up only getting 32 bits?
> >
> > Can you elaborate?
>
> When you write to MSR_IA32_SYSENTER_EIP on AMD, that MSR will be
> directly passed through to the hardware (search for that MSR in
> svm.c). This is because SVM automatically writes the SYSENTER MSRs to
> the SYSENTER fields in the VMCB.

My patch just handles the case when a sysenter is attempted on a system which 
doesn't have that instruction. So I just emulate it. Accessing the MSR and 
setting values is done at boot-time by the OS, and any migrations at that 
instant is a corner case and not too critical.

Now, the patch.

>From e1b760d8e596811081c282484621b49c674f1c22 Mon Sep 17 00:00:00 2001
From: Amit Shah <amit.shah@redhat.com>
Date: Wed, 12 Nov 2008 11:31:05 +0530
Subject: [PATCH] KVM: SVM: Emulate SYSENTER/SYSEXIT on AMD processors

This patch enables emulation of the sysenter/sysexit instructions in
AMD long mode. This will enable a guest started on an Intel machine to
be migrated to an AMD machine.

Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 arch/x86/kvm/svm.c         |   13 ++++
 arch/x86/kvm/x86_emulate.c |  137 
+++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 149 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index f0ad4d4..4e6e1dc 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -1155,6 +1155,19 @@ static int vmmcall_interception(struct vcpu_svm *svm, 
struct kvm_run *kvm_run)
 static int invalid_op_interception(struct vcpu_svm *svm,
 				   struct kvm_run *kvm_run)
 {
+	/*
+	 * If we're running in long mode on x86_64, check if we can
+	 * emulate sysenter / sysexit
+	 */
+	if (!is_long_mode(&svm->vcpu))
+		goto out;
+
+	if (emulate_instruction(&svm->vcpu, NULL, 0, 0, 0) == EMULATE_DONE) {
+		/* We could emulate it. */
+		return 1;
+	}
+
+ out:
 	kvm_queue_exception(&svm->vcpu, UD_VECTOR);
 	return 1;
 }
diff --git a/arch/x86/kvm/x86_emulate.c b/arch/x86/kvm/x86_emulate.c
index 8f60ace..c8afd20 100644
--- a/arch/x86/kvm/x86_emulate.c
+++ b/arch/x86/kvm/x86_emulate.c
@@ -205,7 +205,9 @@ static u16 twobyte_table[256] = {
 	ModRM | ImplicitOps, ModRM, ModRM | ImplicitOps, ModRM, 0, 0, 0, 0,
 	0, 0, 0, 0, 0, 0, 0, 0,
 	/* 0x30 - 0x3F */
-	ImplicitOps, 0, ImplicitOps, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	ImplicitOps, 0, ImplicitOps, 0,
+	ImplicitOps, ImplicitOps, 0, 0,
+	0, 0, 0, 0, 0, 0, 0, 0,
 	/* 0x40 - 0x47 */
 	DstReg | SrcMem | ModRM | Mov, DstReg | SrcMem | ModRM | Mov,
 	DstReg | SrcMem | ModRM | Mov, DstReg | SrcMem | ModRM | Mov,
@@ -305,8 +307,11 @@ static u16 group2_table[] = {
 };
 
 /* EFLAGS bit definitions. */
+#define EFLG_VM (1<<17)
+#define EFLG_RF (1<<16)
 #define EFLG_OF (1<<11)
 #define EFLG_DF (1<<10)
+#define EFLG_IF (1<<9)
 #define EFLG_SF (1<<7)
 #define EFLG_ZF (1<<6)
 #define EFLG_AF (1<<4)
@@ -1959,6 +1964,136 @@ twobyte_insn:
 		rc = X86EMUL_CONTINUE;
 		c->dst.type = OP_NONE;
 		break;
+	case 0x34: { /* sysenter */
+		/* Vol 2b */
+		unsigned long cr0 = ctxt->vcpu->arch.cr0;
+		struct kvm_segment cs, ss;
+		u64 data;
+
+		if (cr0 & X86_CR0_PE) {
+			kvm_inject_gp(ctxt->vcpu, 0);
+			goto cannot_emulate;
+		}
+
+		kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_CS, &data);
+		if (!(data & 0xFFFC)) {
+			kvm_inject_gp(ctxt->vcpu, 0);
+			goto cannot_emulate;
+		}
+
+		ctxt->eflags &= ~(EFLG_VM | EFLG_IF | EFLG_RF);
+
+		kvm_x86_ops->get_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
+		cs.selector = (__u16) data;
+		cs.base = 0;
+		cs.limit = 0xfffff;
+		cs.g = 1;
+		cs.s = 1;
+		cs.type = 0x0b;
+		cs.db = 1;
+		cs.dpl = 0;
+		cs.selector &= ~SELECTOR_RPL_MASK;
+		cs.present = 1;
+		/* The CPL should be set to 0 */
+
+		if (ctxt->mode == X86EMUL_MODE_PROT64) {
+			cs.l = 1;
+			cs.limit = 0xffffffff;
+		}
+
+		ss.selector = cs.selector + 8;
+		ss.base = 0;
+		ss.limit = 0xfffff;
+		ss.g = 1;
+		ss.s = 1;
+		ss.type = 0x03;
+		ss.db = 1;
+		ss.dpl = 0;
+		ss.selector &= ~SELECTOR_RPL_MASK;
+		ss.present = 1;
+		if (ctxt->mode == X86EMUL_MODE_PROT64) {
+			ss.limit = 0xffffffff;
+		}
+
+		kvm_x86_ops->set_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
+		kvm_x86_ops->set_segment(ctxt->vcpu, &ss, VCPU_SREG_SS);
+
+		kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_EIP, &data);
+		c->eip = data;
+
+		kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_ESP, &data);
+		c->regs[VCPU_REGS_RSP] = data;
+
+		goto writeback;
+		break;
+	}
+	case 0x35: { /* sysexit */
+		/* Vol 2b */
+		u64 data;
+		unsigned long cr0 = ctxt->vcpu->arch.cr0;
+		struct kvm_segment cs, ss;
+
+		if (cr0 & X86_CR0_PE) {
+			kvm_inject_gp(ctxt->vcpu, 0);
+			goto cannot_emulate;
+		}
+
+		kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_CS, &data);
+		if (!(data & 0xFFFC) ||
+		    ((ctxt->mode == X86EMUL_MODE_PROT64) && !data)) {
+			kvm_inject_gp(ctxt->vcpu, 0);
+			goto cannot_emulate;
+		}
+
+		/* Check if CPL is 0. If not, inject_gp */
+
+		kvm_x86_ops->get_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
+		cs.selector = (u16)(data +
+				    (ctxt->mode == X86EMUL_MODE_PROT64 ? 32 : 16));
+		cs.base = 0;
+		cs.limit = 0xfffff;
+		cs.g = 1;
+		cs.s = 1;
+		cs.type = 0x0b;
+		cs.db = 1;
+		cs.dpl = 3;
+		cs.selector |= SELECTOR_RPL_MASK;
+		cs.present = 1;
+		cs.l = 0; /* For return to compatibility mode */
+		/* The CPL should be set to 3 */
+
+		if (ctxt->mode == X86EMUL_MODE_PROT64) {
+			cs.l = 1;
+			/* The manual doesn't talk about CS limit */
+		}
+
+		ss.selector = cs.selector +
+			(ctxt->mode == X86EMUL_MODE_PROT64 ? 16 : 8);
+		ss.base = 0;
+		ss.limit = 0xfffff;
+		ss.g = 1;
+		ss.s = 1;
+		ss.type = 0x03;
+		ss.db = 1;
+		ss.dpl = 3;
+		ss.selector |= SELECTOR_RPL_MASK;
+		ss.present = 1;
+		if (ctxt->mode == X86EMUL_MODE_PROT64) {
+			ss.base = 0;
+			ss.limit = 0xffffffff;
+		}
+
+		kvm_x86_ops->set_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
+		kvm_x86_ops->set_segment(ctxt->vcpu, &ss, VCPU_SREG_SS);
+
+		c->eip = ctxt->vcpu->arch.regs[VCPU_REGS_RDX];
+		c->regs[VCPU_REGS_RSP] = c->regs[VCPU_REGS_RCX];
+
+		/* TODO: Check if rip and rsp are canonical. inject_gp() if not */
+
+		goto writeback;
+		break;
+	}
 	case 0x40 ... 0x4f:	/* cmov */
 		c->dst.val = c->dst.orig_val = c->src.val;
 		if (!test_cc(c->b, ctxt->eflags))
-- 
1.5.4.3

Re: Cross vendor migration ideas

[Nitin A Kamble]

[-- Attachment #1: Type: text/plain, Size: 19221 bytes --]

Amit, Alex,
  Please see my comments bellow.

Avi, 
  Please have a look at the patches, and let me know the parts you think
can be done better.

On Fri, 2008-11-14 at 06:07 -0700, Amit Shah wrote:
> * On Thursday 13 Nov 2008 19:08:14 Alexander Graf wrote:
> > On 13.11.2008, at 05:35, Amit Shah wrote:
> > > * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote:
> > >> On 12.11.2008, at 17:52, Amit Shah wrote:
> > >>> Hi Alex,
> > >>>
> > >>> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I was thinking a bit about cross vendor migration recently and
> > >>>> since
> > >>>> we're doing open source development, I figured it might be a good
> > >>>> idea
> > >>>> to talk to everyone about this.
> > >>>>
> > >>>> So why are we having a problem?
> > >>>>
> > >>>> In normal operation we don't. If we're running a 32-bit kernel, we
> > >>>> can
> > >>>> use SYSENTER to jump from kernel<->userspace. If we're on a 64-bit
> > >>>> kernel with 64-bit userspace, every CPU supports SYSCALL. At least
> > >>>> Linux is being smart on this and does use exactly these two
> > >>>> capabilities in these two cases.
> > >>>> But if we're running in compat mode (64-bit kernel with 32-bit
> > >>>> userspace), things differ. Intel supports only SYSENTER here, while
> > >>>> AMD only supports SYSCALL. Both can still use int80.
> > >>>>
> > >>>> Operating systems detect usage of SYSCALL or SYSENTER pretty
> > >>>> early on
> > >>>> (Linux does this on vdso). So when we boot up on an Intel machine,
> > >>>> Linux assumes that using SYSENTER in compat mode is fine. Migrating
> > >>>> that machine to an AMD machine breaks this assumption though, since
> > >>>> SYSENTER can't be used in compat mode.
> > >>>> On LInux, this detection is based on the CPU vendor string. If
> > >>>> Linux
> > >>>> finds a "GenuineIntel", SYSENTER is used in compat mode, if it's
> > >>>> "AuthenticAMD", SYSCALL is used and if none of these two is found,
> > >>>> int80 is used.
> > >>>>
> > >>>> I tried modifying the vendor string, removed the "overwrite the
> > >>>> vendor
> > >>>> string with the native string" hack and things look like they work
> > >>>> just fine with Linux.
> > >>>>
> > >>>> Unfortunately right now I don't have a 64-bit Windows installation
> > >>>> around to check if that approach works there too, but if it does
> > >>>> and
> > >>>> no known OS breaks due to the invalid vendor string, we can just
> > >>>> create our own virtual CPU string, no?
> > >>>
> > >>> qemu has an option for that, -cpu qemu64 IIRC. As long as we expose
> > >>> practically correct cpuids and MSRs, this should be fine. I've not
> > >>> tested
> > >>> qemu64 with winxp x64 though. Also, last I knew, winxp x64
> > >>> installation
> > >>> didn't succeed with --no-kvm. qemu by default exposes an AMD CPU
> > >>> type.
> > >>
> > >> I wasn't talking about CPUID features, but the vendor string. Qemu64
> > >> provides the AuthenticAMD string, so we don't run into any issues I'm
> > >> presuming.
> > >
> > > Right -- the thing is, with the default AuthenticAMD string, winp x64
> > > installation fails. That has to be because of some missing cpuids.
> > > That's one
> > > of the drawbacks of exposing a well-known CPU type. I was suggesting
> > > we
> > > should try out the -cpu qemu64 CPU type since it exposes a non-
> > > standard CPU
> > > to see if guests and most userspace programs work fine without any
> > > further
> > > tweaking -- see the 'cons' below for why this might be a problem.
> >
> > I still don't really understand what you're trying to say - qemu64 is
> > the default in KVM right now. You mean winxp64 installation doesn't


> No, the default for KVM is the host CPU type.

Amit, Aliex is correct. the default cpu for kvm is qemu64 not the host.
I have sent the patches to add an options -cpu host. Some of the patches
are gone in, But All the patches are not in yet. Also my patches does
not make the host option as default. I have attached the remaining two
patches. 

Alex, can you try these patches with "-cpu host" option and see if you
can get the host vendor string in the guest for AMD box. I have already
tested it on the latest Intel system.

> 
> > work as is and we should fix it? This has nothing to do with the
> > migration problems, right?
> 
> Solutions shouldn't involve adding known regressions. If our default cpu type
> changes to one that renders some of the OSes we support right now to become
> nonfunctional, such changes won't be accepted. Of course, we can improve the
> qemu64 cpu type to ensure the popular OS types work properly at the least.
> 
> > >>> There are pros and cons to expose a custom vendor ID:
> > >>>
> > >>> pros:
> > >>> - We don't need to have all the cpuid features exposed which are
> > >>> expected of a
> > >>> physically available CPU in the market, for example, badly-coded
> > >>> applications
> > >>> might crash if we don't have SSSE3 on a Core2Duo. But badly-coded or
> > >>> not, not
> > >>> exposing what's actually available on every C2D out there is bad.
> > >>>
> > >>> cons:
> > >>> - To expose the "correct" set of feature bits for a known processor,
> > >>> we also
> > >>> need to check the family/model/stepping to support the exact same
> > >>> feature
> > >>> bits that were present in the CPU.
> > >>> - We might not get some optimizations that OSes might have based on
> > >>> CPU type,
> > >>> even if the host CPU qualifies for such optimizations
> > >>> - Standard programs like benchmarking tools, etc., might fail if
> > >>> they depend
> > >>> on the vendor string for their functionality
> > >>>
> > >>> For 32-bit guests, I think exposing a pentium4 or Athlon CPU type
> > >>> should be
> > >>> fine. For 64-bit guests, the newer the better.
> > >>
> > >> Well, we could create different CPU definitions:
> > >>
> > >> - migration safe (do what is safe for migration)
> > >
> > > There are multiple ways of approaching this: peg to a least-known
> > > good CPU
> > > type, all of whose instructions will work on processors from both
> > > the major
> > > vendors. However, you never know how the server pools change and
> > > you'd want
> > > to upgrade the CPU type once you know the CPUs that are installed in
> > > servers.
> > > This has to be dynamic and the management application has to take
> > > care of
> > > exposing a CPU that's of a "safe" type for the particular server
> > > pool. We
> > > have to provide ways to mask off CPUID bits as requested by the
> > > management
> > > application. (Each server sends its cpuid to the management
> > > application,
> > > which calculates the safest bits and then conveys this to each
> > > server before
> > > starting a VM.)
> >
> > IMHO we shouldn't really start to be smart here. There's only so much
> > benefit in using the least common dominator between all CPUs in the
> > datacenter vs. using the least common dominator between all possible
> > CPUs. You'll basically end up enabling some newer SSE instructions.
> 
> I'm just saying the management application will do it. So it'll be local to
> the server pool the management app caters to. Not a common denominator for
> all deployments.
> 
> > So I don't think we need to go through the hassle of making this
> > dynamic. If you want to migrate your machines - use the migrate
> > preset. That won't give you the 150% speed boost on video encoding,
> > but should not really be any slower on normal workloads. It does make
> > things a lot more transparent to us and the admin of a network though,
> > because you know what you'll end up with "-cpu migration".
> 
> We hardly know what uses KVM will be put to. Server virtualisation, desktop
> virtualisation, combination, what not. If we provide with the flexibility to
> the admin to tune as necessary, it's not a bad option at all. All the
> userspace needs is one tool that can calculate the max. features supported by
> the current CPU and send it over to the management app when asked for. The
> management app does the rest. KVM is not involved at all.
> 
> > >> - CPU specific (like a Core2Duo, necessary to run Mac OS X)
> > >
> > > This doesn't need any more work -- we already have the ability to
> > > select CPU
> > > types. If the management application has knowledge of the kind of OS
> > > being
> > > installed in a VM (which these days is true), exposing a Core2Duo
> > > for a
> > > Mac-based OS isn't difficult.
> >
> > There is no sysenter emulation for IA-32e on AMD yet, right? That's
> > the only issue I see here and your emulation patch should address that.
> 
> As I've mentioned before, I've not yet been able to test my patch because I've
> not found the sysenter/sysexit calls being used at all. It's included at the
> end of this mail for review; hopefully someone finds a use-case and we can
> take it forward.
> 
> > >> - host (fastest possible, but no migration)
> > >
> > > This should be the default.
> >
> > I'm not sure. Either host or migration should be the default. This
> 
> I'm suggesting that 'host' should be default. Where do we disagree?

Well I believe the corss-architecture migration is not a common case. So
the host should be default in IMHO too. But I will not press for it.

> 
> > actually depends on the workload you have on KVM. For servers you'll
> > probably want to have migration be the default. For desktop usage it's
> > host. I can't think of a way we can be smart about that on the KVM
> > level.
> 
> For individual runs from the command line, I'd prefer the host to be the
> default. For a wider deployment, the management app will set the defaults as
> necessary (admin-chosen).
> 
> > >> I don't think we could find one definition that fits all, so the user
> > >> would have to define what the usage pattern will be.
> > >>
> > >>>> I'd love to hear comments and suggestions on this and hope we'll
> > >>>> end
> > >>>> up in a fruitful discussion on how to improve the current
> > >>>> situation.
> > >>>
> > >>> I have a patch ready for emulating sysenter/sysexit on AMD systems
> > >>> (needs
> > >>> testing). Patching the guest was an option that was discouraged; I
> > >>> had a hack
> > >>> ready but it was quickly shelved (again, untested).
> > >>
> > >> That sounds useful for misbehaving guests or cases I haven't thought
> > >> of yet. Are you sure you're intercepting the SYSENTER MSRs on AMD, so
> > >> you don't end up only getting 32 bits?
> > >
> > > Can you elaborate?
> >
> > When you write to MSR_IA32_SYSENTER_EIP on AMD, that MSR will be
> > directly passed through to the hardware (search for that MSR in
> > svm.c). This is because SVM automatically writes the SYSENTER MSRs to
> > the SYSENTER fields in the VMCB.
> 
> My patch just handles the case when a sysenter is attempted on a system which
> doesn't have that instruction. So I just emulate it. Accessing the MSR and
> setting values is done at boot-time by the OS, and any migrations at that
> instant is a corner case and not too critical.
> 
> Now, the patch.
> 
> From e1b760d8e596811081c282484621b49c674f1c22 Mon Sep 17 00:00:00 2001
> From: Amit Shah <amit.shah@redhat.com>
> Date: Wed, 12 Nov 2008 11:31:05 +0530
> Subject: [PATCH] KVM: SVM: Emulate SYSENTER/SYSEXIT on AMD processors
> 
> This patch enables emulation of the sysenter/sysexit instructions in
> AMD long mode. This will enable a guest started on an Intel machine to
> be migrated to an AMD machine.
> 
> Signed-off-by: Amit Shah <amit.shah@redhat.com>
> ---
>  arch/x86/kvm/svm.c         |   13 ++++
>  arch/x86/kvm/x86_emulate.c |  137
> +++++++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 149 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
> index f0ad4d4..4e6e1dc 100644
> --- a/arch/x86/kvm/svm.c
> +++ b/arch/x86/kvm/svm.c
> @@ -1155,6 +1155,19 @@ static int vmmcall_interception(struct vcpu_svm *svm,
> struct kvm_run *kvm_run)
>  static int invalid_op_interception(struct vcpu_svm *svm,
>                                    struct kvm_run *kvm_run)
>  {
> +       /*
> +        * If we're running in long mode on x86_64, check if we can
> +        * emulate sysenter / sysexit
> +        */
> +       if (!is_long_mode(&svm->vcpu))
> +               goto out;
> +
> +       if (emulate_instruction(&svm->vcpu, NULL, 0, 0, 0) == EMULATE_DONE) {
> +               /* We could emulate it. */
> +               return 1;
> +       }
> +
> + out:
>         kvm_queue_exception(&svm->vcpu, UD_VECTOR);
>         return 1;
>  }
> diff --git a/arch/x86/kvm/x86_emulate.c b/arch/x86/kvm/x86_emulate.c
> index 8f60ace..c8afd20 100644
> --- a/arch/x86/kvm/x86_emulate.c
> +++ b/arch/x86/kvm/x86_emulate.c
> @@ -205,7 +205,9 @@ static u16 twobyte_table[256] = {
>         ModRM | ImplicitOps, ModRM, ModRM | ImplicitOps, ModRM, 0, 0, 0, 0,
>         0, 0, 0, 0, 0, 0, 0, 0,
>         /* 0x30 - 0x3F */
> -       ImplicitOps, 0, ImplicitOps, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> +       ImplicitOps, 0, ImplicitOps, 0,
> +       ImplicitOps, ImplicitOps, 0, 0,
> +       0, 0, 0, 0, 0, 0, 0, 0,
>         /* 0x40 - 0x47 */
>         DstReg | SrcMem | ModRM | Mov, DstReg | SrcMem | ModRM | Mov,
>         DstReg | SrcMem | ModRM | Mov, DstReg | SrcMem | ModRM | Mov,
> @@ -305,8 +307,11 @@ static u16 group2_table[] = {
>  };
> 
>  /* EFLAGS bit definitions. */
> +#define EFLG_VM (1<<17)
> +#define EFLG_RF (1<<16)
>  #define EFLG_OF (1<<11)
>  #define EFLG_DF (1<<10)
> +#define EFLG_IF (1<<9)
>  #define EFLG_SF (1<<7)
>  #define EFLG_ZF (1<<6)
>  #define EFLG_AF (1<<4)
> @@ -1959,6 +1964,136 @@ twobyte_insn:
>                 rc = X86EMUL_CONTINUE;
>                 c->dst.type = OP_NONE;
>                 break;
> +       case 0x34: { /* sysenter */
> +               /* Vol 2b */
> +               unsigned long cr0 = ctxt->vcpu->arch.cr0;
> +               struct kvm_segment cs, ss;
> +               u64 data;
> +
> +               if (cr0 & X86_CR0_PE) {
> +                       kvm_inject_gp(ctxt->vcpu, 0);
> +                       goto cannot_emulate;
> +               }
> +
> +               kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_CS, &data);
> +               if (!(data & 0xFFFC)) {
> +                       kvm_inject_gp(ctxt->vcpu, 0);
> +                       goto cannot_emulate;
> +               }
> +
> +               ctxt->eflags &= ~(EFLG_VM | EFLG_IF | EFLG_RF);
> +
> +               kvm_x86_ops->get_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
> +               cs.selector = (__u16) data;
> +               cs.base = 0;
> +               cs.limit = 0xfffff;
> +               cs.g = 1;
> +               cs.s = 1;
> +               cs.type = 0x0b;
> +               cs.db = 1;
> +               cs.dpl = 0;
> +               cs.selector &= ~SELECTOR_RPL_MASK;
> +               cs.present = 1;
> +               /* The CPL should be set to 0 */
> +
> +               if (ctxt->mode == X86EMUL_MODE_PROT64) {
> +                       cs.l = 1;
> +                       cs.limit = 0xffffffff;
> +               }
> +
> +               ss.selector = cs.selector + 8;
> +               ss.base = 0;
> +               ss.limit = 0xfffff;
> +               ss.g = 1;
> +               ss.s = 1;
> +               ss.type = 0x03;
> +               ss.db = 1;
> +               ss.dpl = 0;
> +               ss.selector &= ~SELECTOR_RPL_MASK;
> +               ss.present = 1;
> +               if (ctxt->mode == X86EMUL_MODE_PROT64) {
> +                       ss.limit = 0xffffffff;
> +               }
> +
> +               kvm_x86_ops->set_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
> +               kvm_x86_ops->set_segment(ctxt->vcpu, &ss, VCPU_SREG_SS);
> +
> +               kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_EIP, &data);
> +               c->eip = data;
> +
> +               kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_ESP, &data);
> +               c->regs[VCPU_REGS_RSP] = data;
> +
> +               goto writeback;
> +               break;
> +       }
> +       case 0x35: { /* sysexit */
> +               /* Vol 2b */
> +               u64 data;
> +               unsigned long cr0 = ctxt->vcpu->arch.cr0;
> +               struct kvm_segment cs, ss;
> +
> +               if (cr0 & X86_CR0_PE) {
> +                       kvm_inject_gp(ctxt->vcpu, 0);
> +                       goto cannot_emulate;
> +               }
> +
> +               kvm_x86_ops->get_msr(ctxt->vcpu, MSR_IA32_SYSENTER_CS, &data);
> +               if (!(data & 0xFFFC) ||
> +                   ((ctxt->mode == X86EMUL_MODE_PROT64) && !data)) {
> +                       kvm_inject_gp(ctxt->vcpu, 0);
> +                       goto cannot_emulate;
> +               }
> +
> +               /* Check if CPL is 0. If not, inject_gp */
> +
> +               kvm_x86_ops->get_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
> +               cs.selector = (u16)(data +
> +                                   (ctxt->mode == X86EMUL_MODE_PROT64 ? 32 : 16));
> +               cs.base = 0;
> +               cs.limit = 0xfffff;
> +               cs.g = 1;
> +               cs.s = 1;
> +               cs.type = 0x0b;
> +               cs.db = 1;
> +               cs.dpl = 3;
> +               cs.selector |= SELECTOR_RPL_MASK;
> +               cs.present = 1;
> +               cs.l = 0; /* For return to compatibility mode */
> +               /* The CPL should be set to 3 */
> +
> +               if (ctxt->mode == X86EMUL_MODE_PROT64) {
> +                       cs.l = 1;
> +                       /* The manual doesn't talk about CS limit */
> +               }
> +
> +               ss.selector = cs.selector +
> +                       (ctxt->mode == X86EMUL_MODE_PROT64 ? 16 : 8);
> +               ss.base = 0;
> +               ss.limit = 0xfffff;
> +               ss.g = 1;
> +               ss.s = 1;
> +               ss.type = 0x03;
> +               ss.db = 1;
> +               ss.dpl = 3;
> +               ss.selector |= SELECTOR_RPL_MASK;
> +               ss.present = 1;
> +               if (ctxt->mode == X86EMUL_MODE_PROT64) {
> +                       ss.base = 0;
> +                       ss.limit = 0xffffffff;
> +               }
> +
> +               kvm_x86_ops->set_segment(ctxt->vcpu, &cs, VCPU_SREG_CS);
> +               kvm_x86_ops->set_segment(ctxt->vcpu, &ss, VCPU_SREG_SS);
> +
> +               c->eip = ctxt->vcpu->arch.regs[VCPU_REGS_RDX];
> +               c->regs[VCPU_REGS_RSP] = c->regs[VCPU_REGS_RCX];
> +
> +               /* TODO: Check if rip and rsp are canonical. inject_gp() if not */
> +
> +               goto writeback;
> +               break;
> +       }
>         case 0x40 ... 0x4f:     /* cmov */
>                 c->dst.val = c->dst.orig_val = c->src.val;
>                 if (!test_cc(c->b, ctxt->eflags))
> --
> 1.5.4.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Thanks & Regards,
Nitin
Open Source Technology Center, Intel Corporation
-----------------------------------------------------------------
The mind is like a parachute; it works much better when it's open

[-- Attachment #2: kvm_kernel_cpuid_patch3.diff --]
[-- Type: text/x-patch, Size: 2656 bytes --]

commit 70e4e65bc591eb9cf25c1cbc0d16b2cbdb089a6f
Author: Nitin A Kamble <nitin.a.kamble@intel.com>
Date:   Wed Nov 5 16:17:46 2008 -0800

    Change the ioctl KVM_GET_SUPPORTED_CPUID, such that it will return the
    no of entries in the list when requested no of entries (nent) is 0.
    
    Also add another KVM_CHECK_EXTENSION, KVM_CAP_CPUID_SIZER to determine
    if the running kernel supports the above changed ABI.
    
    Signed-Off-By: Nitin A Kamble <nitin.a.kamble@intel.com>

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 09e6c56..e50db11 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -86,7 +86,7 @@
 #define KVM_NUM_MMU_PAGES (1 << KVM_MMU_HASH_SHIFT)
 #define KVM_MIN_FREE_MMU_PAGES 5
 #define KVM_REFILL_PAGES 25
-#define KVM_MAX_CPUID_ENTRIES 40
+#define KVM_MAX_CPUID_ENTRIES 100
 #define KVM_NR_FIXED_MTRR_REGION 88
 #define KVM_NR_VAR_MTRR 8
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index bf7461b..52e6207 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -969,6 +969,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_NOP_IO_DELAY:
 	case KVM_CAP_MP_STATE:
 	case KVM_CAP_SYNC_MMU:
+	case KVM_CAP_CPUID_SIZER:
 		r = 1;
 		break;
 	case KVM_CAP_COALESCED_MMIO:
@@ -1303,10 +1304,14 @@ static int kvm_dev_ioctl_get_supported_cpuid(struct kvm_cpuid2 *cpuid,
 {
 	struct kvm_cpuid_entry2 *cpuid_entries;
 	int limit, nent = 0, r = -E2BIG;
+	int sizer = 0;
 	u32 func;
 
-	if (cpuid->nent < 1)
-		goto out;
+	if (cpuid->nent == 0) {
+		sizer = 1;
+		cpuid->nent = KVM_MAX_CPUID_ENTRIES; 
+        }
+
 	r = -ENOMEM;
 	cpuid_entries = vmalloc(sizeof(struct kvm_cpuid_entry2) * cpuid->nent);
 	if (!cpuid_entries)
@@ -1327,9 +1332,11 @@ static int kvm_dev_ioctl_get_supported_cpuid(struct kvm_cpuid2 *cpuid,
 		do_cpuid_ent(&cpuid_entries[nent], func, 0,
 			       &nent, cpuid->nent);
 	r = -EFAULT;
-	if (copy_to_user(entries, cpuid_entries,
+	if (!sizer) {
+		if (copy_to_user(entries, cpuid_entries,
 			nent * sizeof(struct kvm_cpuid_entry2)))
-		goto out_free;
+			goto out_free;
+        }
 	cpuid->nent = nent;
 	r = 0;
 
diff --git a/include/linux/kvm.h b/include/linux/kvm.h
index 44fd7fa..d4cb8b1 100644
--- a/include/linux/kvm.h
+++ b/include/linux/kvm.h
@@ -392,6 +392,9 @@ struct kvm_trace_rec {
 #endif
 #define KVM_CAP_IOMMU 18
 #define KVM_CAP_NMI 19
+#define KVM_CAP_CPUID_SIZER 20 /* return of 1 means the KVM_GET_SUPPORTED_CPUID */
+                               /* ioctl will return the size of list when input */
+                               /* list size (nent) is 0 */
 
 /*
  * ioctls for VM fds

[-- Attachment #3: kvm_userspace_patch10.diff --]
[-- Type: text/x-patch, Size: 11297 bytes --]

diff --git a/libkvm/libkvm-x86.c b/libkvm/libkvm-x86.c
index a8cca15..7aafa20 100644
--- a/libkvm/libkvm-x86.c
+++ b/libkvm/libkvm-x86.c
@@ -379,6 +379,34 @@ int kvm_set_msrs(kvm_context_t kvm, int vcpu, struct kvm_msr_entry *msrs,
     return r;
 }
 
+/*
+ * Returns available host cpuid entries.  User must free.
+ */
+struct kvm_cpuid2 *kvm_get_host_cpuid_entries(kvm_context_t kvm)
+{
+	struct kvm_cpuid2 sizer, *cpuids;
+	int r, e;
+
+	sizer.nent = 0;
+	r = ioctl(kvm->fd, KVM_GET_SUPPORTED_CPUID, &sizer);
+	if (r == -1 && errno != E2BIG)
+		return NULL;
+	cpuids = malloc(sizeof *cpuids + sizer.nent * sizeof *cpuids->entries);
+	if (!cpuids) {
+		errno = ENOMEM;
+		return NULL;
+	}
+	cpuids->nent = sizer.nent;
+	r = ioctl(kvm->fd, KVM_GET_SUPPORTED_CPUID, cpuids);
+	if (r == -1) {
+		e = errno;
+		free(cpuids);
+		errno = e;
+		return NULL;
+	}
+	return cpuids;
+}
+
 static void print_seg(FILE *file, const char *name, struct kvm_segment *seg)
 {
     	fprintf(stderr,
@@ -458,9 +486,9 @@ __u64 kvm_get_cr8(kvm_context_t kvm, int vcpu)
 }
 
 int kvm_setup_cpuid(kvm_context_t kvm, int vcpu, int nent,
-		    struct kvm_cpuid_entry *entries)
+		    struct kvm_cpuid_entry2 *entries)
 {
-	struct kvm_cpuid *cpuid;
+	struct kvm_cpuid2 *cpuid;
 	int r;
 
 	cpuid = malloc(sizeof(*cpuid) + nent * sizeof(*entries));
@@ -469,7 +497,7 @@ int kvm_setup_cpuid(kvm_context_t kvm, int vcpu, int nent,
 
 	cpuid->nent = nent;
 	memcpy(cpuid->entries, entries, nent * sizeof(*entries));
-	r = ioctl(kvm->vcpu_fd[vcpu], KVM_SET_CPUID, cpuid);
+	r = ioctl(kvm->vcpu_fd[vcpu], KVM_SET_CPUID2, cpuid);
 
 	free(cpuid);
 	return r;
diff --git a/libkvm/libkvm.h b/libkvm/libkvm.h
index 423ce31..f84d524 100644
--- a/libkvm/libkvm.h
+++ b/libkvm/libkvm.h
@@ -27,6 +27,9 @@ typedef struct kvm_context *kvm_context_t;
 struct kvm_msr_list *kvm_get_msr_list(kvm_context_t);
 int kvm_get_msrs(kvm_context_t, int vcpu, struct kvm_msr_entry *msrs, int n);
 int kvm_set_msrs(kvm_context_t, int vcpu, struct kvm_msr_entry *msrs, int n);
+struct kvm_cpuid2 *kvm_get_host_cpuid_entries(kvm_context_t);
+void get_host_cpuid_entry(uint32_t function, uint32_t index, 
+                struct kvm_cpuid_entry2 * e);
 #endif
 
 /*!
@@ -374,7 +377,7 @@ int kvm_guest_debug(kvm_context_t, int vcpu, struct kvm_debug_guest *dbg);
  * \return 0 on success, or -errno on error
  */
 int kvm_setup_cpuid(kvm_context_t kvm, int vcpu, int nent,
-		    struct kvm_cpuid_entry *entries);
+		    struct kvm_cpuid_entry2 *entries);
 
 /*!
  * \brief Setting the number of shadow pages to be allocated to the vm
diff --git a/qemu/qemu-kvm-x86.c b/qemu/qemu-kvm-x86.c
index bf62e18..b776ebf 100644
--- a/qemu/qemu-kvm-x86.c
+++ b/qemu/qemu-kvm-x86.c
@@ -21,6 +21,7 @@
 #define MSR_IA32_TSC		0x10
 
 static struct kvm_msr_list *kvm_msr_list;
+static struct kvm_cpuid2 *kvm_host_cpuid_entries;
 extern unsigned int kvm_shadow_memory;
 extern kvm_context_t kvm_context;
 static int kvm_has_msr_star;
@@ -52,11 +53,17 @@ int kvm_arch_qemu_create_context(void)
 
     kvm_msr_list = kvm_get_msr_list(kvm_context);
     if (!kvm_msr_list)
-		return -1;
+        return -1;
+
     for (i = 0; i < kvm_msr_list->nmsrs; ++i)
-	if (kvm_msr_list->indices[i] == MSR_STAR)
-	    kvm_has_msr_star = 1;
-	return 0;
+        if (kvm_msr_list->indices[i] == MSR_STAR)
+        kvm_has_msr_star = 1;
+
+    kvm_host_cpuid_entries = kvm_get_host_cpuid_entries(kvm_context);
+    if (!kvm_host_cpuid_entries)
+        return -1;
+
+    return 0;
 }
 
 static void set_msr_entry(struct kvm_msr_entry *entry, uint32_t index,
@@ -476,13 +483,61 @@ static void host_cpuid(uint32_t function, uint32_t *eax, uint32_t *ebx,
 	*edx = vec[3];
 }
 
+void get_host_cpuid_entry(uint32_t function, uint32_t index, 
+                struct kvm_cpuid_entry2 * e)
+{
+    int i;
+    struct kvm_cpuid_entry2 *entries;
+
+    memset(e, 0, (sizeof *e));
+    e->function = function;
+    e->index = index;
+    
+    if (!kvm_host_cpuid_entries) 
+        return;
+
+    entries = kvm_host_cpuid_entries->entries;
+
+    for (i=0; i<kvm_host_cpuid_entries->nent; i++) {
+        struct kvm_cpuid_entry2 *ent = &entries[i];
+        if (ent->function != function)
+           continue; 
+        if ((ent->flags & KVM_CPUID_FLAG_SIGNIFCANT_INDEX) && 
+                (ent->index != index))
+           continue;
+        if ((ent->flags & KVM_CPUID_FLAG_STATEFUL_FUNC) &&
+                !(ent->flags & KVM_CPUID_FLAG_STATE_READ_NEXT))
+           continue;
+
+        memcpy(e, ent, sizeof (*e));
+
+        if (ent->flags & KVM_CPUID_FLAG_STATEFUL_FUNC) {
+                int j;
+		ent->flags &= ~KVM_CPUID_FLAG_STATE_READ_NEXT;
+                for (j=i+1; ; j=(j+1)%(kvm_host_cpuid_entries->nent)) {
+                    struct kvm_cpuid_entry2 *entj = &entries[j];
+                    if (entj->function == ent->function) {
+                        entj->flags |= KVM_CPUID_FLAG_STATE_READ_NEXT;
+                        break;
+                    }
+                }
+        }
+        break;
+    }
+}
 
-static void do_cpuid_ent(struct kvm_cpuid_entry *e, uint32_t function,
-			 CPUState *env)
+static void do_cpuid_ent(struct kvm_cpuid_entry2 *e, uint32_t function,
+			 uint32_t index, CPUState *env)
 {
+    if (env->cpuid_host_cpu) {
+        get_host_cpuid_entry(function, index, e);
+	return;
+    }
+    e->function = function;
+    e->index = index;
     env->regs[R_EAX] = function;
+    env->regs[R_ECX] = index;
     qemu_kvm_cpuid_on_env(env);
-    e->function = function;
     e->eax = env->regs[R_EAX];
     e->ebx = env->regs[R_EBX];
     e->ecx = env->regs[R_ECX];
@@ -521,6 +576,11 @@ static void do_cpuid_ent(struct kvm_cpuid_entry *e, uint32_t function,
     if (function == 1)
 	e->ecx |= (1u << 31);
 
+    if ((function == 4) || (function == 0xb)) 
+       e->flags = KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
+    else
+       e->flags = 0;
+
     // 3dnow isn't properly emulated yet
     if (function == 0x80000001)
 	e->edx &= ~0xc0000000;
@@ -559,17 +619,30 @@ static int get_para_features(kvm_context_t kvm_context)
 
 int kvm_arch_qemu_init_env(CPUState *cenv)
 {
-    struct kvm_cpuid_entry cpuid_ent[100];
+    struct kvm_cpuid_entry2 *cpuid_ent, entry, *e;
+    int cpuid_nent = 0, malloc_size = 0;
+    CPUState copy;
+    uint32_t i, limit; 
 #ifdef KVM_CPUID_SIGNATURE
-    struct kvm_cpuid_entry *pv_ent;
+    struct kvm_cpuid_entry2 *pv_ent;
     uint32_t signature[3];
+
+    malloc_size += 2;
 #endif
-    int cpuid_nent = 0;
-    CPUState copy;
-    uint32_t i, limit;
 
     copy = *cenv;
 
+    if (copy.cpuid_host_cpu) {
+        if (!kvm_host_cpuid_entries)
+	   return -EINVAL;
+        malloc_size += kvm_host_cpuid_entries->nent;
+    } else
+           malloc_size += 100;
+
+    cpuid_ent = malloc(malloc_size * sizeof (struct kvm_cpuid_entry2));
+    if (!cpuid_ent)
+       return -ENOMEM;
+
 #ifdef KVM_CPUID_SIGNATURE
     /* Paravirtualization CPUIDs */
     memcpy(signature, "KVMKVMKVM", 12);
@@ -587,21 +660,48 @@ int kvm_arch_qemu_init_env(CPUState *cenv)
     pv_ent->eax = get_para_features(kvm_context);
 #endif
 
-    copy.regs[R_EAX] = 0;
-    qemu_kvm_cpuid_on_env(&copy);
-    limit = copy.regs[R_EAX];
+    limit = copy.cpuid_level;
+    for (i=0; ((i<2) && (i<limit)) ; i++) {
+        e = &cpuid_ent[cpuid_nent++];
+	do_cpuid_ent(e, i, 0, &copy);
+    }
+
+    if (limit >= 2) { /* get the multiple stateful leaf values */
+        do_cpuid_ent(&entry, 2, 0, &copy);
+	cpuid_ent[cpuid_nent++] = entry;
+        for (i = 1; i<(entry.eax & 0xff); i++) {
+            e = &cpuid_ent[cpuid_nent++];
+            do_cpuid_ent(e, 2, 0, &copy);
+        } 
+    }
 
-    for (i = 0; i <= limit; ++i)
-	do_cpuid_ent(&cpuid_ent[cpuid_nent++], i, &copy);
+    for (i = 3; i <= limit; i++) {
+        e = &cpuid_ent[cpuid_nent++];
+	do_cpuid_ent(e, i, 0, &copy);
+    }
 
-    copy.regs[R_EAX] = 0x80000000;
-    qemu_kvm_cpuid_on_env(&copy);
-    limit = copy.regs[R_EAX];
+    if (limit >= 4) { /* get the per index values */
+        int i = 1;
+        do {
+            e = &cpuid_ent[cpuid_nent++];
+            do_cpuid_ent(e, 4, i++, &copy);
+        } while(e->eax & 0x1f); /* until the last index */
+    }
+
+    if (limit >= 0xb) { /* get the per index values */
+        int i = 1;
+        do {
+            e = &cpuid_ent[cpuid_nent++];
+            do_cpuid_ent(e, 0xb, i++, &copy);
+        } while(e->ecx & 0xff00); /* until the last index */
+    }
 
+    limit = copy.cpuid_xlevel;
     for (i = 0x80000000; i <= limit; ++i)
-	do_cpuid_ent(&cpuid_ent[cpuid_nent++], i, &copy);
+	do_cpuid_ent(&cpuid_ent[cpuid_nent++], i, 0, &copy);
 
     kvm_setup_cpuid(kvm_context, cenv->cpu_index, cpuid_nent, cpuid_ent);
+    free(cpuid_ent);
     return 0;
 }
 
diff --git a/qemu/target-i386/cpu.h b/qemu/target-i386/cpu.h
index 11bc2c1..42d646a 100644
--- a/qemu/target-i386/cpu.h
+++ b/qemu/target-i386/cpu.h
@@ -612,6 +612,7 @@ typedef struct CPUX86State {
     uint32_t cpuid_ext2_features;
     uint32_t cpuid_ext3_features;
     uint32_t cpuid_apic_id;
+    uint32_t cpuid_host_cpu;
 
 #ifdef USE_KQEMU
     int kqemu_enabled;
diff --git a/qemu/target-i386/helper.c b/qemu/target-i386/helper.c
index 68efd4d..c23e16e 100644
--- a/qemu/target-i386/helper.c
+++ b/qemu/target-i386/helper.c
@@ -152,6 +152,9 @@ typedef struct x86_def_t {
 static x86_def_t x86_defs[] = {
 #ifdef TARGET_X86_64
     {
+        .name = "host",
+    },
+    {
         .name = "qemu64",
         .level = 2,
         .vendor1 = CPUID_VENDOR_AMD_1,
@@ -405,10 +408,59 @@ void x86_cpu_list (FILE *f, int (*cpu_fprintf)(FILE *f, const char *fmt, ...))
         (*cpu_fprintf)(f, "x86 %16s\n", x86_defs[i].name);
 }
 
+int fill_x86_defs_for_host(CPUX86State *env, x86_def_t * def)
+{
+    struct kvm_cpuid_entry2 e;
+
+    get_host_cpuid_entry(0, 0, &e);
+    env->cpuid_level = e.eax;
+    env->cpuid_vendor1 = e.ebx;
+    env->cpuid_vendor2 = e.ecx;
+    env->cpuid_vendor3 = e.edx;
+
+    get_host_cpuid_entry(1, 0, &e);
+    env->cpuid_version = e.eax;
+    env->cpuid_features = e.edx;
+    env->cpuid_ext_features = e.ecx;
+
+    get_host_cpuid_entry(0x80000000, 0, &e);
+    env->cpuid_xlevel = e.eax;
+
+    get_host_cpuid_entry(0x80000001, 0, &e);
+    env->cpuid_ext3_features = e.ecx;
+    env->cpuid_ext2_features = e.edx;
+
+    get_host_cpuid_entry(0x80000002, 0, &e);
+    env->cpuid_model[0] = e.eax;
+    env->cpuid_model[1] = e.ebx;
+    env->cpuid_model[2] = e.ecx;
+    env->cpuid_model[3] = e.edx;
+
+    get_host_cpuid_entry(0x80000003, 0, &e);
+    env->cpuid_model[4] = e.eax;
+    env->cpuid_model[5] = e.ebx;
+    env->cpuid_model[6] = e.ecx;
+    env->cpuid_model[7] = e.edx;
+
+    get_host_cpuid_entry(0x80000004, 0, &e);
+    env->cpuid_model[8] = e.eax;
+    env->cpuid_model[9] = e.ebx;
+    env->cpuid_model[10] = e.ecx;
+    env->cpuid_model[11] = e.edx;
+
+    return 0;
+}
+
 static int cpu_x86_register (CPUX86State *env, const char *cpu_model)
 {
     x86_def_t def1, *def = &def1;
 
+    if (strcmp(cpu_model, "host") == 0) {
+        env->cpuid_host_cpu = 1;
+        fill_x86_defs_for_host(env, def);
+        return 0;
+    } /* else follow through */
+    env->cpuid_host_cpu = 0;
     if (cpu_x86_find_by_name(def, cpu_model) < 0)
         return -1;
     if (def->vendor1) {

Re: Cross vendor migration ideas

[Amit Shah]

----- "Nitin A Kamble" <nitin.a.kamble@intel.com> wrote:

> Amit, Alex,
>   Please see my comments bellow.
> 
> Avi, 
>   Please have a look at the patches, and let me know the parts you
> think
> can be done better.
> 
> On Fri, 2008-11-14 at 06:07 -0700, Amit Shah wrote:
> > * On Thursday 13 Nov 2008 19:08:14 Alexander Graf wrote:
> > > On 13.11.2008, at 05:35, Amit Shah wrote:
> > > > * On Wednesday 12 Nov 2008 22:49:16 Alexander Graf wrote:
> > > >> On 12.11.2008, at 17:52, Amit Shah wrote:
> > > >>> Hi Alex,
> > > >>>
> > > >>> * On Wednesday 12 Nov 2008 21:09:43 Alexander Graf wrote:
> > > >>>> Hi,
> > > >>>>
> > > >>>> I was thinking a bit about cross vendor migration recently
> and
> > > >>>> since
> > > >>>> we're doing open source development, I figured it might be a
> good
> > > >>>> idea
> > > >>>> to talk to everyone about this.
> > > >>>>
> > > >>>> So why are we having a problem?
> > > >>>>
> > > >>>> In normal operation we don't. If we're running a 32-bit
> kernel, we
> > > >>>> can
> > > >>>> use SYSENTER to jump from kernel<->userspace. If we're on a
> 64-bit
> > > >>>> kernel with 64-bit userspace, every CPU supports SYSCALL. At
> least
> > > >>>> Linux is being smart on this and does use exactly these two
> > > >>>> capabilities in these two cases.
> > > >>>> But if we're running in compat mode (64-bit kernel with
> 32-bit
> > > >>>> userspace), things differ. Intel supports only SYSENTER here,
> while
> > > >>>> AMD only supports SYSCALL. Both can still use int80.
> > > >>>>
> > > >>>> Operating systems detect usage of SYSCALL or SYSENTER pretty
> > > >>>> early on
> > > >>>> (Linux does this on vdso). So when we boot up on an Intel
> machine,
> > > >>>> Linux assumes that using SYSENTER in compat mode is fine.
> Migrating
> > > >>>> that machine to an AMD machine breaks this assumption though,
> since
> > > >>>> SYSENTER can't be used in compat mode.
> > > >>>> On LInux, this detection is based on the CPU vendor string.
> If
> > > >>>> Linux
> > > >>>> finds a "GenuineIntel", SYSENTER is used in compat mode, if
> it's
> > > >>>> "AuthenticAMD", SYSCALL is used and if none of these two is
> found,
> > > >>>> int80 is used.
> > > >>>>
> > > >>>> I tried modifying the vendor string, removed the "overwrite
> the
> > > >>>> vendor
> > > >>>> string with the native string" hack and things look like they
> work
> > > >>>> just fine with Linux.
> > > >>>>
> > > >>>> Unfortunately right now I don't have a 64-bit Windows
> installation
> > > >>>> around to check if that approach works there too, but if it
> does
> > > >>>> and
> > > >>>> no known OS breaks due to the invalid vendor string, we can
> just
> > > >>>> create our own virtual CPU string, no?
> > > >>>
> > > >>> qemu has an option for that, -cpu qemu64 IIRC. As long as we
> expose
> > > >>> practically correct cpuids and MSRs, this should be fine. I've
> not
> > > >>> tested
> > > >>> qemu64 with winxp x64 though. Also, last I knew, winxp x64
> > > >>> installation
> > > >>> didn't succeed with --no-kvm. qemu by default exposes an AMD
> CPU
> > > >>> type.
> > > >>
> > > >> I wasn't talking about CPUID features, but the vendor string.
> Qemu64
> > > >> provides the AuthenticAMD string, so we don't run into any
> issues I'm
> > > >> presuming.
> > > >
> > > > Right -- the thing is, with the default AuthenticAMD string,
> winp x64
> > > > installation fails. That has to be because of some missing
> cpuids.
> > > > That's one
> > > > of the drawbacks of exposing a well-known CPU type. I was
> suggesting
> > > > we
> > > > should try out the -cpu qemu64 CPU type since it exposes a non-
> > > > standard CPU
> > > > to see if guests and most userspace programs work fine without
> any
> > > > further
> > > > tweaking -- see the 'cons' below for why this might be a
> problem.
> > >
> > > I still don't really understand what you're trying to say - qemu64
> is
> > > the default in KVM right now. You mean winxp64 installation
> doesn't
> 
> 
> > No, the default for KVM is the host CPU type.
> 
> Amit, Aliex is correct. the default cpu for kvm is qemu64 not the
> host.

There are two things -- the Vendor ID and the cpuid bits. I'm talking about the vendor ID here.

> I have sent the patches to add an options -cpu host. Some of the
> patches
> are gone in, But All the patches are not in yet. Also my patches does
> not make the host option as default. I have attached the remaining
> two
> patches.

Amit.

RE: Cross vendor migration ideas

[Skywing]

Comments inline.

-----Original Message-----
From: Andi Kleen <andi@firstfloor.org>
Sent: Saturday, November 15, 2008 07:03
To: Serebrin, Benjamin (Calendar) <Benjamin.Serebrin@exchange.amd.com>
Cc: Skywing <Skywing@valhallalegends.com>; Alexander Graf <agraf@suse.de>; Anthony Liguori <anthony@codemonkey.ws>; kvm@vger.kernel.org <kvm@vger.kernel.org>; Amit Shah <amit.shah@redhat.com>; Avi Kivity <avi@redhat.com>; Wahlig, Elsie <elsie.wahlig@amd.com>; Nakajima, Jun <jun.nakajima@intel.com>
Subject: Re: Cross vendor migration ideas

> Also it might break user space, unless you key the fake vendor CPUID
> intercept on ring 0 vs ring 3 (but even if that might not be enough
> because some kernel modules can call CPUID on their own)
> 
> I think just emulating SYSCALL/SYSENTER would be safer. It shouldn't
> be that much slower than int 0x80 hopefully.
> 
> -Andi

If I understand that idea correctly, I think that trying to present a differing view of the CPU vendor and/or feature set is likely to be fraught with peril and highly fragile.

There are many systems where the kernel gets processor feature data in CPL=0 and makes it available to user mode.  This creates an inconsistency when user mode uses cpuid directly and comes up with differing results.

As a real world example, in Windows, the kernel is responsible for examining the current processors installed on the box, mapping processor feature codes to a standard encoding, and making this available to all user mode processes via a shared memory region (read only to user mode) at a well known location (SharedUserData->ProcessorFeatures).

However, user mode programs are free to make their own determinations via cpuid calls if they so wish.  Many programs use the standard abstracted processor feature determination mechanism (figured by the kernel), while others, for varying reasons (some good and some foolish) just call cpuid directly at CPL=3.  There can even be a mix within different code modules loaded into a process.

In the specific case of Windows and system services, IIRC the old-style int 2e system service interface is always initialized on 32-bit x86 kernels, even if sysenter or syscall will be preferred.  As far as I know, this is just for backcompat with programs issuing system services directly as opposed to using the correct C-level abstraction layer.

For Windows, it is CPL=0 inspection of processor attributes via cpuid (assuming the "disable fast system service interface" registry value is not set) that determines which is used.  This occurs once, at system boot, as one would expect.

That being said, I still think that it will break things in difficult to debug ways should you introduce an inconsistent view of cpuid features into the system.

- S