Why exit on MSR_STAR and friends?

Why exit on MSR_STAR and friends?

[Nadav Har'El]

Hi, In the recent KVM forum, Marcelo Tosatti presented some KVM performance
improvements. One of them (page 17 of the presentation) was about the MSRs
used by SYSCALL: MSR_STAR, MSR_LSTAR, MSR_CSTAR (and also MSR_SYSCALL_MASK).

He said that "Guests have direct access to these MSRs", and this is why KVM
needs to restore their original host value when returning to user space
(they aren't used in the kernel, so there's no reason to restore them earlier).

I was suprised, then, to discover that KVM doesn't add these MSRs to the
MSR bitmap, so when the guest changes these MSR values, or even reads them,
we cause an exit - both on read and on write of these MSRs.

I was wondering why these exits are needed. 
I can maybe guess why an exit is needed on write - just to save the guest
value so we don't need to read it when going back to user space. Is this
a good optimization because we assume that the guest very rarely changes
these MSRs? Or is there another explanation as to why these exits are
needed?
But I can't even guess why an exit is needed on read...


If you're curious why I noticed these exits - I was running a nested KVM
(L0's guest L1 is KVM, that itself has a guest L2). Whenever L2 does something
that L1 needs to handle in user space (e.g., PIO), L1 does all these MSR
reads and writes, and we get exits for each of them - many of those exits
for each L2 PIO :(

Thanks,
Nadav Har'El.


-- 
Nadav Har'El                        |   Wednesday, Feb 16 2011, 12 Adar I 5771
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Support bacteria - they're the only
http://nadav.harel.org.il           |culture some people have!

Re: Why exit on MSR_STAR and friends?

[Avi Kivity]

On 02/16/2011 05:17 PM, Nadav Har'El wrote:
> Hi, In the recent KVM forum, Marcelo Tosatti presented some KVM performance
> improvements. One of them (page 17 of the presentation) was about the MSRs
> used by SYSCALL: MSR_STAR, MSR_LSTAR, MSR_CSTAR (and also MSR_SYSCALL_MASK).
>
> He said that "Guests have direct access to these MSRs", and this is why KVM
> needs to restore their original host value when returning to user space
> (they aren't used in the kernel, so there's no reason to restore them earlier).
>
> I was suprised, then, to discover that KVM doesn't add these MSRs to the
> MSR bitmap, so when the guest changes these MSR values, or even reads them,
> we cause an exit - both on read and on write of these MSRs.
>
> I was wondering why these exits are needed.
> I can maybe guess why an exit is needed on write - just to save the guest
> value so we don't need to read it when going back to user space. Is this
> a good optimization because we assume that the guest very rarely changes
> these MSRs?

Exactly.

> Or is there another explanation as to why these exits are
> needed?

No.

> But I can't even guess why an exit is needed on read...

It isn't needed.  The code doesn't distinguish between the read and 
write bitmaps, and so far no guest issues rdmsr for these msrs with any 
frequency (kvm as a guest will write those msrs, but it shouldn't read 
them on Intel).  Do you see frequent reads on some guest?

> If you're curious why I noticed these exits - I was running a nested KVM
> (L0's guest L1 is KVM, that itself has a guest L2). Whenever L2 does something
> that L1 needs to handle in user space (e.g., PIO), L1 does all these MSR
> reads and writes, and we get exits for each of them - many of those exits
> for each L2 PIO :(

It shouldn't be doing MSR reads.  The only MSR read I can see is for 
SYSENTER_ESP when a vcpu is migrated (easily avoided).  But maybe I'm 
missing something.

-- 
error compiling committee.c: too many arguments to function

Re: Why exit on MSR_STAR and friends?

[Nadav Har'El]

On Thu, Feb 17, 2011, Avi Kivity wrote about "Re: Why exit on MSR_STAR and friends?":
> It isn't needed.  The code doesn't distinguish between the read and 
> write bitmaps, and so far no guest issues rdmsr for these msrs with any 
> frequency (kvm as a guest will write those msrs, but it shouldn't read 
> them on Intel).  Do you see frequent reads on some guest?

I saw reads of all these MSRs (STAR, LSTAR, CSTAR and SYSCALL_MASK) at about
half the number of the writes.

Looking deeper now, I realize why I saw these and you didn't. I happened
to run some old L1 image, with apparently a 1 year old Linux and KVM.
In that version, __vmx_load_host_state() called:
	save_msrs(vmx->guest_msrs, vmx->save_nmsrs);
        load_msrs(vmx->host_msrs, vmx->save_nmsrs);
and save_msrs read all those MSRs. Looking at the current code, indeed
this is no loger done.

So you can say that there are some guests which frequently reads those MSRs -
old versions of KVM :-) But I agree, this is nothing to worry about.

I guess I should be more worried why I got all these PIOs in L2 in the
first place - the MSR reads and writes in L1 were just an odd consequence
of that. It turns out that the "ping -f" workload I was running in L2
insisted to get accurate timings of each packet, and this, I'm not still
sure why, caused a ACPI PM_TIMER PIO for each packet. I guess that normal
workloads won't use the timer on every packet, so that shouldn't matter.

-- 
Nadav Har'El                        |    Thursday, Feb 17 2011, 13 Adar I 5771
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Knowledge is knowing a tomato is a fruit.
http://nadav.harel.org.il           |Wisdom is not putting it in a fruit salad.

Re: Why exit on MSR_STAR and friends?

[Avi Kivity]

On 02/17/2011 12:29 PM, Nadav Har'El wrote:
> I guess I should be more worried why I got all these PIOs in L2 in the
> first place - the MSR reads and writes in L1 were just an odd consequence
> of that. It turns out that the "ping -f" workload I was running in L2
> insisted to get accurate timings of each packet, and this, I'm not still
> sure why, caused a ACPI PM_TIMER PIO for each packet. I guess that normal
> workloads won't use the timer on every packet, so that shouldn't matter.

Strange, isn't kvmclock exposed to L2?


-- 
error compiling committee.c: too many arguments to function