questions about ack_intr_on_exit and preemption hooks

questions about ack_intr_on_exit and preemption hooks

[Stephane Eranian]

Avi,

A couple of months back, we had a discussion about PMU virtualization
and the difficulty I encountered trying to catch the PMU interrupt
vector in kvm on VM-exit. KVM does not set ack_intr_on_intr. Would
you mind reminding me of the reason for this?

On the topic of scheduler hooks for use by KVM, I think this
would help me also because it means I would not have to save
and restore the PMU registers on all VM-exit/VM-entry. At least
that is what I understood based on the discussion. Those registers
only need to be saved/restore when the KVM process is context switched.

Thanks.

-- 
-Stephane

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: questions about ack_intr_on_exit and preemption hooks

[Dong, Eddie]

kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> Avi,
> 
> A couple of months back, we had a discussion about PMU virtualization
> and the difficulty I encountered trying to catch the PMU interrupt
> vector in kvm on VM-exit. KVM does not set ack_intr_on_intr. Would
> you mind reminding me of the reason for this?
> 
> On the topic of scheduler hooks for use by KVM, I think this
> would help me also because it means I would not have to save
> and restore the PMU registers on all VM-exit/VM-entry. At least

W/o the hook, the PMU save/restore can be done in heavy weight 
VM Exit path, the majority of VM Exit are light weight which doesn't
need to do if you don't care the cycles spent in KVM.
refer vmx_vcpu_put & vmx_vcpu_load.

With hook, then we can further reduce the save/restore effort :-)

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: questions about ack_intr_on_exit and preemption hooks

[Stephane Eranian]

Eddie,

On Fri, Jul 13, 2007 at 08:46:06AM +0800, Dong, Eddie wrote:
> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> > Avi,
> > 
> > A couple of months back, we had a discussion about PMU virtualization
> > and the difficulty I encountered trying to catch the PMU interrupt
> > vector in kvm on VM-exit. KVM does not set ack_intr_on_intr. Would
> > you mind reminding me of the reason for this?
> > 
> > On the topic of scheduler hooks for use by KVM, I think this
> > would help me also because it means I would not have to save
> > and restore the PMU registers on all VM-exit/VM-entry. At least
> 
> W/o the hook, the PMU save/restore can be done in heavy weight 
> VM Exit path, the majority of VM Exit are light weight which doesn't
> need to do if you don't care the cycles spent in KVM.
> refer vmx_vcpu_put & vmx_vcpu_load.
> 
I am not sure I understand your terminology about heavy vs. light. 

You do need to stop monitoring on every VM-exit, because you do not
want to continue measuring while in KVM. On VM-entry, then you need
to re-activate. So you can use explcit code to clear a bunch of 
MSR on VM-exit and restore them on VM-entry. Alternatively, you
can use (on VT-x, not on AMD-v) the VT-x feature to automatically
save and restore MSRs on VM-exit/VM-entry. All of this can be done
lazily, i.e., only once you start using the PMU.

On VM-exit, which leads to context switch of the KVM thread, then
you do need to do more work and save the PMU counters. That would
have to be done by software given that VT-x does not know about
Linux context switching. This is where the hooks come in handy.
Same thing on context switch in.

> With hook, then we can further reduce the save/restore effort :-)

-- 

-Stephane

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: questions about ack_intr_on_exit and preemption hooks

[Dong, Eddie]

Stephane Eranian wrote:
> Eddie,
> 
> On Fri, Jul 13, 2007 at 08:46:06AM +0800, Dong, Eddie wrote:
>> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
>>> Avi,
>>> 
>>> A couple of months back, we had a discussion about PMU
>>> virtualization and the difficulty I encountered trying to catch the
>>> PMU interrupt vector in kvm on VM-exit. KVM does not set
>>> ack_intr_on_intr. Would you mind reminding me of the reason for
>>> this? 
>>> 
>>> On the topic of scheduler hooks for use by KVM, I think this
>>> would help me also because it means I would not have to save
>>> and restore the PMU registers on all VM-exit/VM-entry. At least
>> 
>> W/o the hook, the PMU save/restore can be done in heavy weight
>> VM Exit path, the majority of VM Exit are light weight which doesn't
>> need to do if you don't care the cycles spent in KVM.
>> refer vmx_vcpu_put & vmx_vcpu_load.
>> 
> I am not sure I understand your terminology about heavy vs. light.

Hi, Stephane:

Sorry for the confuse. Heavy weight VM exit here means a VM Exit which 
may cause context switch, while light weight VM Exit doesn't.
We already have some lazy MSR save/restore stuff which is only for 
heavy weight VM Exit such as MSR_CSTAR save/restore.


> 
> You do need to stop monitoring on every VM-exit, because you do not
> want to continue measuring while in KVM. On VM-entry, then you need
> to re-activate. So you can use explcit code to clear a bunch of
> MSR on VM-exit and restore them on VM-entry. Alternatively, you
> can use (on VT-x, not on AMD-v) the VT-x feature to automatically
> save and restore MSRs on VM-exit/VM-entry. All of this can be done
> lazily, i.e., only once you start using the PMU.

Yes.

> 
> On VM-exit, which leads to context switch of the KVM thread, then
> you do need to do more work and save the PMU counters. That would
> have to be done by software given that VT-x does not know about
> Linux context switching. This is where the hooks come in handy.
> Same thing on context switch in.

Yes. But without the hook, this save/restore is only needed for heavy
weight
VM Exit.

> 
>> With hook, then we can further reduce the save/restore effort :-)

Yes, that serve us much better :-)

Thx, eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: questions about ack_intr_on_exit and preemption hooks

[Avi Kivity]

Stephane Eranian wrote:
> Avi,
>
> A couple of months back, we had a discussion about PMU virtualization
> and the difficulty I encountered trying to catch the PMU interrupt
> vector in kvm on VM-exit. KVM does not set ack_intr_on_intr. Would
> you mind reminding me of the reason for this?
>
>   

If we ack the interrupt, we have to dispatch it ourselves, most likely
by faking an 'int $xx' instruction.

By not acking the interrupt, we allow it do be delivered through the
host IDT whether we are in guest mode or not.

> On the topic of scheduler hooks for use by KVM, I think this
> would help me also because it means I would not have to save
> and restore the PMU registers on all VM-exit/VM-entry. At least
> that is what I understood based on the discussion. Those registers
> only need to be saved/restore when the KVM process is context switched.
>   

If you want to monitor the guest, then I think you need to switch
immediately on exit/entry, using the hardware msr switching mechanism. 
Otherwise, host events will be counted against the guest counters.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: questions about ack_intr_on_exit and preemption hooks

[Stephane Eranian]

Avi,

On Fri, Jul 13, 2007 at 09:53:36AM +0300, Avi Kivity wrote:
> Stephane Eranian wrote:
> > Avi,
> >
> > A couple of months back, we had a discussion about PMU virtualization
> > and the difficulty I encountered trying to catch the PMU interrupt
> > vector in kvm on VM-exit. KVM does not set ack_intr_on_intr. Would
> > you mind reminding me of the reason for this?
> >
> >   
> 
> If we ack the interrupt, we have to dispatch it ourselves, most likely
> by faking an 'int $xx' instruction.
> 
> By not acking the interrupt, we allow it do be delivered through the
> host IDT whether we are in guest mode or not.
> 
Doesn't that assume that any interrupt generated by the guest has a
host IDT entry?

On a related issue, you set NMI_EXITING. Why is that? Assuming the
guest does not use the NMI watchdog. Are you doing this to ensure
that the host NMI watchdog keeps on going while you're executing 
in the guest?

Thanks.

-- 
-Stephane

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: questions about ack_intr_on_exit and preemption hooks

[Avi Kivity]

Stephane Eranian wrote:
> Avi,
>
> On Fri, Jul 13, 2007 at 09:53:36AM +0300, Avi Kivity wrote:
>   
>> Stephane Eranian wrote:
>>     
>>> Avi,
>>>
>>> A couple of months back, we had a discussion about PMU virtualization
>>> and the difficulty I encountered trying to catch the PMU interrupt
>>> vector in kvm on VM-exit. KVM does not set ack_intr_on_intr. Would
>>> you mind reminding me of the reason for this?
>>>
>>>   
>>>       
>> If we ack the interrupt, we have to dispatch it ourselves, most likely
>> by faking an 'int $xx' instruction.
>>
>> By not acking the interrupt, we allow it do be delivered through the
>> host IDT whether we are in guest mode or not.
>>
>>     
> Doesn't that assume that any interrupt generated by the guest has a
> host IDT entry?
>
>   

Right now we don't allow the guest access to the hardware, so the guest 
can't generate any interrupt.

> On a related issue, you set NMI_EXITING. Why is that? Assuming the
> guest does not use the NMI watchdog. Are you doing this to ensure
> that the host NMI watchdog keeps on going while you're executing 
> in the guest?
>   

Yes.  Or oprofile.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/