[PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Boris Ostrovsky]

Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
will likely perform same IPIs as would have the guest.

More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
guest's address on remote CPU (when, for example, VCPU from another guest
is running there).

Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Suggested-by: Jan Beulich <jbeulich@suse.com>
Cc: stable@vger.kernel.org # 3.14+
---
 arch/x86/xen/mmu.c |    9 ++-------
 1 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 9c479fe..9ed7eed 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -2495,14 +2495,9 @@ void __init xen_init_mmu_ops(void)
 {
 	x86_init.paging.pagetable_init = xen_pagetable_init;
 
-	/* Optimization - we can use the HVM one but it has no idea which
-	 * VCPUs are descheduled - which means that it will needlessly IPI
-	 * them. Xen knows so let it do the job.
-	 */
-	if (xen_feature(XENFEAT_auto_translated_physmap)) {
-		pv_mmu_ops.flush_tlb_others = xen_flush_tlb_others;
+	if (xen_feature(XENFEAT_auto_translated_physmap))
 		return;
-	}
+
 	pv_mmu_ops = xen_mmu_ops;
 
 	memset(dummy_mapping, 0xff, PAGE_SIZE);
-- 
1.7.1

Re: [Xen-devel] [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Roger Pau Monné]

El 14/12/15 a les 16.27, Konrad Rzeszutek Wilk ha escrit:
> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>> will likely perform same IPIs as would have the guest.
>>
> 
> But if the VCPU is asleep, doing it via the hypervisor will save us waking
> up the guest VCPU, sending an IPI - just to do an TLB flush
> of that CPU. Which is pointless as the CPU hadn't been running the
> guest in the first place.
> 
>>
>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>> guest's address on remote CPU (when, for example, VCPU from another
>> guest
>> is running there).
> 
> Right, so the hypervisor won't even send an IPI there.
> 
> But if you do it via the normal guest IPI mechanism (which are opaque
> to the hypervisor) you and up scheduling the guest VCPU to do
> send an hypervisor callback. And the callback will go the IPI routine
> which will do an TLB flush. Not necessary.
> 
> This is all in case of oversubscription of course. In the case where
> we are fine on vCPU resources it does not matter.
> 
> Perhaps if we have PV aware TLB flush it could do this differently?

Why don't HVM/PVH just uses the HVMOP_flush_tlbs hypercall?

Roger.

Re: [Xen-devel] [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Boris Ostrovsky]

On 12/14/2015 10:35 AM, Roger Pau Monn� wrote:
> El 14/12/15 a les 16.27, Konrad Rzeszutek Wilk ha escrit:
>> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>>> will likely perform same IPIs as would have the guest.
>>>
>> But if the VCPU is asleep, doing it via the hypervisor will save us waking
>> up the guest VCPU, sending an IPI - just to do an TLB flush
>> of that CPU. Which is pointless as the CPU hadn't been running the
>> guest in the first place.

OK, I then mis-read the hypervisor code, I didn't realize that 
vcpumask_to_pcpumask() takes into account vcpu_dirty_cpumask.


>>
>>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>>> guest's address on remote CPU (when, for example, VCPU from another
>>> guest
>>> is running there).
>> Right, so the hypervisor won't even send an IPI there.
>>
>> But if you do it via the normal guest IPI mechanism (which are opaque
>> to the hypervisor) you and up scheduling the guest VCPU to do
>> send an hypervisor callback. And the callback will go the IPI routine
>> which will do an TLB flush. Not necessary.
>>
>> This is all in case of oversubscription of course. In the case where
>> we are fine on vCPU resources it does not matter.
>>
>> Perhaps if we have PV aware TLB flush it could do this differently?
> Why don't HVM/PVH just uses the HVMOP_flush_tlbs hypercall?

It doesn't take any parameters so it will invalidate TLBs for all VCPUs, 
which is more than is being asked for. Especially in the case of 
MMUEXT_INVLPG_MULTI.

(That's in addition to the fact that it currently doesn't work for PVH 
as it has a test for is_hvm_domain() instead of has_hvm_container_domain()).

-boris

Re: [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Boris Ostrovsky]

On 12/14/2015 10:27 AM, Konrad Rzeszutek Wilk wrote:
> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>> will likely perform same IPIs as would have the guest.
>>
> But if the VCPU is asleep, doing it via the hypervisor will save us waking
> up the guest VCPU, sending an IPI - just to do an TLB flush
> of that CPU. Which is pointless as the CPU hadn't been running the
> guest in the first place.
>
>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>> guest's address on remote CPU (when, for example, VCPU from another
>> guest
>> is running there).
> Right, so the hypervisor won't even send an IPI there.
>
> But if you do it via the normal guest IPI mechanism (which are opaque
> to the hypervisor) you and up scheduling the guest VCPU to do
> send an hypervisor callback. And the callback will go the IPI routine
> which will do an TLB flush. Not necessary.
>
> This is all in case of oversubscription of course. In the case where
> we are fine on vCPU resources it does not matter.


So then should we keep these two operations (MMUEXT_INVLPG_MULTI and 
MMUEXT_TLB_FLUSH_MULT) available to HVM/PVH guests? If the guest's VCPU 
is not running then TLBs must have been flushed.

Jan?

-boris


>
> Perhaps if we have PV aware TLB flush it could do this differently?
>
>> Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
>> Suggested-by: Jan Beulich <jbeulich@suse.com>
>> Cc: stable@vger.kernel.org # 3.14+
>> ---
>>   arch/x86/xen/mmu.c |    9 ++-------
>>   1 files changed, 2 insertions(+), 7 deletions(-)
>>
>> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
>> index 9c479fe..9ed7eed 100644
>> --- a/arch/x86/xen/mmu.c
>> +++ b/arch/x86/xen/mmu.c
>> @@ -2495,14 +2495,9 @@ void __init xen_init_mmu_ops(void)
>>   {
>>   	x86_init.paging.pagetable_init = xen_pagetable_init;
>>   
>> -	/* Optimization - we can use the HVM one but it has no idea which
>> -	 * VCPUs are descheduled - which means that it will needlessly IPI
>> -	 * them. Xen knows so let it do the job.
>> -	 */
>> -	if (xen_feature(XENFEAT_auto_translated_physmap)) {
>> -		pv_mmu_ops.flush_tlb_others = xen_flush_tlb_others;
>> +	if (xen_feature(XENFEAT_auto_translated_physmap))
>>   		return;
>> -	}
>> +
>>   	pv_mmu_ops = xen_mmu_ops;
>>   
>>   	memset(dummy_mapping, 0xff, PAGE_SIZE);
>> -- 
>> 1.7.1
>>

Re: [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Jan Beulich]

>>> On 15.12.15 at 15:36, <boris.ostrovsky@oracle.com> wrote:
> On 12/14/2015 10:27 AM, Konrad Rzeszutek Wilk wrote:
>> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>>> will likely perform same IPIs as would have the guest.
>>>
>> But if the VCPU is asleep, doing it via the hypervisor will save us waking
>> up the guest VCPU, sending an IPI - just to do an TLB flush
>> of that CPU. Which is pointless as the CPU hadn't been running the
>> guest in the first place.
>>
>>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>>> guest's address on remote CPU (when, for example, VCPU from another
>>> guest
>>> is running there).
>> Right, so the hypervisor won't even send an IPI there.
>>
>> But if you do it via the normal guest IPI mechanism (which are opaque
>> to the hypervisor) you and up scheduling the guest VCPU to do
>> send an hypervisor callback. And the callback will go the IPI routine
>> which will do an TLB flush. Not necessary.
>>
>> This is all in case of oversubscription of course. In the case where
>> we are fine on vCPU resources it does not matter.
> 
> 
> So then should we keep these two operations (MMUEXT_INVLPG_MULTI and 
> MMUEXT_TLB_FLUSH_MULT) available to HVM/PVH guests? If the guest's VCPU 
> is not running then TLBs must have been flushed.

While I followed the discussion, it didn't become clear to me what
uses these are for HVM guests considering the separate address
spaces. As long as they're useless if called, I'd still favor making
them inaccessible.

Jan

Re: [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Boris Ostrovsky]

On 12/15/2015 10:03 AM, Jan Beulich wrote:
>>>> On 15.12.15 at 15:36, <boris.ostrovsky@oracle.com> wrote:
>> On 12/14/2015 10:27 AM, Konrad Rzeszutek Wilk wrote:
>>> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>>>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>>>> will likely perform same IPIs as would have the guest.
>>>>
>>> But if the VCPU is asleep, doing it via the hypervisor will save us waking
>>> up the guest VCPU, sending an IPI - just to do an TLB flush
>>> of that CPU. Which is pointless as the CPU hadn't been running the
>>> guest in the first place.
>>>
>>>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>>>> guest's address on remote CPU (when, for example, VCPU from another
>>>> guest
>>>> is running there).
>>> Right, so the hypervisor won't even send an IPI there.
>>>
>>> But if you do it via the normal guest IPI mechanism (which are opaque
>>> to the hypervisor) you and up scheduling the guest VCPU to do
>>> send an hypervisor callback. And the callback will go the IPI routine
>>> which will do an TLB flush. Not necessary.
>>>
>>> This is all in case of oversubscription of course. In the case where
>>> we are fine on vCPU resources it does not matter.
>>
>> So then should we keep these two operations (MMUEXT_INVLPG_MULTI and
>> MMUEXT_TLB_FLUSH_MULT) available to HVM/PVH guests? If the guest's VCPU
>> is not running then TLBs must have been flushed.
> While I followed the discussion, it didn't become clear to me what
> uses these are for HVM guests considering the separate address
> spaces.

To avoid unnecessary IPIs to VCPUs that are not currently scheduled (my 
mistake was that I didn't realize that IPIs to those pCPUs will be 
filtered out by the hypervisor).

> As long as they're useless if called, I'd still favor making
> them inaccessible.


VCPUs that are scheduled will receive the required flush requests.

-boris

Re: [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Jan Beulich]

>>> On 15.12.15 at 16:14, <boris.ostrovsky@oracle.com> wrote:
> On 12/15/2015 10:03 AM, Jan Beulich wrote:
>>>>> On 15.12.15 at 15:36, <boris.ostrovsky@oracle.com> wrote:
>>> On 12/14/2015 10:27 AM, Konrad Rzeszutek Wilk wrote:
>>>> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>>>>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>>>>> will likely perform same IPIs as would have the guest.
>>>>>
>>>> But if the VCPU is asleep, doing it via the hypervisor will save us waking
>>>> up the guest VCPU, sending an IPI - just to do an TLB flush
>>>> of that CPU. Which is pointless as the CPU hadn't been running the
>>>> guest in the first place.
>>>>
>>>>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>>>>> guest's address on remote CPU (when, for example, VCPU from another
>>>>> guest
>>>>> is running there).
>>>> Right, so the hypervisor won't even send an IPI there.
>>>>
>>>> But if you do it via the normal guest IPI mechanism (which are opaque
>>>> to the hypervisor) you and up scheduling the guest VCPU to do
>>>> send an hypervisor callback. And the callback will go the IPI routine
>>>> which will do an TLB flush. Not necessary.
>>>>
>>>> This is all in case of oversubscription of course. In the case where
>>>> we are fine on vCPU resources it does not matter.
>>>
>>> So then should we keep these two operations (MMUEXT_INVLPG_MULTI and
>>> MMUEXT_TLB_FLUSH_MULT) available to HVM/PVH guests? If the guest's VCPU
>>> is not running then TLBs must have been flushed.
>> While I followed the discussion, it didn't become clear to me what
>> uses these are for HVM guests considering the separate address
>> spaces.
> 
> To avoid unnecessary IPIs to VCPUs that are not currently scheduled (my 
> mistake was that I didn't realize that IPIs to those pCPUs will be 
> filtered out by the hypervisor).
> 
>> As long as they're useless if called, I'd still favor making
>> them inaccessible.
> 
> VCPUs that are scheduled will receive the required flush requests.

I don't follow - an INVLPG done by the hypervisor won't do any
flushing for a HVM guest.

Jan

Re: [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Boris Ostrovsky]

On 12/15/2015 10:24 AM, Jan Beulich wrote:
>>>> On 15.12.15 at 16:14, <boris.ostrovsky@oracle.com> wrote:
>> On 12/15/2015 10:03 AM, Jan Beulich wrote:
>>>>>> On 15.12.15 at 15:36, <boris.ostrovsky@oracle.com> wrote:
>>>> On 12/14/2015 10:27 AM, Konrad Rzeszutek Wilk wrote:
>>>>> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>>>>>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>>>>>> will likely perform same IPIs as would have the guest.
>>>>>>
>>>>> But if the VCPU is asleep, doing it via the hypervisor will save us waking
>>>>> up the guest VCPU, sending an IPI - just to do an TLB flush
>>>>> of that CPU. Which is pointless as the CPU hadn't been running the
>>>>> guest in the first place.
>>>>>
>>>>>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>>>>>> guest's address on remote CPU (when, for example, VCPU from another
>>>>>> guest
>>>>>> is running there).
>>>>> Right, so the hypervisor won't even send an IPI there.
>>>>>
>>>>> But if you do it via the normal guest IPI mechanism (which are opaque
>>>>> to the hypervisor) you and up scheduling the guest VCPU to do
>>>>> send an hypervisor callback. And the callback will go the IPI routine
>>>>> which will do an TLB flush. Not necessary.
>>>>>
>>>>> This is all in case of oversubscription of course. In the case where
>>>>> we are fine on vCPU resources it does not matter.
>>>> So then should we keep these two operations (MMUEXT_INVLPG_MULTI and
>>>> MMUEXT_TLB_FLUSH_MULT) available to HVM/PVH guests? If the guest's VCPU
>>>> is not running then TLBs must have been flushed.
>>> While I followed the discussion, it didn't become clear to me what
>>> uses these are for HVM guests considering the separate address
>>> spaces.
>> To avoid unnecessary IPIs to VCPUs that are not currently scheduled (my
>> mistake was that I didn't realize that IPIs to those pCPUs will be
>> filtered out by the hypervisor).
>>
>>> As long as they're useless if called, I'd still favor making
>>> them inaccessible.
>> VCPUs that are scheduled will receive the required flush requests.
> I don't follow - an INVLPG done by the hypervisor won't do any
> flushing for a HVM guest.

I thought that this would be done with VPID of intended VCPU still 
loaded and so INVLPG would flush guest's address?

-boris

Re: [PATCH] xen/x86/pvh: Use HVM's flush_tlb_others op

[Jan Beulich]

>>> On 15.12.15 at 16:37, <boris.ostrovsky@oracle.com> wrote:
> On 12/15/2015 10:24 AM, Jan Beulich wrote:
>>>>> On 15.12.15 at 16:14, <boris.ostrovsky@oracle.com> wrote:
>>> On 12/15/2015 10:03 AM, Jan Beulich wrote:
>>>>>>> On 15.12.15 at 15:36, <boris.ostrovsky@oracle.com> wrote:
>>>>> On 12/14/2015 10:27 AM, Konrad Rzeszutek Wilk wrote:
>>>>>> On Sat, Dec 12, 2015 at 07:25:55PM -0500, Boris Ostrovsky wrote:
>>>>>>> Using MMUEXT_TLB_FLUSH_MULTI doesn't buy us much since the hypervisor
>>>>>>> will likely perform same IPIs as would have the guest.
>>>>>>>
>>>>>> But if the VCPU is asleep, doing it via the hypervisor will save us waking
>>>>>> up the guest VCPU, sending an IPI - just to do an TLB flush
>>>>>> of that CPU. Which is pointless as the CPU hadn't been running the
>>>>>> guest in the first place.
>>>>>>
>>>>>>> More importantly, using MMUEXT_INVLPG_MULTI may not to invalidate the
>>>>>>> guest's address on remote CPU (when, for example, VCPU from another
>>>>>>> guest
>>>>>>> is running there).
>>>>>> Right, so the hypervisor won't even send an IPI there.
>>>>>>
>>>>>> But if you do it via the normal guest IPI mechanism (which are opaque
>>>>>> to the hypervisor) you and up scheduling the guest VCPU to do
>>>>>> send an hypervisor callback. And the callback will go the IPI routine
>>>>>> which will do an TLB flush. Not necessary.
>>>>>>
>>>>>> This is all in case of oversubscription of course. In the case where
>>>>>> we are fine on vCPU resources it does not matter.
>>>>> So then should we keep these two operations (MMUEXT_INVLPG_MULTI and
>>>>> MMUEXT_TLB_FLUSH_MULT) available to HVM/PVH guests? If the guest's VCPU
>>>>> is not running then TLBs must have been flushed.
>>>> While I followed the discussion, it didn't become clear to me what
>>>> uses these are for HVM guests considering the separate address
>>>> spaces.
>>> To avoid unnecessary IPIs to VCPUs that are not currently scheduled (my
>>> mistake was that I didn't realize that IPIs to those pCPUs will be
>>> filtered out by the hypervisor).
>>>
>>>> As long as they're useless if called, I'd still favor making
>>>> them inaccessible.
>>> VCPUs that are scheduled will receive the required flush requests.
>> I don't follow - an INVLPG done by the hypervisor won't do any
>> flushing for a HVM guest.
> 
> I thought that this would be done with VPID of intended VCPU still 
> loaded and so INVLPG would flush guest's address?

Again - we're talking about separate address spaces here. INVLPG
can only act on the current one.

Jan