Re: [PATCH] nestedhvm: ASID emulation

[Christoph Egger <Christoph.Egger@amd.com>]

On 04/13/11 18:22, Keir Fraser wrote:
> On 13/04/2011 16:19, "Christoph Egger"<Christoph.Egger@amd.com>  wrote:
>
>> On 04/13/11 17:05, Keir Fraser wrote:
>>> On 13/04/2011 15:26, "Christoph Egger"<Christoph.Egger@amd.com>   wrote:
>>>
>>> Is this measurable on a macro benchmark?
>>
>> I measured this with xentrace while l2 guest is booting.
>>
>> The speedup is noticable on the end-user side just by the feeling on how
>> fast the l2 guest reacts on user input.
>
> Yikes. Does nestedhvm suck really badly then? I can't believe this patch
> alone gets you from sucky to good performance. Is it an improvement from
> sucky to not-quite-as-sucky?

The very first patch series I sent to xen-devel, the cost of a
VMRUN emulation was about 25000 cycles with Xen-on-Xen while hvmloader
was coming up as l2 guest. With every new patch series there were also
performance optimizations done. Now the cost of a VMRUN
emulation is about 11000 cycles and with this ASID emulation patch
the cost of a VMRUN emulation goes down to about 10000 cycles.

There is still room for more performance, I am working on.

A cost of 15000 cycles for VMRUN emulation is fast enough that Solitair
in XP mode of Windows 7 guest recognizes a double click ;-)

Solitair in XP mode runs faster than Solitair in Windows 7 ;-)


>>> I mean this looks like a micro-optimisation on a feature that noone is going
>>> to use for serious performance work anyway.
>>>
>>>> 4. nestedhvm is enabled and we are going to run l2 guest
>>>>
>>>> We run the l1 guest in the last call. The asid generation may have
>>>> changed by then. In this case the current nv_n2asid number is stale
>>>> and the value of data->next_asid is<= of nv->nv_n2asid.
>>>
>>> How do you know for sure that next_asid will be<= nv_n2asid in this case?
>>> What if other VCPUs have run meanwhile, and next_asid has been incremented
>>> multiple times until it is greather than nv_n2asid?
>>
>> In that case  curr->arch.hvm_vcpu.asid_generation is not equal to
>> data->core_asid_generation  and a new hw ASID number is assigned
>> regardless of the values of data->next_asid and nv_n2asid.
>
> What if nv_n2asid wast last assigned on a previous generation, but nv_n1asid
> was assigned from the current generation? Then you'd have next_asid>
> nv_n2asid, but nv_n2asid is stale.

When the generation changes, next_asid is set back to 1 and the hw TLB 
flushed.

If next_asid is larger than nv_n2asid then it is reused and this is ok
since the hw TLB got flushed. It is just important that nv_n1asid never
gets the same hw ASID assigned and that is what the do { } while loop
ensures.

The hardware doesn't enforce an incremental use of the hw ASID numbers.

The point of ASID emulation is to reduce the number of
TLB invalidations by switching forth and back between two
hw ASID numbers when switching between l1 and l2 guest.
This reduces pagetable walks of the hw MMU so much that the
performance win is noticable by the end-user.

Without this patch the ASID is invalidated on every VMRUN and VMEXIT 
emulation.

The Flush-by-ASID feature even let me reuse the same hw ASID number
when the l2 guest wants to flush its TLB.

Christoph


>   -- Keir
>
>> Christoph
>>
>>>
>>>    -- Keir
>>>
>>>> The the value of nv->nv_n2asid is valid if l1 guest doesn't change
>>>> the virtual asid (= asid number in the virtual vmcb) and
>>>> data->next_asid is larger than nv->nv_n2asid. In this case
>>>> just reuse the same hw ASID that has been used from the last
>>>> VMRUN emulation.
>>>>
>>>> 5. nestedhvm is enabled and we are going to run l2 guest again
>>>>
>>>> The same hw ASID should be reused unless the generation changed because
>>>> the nestedp2m got flushed or the vcpu moved to a different physical cpu,
>>>> for example.
>>>> But the hw ASID number may never match the hw ASID used to run the l1 guest.
>>>>
>>>>
>>>> In all cases we have to verify that the
>>>>
>>>>> I wouldn't bother fixing #2 unless there's a convincing answer for #1.



-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85689 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632