Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

["David S. Ahern" <daahern@cisco.com>]

Avi Kivity wrote:
> David S. Ahern wrote:
>>> Oh!  Only 45K pages were direct, so the other 45K were shared, with
>>> perhaps many ptes.  We shoud count ptes, not pages.
>>>
>>> Can you modify page_referenced() to count the numbers of ptes mapped (1
>>> for direct pages, nr_chains for indirect pages) and print the total
>>> deltas in active_anon_scan?
>>>
>>>     
>>
>> Here you go. I've shortened the line lengths to get them to squeeze into
>> 80 columns:
>>
>> anon_scan, all HighMem zone, 187,910 active pages at loop start:
>>   count[12] 21462 -> 230,   direct 20469, chains 3479,   dj 58
>>   count[11] 1338  -> 1162,  direct 227,   chains 26144,  dj 59
>>   count[8] 29397  -> 5410,  direct 26115, chains 27617,  dj 117
>>   count[4] 35804  -> 25556, direct 31508, chains 82929,  dj 256
>>   count[3] 2738   -> 2207,  direct 2680,  chains 58,     dj 7
>>   count[0] 92580  -> 89509, direct 75024, chains 262834, dj 726
>> (age number is the index in [])
>>
>>   
> 
> Where do all those ptes come from?  that's 180K pages (most of highmem),
> but with 550K ptes.
> 
> The memuser workload doesn't use fork(), so there shouldn't be any
> indirect ptes.
> 
> We might try to unshadow the fixmap page; that means we don't have to do
> 4 fixmap pte accesses per pte scanned.
> 
> The kernel uses two methods for clearing the accessed bit:
> 
> For direct pages:
> 
>                if (pte_young(*pte) && ptep_test_and_clear_young(pte))
>                        referenced++;
> 
> (two accesses)
> 
> For indirect pages:
> 
>                                if (ptep_test_and_clear_young(pte))
>                                        referenced++;
> 
> (one access)
> 
> Which have to be emulated if we don't shadow the fixmap.  With the data
> above, that translates to 700K emulations with your numbers above, vs
> 2200K emulations, a 3X improvement.  I'm not sure it will be sufficient
> given that we're reducing a 10-second kscand scan into a 3-second scan.
> 

A 3-second scan is much better and incomparison to where kvm was when I
started this e-mail thread (as high as 30-seconds for a scan) it's a
10-fold improvement.

I gave a shot at implementing your suggestion, but evidently I am still
not understanding the shadow implementation. Can you suggest a patch to
try this out?

david