Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

David S. Ahern
2008-06-05T16:20:37Z

Avi Kivity wrote:
> David S. Ahern wrote:
>>> Oh! Only 45K pages were direct, so the other 45K were shared, with
>>> perhaps many ptes. We shoud count ptes, not pages.
>>>
>>> Can you modify page_referenced() to count the numbers of ptes mapped (1
>>> for direct pages, nr_chains for indirect pages) and print the total
>>> deltas in active_anon_scan?
>>>
>>>
>>
>> Here you go. I've shortened the line lengths to get them to squeeze into
>> 80 columns:
>>
>> anon_scan, all HighMem zone, 187,910 active pages at loop start:
>> count[12] 21462 -> 230, direct 20469, chains 3479, dj 58
>> count[11] 1338 -> 1162, direct 227, chains 26144, dj 59
>> count[8] 29397 -> 5410, direct 26115, chains 27617, dj 117
>> count[4] 35804 -> 25556, direct 31508, chains 82929, dj 256
>> count[3] 2738 -> 2207, direct 2680, chains 58, dj 7
>> count[0] 92580 -> 89509, direct 75024, chains 262834, dj 726
>> (age number is the index in [])
>>
>>
>
> Where do all those ptes come from? that's 180K pages (most of highmem),
> but with 550K ptes.
>
> The memuser workload doesn't use fork(), so there shouldn't be any
> indirect ptes.
>
> We might try to unshadow the fixmap page; that means we don't have to do
> 4 fixmap pte accesses per pte scanned.
>
> The kernel uses two methods for clearing the accessed bit:
>
> For direct pages:
>
> if (pte_young(*pte) && ptep_test_and_clear_young(pte))
> referenced++;
>
> (two accesses)
>
> For indirect pages:
>
> if (ptep_test_and_clear_young(pte))
> referenced++;
>
> (one access)
>
> Which have to be emulated if we don't shadow the fixmap. With the data
> above, that translates to 700K emulations with your numbers above, vs
> 2200K emulations, a 3X improvement. I'm not sure it will be sufficient
> given that we're reducing a 10-second kscand scan into a 3-second scan.
>

A 3-second scan is much better and incomparison to where kvm was when I
started this e-mail thread (as high as 30-seconds for a scan) it's a
10-fold improvement.

I gave a shot at implementing your suggestion, but evidently I am still
not understanding the shadow implementation. Can you suggest a patch to
try this out?

david