Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

[Avi Kivity <avi@qumranet.com>]

David S. Ahern wrote:
>> Oh!  Only 45K pages were direct, so the other 45K were shared, with
>> perhaps many ptes.  We shoud count ptes, not pages.
>>
>> Can you modify page_referenced() to count the numbers of ptes mapped (1
>> for direct pages, nr_chains for indirect pages) and print the total
>> deltas in active_anon_scan?
>>
>>     
>
> Here you go. I've shortened the line lengths to get them to squeeze into
> 80 columns:
>
> anon_scan, all HighMem zone, 187,910 active pages at loop start:
>   count[12] 21462 -> 230,   direct 20469, chains 3479,   dj 58
>   count[11] 1338  -> 1162,  direct 227,   chains 26144,  dj 59
>   count[8] 29397  -> 5410,  direct 26115, chains 27617,  dj 117
>   count[4] 35804  -> 25556, direct 31508, chains 82929,  dj 256
>   count[3] 2738   -> 2207,  direct 2680,  chains 58,     dj 7
>   count[0] 92580  -> 89509, direct 75024, chains 262834, dj 726
> (age number is the index in [])
>
>   

Where do all those ptes come from?  that's 180K pages (most of highmem), 
but with 550K ptes.

The memuser workload doesn't use fork(), so there shouldn't be any 
indirect ptes.

We might try to unshadow the fixmap page; that means we don't have to do 
4 fixmap pte accesses per pte scanned.

The kernel uses two methods for clearing the accessed bit:

For direct pages:

                if (pte_young(*pte) && ptep_test_and_clear_young(pte))
                        referenced++;

(two accesses)

For indirect pages:

                                if (ptep_test_and_clear_young(pte))
                                        referenced++;

(one access)

Which have to be emulated if we don't shadow the fixmap.  With the data 
above, that translates to 700K emulations with your numbers above, vs 
2200K emulations, a 3X improvement.  I'm not sure it will be sufficient 
given that we're reducing a 10-second kscand scan into a 3-second scan.

> If you sum the direct pages and the chains count for each row, convert
> dj into dt (divided by HZ = 100) you get:
>
> ( 20469 + 3479 )   / 0.58 = 41289
> ( 227 + 26144 )    / 0.59 = 44696
> ( 26115 + 27617 )  / 1.17 = 45924
> ( 31508 + 82929 )  / 2.56 = 44701
> ( 2680 + 58 )      / 0.07 = 39114
> ( 75024 + 262834 ) / 7.26 = 46536
> ( 499 + 20022 )    / 0.44 = 46638
> ( 7189 + 9854 )    / 0.37 = 46062
> ( 5071 + 9388 )    / 0.31 = 46641
>
> For 4 pte writes per direct page or chain entry comes to ~187,000/sec
> which is close to the total collected by kvm_stat (data width shrunk to
> fit in e-mail; hope this is readable still):
>
>
> |----------         mmu_          ----------|-----  pf_  -----|
>  cache  flood  pde_z    pte_u    pte_w  shado    fixed    guest
>    267    271     95    21455    21842    285    22840      165
>     66     88      0    12102    12224     88    12458        0
>   2042   2133      0   178146   180515   2133   188089      387
>   1053   1212      0   187067   188485   1212   193011        8
>   4771   4811     88   185129   190998   4825   207490      448
>    910    824      7   183066   184050    824   195836       12
>    707    785      0   176381   177300    785   180350        6
>   1167   1144      0   189618   191014   1144   195902       10
>   4238   4193     87   188381   193590   4206   207030      465
>   1448   1400      7   187786   189509   1400   198688       21
>    982    971      0   187880   189076    971   198405        2
>   1165   1208      0   190007   191503   1208   195746       13
>   1106   1146      0   189144   190550   1146   195143        0
>   4767   4788     96   185802   191704   4802   206362      477
>   1388   1431      0   187387   188991   1431   195115        3
>    584    551      0    77176    77802    551    84829       10
>     12      7      0     3601     3609      7    13497        4
>    243    153     91    31085    31333    167    35059      879
>     21     18      6     3130     3155     18     3827        2
>     21      4      1     4665     4670      4     6825        9
>
>   
>>> The kvm_stat data for this time period is attached due to line lengths.
>>>
>>>
>>> Also, I forgot to mention this before, but there is a bug in the
>>> kscand code in the RHEL3U8 kernel. When it scans the cache list it
>>> uses the count from the anonymous list:
>>>
>>>             if (need_active_cache_scan(zone)) {
>>>                 for (age = MAX_AGE-1; age >= 0; age--)  {
>>>                     scan_active_list(zone, age,
>>>                         &zone->active_cache_list[age],
>>>                         zone->active_anon_count[age]);
>>>                               ^^^^^^^^^^^^^^^^^
>>>                     if (current->need_resched)
>>>                         schedule();
>>>                 }
>>>             }
>>>
>>> When the anonymous count is higher it is scanning the cache list
>>> repeatedly. An example of that was captured here:
>>>
>>> active_cache_scan: HighMem, age 7, count[age] 222 -> 179, count anon
>>> 111967, direct 626, dj 3
>>>
>>> count anon is active_anon_count[age] which at this moment was 111,967.
>>> There were only 222 entries in the cache list, but the count value
>>> passed to scan_active_list was 111,967. When the cache list has a lot
>>> of direct pages, that causes a larger hit on kvm than needed. That
>>> said, I have to live with the bug in the guest.
>>>   
>>>       
>> For debugging, can you fix it?  It certainly has a large impact.
>>
>>     
> yes, I have run a few tests with it fixed to get a ballpark on the
> impact. The fix is included in the number above.
>
>   
>> Perhaps it is fixed in an update kernel.  There's a 2.4.21-50.EL in the
>> centos 3.8 update repos.
>>
>>     

It seems to have been fixed there.

-- 
error compiling committee.c: too many arguments to function