Re: [kvm-devel] performance with guests running 2.4 kernels (specifically RHEL3)

Avi Kivity
2008-06-05T08:37:14Z

David S. Ahern wrote:
>> Oh! Only 45K pages were direct, so the other 45K were shared, with
>> perhaps many ptes. We shoud count ptes, not pages.
>>
>> Can you modify page_referenced() to count the numbers of ptes mapped (1
>> for direct pages, nr_chains for indirect pages) and print the total
>> deltas in active_anon_scan?
>>
>>
>
> Here you go. I've shortened the line lengths to get them to squeeze into
> 80 columns:
>
> anon_scan, all HighMem zone, 187,910 active pages at loop start:
> count[12] 21462 -> 230, direct 20469, chains 3479, dj 58
> count[11] 1338 -> 1162, direct 227, chains 26144, dj 59
> count[8] 29397 -> 5410, direct 26115, chains 27617, dj 117
> count[4] 35804 -> 25556, direct 31508, chains 82929, dj 256
> count[3] 2738 -> 2207, direct 2680, chains 58, dj 7
> count[0] 92580 -> 89509, direct 75024, chains 262834, dj 726
> (age number is the index in [])
>
>

Where do all those ptes come from? that's 180K pages (most of highmem),
but with 550K ptes.

The memuser workload doesn't use fork(), so there shouldn't be any
indirect ptes.

We might try to unshadow the fixmap page; that means we don't have to do
4 fixmap pte accesses per pte scanned.

The kernel uses two methods for clearing the accessed bit:

For direct pages:

if (pte_young(*pte) && ptep_test_and_clear_young(pte))
referenced++;

(two accesses)

For indirect pages:

if (ptep_test_and_clear_young(pte))
referenced++;

(one access)

Which have to be emulated if we don't shadow the fixmap. With the data
above, that translates to 700K emulations with your numbers above, vs
2200K emulations, a 3X improvement. I'm not sure it will be sufficient
given that we're reducing a 10-second kscand scan into a 3-second scan.

> If you sum the direct pages and the chains count for each row, convert
> dj into dt (divided by HZ = 100) you get:
>
> ( 20469 + 3479 ) / 0.58 = 41289
> ( 227 + 26144 ) / 0.59 = 44696
> ( 26115 + 27617 ) / 1.17 = 45924
> ( 31508 + 82929 ) / 2.56 = 44701
> ( 2680 + 58 ) / 0.07 = 39114
> ( 75024 + 262834 ) / 7.26 = 46536
> ( 499 + 20022 ) / 0.44 = 46638
> ( 7189 + 9854 ) / 0.37 = 46062
> ( 5071 + 9388 ) / 0.31 = 46641
>
> For 4 pte writes per direct page or chain entry comes to ~187,000/sec
> which is close to the total collected by kvm_stat (data width shrunk to
> fit in e-mail; hope this is readable still):
>
>
> |---------- mmu_ ----------|----- pf_ -----|
> cache flood pde_z pte_u pte_w shado fixed guest
> 267 271 95 21455 21842 285 22840 165
> 66 88 0 12102 12224 88 12458 0
> 2042 2133 0 178146 180515 2133 188089 387
> 1053 1212 0 187067 188485 1212 193011 8
> 4771 4811 88 185129 190998 4825 207490 448
> 910 824 7 183066 184050 824 195836 12
> 707 785 0 176381 177300 785 180350 6
> 1167 1144 0 189618 191014 1144 195902 10
> 4238 4193 87 188381 193590 4206 207030 465
> 1448 1400 7 187786 189509 1400 198688 21
> 982 971 0 187880 189076 971 198405 2
> 1165 1208 0 190007 191503 1208 195746 13
> 1106 1146 0 189144 190550 1146 195143 0
> 4767 4788 96 185802 191704 4802 206362 477
> 1388 1431 0 187387 188991 1431 195115 3
> 584 551 0 77176 77802 551 84829 10
> 12 7 0 3601 3609 7 13497 4
> 243 153 91 31085 31333 167 35059 879
> 21 18 6 3130 3155 18 3827 2
> 21 4 1 4665 4670 4 6825 9
>
>
>>> The kvm_stat data for this time period is attached due to line lengths.
>>>
>>>
>>> Also, I forgot to mention this before, but there is a bug in the
>>> kscand code in the RHEL3U8 kernel. When it scans the cache list it
>>> uses the count from the anonymous list:
>>>
>>> if (need_active_cache_scan(zone)) {
>>> for (age = MAX_AGE-1; age >= 0; age--) {
>>> scan_active_list(zone, age,
>>> &zone->active_cache_list[age],
>>> zone->active_anon_count[age]);
>>> ^^^^^^^^^^^^^^^^^
>>> if (current->need_resched)
>>> schedule();
>>> }
>>> }
>>>
>>> When the anonymous count is higher it is scanning the cache list
>>> repeatedly. An example of that was captured here:
>>>
>>> active_cache_scan: HighMem, age 7, count[age] 222 -> 179, count anon
>>> 111967, direct 626, dj 3
>>>
>>> count anon is active_anon_count[age] which at this moment was 111,967.
>>> There were only 222 entries in the cache list, but the count value
>>> passed to scan_active_list was 111,967. When the cache list has a lot
>>> of direct pages, that causes a larger hit on kvm than needed. That
>>> said, I have to live with the bug in the guest.
>>>
>>>
>> For debugging, can you fix it? It certainly has a large impact.
>>
>>
> yes, I have run a few tests with it fixed to get a ballpark on the
> impact. The fix is included in the number above.
>
>
>> Perhaps it is fixed in an update kernel. There's a 2.4.21-50.EL in the
>> centos 3.8 update repos.
>>
>>

It seems to have been fixed there.

--
error compiling committee.c: too many arguments to function