Re: [RFT] mmu optimizations branch

Avi Kivity
2007-01-02T16:32:09Z

Ingo Molnar wrote:
> * Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
>
>
>> This is a request for testing of the mmu optimizations branch.
>>
>> Currently the shadow page tables are discarded every time the guest
>> performs a context switch. The mmu branch allows shadow page tables
>> to be cached across context switches, greatly reducing the cpu
>> utilization on multi process workloads. It is now stable enough for
>> testing (though perhaps not for general use).
>>
>
> i have tested it with Fedora Core 6 guest (32-bit, nopae), under a FC6
> host (32-bit CoreDuo2, nopae, enough RAM), and it's working great!
>
> Here are some quick numbers. Context-switch overhead with lmbench
> lat_ctx -s 0 [zero memory footprint]:
>
> -------------------------------------------------
> #tasks native kvm-r4204 kvm-r4232(mmu)
> -------------------------------------------------
> 2: 2.02 180.91 9.19
> 20: 4.04 183.21 10.01
> 50: 4.30 185.95 11.27
>
> so here it's a /massive/, almost 20 times speedup!
>

Excellent. 10us is approximately the vmexit overhead on intel (we
regularly see 100-120k exits/sec), so it means a context switch is
exactly one exit. Hard to beat without nested page tables.

> Context-switch overhead with -s 1000 (1MB memory footprint):
>
> -------------------------------------------------
> #tasks native kvm-r4204 kvm-r4232(mmu)
> -------------------------------------------------
> 2: 150.5 1032.97 295.16
> 20: 216.6 1020.34 393.01
> 50: 218.1 1015.58 2335.99[*]
>
> the speedup is nice here too. Note the outlier at 50 tasks: it's
> consistently reproducable. Could KVM be trashing the pagetable cache due
> to some sort of internal limit? It's not due to guest size
>

kvm now caches 256 page tables; so if every process uses 5 page tables,
plus some for the kernel, you'd get thrashing. I don't understand why
we're lower than native with 2 processes. Maybe background work causes
page tables to be evicted (see page replacement, below).

I plan to add a tunable for the cache size, and autotuning later on.

The shadow page replacement algorithm can also use some work, currently
it's FIFO. It can be easily made to mimic the Linux active/inactive
lists to approximate LRU by examining the accessed bits on parent page
tables.

> The -mmu FC6 guest is visibly faster, so it's not just microbenchmarks
> that benefit from this change. KVM got /massively/ faster in every
> aspect, kudos Avi! (Note that r4204 already included the interactivity
> IRQ fixes so the improvements are i think purely due to pagetable
> caching speedups.)
>
> on a related note, i also got:
>
> vmwrite error: reg 6802 value cfd3c4a4 (err 17408)
>
>

This is already fixed on the trunk (which now has mmu merged).

> and:
>
> kvm: unhandled wrmsr: 0xc1
> inject_general_protection: rip 0xc011f7f3
> kvm: unhandled wrmsr: 0x186
> inject_general_protection: rip 0xc011f7f3
> kvm: unhandled wrmsr: 0xc1
> inject_general_protection: rip 0xc011f7f3
> kvm: unhandled wrmsr: 0x186
> inject_general_protection: rip 0xc011f7f3
>
> unfortunately 0xc011f7f3 is in native_write_msr(), which isnt very
> helpful. (i have CONFIG_PARAVIRT enabled in the -rt guest and host
> kernels) But the MSR values suggest that this is the NMI watchdog thing
> again, trying to program MSR_ARCH_PERFMON_EVENTSEL0 and
> MSR_ARCH_PERFMON_PERFCTR0, but this time Linux recovered due to a more
> robust MSR handling. The guest disabled the NMI watchdog with:
>
> Testing NMI watchdog ... CPU#0: NMI appears to be stuck (0->0)!
>
> the FC6 installer hang that i saw with earlier MMU-branch snapshots is
> fixed.
>

Good. Handling the counter well would have been very difficult,
especially if attempting to support cross migration.