Re: [RFC] concurrent guest walker and instruction emulation

Avi Kivity
2007-12-18T15:49:51Z

Marcelo Tosatti wrote:
> Hi,
>
> The following is an improvement on top of an earlier patch by Izik. It
> increases pagefault scalability for SMP guests by allowing concurrent
> guest walking, allocation and instruction emulation on the fault path.
>
> The test being used is pft, which starts a number of threads
> allocating and writing malloc()'ed memory. pft.c can be found at
> http://lkml.org/lkml/2004/8/15/58
>
> The script being used is:
>
> bytes=$((400*1024*1024))
> ./pft -t -b$bytes -r10 -f1
> ./pft -b$bytes -r10 -f2
> ./pft -b$bytes -r10 -f3
> ./pft -b$bytes -r10 -f4
> ./pft -b$bytes -r10 -f8
>
> This is a 4-way guest.
>
> One important detail from the results is that there is no difference
> for the two threads case, but beyond that we see a clear improvement.
> follow_page() is showing up high in profiling, so I believe this
> is partly due to the fact that it does follow_page() twice while
> holding the lock, once in mmu_set_spte() from walk_addr() and again in
> mmu_set_spte() in fetch() - I'm looking into removing those duplicated
> calls for the same gfn.
>
> The patch still lacks the copy_from_user_inatomic() change in
> prefetch_page() to avoid a potential sleep in case the page is swapped
> out.
>
> Another issue is that now fetch() will re-read the pte's after
> instantiating a shadow page, but in theory they could be swapped out. I
> believe that is safe since walk_addr() just touched the pte's bringing
> them in from swap.
>

We need to convert that to kvm_read_guest_atomic() to avoid even that
theoretical race. If the read fails, we can simply return and let the
guest retry the faulting instruction.

> Switching the shadow lock to a spinlock also reduces the pagefault
> latency in > 2 CPU's case to 50%.
>
>
> stock KVM:
> Gb Rep Threads User System Wall flt/cpu/s fault/wsec
> 0 10 1 0.292s 5.744s 6.006s169638.179 168754.614
> 0 10 2 0.476s 7.340s 4.032s131005.143 236904.678
> 0 10 3 0.620s 10.972s 4.058s 88331.275 223355.103
> 0 10 4 0.588s 14.784s 4.062s 66610.464 221352.545
> 0 10 8 0.616s 15.100s 4.074s 65152.462 215861.148
>
> KVM + kvm-scale.patch:
> Gb Rep Threads User System Wall flt/cpu/s fault/wsec
> 0 10 1 0.328s 6.668s 7.002s146360.233 145746.022
> 0 10 2 0.472s 7.708s 4.036s125175.570 234513.121
> 0 10 3 0.500s 8.480s 3.038s114024.057 302227.769
> 0 10 4 0.508s 9.156s 3.012s105953.654 328156.474
> 0 10 8 1.144s 9.260s 3.028s 98417.544 311395.745
>
>
>

Very impressive results (and patch).

--
error compiling committee.c: too many arguments to function