Re: [RFC] concurrent guest walker and instruction emulation

[Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>]

Marcelo Tosatti wrote:
> Hi, 
>
> The following is an improvement on top of an earlier patch by Izik. It
> increases pagefault scalability for SMP guests by allowing concurrent
> guest walking, allocation and instruction emulation on the fault path.
>
> The test being used is pft, which starts a number of threads
> allocating and writing malloc()'ed memory. pft.c can be found at
> http://lkml.org/lkml/2004/8/15/58
>
> The script being used is:
>
> bytes=$((400*1024*1024))
> ./pft -t -b$bytes -r10 -f1
> ./pft -b$bytes -r10 -f2
> ./pft -b$bytes -r10 -f3
> ./pft -b$bytes -r10 -f4
> ./pft -b$bytes -r10 -f8
>
> This is a 4-way guest.
>
> One important detail from the results is that there is no difference
> for the two threads case, but beyond that we see a clear improvement.
> follow_page() is showing up high in profiling, so I believe this
> is partly due to the fact that it does follow_page() twice while
> holding the lock, once in mmu_set_spte() from walk_addr() and again in
> mmu_set_spte() in fetch() - I'm looking into removing those duplicated
> calls for the same gfn.
>
> The patch still lacks the copy_from_user_inatomic() change in
> prefetch_page() to avoid a potential sleep in case the page is swapped
> out. 
>
> Another issue is that now fetch() will re-read the pte's after
> instantiating a shadow page, but in theory they could be swapped out. I
> believe that is safe since walk_addr() just touched the pte's bringing
> them in from swap.
>   

We need to convert that to kvm_read_guest_atomic() to avoid even that 
theoretical race.  If the read fails, we can simply return and let the 
guest retry the faulting instruction.

> Switching the shadow lock to a spinlock also reduces the pagefault
> latency in > 2 CPU's case to 50%.
>
>
> stock KVM:
>  Gb Rep Threads   User      System     Wall flt/cpu/s fault/wsec
>   0  10    1    0.292s      5.744s   6.006s169638.179 168754.614
>   0  10    2    0.476s      7.340s   4.032s131005.143 236904.678
>   0  10    3    0.620s     10.972s   4.058s 88331.275 223355.103
>   0  10    4    0.588s     14.784s   4.062s 66610.464 221352.545
>   0  10    8    0.616s     15.100s   4.074s 65152.462 215861.148
>
> KVM + kvm-scale.patch:
>  Gb Rep Threads   User      System     Wall flt/cpu/s fault/wsec   
>   0  10    1    0.328s      6.668s   7.002s146360.233 145746.022
>   0  10    2    0.472s      7.708s   4.036s125175.570 234513.121
>   0  10    3    0.500s      8.480s   3.038s114024.057 302227.769
>   0  10    4    0.508s      9.156s   3.012s105953.654 328156.474
>   0  10    8    1.144s      9.260s   3.028s 98417.544 311395.745
>
>
>   

Very impressive results (and patch).

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace