Re: [PATCH 0/2] Simplify RCU freeing of shadow pages

[Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>]

On 04/22/2012 09:24 PM, Avi Kivity wrote:

> On 04/20/2012 07:05 AM, Xiao Guangrong wrote:
>> On 04/20/2012 12:26 AM, Avi Kivity wrote:
>>
>>> This patchset simplifies the freeing by RCU of mmu pages.
>>>
>>> Xiao, I'm sure you thought of always freeing by RCU.  Why didn't you choose
>>> this way?  I saves a couple of atomics in the fast path.
>>>
>>
>>
>> Avi, we have discussed it last year:
>>
>> https://lkml.org/lkml/2011/6/29/177
> 
> Here are my own measurements (4 way guest on 24-way host):
> 
> time make -j8
> 
> ept=0, with patch
> 
> real    3m54.526s
> user    11m7.074s
> sys    3m29.678s
> 
> real    3m53.806s
> user    11m13.654s
> sys    3m26.577s
> 
> real    3m53.578s
> user    11m13.491s
> sys    3m26.095s
> 
> real    3m53.956s
> user    11m14.866s
> sys    3m25.075s
> 
> ept=1, with patch
> 
> 
> real    2m36.510s
> user    8m18.905s
> sys    1m31.712s
> 
> real    2m36.642s
> user    8m18.706s
> sys    1m31.898s
> 
> real    2m36.952s
> user    8m20.041s
> sys    1m31.929s
> 
> real    2m37.036s
> user    8m20.204s
> sys    1m31.947s
> 
> ept=0, unpatched
> 
> real    3m51.299s
> user    11m23.770s
> sys    3m28.049s
> 
> real    3m52.982s
> user    11m28.419s
> sys    3m29.645s
> 
> real    3m52.634s
> user    11m27.325s
> sys    3m29.752s
> 
> real    3m53.696s
> user    11m30.215s
> sys    3m30.146s
> 
> ept=1, unpatched
> 
> real    2m37.134s
> user    8m26.707s
> sys    1m33.152s
> 
> real    2m36.838s
> user    8m26.255s
> sys    1m33.078s
> 
> real    2m36.826s
> user    8m26.917s
> sys    1m32.449s
> 
> real    2m36.877s
> user    8m26.226s
> sys    1m32.074s
> 
> 
> So there is a small degradation with ept=0 and an even smaller
> improvement with ept=1.  But certainly we should give more weight to
> ept=1, and also factor in the code simplification.
> 


I agree.

>> I have optimized/simplified for "write flood" a lot, but, unfortunately,
>> the zapping sp is still frequently.
>>
>> Maybe we can cache the zapped sp in a invalid_sp_list to reduce the
>> frequency.
> 
> To recover the time for re-initializing sp->spt?


Batch free shadow page, for example, list 10 sp in the invalid_list, then
just need one rcu_free to free them.

> 
>> Or, we may use SLAB_DESTROY_BY_RCU to free the shadow page.
> 
> IIUC a kmem_cache_alloc() can reuse an allocation freed using
> SLAB_DESTROY_BY_RCU before a grace period has elapsed.
> 


Yes. So we need a refcount out of mmu-lock, and see the shadow page
is stil pointed the upper level (prevent not re-alloced). This way
may make code more complex.