Re: [PATCH v3 07/15] KVM: MMU: introduce nulls desc

Xiao Guangrong
2013-12-05T15:31:00Z

On Dec 5, 2013, at 9:50 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:

> GOn Tue, Dec 03, 2013 at 03:10:48PM +0800, Xiao Guangrong wrote:
>> On 11/28/2013 04:53 PM, Xiao Guangrong wrote:
>>> On 11/27/2013 03:31 AM, Marcelo Tosatti wrote:
>>>> On Tue, Nov 26, 2013 at 11:21:37AM +0800, Xiao Guangrong wrote:
>>>>> On 11/26/2013 02:12 AM, Marcelo Tosatti wrote:
>>>>>> On Mon, Nov 25, 2013 at 02:29:03PM +0800, Xiao Guangrong wrote:
>>>>>>>>> Also, there is no guarantee of termination (as long as sptes are
>>>>>>>>> deleted with the correct timing). BTW, can't see any guarantee of
>>>>>>>>> termination for rculist nulls either (a writer can race with a lockless
>>>>>>>>> reader indefinately, restarting the lockless walk every time).
>>>>>>>>
>>>>>>>> Hmm, that can be avoided by checking dirty-bitmap before rewalk,
>>>>>>>> that means, if the dirty-bitmap has been set during lockless write-protection,
>>>>>>>> it�s unnecessary to write-protect its sptes. Your idea?
>>>>>>> This idea is based on the fact that the number of rmap is limited by
>>>>>>> RMAP_RECYCLE_THRESHOLD. So, in the case of adding new spte into rmap,
>>>>>>> we can break the rewalk at once, in the case of deleting, we can only
>>>>>>> rewalk RMAP_RECYCLE_THRESHOLD times.
>>>>>>
>>>>>> Please explain in more detail.
>>>>>
>>>>> Okay.
>>>>>
>>>>> My proposal is like this:
>>>>>
>>>>> pte_list_walk_lockless()
>>>>> {
>>>>> restart:
>>>>>
>>>>> + if (__test_bit(slot->arch.dirty_bitmap, gfn-index))
>>>>> + return;
>>>>>
>>>>> code-doing-lockless-walking;
>>>>> ......
>>>>> }
>>>>>
>>>>> Before do lockless-walking, we check the dirty-bitmap first, if
>>>>> it is set we can simply skip write-protection for the gfn, that
>>>>> is the case that new spte is being added into rmap when we lockless
>>>>> access the rmap.
>>>>
>>>> The dirty bit could be set after the check.
>>>>
>>>>> For the case of deleting spte from rmap, the number of entry is limited
>>>>> by RMAP_RECYCLE_THRESHOLD, that is not endlessly.
>>>>
>>>> It can shrink and grow while lockless walk is performed.
>>>
>>> Yes, indeed.
>>>
>>> Hmmm, another idea in my mind to fix this is encoding the position into
>>> the reserved bits of desc->more pointer, for example:
>>>
>>> +------+ +------+ +------+
>>> rmapp -> |Desc 0| -> |Desc 1| -> |Desc 2|
>>> +------+ +------+ +------+
>>>
>>> There are 3 descs on the rmap, and:
>>> rmapp = &desc0 | 1UL | 3UL << 50;
>>> desc0->more = desc1 | 2UL << 50;
>>> desc1->more = desc0 | 1UL << 50
>>> desc2->more = &rmapp | 1UL; (The nulls pointer)
>>>
>>> We will walk to the next desc only if the "position" of current desc
>>> is >= the position of next desc. That can make sure we can reach the
>>> last desc anyway.
>>>
>>> And in order to avoiding doing too many "rewalk", we will goto the
>>> slow path (do walk with holding the lock) instead when retry the walk
>>> more that N times.
>>
>> How about this idea? Or you guys still prefer to the idea of lockless on
>> first-level?
>
> Xiao,
>
> Is it not the case that simply moving to the slow path once a maximum of
> rewalks has been reached enough? (looks a like a good solution).

In some cases, the lockless walker will do endless-walking on desc and
without rewalk, consider this case:

there are two descs: desc1 and desc2 who is pointed by desc1->next：
desc1->next = desc2.

CPU 0 CPU 1

lockless walk on desc1
deleting desc1 from the rmap
lockless walk on desc2 (desc1->next)
delete desc2 from the rmap
add desc1
add desc2, then desc2->next = desc1

lockless walk on desc1
delete desc2
delete desc1
add desc2
add desc1; the desc1->next = desc2
lockless walk on desc2

……

Then, the walker is endlessly walking on desc1 and desc2 without any rewalk.

>
> Please move lockless rcu walking code to generic code where it belongs.

Okay.