Re: [PATCH 2/2] xen: spread page scrubbing across all idle CPU

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 11/06/14 11:36, Bob Liu wrote:
> On 06/11/2014 06:13 PM, George Dunlap wrote:
>> On Wed, Jun 11, 2014 at 9:13 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 11.06.14 at 04:51, <bob.liu@oracle.com> wrote:
>>>> On 06/10/2014 10:12 PM, Jan Beulich wrote:
>>>>>>>> On 10.06.14 at 14:18, <lliubbo@gmail.com> wrote:
>>>>>> +    if( is_tasklet )
>>>>>> +        tasklet_schedule_on_cpu(&global_scrub_tasklet, cpu);
>>>>> So you re-schedule this tasklet immediately - while this may be
>>>>> acceptable inside the hypervisor, did you consider the effect this
>>>>> will have on the guest (normally Dom0)? Its respective vCPU won't
>>>>> get run _at all_ until you're done scrubbing.
>>>>>
>>>> Yes, that's a problem. I don't have any better idea right now.
>>>>
>>>> What I'm trying is doing the scrubbing on current CPU as well as on all
>>>> idle vcpus in parallel.
>>>> I also considered your suggestion about doing the scrubbing in the
>>>> background as well as on the allocation path. But I think it's more
>>>> unacceptable for users to get blocked randomly for a uncertain time when
>>>> allocating a large mount of memory.
>>>> That's why I still chose the sync way that once 'xl destroy' return all
>>>> memory are scrubbed.
>>> But I hope you realize that in the current shape, with the shortcomings
>>> pointed out un-addressed, there's no way for this to go in.
>> Would it make more sense to do something like the following:
>> * Have a "clean" freelist and a "dirty" freelist
>> * When destroying a domain, simply move pages to the dirty freelist
>> * Have idle vcpus scrub the dirty freelist before going to sleep
>>  - ...and wake up idle vcpus to do some scrubbing when adding pages to
>> the dirty freelist
>> * In alloc_domheap_pages():
>>  - If there are pages on the "clean" freelist, allocate them
>>  - If there are no pages on the "clean" freelist but there are on the
>> "dirty" freelist, scrub pages from the "dirty" freelist synchronously.
>>
> Thank you very much for your suggestion, it's similar as Jan suggested.
>
> My concern of this approach is in some bad situation the allocation path
> may be blocked for a long time waiting for scrubbing "dirty" freelist.
>
> What the users see is it's much faster to destroy a domain but may
> slower to create an new one(or slow down other routines need to alloc
> large memory).
>
> If you and Jan think it's fine I can make a patch towards this way.
>

One way or another the pages need to be scrubbed.  All you are doing is
juggling when this time hit gets taken.  I would agree that having the
toolstack take the time hit as part of repeated domain_kill() hypercalls
is antisocial, especially as the toolstack/dom0 has far more important
things to be doing than being blocked in Xen.

I too would agree with Jan's suggestion and think it is the best
solution to the problem at hand.

~Andrew