Re: [PATCH 2/2] xen: spread page scrubbing across all idle CPU

[Bob Liu <bob.liu@oracle.com>]

On 06/11/2014 06:45 PM, George Dunlap wrote:
> On Wed, Jun 11, 2014 at 11:36 AM, Bob Liu <bob.liu@oracle.com> wrote:
>>
>> On 06/11/2014 06:13 PM, George Dunlap wrote:
>>> On Wed, Jun 11, 2014 at 9:13 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 11.06.14 at 04:51, <bob.liu@oracle.com> wrote:
>>>>> On 06/10/2014 10:12 PM, Jan Beulich wrote:
>>>>>>>>> On 10.06.14 at 14:18, <lliubbo@gmail.com> wrote:
>>>>>>> +    if( is_tasklet )
>>>>>>> +        tasklet_schedule_on_cpu(&global_scrub_tasklet, cpu);
>>>>>>
>>>>>> So you re-schedule this tasklet immediately - while this may be
>>>>>> acceptable inside the hypervisor, did you consider the effect this
>>>>>> will have on the guest (normally Dom0)? Its respective vCPU won't
>>>>>> get run _at all_ until you're done scrubbing.
>>>>>>
>>>>>
>>>>> Yes, that's a problem. I don't have any better idea right now.
>>>>>
>>>>> What I'm trying is doing the scrubbing on current CPU as well as on all
>>>>> idle vcpus in parallel.
>>>>> I also considered your suggestion about doing the scrubbing in the
>>>>> background as well as on the allocation path. But I think it's more
>>>>> unacceptable for users to get blocked randomly for a uncertain time when
>>>>> allocating a large mount of memory.
>>>>> That's why I still chose the sync way that once 'xl destroy' return all
>>>>> memory are scrubbed.
>>>>
>>>> But I hope you realize that in the current shape, with the shortcomings
>>>> pointed out un-addressed, there's no way for this to go in.
>>>
>>> Would it make more sense to do something like the following:
>>> * Have a "clean" freelist and a "dirty" freelist
>>> * When destroying a domain, simply move pages to the dirty freelist
>>> * Have idle vcpus scrub the dirty freelist before going to sleep
>>>  - ...and wake up idle vcpus to do some scrubbing when adding pages to
>>> the dirty freelist
>>> * In alloc_domheap_pages():
>>>  - If there are pages on the "clean" freelist, allocate them
>>>  - If there are no pages on the "clean" freelist but there are on the
>>> "dirty" freelist, scrub pages from the "dirty" freelist synchronously.
>>>
>>
>> Thank you very much for your suggestion, it's similar as Jan suggested.
>>
>> My concern of this approach is in some bad situation the allocation path
>> may be blocked for a long time waiting for scrubbing "dirty" freelist.
>>
>> What the users see is it's much faster to destroy a domain but may
>> slower to create an new one(or slow down other routines need to alloc
>> large memory).
> 
> Yes, so on a system with near 100% memory usage, a reboot of a large
> VM would mean short destroy, long start-up, rather than long destroy,
> short start-up.  End-to-end it's basically the same; but on a system
> with less memory usage, both operations are significantly faster.
> 

Okay, I'll follow your suggestion.

And thanks everyone!

-- 
Regards,
-Bob