Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

[Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>]

On 2015/08/25 8:15, Paul Turner wrote:
> On Mon, Aug 24, 2015 at 3:49 PM, Tejun Heo <tj@kernel.org> wrote:
>> Hello,
>>
>> On Mon, Aug 24, 2015 at 03:03:05PM -0700, Paul Turner wrote:
>>>> Hmm... I was hoping for an actual configurations and usage scenarios.
>>>> Preferably something people can set up and play with.
>>>
>>> This is much easier to set up and play with synthetically.  Just
>>> create the 10 threads and 100 threads above then experiment with
>>> configurations designed at guaranteeing the set of 100 threads
>>> relatively uniform throughput regardless of how many are active.   I
>>> don't think trying to run a VM stack adds anything except complexity
>>> of reproduction here.
>>
>> Well, but that loses most of details and why such use cases matter to
>> begin with.  We can imagine up stuff to induce arbitrary set of
>> requirements.
>
> All that's being proved or disproved here is that it's difficult to
> coordinate the consumption of asymmetric thread pools using nice.  The
> constraints here were drawn from a real-world example.
>
>>
>>>> I take that the
>>>> CPU intensive helper threads are usually IO workers?  Is the scenario
>>>> where the VM is set up with a lot of IO devices and different ones may
>>>> consume large amount of CPU cycles at any given point?
>>>
>>> Yes, generally speaking there are a few major classes of IO (flash,
>>> disk, network) that a guest may invoke.  Each of these backends is
>>> separate and chooses its own threading.
>>
>> Hmmm... if that's the case, would limiting iops on those IO devices
>> (or classes of them) work?  qemu already implements IO limit mechanism
>> after all.
>
> No.
>
> 1) They should proceed at the maximum rate that they can that's still
> within their provisioning budget.
> 2) The cost/IO is both inconsistent and changes over time.  Attempting
> to micro-optimize every backend for this is infeasible, this is
> exactly the type of problem that the scheduler can usefully help
> arbitrate.
> 3) Even pretending (2) is fixable, dynamically dividing these
> right-to-work tokens between different I/O device backends is
> extremely complex.
>

I think I should explain my customer's use case of qemu + cpuset/cpu (via libvirt)

(1) Isolating hypervisor thread.
    As already discussed, hypervisor threads are isolated by cpuset. But their purpose
    is to avoid _latency_ spike caused by hypervisor behavior. So, "nice" cannot be solution
    as already discussed.

(2) Fixed rate vcpu service.
    With using cpu controller's quota/period feature, my customer creates  vcpu models like
    Low(1GHz), Mid(2GHz), High(3GHz) for IaaS system.

    To do this, each vcpus should be quota-limited independently, with per-thread cpu control.

Especially, the method (1) is used in several enterprise customers for stabilizing their system.

Sub-process control should be provided by some way.

Thanks,
-Kame



>>
>> Anyways, a point here is that threads of the same process competing
>> isn't a new problem.  There are many ways to make those threads play
>> nice as the application itself often has to be involved anyway,
>> especially for something like qemu which is heavily involved in
>> provisioning resources.
>
> It's certainly not a new problem, but it's a real one, and it's
> _hard_.  You're proposing removing the best known solution.
>
>>
>> cgroups can be a nice brute-force add-on which lets sysadmins do wild
>> things but it's inherently hacky and incomplete for coordinating
>> threads.  For example, what is it gonna do if qemu cloned vcpus and IO
>> helpers dynamically off of the same parent thread?
>
> We're talking about sub-process usage here.  This is the application
> coordinating itself, NOT the sysadmin.  Processes are becoming larger
> and larger, we need many of the same controls within them that we have
> between them.
>
>>   It requires
>> application's cooperation anyway but at the same time is painful to
>> actually interact from those applications.
>
> As discussed elsewhere on thread this is really not a problem if you
> define consistent rules with respect to which parts are managed by
> who.  The argument of potential interference is no different to
> messing with an application's on-disk configuration behind its back.
> Alternate strawmen which greatly improve this from where we are today
> have also been proposed.
>
>>
>> Thanks.
>>
>> --
>> tejun
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>