Re: CFS Bandwidth Control - Test results of cgroups tasks pinned vs unpinnede

[Paul Turner <pjt@google.com>]

On 09/07/11 08:20, Srivatsa Vaddagiri wrote:
> [Apologies if you get this email multiple times - there is some email
> client config issue that I am fixing up]
>
> * Paul Turner<pjt@google.com>  [2011-06-21 12:48:17]:
>
>> Hi Kamalesh,
>>
>> Can you see what things look like under v7?
>>
>> There's been a few improvements to quota re-distribution that should
>> hopefully help your test case.
>>
>> The remaining idle% I see on my machines appear to be a product of
>> load-balancer inefficiency.
>

Hey Srivatsa,

Thanks for taking another look at this -- sorry for the delayed reply!

> which is quite a complex problem to solve! I am still surprised that
> we can't handle 32 cpuhogs on a 16-cpu system very easily. The tasks seem to
> hop around madly rather than settle down as 2 tasks/cpu. Kamalesh, can you post
> the exact count of migrations we saw on latest tip over a 20-sec window?
>
> Anyway, here's a "hack" to minimize the idle time induced due to load-balance
> issues. It brings down idle time from 7+% to ~0% ..I am not too happy about
> this, but I don't see any other simpler solutions to solve the idle time issue
> completely (other than making load-balancer completely fair!).

Hum,

So BWC returns bandwidth on voluntary sleep to the parent, so the most 
we can really lose is NR_CPUS * 1ms (how much a cpu keeps in case the 
entity re-wakes up quickly).  Technically we could lose another few ms 
if there's not enough BW left to bother distributing and we're near the 
end of the period; but I think that works out to another 6ms or so at 
worst.

As discussed in the long thread dangling off this; it's load-balance 
that's at fault -- allowing steal time is just hiding this by instead 
letting cpus run over quota within a period.

If you for example set-up a deadline oriented test that tried to 
accomplish the same amount of work (without bandwidth limits) and threw 
away the rest of the work when it reached period expiration (a benchmark 
I've been meaning to write and publish as a more general load-balance 
test actually); then I suspect we'd see similar problems; and sadly, 
this case is both more representative of real-world performance and not 
fixable by something like steal-time.

So... we're probably better off trying to improve LB; I raised it in 
another reply on the chain but the NOHZ vs ticks ilb numbers look pretty 
compelling as an area for improvement in this regard.

Thanks!

- Paul
>
> --
>
> Fix excessive idle time reported when cgroups are capped.  The patch
> introduces the notion of "steal" (or "grace") time which is the surplus
> time/bandwidth each cgroup is allowed to consume, subject to a maximum
> steal time (sched_cfs_max_steal_time_us). Cgroups are allowed this "steal"
> or "grace" time when the lone task running on a cpu is about to be throttled.