Re: [RFC] CPU hard limits

Srivatsa Vaddagiri
2009-06-07T10:13:19Z

On Fri, Jun 05, 2009 at 05:18:13AM -0700, Paul Menage wrote:
> Well yes, it's true that you *could* just enforce shares over a
> granularity of minutes, and limits over a granularity of milliseconds.
> But why would you? It could well make sense that you can adjust the
> granularity over which shares are enforced - e.g. for batch jobs, only
> enforcing over minutes or tens of seconds might be fine. But if you're
> doing the fine-grained accounting and scheduling required for the
> tight hard limit enforcement, it doesn't seem as though it should be
> much harder to enforce shares at the same granularity for those
> cgroups that matter. In fact I thought that's what CFS already did -
> updated the virtual time accounting at each context switch, and picked
> the runnable child with the oldest virtual time. (Maybe someone like
> Ingo or Peter who's more familiar than I with the CFS implementation
> could comment here?)

Using shares to guarantee resources over short period (<2-3 seconds) works
just well on a single CPU. The complexity is with multi-cpu case, where CFS can
take a long time to converge to a fair point. This is because fairness is based
on rebalancing tasks equally across all CPUs.

For something like 4 tasks on 4 CPUs, it will converge pretty quickly
(2-3 seconds):

[top o/p refreshed every 2sec on 2.6.30-rc5-tip]

14753 vatsa 20 0 63812 1072 924 R 99.9 0.0 0:39.54 hog
14754 vatsa 20 0 63812 1072 924 R 99.9 0.0 0:38.69 hog
14756 vatsa 20 0 63812 1076 924 R 99.9 0.0 0:38.27 hog
14755 vatsa 20 0 63812 1072 924 R 99.6 0.0 0:38.27 hog

whereas for something like 5 tasks on 4 CPUs, it will take a sufficiently
longer time (>30 seconds)

[top o/p refreshed every 2sec]:

14754 vatsa 20 0 63812 1072 924 R 86.0 0.0 2:06.45 hog
14766 vatsa 20 0 63812 1072 924 R 83.0 0.0 0:07.95 hog
14756 vatsa 20 0 63812 1076 924 R 81.7 0.0 2:06.48 hog
14753 vatsa 20 0 63812 1072 924 R 78.7 0.0 2:07.10 hog
14755 vatsa 20 0 63812 1072 924 R 69.4 0.0 2:05.62 hog

[top o/p refreshed every 120sec]:

14766 vatsa 20 0 63812 1072 924 R 90.1 0.0 5:57.22 hog
14755 vatsa 20 0 63812 1072 924 R 84.8 0.0 8:01.61 hog
14754 vatsa 20 0 63812 1072 924 R 77.3 0.0 7:52.04 hog
14753 vatsa 20 0 63812 1072 924 R 74.1 0.0 7:29.01 hog
14756 vatsa 20 0 63812 1076 924 R 73.5 0.0 7:34.69 hog

[Note that even over 2min, we haven't achieved perfect fairness]

> > By having hard-limits, we are
> > "reserving" (potentially idle) slots where the high-priority group can run and
> > claim its guaranteed share almost immediately.

On further thinking, this is not as simple as that. In above example of
5 tasks on 4 CPUs, we could cap each task at a hard limit of 80%
(4 CPUs/5 tasks), which is still not sufficient to ensure that each
task gets the perfect fairness of 80%! Not just that, hard-limit
for a group (on each CPU) will have to be adjusted based on its task
distribution. For ex: a group that has a hard-limit of 25% on a 4-cpu
system and that has a single task, is entitled to claim a whole CPU. So
the per-cpu hard-limit for the group should be 100% on whatever CPU the
task is running. This adjustment of per-cpu hard-limit should happen
whenever the task distribution of the group across CPUs change - which
in theory would require you to monitor every task exit/migration
event and readjust limits, making it very complex and high-overhead.

Balbir,
I dont think guarantee can be met easily thr' hard-limits in
case of CPU resource. Atleast its not as straightforward as in case of
memory!

- vatsa