unexpected behaviour of cgroups v1 on 6.12 kernel

[Chris Friesen <chris.friesen@windriver.com>]

Hi all,

I'm not subscribed to the list, so please CC me on replies.

I'm seeing some unexpected behaviour with the cpu/cpuset cgroups 
controllers (with cgroups v1) on 6.12.18 with PREEMPT_RT enabled.

I set up the following cgroup hierarchy for both cpu and cpuset cgroups:

foo:   cpu 15, shares 1024
foo/a: cpu 15, shares 1024

bar:   cpu 15-19, shares 1024
bar/a: cpu 15, shares 1024
bar/b: cpu 16, shares 1024
bar/c: cpu 17, shares 1024
bar/d: cpu 18, shares 1024
bar/e: cpu 19, shares 1024

I then ran a single cpu hog in each of the leaf-node cgroups in the 
default SCHED_OTHER class.

As expected, the tasks in bar/b, bar/c, bar/d, and bar/e each got 100% 
of their CPU.  What I didn't expect was that the task running in foo/a 
got 83.3%, while the task in bar/a got 16.7%.  Is this expected?

I guess what I'm asking here is whether the cgroups CPU share 
calculation is supposed to be performed separately per CPU, or whether 
it's global but somehow scaled by the number of CPUs that the cgroup is 
runnable on, so that the total CPU time of group "bar" is expected to be 
5x the total CPU time of group "foo".

I then killed all the tasks in bar/b, bar/c, bar/d, and bar/e.  The 
tasks in foo/a and bar/a continued for a while at 83/16, then moved to 
80/20, and only about 75 seconds later finally moved to 50/50.    Is 
this long time to "rebalance" expected?  If so, can this time be 
modified by the admin user at runtime or is it inherent in the code?

As further data, if I have tasks in foo/a, bar/a, bar/b, bar/c then 
foo/a gets 75%, bar/a gets 25%, bar/b and bar/c both get 100%.

If I have tasks in foo/a, bar/a, bar/b then foo/a gets 66%, bar/a gets 
33%, bar/b gets 100%.  (But it started out with foo/a getting 75% and 
switched 10s of seconds later, which seems odd.)

Thanks,
Chris