Re: [PATCH v2] sched/topology, cpuset: Account for housekeeping CPUs to avoid empty cpumasks

[Valentin Schneider <valentin.schneider@arm.com>]

On 15/11/2019 17:18, Michal Koutný wrote:
> Hello.
> 
> On Thu, Nov 14, 2019 at 04:03:50PM +0000, Valentin Schneider <valentin.schneider@arm.com> wrote:
>> Michal, could I nag you for a reviewed-by? I'd feel a bit more confident
>> with any sort of approval from folks who actually do use cpusets.
> TL;DR I played with the v5.4-rc6 _without_ this fixup and I conclude it
> unnecessary (IOW my previous theoretical observation was wrong).
> 

Thanks for going through the trouble of testing the thing.

> 
> The original problem is non-issue with v2 cpuset controller, because
> effective_cpus are never empty. isolcpus doesn't take out cpuset CPUs,
> hotplug does. In the case, no online CPU remains in the cpuset, it
> inherits ancestor's non-empty cpuset.
> 

But we still take out the isolcpus from the domain span before handing it
over to the scheduler:

	cpumask_or(dp, dp, b->effective_cpus);                               
	cpumask_and(dp, dp, housekeeping_cpumask(HK_FLAG_DOMAIN));

But...

> I reproduced the problem with v1 (before your fix). However, in v1
> effective == allowed (we're destructive and overwrite allowed on
> hotunplug) and we already check the emptiness of 
> 
>   cpumask_intersects(cp->cpus_allowed, housekeeping_cpumask(HK_FLAG_DOMAIN)
> 
> few lines higher. I.e. the fixup adds redundant check against the empty
> sched domain production.
> 

...You're right, I've been misreading that as a '!is_sched_load_balance()'
condition ever since. Duh. So this condition will always catch cpusets than
only span outside the housekeeping domain, and my previous fixup will
catch newly-empty cpusets (due to HP). Perhaps it would've been cleaner to
merge the two, but as things stand this patch isn't needed (as you say).


I tried this out to really be sure (8 CPU SMP aarch64 qemu target):

  cd /sys/fs/cgroup/cpuset                                                                             
                                                                                                     
  mkdir cs1                                                                                            
  echo 1 > cs1/cpuset.cpu_exclusive                                                                    
  echo 0 > cs1/cpuset.mems                                                                             
  echo 0-4 > cs1/cpuset.cpus                                                                           
                                                                                                     
  mkdir cs2                                                                                            
  echo 1 > cs2/cpuset.cpu_exclusive                                                                    
  echo 0 > cs2/cpuset.mems                                                                             
  echo 5-7 > cs2/cpuset.cpus                                                                           
                                                                                                     
  echo 0 > cpuset.sched_load_balance

booted with
  
  isolcpus=6-7

It seems that creating a cpuset with CPUs only outside the housekeeping
domain is forbidden, so I'm creating cs2 with *one* CPU in the domain. When
I hotplug it out, nothing dies horribly:

  echo 0 > /sys/devices/system/cpu/cpu5/online
  [   24.688145] CPU5: shutdown
  [   24.689438] psci: CPU5 killed.
  [   24.714168] allowed=0-4 effective=0-4 housekeeping=0-5
  [   24.714642] allowed=6-7 effective=6-7 housekeeping=0-5
  [   24.715416] CPU5 attaching NULL sched-domain.

> Sorry for the noise and HTH,

Sure does, thanks!

> Michal
>