Re: [PATCH 02/27] sched/isolation: Introduce housekeeping per-cpu rwsem

[Phil Auld <pauld@redhat.com>]

Hi Waiman,

On Mon, Jun 23, 2025 at 01:34:58PM -0400 Waiman Long wrote:
> On 6/20/25 11:22 AM, Frederic Weisbecker wrote:
> > The HK_TYPE_DOMAIN isolation cpumask, and further the
> > HK_TYPE_KERNEL_NOISE cpumask will be made modifiable at runtime in the
> > future.
> > 
> > The affected subsystems will need to synchronize against those cpumask
> > changes so that:
> > 
> > * The reader get a coherent snapshot
> > * The housekeeping subsystem can safely propagate a cpumask update to
> >    the susbsytems after it has been published.
> > 
> > Protect against readsides that can sleep with per-cpu rwsem. Updates are
> > expected to be very rare given that CPU isolation is a niche usecase and
> > related cpuset setup happen only in preparation work. On the other hand
> > read sides can occur in more frequent paths.
> > 
> > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> 
> Thanks for the patch series and it certainly has some good ideas. However I
> am a bit concern about the overhead of using percpu-rwsem for
> synchronization especially when the readers have to wait for the completion
> on the writer side. From my point of view, during the transition period when
> new isolated CPUs are being added or old ones being removed, the reader will
> either get the old CPU data or the new one depending on the exact timing.
> The effect the CPU selection may persist for a while after the end of the
> critical section.
> 
> Can we just rely on RCU to make sure that it either get the new one or the
> old one but nothing in between without the additional overhead?
> 
> My current thinking is to make use CPU hotplug to enable better CPU
> isolation. IOW, I would shut down the affected CPUs, change the housekeeping
> masks and then bring them back online again. That means the writer side will
> take a while to complete.

The problem with this approach is that offlining a cpu effects all the other
cpus and causes latency spikes on other low latency tasks which may already be
running on other parts of the system.

I just don't want us to finally get to dynamic isolation and have it not
usable for the usecases asking for it. 

Cheers,
Phil

> 
> Cheers,
> Longman
> 
> > ---
> >   include/linux/sched/isolation.h |  7 +++++++
> >   kernel/sched/isolation.c        | 12 ++++++++++++
> >   kernel/sched/sched.h            |  1 +
> >   3 files changed, 20 insertions(+)
> > 
> > diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h
> > index f98ba0d71c52..8de4f625a5c1 100644
> > --- a/include/linux/sched/isolation.h
> > +++ b/include/linux/sched/isolation.h
> > @@ -41,6 +41,9 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type)
> >   		return true;
> >   }
> > +extern void housekeeping_lock(void);
> > +extern void housekeeping_unlock(void);
> > +
> >   extern void __init housekeeping_init(void);
> >   #else
> > @@ -73,6 +76,8 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type)
> >   	return true;
> >   }
> > +static inline void housekeeping_lock(void) { }
> > +static inline void housekeeping_unlock(void) { }
> >   static inline void housekeeping_init(void) { }
> >   #endif /* CONFIG_CPU_ISOLATION */
> > @@ -84,4 +89,6 @@ static inline bool cpu_is_isolated(int cpu)
> >   	       cpuset_cpu_is_isolated(cpu);
> >   }
> > +DEFINE_LOCK_GUARD_0(housekeeping, housekeeping_lock(), housekeeping_unlock())
> > +
> >   #endif /* _LINUX_SCHED_ISOLATION_H */
> > diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
> > index 83cec3853864..8c02eeccea3b 100644
> > --- a/kernel/sched/isolation.c
> > +++ b/kernel/sched/isolation.c
> > @@ -18,12 +18,24 @@ static cpumask_var_t housekeeping_cpumasks[HK_TYPE_MAX];
> >   unsigned long housekeeping_flags;
> >   EXPORT_SYMBOL_GPL(housekeeping_flags);
> > +DEFINE_STATIC_PERCPU_RWSEM(housekeeping_pcpu_lock);
> > +
> >   bool housekeeping_enabled(enum hk_type type)
> >   {
> >   	return !!(housekeeping_flags & BIT(type));
> >   }
> >   EXPORT_SYMBOL_GPL(housekeeping_enabled);
> > +void housekeeping_lock(void)
> > +{
> > +	percpu_down_read(&housekeeping_pcpu_lock);
> > +}
> > +
> > +void housekeeping_unlock(void)
> > +{
> > +	percpu_up_read(&housekeeping_pcpu_lock);
> > +}
> > +
> >   int housekeeping_any_cpu(enum hk_type type)
> >   {
> >   	int cpu;
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index 475bb5998295..0cdb560ef2f3 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -46,6 +46,7 @@
> >   #include <linux/mm.h>
> >   #include <linux/module.h>
> >   #include <linux/mutex_api.h>
> > +#include <linux/percpu-rwsem.h>
> >   #include <linux/plist.h>
> >   #include <linux/poll.h>
> >   #include <linux/proc_fs.h>
> 
> 

--