Re: [PATCH RFC] rcu/nocb: Provide default all-CPUs mask for RCU_NOCB_CPU=y

Paul E. McKenney
2022-04-11T15:41:17Z

On Mon, Apr 11, 2022 at 11:17:02AM -0400, Joel Fernandes wrote:
> Hi Paul,
>
> On Fri, Apr 8, 2022 at 4:54 PM Paul E. McKenney <paulmck@kernel.org> wrote:
> [...]
> > > > > > > > > And is it really all -that- hard to specify an additional boot parameter
> > > > > > > > > across ChromeOS devices? Android seems to manage it. ;-)
> > > > > > > >
> > > > > > > > That's not the hard part I think. The hard part is to make sure a
> > > > > > > > future Linux user who is not an RCU expert does not forget to turn it
> > > > > > > > on. ChromeOS is not the only OS that I've seen someone forget to do it
> > > > > > > > ;-D. AFAIR, there were Android devices too in the past where I saw
> > > > > > > > this forgotten. I don't think we should rely on the users doing the
> > > > > > > > right thing (as much as possible).
> > > > > > > >
> > > > > > > > The single kernel binary point makes sense but in this case, I think
> > > > > > > > the bigger question that I'd have is what is the default behavior and
> > > > > > > > what do *most* users of RCU want. So we can keep sane defaults for the
> > > > > > > > majority and reduce human errors related to configuration.
> > > > > > >
> > > > > > > If both the ChromeOS and Android guys need it, I could reinstate the
> > > > > > > old RCU_NOCB_CPU_ALL Kconfig option. This was removed due to complaints
> > > > > > > about RCU Kconfig complexity, but I believe that Reviewed-by from ChromeOS
> > > > > > > and Android movers and shakers would overcome lingering objections.
> > > > > > >
> > > > > > > Would that help?
> > > > > >
> > > > > > Yes, I think I would love for such a change. I am planning to add a
> > > > > > test to ChromeOS to check whether config options were correctly set
> > > > > > up. So I can test for both the RCU_NOCB_CPU options.
> > > > >
> > > > > Very good!
> > > > >
> > > > > Do you love such a change enough to create the patch and to collect
> > > > > convincing Reviewed-by tags?
> > > >
> > > > Yes sure, just so I understand - basically I have to make the code in
> > > > my patch run when RCU_NOCB_CPU_ALL option is passed (and keep the
> > > > option default disabled), but otherwise default to the current
> > > > behavior, right?
> > >
> > > Sorry rephrasing, "make the code in my patch run when the new
> > > CONFIG_RCU_NOCB_CPU_ALL is enabled".
> >
> > Here is what I believe you are proposing:
> >
> >
> > --- rcu_nocbs rcu_nocbs=???
> >
> > CONFIG_RCU_NOCB_CPU_ALL=n [1] [2] [3]
> >
> > CONFIG_RCU_NOCB_CPU_ALL=y [4] [4] [3]
>
> It is always a pleasure to read your well thought out emails ;-)
>
> >
> > [1] No CPUs are offloaded at boot. CPUs cannot be offloaded at
> > runtime.
> >
> > [2] No CPUs are offloaded at boot, but any CPU can be offloaded
> > (and later de-offloaded) at runtime.
> >
> > [3] The set of CPUs that are offloaded at boot are specified by the
> > mask, represented above with "???". The CPUs that are offloaded
> > at boot can be de-offloaded and offloaded at runtime. The CPUs
> > not offloaded at boot cannot be offloaded at runtime.
>
> Hmm, in other words you are saying that in current code, if only
> select CPUs are offloaded at boot - then only those can be toggled,
> but the others are deemed not offload-able? I am happy to leave that
> quirk/behavior alone as I don't care much right now (for our use
> cases) for runtime toggling.

That is intentional behavior. To see why, suppose that only CPU 0 was
offloaded at boot. Now try offloading some other CPU. A large quantity
of previously read-only data becomes read-write. Synchronization is
not pretty.

On the other hand, default-enabling (de-)offloading on all CPUs creates
lots of unneeded rcuo kthreads.

So we didn't get here by accident. ;-)

If this becomes a problem, I would be thinking in terms of an additional
kernel-boot parameter that made all CPUs offloadable by default. But if
you have a better idea, please do not keep it a secret!

> > [4] All CPUs are offloaded at boot, and any CPU can be de-offloaded
> > and offloaded at runtime. This is the same behavior that
> > you would currently get with CONFIG_RCU_NOCB_CPU_ALL=n and
> > rcu_nocbs=0-N.
>
> Yes, this is the behavior I intend. So then there would not be a need
> to pass a mask (and I suspect for a large number of users, it
> simplifies boot params).

Very good, and from what I can see, this should work for everyone.

> > I believe that Steve Rostedt's review would carry weight for ChromeOS,
> > however, I am suffering a senior moment on the right person for Android.
>
> I think for Android, Kalesh Singh is in the kernel team and Tim Murray
> is the performance lead. They could appropriately represent their RCU
> needs.

Sounds good! Please collect a Reviewed-by from one or both of them.

Thanx, Paul