Re: RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

Florent Carli
2022-10-07T15:06:37Z

Thank you all.

I make sure the rcu_preempt kthread affinity was set on a non-isolated
core (as well as all unbounded rcu* threads) so I guess it is not a
competing issue.

What I’m running on those isolated cores are qemu/kvm vcpu tasks, and
I guess they might spend some time in the kernel… in which case I
understand it’s completely normal to see some rcuc activity.
And so I conclude that using isolcpus + nohz_full (which includes
rcu_nocbs) + rcu_nocb_poll + rcutree.kthread_prio is the way to go.

Thanks everyone.

On Thu, Oct 6, 2022 at 4:13 PM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> On Thu, Oct 06, 2022 at 01:31:43PM +0000, Joel Fernandes wrote:
> > On Wed, Oct 05, 2022 at 06:03:39PM +0200, Daniel Bristot de Oliveira wrote:
> > > On 10/5/22 18:01, Florent Carli wrote:
> > > > Hello,
> > > >
> > > > I'm trying to isolate some cores to run a CPU-bound real-time task and
> > > > even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc
> > > > threads wake up, and I get RCU stall warnings on the isolated core.
> >
> > Isolating a CPU does not mean that RCU activity is forbidden on it, even with
> > nohz_full, if the CPU enters the kernel (say syscall or interrupt), then it
> > may enter a read-side critical section so RCU will be watching it, and such
> > CPU will have to report quiescent state.
>
> What Joel said!
>
> Plus, if you keep that CPU out of the kernel (including avoiding it ever
> taking any interrupts), you won't see rcuc activity on that CPU.
>
> > > > As per the documentation, if I use rcutree.kthread_prio with a
> > > > priority > my RT task, then the rcu stall does not happen.
> >
> > Yes, it sounds like the main RCU GP thread (mostly called rcu_preempt thread)
> > in your system is competing with your RT task. Increase kthread_prio is
> > standard procedure to resolve this issue, as you did.
>
> Or affinity the rcu_preempt kthread elsewhere so that it is not competing
> with your real-time workload.
>
> > > > However I find it confusing: why are the rcuc threads woken up on the
> > > > isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very
> > > > likely erroneous) understanding, I shouldn't have to fiddle with rcu
> > > > priority... In other words, how come I get rcu stalls on cores that
> > > > have no rcu callbacks?
> >
> > It is not possible to provide further help without more info, in particular
> > are you using nohz_full and isolcpus options? Can you provide kernel
> > and #CPU configuration?
> >
> > Happy to look further! Also adding some more folks who know a lot about this
> > stuff and +rcu list for archives.
>
> The rcuc kthreads communicate between the corresponding CPU and the
> rcu_preempt kthread (AKA RCU GP kthread). They wake up only when there
> is something for them to do, and letting your nohz_full/rcu_nocbs CPUs
> execute in the kernel is an excellent way to give them something to do.
>
> In particular, if that CPU runs in the kernel, it might have executed
> an RCU read-side critical section. Because RCU grace periods are not
> permitted to complete until all pre-existing RCU read-side critical
> sections finish, communication is required.
>
> So, yes, that rcuc kthread will need to execute if its CPU queues an
> RCU callback. But it might also need to execute if some other CPU queues
> an RCU callback -and- its CPU spends significant time in the kernel.
>
> Thanx, Paul
>
> > thanks,
> >
> > - Joel
> >
> > >
> > > Adding Joel because we had a chat about it during lpc...
> > >
> > > > Thanks.
> > > >
> > > -- Daniel
> > >