RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

[Florent Carli]

Hello,

I'm trying to isolate some cores to run a CPU-bound real-time task and
even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc
threads wake up, and I get RCU stall warnings on the isolated core.
As per the documentation, if I use rcutree.kthread_prio with a
priority > my RT task, then the rcu stall does not happen.

However I find it confusing: why are the rcuc threads woken up on the
isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very
likely erroneous) understanding, I shouldn't have to fiddle with rcu
priority... In other words, how come I get rcu stalls on cores that
have no rcu callbacks?

Thanks.

Re: RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

[Daniel Bristot de Oliveira]

On 10/5/22 18:01, Florent Carli wrote:
> Hello,
> 
> I'm trying to isolate some cores to run a CPU-bound real-time task and
> even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc
> threads wake up, and I get RCU stall warnings on the isolated core.
> As per the documentation, if I use rcutree.kthread_prio with a
> priority > my RT task, then the rcu stall does not happen.
> 
> However I find it confusing: why are the rcuc threads woken up on the
> isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very
> likely erroneous) understanding, I shouldn't have to fiddle with rcu
> priority... In other words, how come I get rcu stalls on cores that
> have no rcu callbacks?

Adding Joel because we had a chat about it during lpc...

> Thanks.
> 
-- Daniel

Re: RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

[Sebastian Andrzej Siewior]

On 2022-10-05 18:03:39 [+0200], Daniel Bristot de Oliveira wrote:
> On 10/5/22 18:01, Florent Carli wrote:
> > Hello,
> > 
> > I'm trying to isolate some cores to run a CPU-bound real-time task and
> > even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc
> > threads wake up, and I get RCU stall warnings on the isolated core.
> > As per the documentation, if I use rcutree.kthread_prio with a
> > priority > my RT task, then the rcu stall does not happen.
> > 
> > However I find it confusing: why are the rcuc threads woken up on the
> > isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very
> > likely erroneous) understanding, I shouldn't have to fiddle with rcu
> > priority... In other words, how come I get rcu stalls on cores that
> > have no rcu callbacks?

There are different things:
- with CONFIG_RCU_NOCB_CPU you get addition threads and can move the
  rcuo* theads away from the isolated CPU. That means the callbacks will
  be handled on a different CPU.

- you still need to track/ follow the grace period. This still happens
  in the rcuc thread. To get around this you need to use NO_HZ_FULL with
  isolcpus/rcu_nocbs.

> Adding Joel because we had a chat about it during lpc...
> 
> > Thanks.
> > 
> -- Daniel

Sebastian

Re: RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

[Joel Fernandes]

On Wed, Oct 05, 2022 at 06:03:39PM +0200, Daniel Bristot de Oliveira wrote:
> On 10/5/22 18:01, Florent Carli wrote:
> > Hello,
> > 
> > I'm trying to isolate some cores to run a CPU-bound real-time task and
> > even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc
> > threads wake up, and I get RCU stall warnings on the isolated core.

Isolating a CPU does not mean that RCU activity is forbidden on it, even with
nohz_full, if the CPU enters the kernel (say syscall or interrupt), then it
may enter a read-side critical section so RCU will be watching it, and such
CPU will have to report quiescent state.

> > As per the documentation, if I use rcutree.kthread_prio with a
> > priority > my RT task, then the rcu stall does not happen.

Yes, it sounds like the main RCU GP thread (mostly called rcu_preempt thread)
in your system is competing with your RT task. Increase kthread_prio is
standard procedure to resolve this issue, as you did.

> > However I find it confusing: why are the rcuc threads woken up on the
> > isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very
> > likely erroneous) understanding, I shouldn't have to fiddle with rcu
> > priority... In other words, how come I get rcu stalls on cores that
> > have no rcu callbacks?

It is not possible to provide further help without more info, in particular
are you using nohz_full and isolcpus options? Can you provide kernel
and #CPU configuration?

Happy to look further! Also adding some more folks who know a lot about this
stuff and +rcu list for archives.

thanks,

 - Joel

> 
> Adding Joel because we had a chat about it during lpc...
> 
> > Thanks.
> > 
> -- Daniel
> 

Re: RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

[Paul E. McKenney]

On Thu, Oct 06, 2022 at 01:31:43PM +0000, Joel Fernandes wrote:
> On Wed, Oct 05, 2022 at 06:03:39PM +0200, Daniel Bristot de Oliveira wrote:
> > On 10/5/22 18:01, Florent Carli wrote:
> > > Hello,
> > > 
> > > I'm trying to isolate some cores to run a CPU-bound real-time task and
> > > even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc
> > > threads wake up, and I get RCU stall warnings on the isolated core.
> 
> Isolating a CPU does not mean that RCU activity is forbidden on it, even with
> nohz_full, if the CPU enters the kernel (say syscall or interrupt), then it
> may enter a read-side critical section so RCU will be watching it, and such
> CPU will have to report quiescent state.

What Joel said!

Plus, if you keep that CPU out of the kernel (including avoiding it ever
taking any interrupts), you won't see rcuc activity on that CPU.

> > > As per the documentation, if I use rcutree.kthread_prio with a
> > > priority > my RT task, then the rcu stall does not happen.
> 
> Yes, it sounds like the main RCU GP thread (mostly called rcu_preempt thread)
> in your system is competing with your RT task. Increase kthread_prio is
> standard procedure to resolve this issue, as you did.

Or affinity the rcu_preempt kthread elsewhere so that it is not competing
with your real-time workload.

> > > However I find it confusing: why are the rcuc threads woken up on the
> > > isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very
> > > likely erroneous) understanding, I shouldn't have to fiddle with rcu
> > > priority... In other words, how come I get rcu stalls on cores that
> > > have no rcu callbacks?
> 
> It is not possible to provide further help without more info, in particular
> are you using nohz_full and isolcpus options? Can you provide kernel
> and #CPU configuration?
> 
> Happy to look further! Also adding some more folks who know a lot about this
> stuff and +rcu list for archives.

The rcuc kthreads communicate between the corresponding CPU and the
rcu_preempt kthread (AKA RCU GP kthread).  They wake up only when there
is something for them to do, and letting your nohz_full/rcu_nocbs CPUs
execute in the kernel is an excellent way to give them something to do.

In particular, if that CPU runs in the kernel, it might have executed
an RCU read-side critical section.  Because RCU grace periods are not
permitted to complete until all pre-existing RCU read-side critical
sections finish, communication is required.

So, yes, that rcuc kthread will need to execute if its CPU queues an
RCU callback.  But it might also need to execute if some other CPU queues
an RCU callback -and- its CPU spends significant time in the kernel.

							Thanx, Paul

> thanks,
> 
>  - Joel
> 
> > 
> > Adding Joel because we had a chat about it during lpc...
> > 
> > > Thanks.
> > > 
> > -- Daniel
> > 

Re: RCU stall warnings even with rcu_nocbs and rcu_nocb_poll

[Florent Carli]

Thank you all.

I make sure the rcu_preempt kthread affinity was set on a non-isolated
core (as well as all unbounded rcu* threads) so I guess it is not a
competing issue.

What I’m running on those isolated cores are qemu/kvm vcpu tasks, and
I guess they might spend some time in the kernel… in which case I
understand it’s completely normal to see some rcuc activity.
And so I conclude that using isolcpus + nohz_full (which includes
rcu_nocbs) + rcu_nocb_poll + rcutree.kthread_prio is the way to go.

Thanks everyone.

On Thu, Oct 6, 2022 at 4:13 PM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> On Thu, Oct 06, 2022 at 01:31:43PM +0000, Joel Fernandes wrote:
> > On Wed, Oct 05, 2022 at 06:03:39PM +0200, Daniel Bristot de Oliveira wrote:
> > > On 10/5/22 18:01, Florent Carli wrote:
> > > > Hello,
> > > >
> > > > I'm trying to isolate some cores to run a CPU-bound real-time task and
> > > > even though I'm using rcu_nocbs and rcu_nocb_poll, I can see the rcuc
> > > > threads wake up, and I get RCU stall warnings on the isolated core.
> >
> > Isolating a CPU does not mean that RCU activity is forbidden on it, even with
> > nohz_full, if the CPU enters the kernel (say syscall or interrupt), then it
> > may enter a read-side critical section so RCU will be watching it, and such
> > CPU will have to report quiescent state.
>
> What Joel said!
>
> Plus, if you keep that CPU out of the kernel (including avoiding it ever
> taking any interrupts), you won't see rcuc activity on that CPU.
>
> > > > As per the documentation, if I use rcutree.kthread_prio with a
> > > > priority > my RT task, then the rcu stall does not happen.
> >
> > Yes, it sounds like the main RCU GP thread (mostly called rcu_preempt thread)
> > in your system is competing with your RT task. Increase kthread_prio is
> > standard procedure to resolve this issue, as you did.
>
> Or affinity the rcu_preempt kthread elsewhere so that it is not competing
> with your real-time workload.
>
> > > > However I find it confusing: why are the rcuc threads woken up on the
> > > > isolated cores despite using rcu_nocbs and rcu_nocb_poll? In my (very
> > > > likely erroneous) understanding, I shouldn't have to fiddle with rcu
> > > > priority... In other words, how come I get rcu stalls on cores that
> > > > have no rcu callbacks?
> >
> > It is not possible to provide further help without more info, in particular
> > are you using nohz_full and isolcpus options? Can you provide kernel
> > and #CPU configuration?
> >
> > Happy to look further! Also adding some more folks who know a lot about this
> > stuff and +rcu list for archives.
>
> The rcuc kthreads communicate between the corresponding CPU and the
> rcu_preempt kthread (AKA RCU GP kthread).  They wake up only when there
> is something for them to do, and letting your nohz_full/rcu_nocbs CPUs
> execute in the kernel is an excellent way to give them something to do.
>
> In particular, if that CPU runs in the kernel, it might have executed
> an RCU read-side critical section.  Because RCU grace periods are not
> permitted to complete until all pre-existing RCU read-side critical
> sections finish, communication is required.
>
> So, yes, that rcuc kthread will need to execute if its CPU queues an
> RCU callback.  But it might also need to execute if some other CPU queues
> an RCU callback -and- its CPU spends significant time in the kernel.
>
>                                                         Thanx, Paul
>
> > thanks,
> >
> >  - Joel
> >
> > >
> > > Adding Joel because we had a chat about it during lpc...
> > >
> > > > Thanks.
> > > >
> > > -- Daniel
> > >