Re: [PATCH] rcu: Delay the RCU-selftests during boot.

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2022-03-03 12:02:37 [-0800], Paul E. McKenney wrote:
> > > Either way, it sounds like that irq_work_queue(&rtpcp->rtp_irq_work) in
> > > call_rcu_tasks_generic() needs some adjustment to work in RT.  This should
> > > be doable.  Given this, and given that the corresponding diagnostic
> > > function rcu_tasks_verify_self_tests() is a late_initcall() function,
> > > you don't need to move the call to rcu_init_tasks_generic(), correct?
> > 
> > #1 ksoftirqd must be spawned first in order to get timer_list timer to
> >    work. I'm going to do that, this should not be a problem.
> 
> I very much appreciate your flexibility on this, but it would be even
> better if there was a good way to avoid the dependency on ksoftirqd,
> at least during boot time.  Spawning ksoftirqd first would narrow the
> window of RCU unavailability in RT, but it would be good to have RCU
> work throughout, as it currently does in !RT.  (Give or take a short
> time during the midst of the scheduler putting itself together.)

During SYSTEM_BOOTING we could do softirqs right away but we lack the
infrastructure. Starting with SYSTEM_SCHEDULING we rely on the thread so
it needs to be spawned earlier. The problem with SYSTEM_SCHEDULING+ is
that we may deadlock if the softirqs and performed in IRQ-context.

> This might seem a bit utopian or even unreasonable, but please keep in
> mind that both the scheduler and the idle loop use RCU.

But the problem is only the usage of synchronize_rcu(). So
rcu_read_lock() and call_rcu() works. Only synchronize_rcu() does not.
Couldn't we make a rule to use at earliest within early_initcall()?

> However, that swait_event_timeout_exclusive() doesn't need exact timing
> during boot.  Anything that let other tasks run for at least a few tens
> of microseconds (up to say a millisecond) could easily work just fine.
> Is there any such thing in RT?

swait_event_timeout_exclusive() appears not to be the culprit. It is
invoked a few times (with a 6.5ms timeout) but returns without setting
up a timer. So either my setup avoids the timer or this happens always
and is not related to my config).

rcu_tasks_wait_gp() does schedule_timeout_idle() and this is the one
that blocks. This could be replaced with schedule_hrtimeout() (just
tested). I hate the idea to use a precise delay in a timeout like
situation. But we could use schedule_hrtimeout_range() with a HZ delta
so it kind of feels like the timer_list timer ;)

Also I have no idea how often this is triggered / under which
circumstances (assuming it is bound synchronize_rcu()).

> >    - if you can't guarantee that there is only _one_ waiter
> >      => spawn the irq-work thread early.
> 
> Spawning the irq-work kthread early still leaves a hole.

Why is spawning ksoftirqd + irq-work before early_initcall() still a
hole? If the definition is _no_ synchronize_rcu() before
early_initcall() then it works as documented.

> Other approaches:
> 
> o	For the swait_event_timeout_exclusive(), I could make early
> 	boot uses instead do swait_event_exclusive() and make early boot
> 	rcu_sched_clock_irq() do an unconditional wakeup.  This would
> 	require a loop around one of the swait_event_exclusive()
> 	calls (cheaper than making rcu_sched_clock_irq() worry about
> 	durations).
> 
> 	RCU would need to be informed of the end of "early boot",
> 	for example, by invoking some TBD RCU function as soon
> 	as the ksoftirqd kthreads are created.
> 
> 	This would also require that rcu_needs_cpu() always return
> 	true during early boot.
> 
> 	Static branches could be used if required, as they might be in
> 	rcu_needs_cpu() and maybe also in rcu_sched_clock_irq().

swait_event_timeout_exclusive() appears innocent.

> o	A similar TBD RCU function could cause call_rcu_tasks_generic()
> 	to avoid invoking irq_work_queue() until after the relevant
> 	kthread was created, but to do any needed wakeup at that point.
> 	If wakeups are needed before that time (which they might),
> 	then perhaps the combination of rcu_sched_clock_irq() and
> 	rcu_needs_cpu() can help out there as well.

IRQ-work has been addressed in a different patch.

> These would be conditioned on IS_ENABLED(CONFIG_PREEMPT_RT).
> 
> But now you are going to tell me that wakeups cannot be done from the
> scheduler tick interrupt handler?  If that is the case, are there other
> approaches?

If you by my irqwork patch then I think we are down to:
- spawn ksoftirqd early
- use during boot schedule_hrtimeout() or the whole time (no I idea how
  often this triggers).

> > > Back over to you so that I can learn what I am still missing.  ;-)
> > 
> > Hope that helps.
> 
> Maybe?  You tell me!  ;-)
> 
> 							Thanx, Paul

Sebastian