Re: slow sync rcu_tasks_trace

["Paul E. McKenney" <paulmck@kernel.org>]

On Wed, Sep 09, 2020 at 10:12:28AM -0700, Alexei Starovoitov wrote:
> On Wed, Sep 09, 2020 at 04:38:58AM -0700, Paul E. McKenney wrote:
> > On Tue, Sep 08, 2020 at 07:34:20PM -0700, Alexei Starovoitov wrote:
> > > Hi Paul,
> > > 
> > > Looks like sync rcu_tasks_trace got slower or we simply didn't notice
> > > it earlier.
> > > 
> > > In selftests/bpf try:
> > > time ./test_progs -t trampoline_count
> > > #101 trampoline_count:OK
> > > Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
> > > 
> > > real    1m17.082s
> > > user    0m0.145s
> > > sys    0m1.369s
> > > 
> > > so it's really something going on with sync rcu_tasks_trace.
> > > Could you please take a look?
> > 
> > I am guessing that your .config has CONFIG_TASKS_TRACE_RCU_READ_MB=n.
> > If I am wrong, please try CONFIG_TASKS_TRACE_RCU_READ_MB=y.
> 
> I've added
> CONFIG_RCU_EXPERT=y
> CONFIG_TASKS_TRACE_RCU_READ_MB=y
> 
> and it helped:
> 
> time ./test_progs -t trampoline_count
> #101 trampoline_count:OK
> Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
> 
> real	0m8.924s
> user	0m0.138s
> sys	0m1.408s
> 
> But this is still bad. It's 4 times slower vs rcu_tasks
> and isn't really usable for bpf, since it adds memory barriers exactly
> where we need them removed.
> 
> In the default configuration rcu_tasks_trace is 40! times slower than rcu_tasks.
> This huge difference in sync times concerns me a lot.
> If bpf has to use memory barriers in rcu_read_lock_trace
> and still be 4 times slower than rcu_tasks in the best case
> then there is no much point in rcu_tasks_trace.
> Converting everything to srcu would be better, but I really hope
> you can find a solution to this tasks_trace issue.
> 
> > Otherwise (or alternatively), could you please try booting with
> > rcupdate.rcu_task_ipi_delay=50?  The default value is 500, or half a
> > second on a HZ=1000 system, which on a busy system could easily result
> > in the grace-period delays that you are seeing.  The value of this
> > kernel boot parameter does interact with the tasklist-scan backoffs,
> > so its effect will not likely be linear.
> 
> The tests were run on freshly booted VM with 4 cpus. The VM is idle.
> The host is idle too.
> 
> Adding rcupdate.rcu_task_ipi_delay=50 boot param sort-of helped:
> time ./test_progs -t trampoline_count
> #101 trampoline_count:OK
> Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
> 
> real	0m25.890s
> user	0m0.124s
> sys	0m1.507s
> It is still awful.
> 
> >From "perf report" there is little time spend in the kernel. The kernel is
> waiting on something. I thought in theory the rcu_tasks_trace should have been
> faster on update side vs rcu_tasks ? Could it be a bug somewhere and some
> missing wakeup? It doesn't feel that it works as intended. Whatever it is
> please try to reproduce it to remove me as a middle man.

On it.

To be fair, I was designing for a nominal one-second grace period,
which was also the rough goal for rcu_tasks.

When do you need this by?

Left to myself, I will aim for the merge window after the upcoming one,
and then backport to the prior -stable versions having RCU tasks trace.

							Thanx, Paul