Re: rcu: performance regression

Alex,Shi
2011-06-14T08:33:33Z

On Tue, 2011-06-14 at 13:26 +0800, Li, Shaohua wrote:
> Commit a26ac2455ffcf3(rcu: move TREE_RCU from softirq to kthread)
> introduced performance regression. In our AIM7 test, this commit caused
> about 40% regression.
> The commit runs rcu callbacks in a kthread instead of softirq. We
> observed high rate of context switch which is caused by this. Out test
> system has 64 CPUs and HZ is 1000, so we saw more than 64k context
> switch per second which is caused by the rcu thread.
> I also did trace and found when rcy thread is woken up, most time the
> thread doesn't handle any callbacks actually, it just initializes new gp
> or end one gp or similar.
> From my understanding, the purpose to make rcu runs in kthread is to
> speed up rcu callbacks run (with help of rtmutex PI), not for end gp and
> so on, which runs pretty fast actually and doesn't need boost.
> To verify my findings, I had below debug patch applied. It still handles
> rcu callbacks in kthread if there is any pending callbacks, but other
> things are still running in softirq. this completely solved our
> regression. I thought this can still boost callbacks run. but I'm not
> expert in the area, so please help.

This commit also cause hackbench process mode performance dropping, and
Shaohua's patch do recovered this. But in hackbench testing, the vmstat
show context switch have some reduce. And perf tool show
root_domain->cpupri->prio_to_cpu[]->lock has contention with the commit.

11.53% hackbench [kernel] [k]
|
--- _raw_spin_lock_irqsave
cpupri_set
__enqueue_rt_entity
enqueue_rt_entity
enqueue_task_rt
enqueue_task
activate_task
ttwu_activate
ttwu_do_activate.clone.3
try_to_wake_up
wake_up_process
invoke_rcu_cpu_kthread
rcu_check_callbacks
update_process_times
tick_sched_timer
__run_hrtimer