Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

Paul E. McKenney
2009-02-17T14:01:44Z

On Tue, Feb 17, 2009 at 10:46:57AM +0100, Ingo Molnar wrote:
>
> * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
>
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > >
> > > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > >
> > > > > Here the calls to rcu_process_callbacks() are only 75
> > > > > microseconds apart, so that this function is consuming more
> > > > > than 10% of a CPU. The strange thing is that I don't see a
> > > > > raise_softirq() in between, though perhaps it gets inlined or
> > > > > something that makes it invisible to ftrace.
> > > >
> > > > look at the latest trace please, that has even the most inline
> > > > raise-softirq method instrumented, so all the raising is
> > > > visible.
> > >
> > > Ah, my apologies! This time looking at:
> > >
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > >
> > >
> > > 799.521187 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.521371 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.521555 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.521738 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.521934 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.522068 | 1) ksoftir-2324 | | rcu_check_callbacks() {
> > > 799.522208 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.522392 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.522575 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.522759 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.522956 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.523074 | 1) ksoftir-2324 | | rcu_check_callbacks() {
> > > 799.523214 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.523397 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.523579 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.523762 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.523960 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.524079 | 1) ksoftir-2324 | | rcu_check_callbacks() {
> > > 799.524220 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.524403 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.524587 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > 799.524770 | 1) <idle>-0 | | rcu_check_callbacks() {
> > > [ . . . ]
> > >
> > > Yikes!!!
> > >
> > > Why is rcu_check_callbacks() being invoked so often? It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> >
> > BTW, the other question I have is "why do we need to call
> > rcu_pending() and rcu_check_callbacks() from the idle loop of
> > 32-bit x86, especially given that no other architecture does
> > this?". Don't get me wrong, it would be good to get rcutree's
> > rcu_pending() to avoid spuriously saying that
> > rcu_check_callbacks() should be invoked, so I would still like
> > the trace with my patch, but...
>
> There's no strong reason - we've been back and forth about RCU
> in the dynticks code. Mind sending a test patch for Damien to
> try?

But of course! ;-)

The following patch removes the call to rcu_pending() and
rcu_check_callbacks() from the x86 32-bit idle loop in order to
reduce the softirq load on idle systems.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---

process_32.c | 3 ---
1 file changed, 3 deletions(-)

diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
index a546f55..bd4da2a 100644
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -104,9 +104,6 @@ void cpu_idle(void)
check_pgt_cache();
rmb();

- if (rcu_pending(cpu))
- rcu_check_callbacks(cpu, 0);
-
if (cpu_is_offline(cpu))
play_dead();