All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ingo Molnar <mingo@elte.hu>
To: Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>
Cc: Dhaval Giani <dhaval@linux.vnet.ibm.com>,
	Jens Axboe <jens.axboe@oracle.com>,
	Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>,
	LKML <linux-kernel@vger.kernel.org>,
	linuxppc-dev@ozlabs.org, Balbir Singh <balbir@linux.vnet.ibm.com>
Subject: Re: [BUG] 2.6.25-rc1-git1 softlockup while bootup on powerpc
Date: Tue, 12 Feb 2008 08:58:12 +0100	[thread overview]
Message-ID: <20080212075812.GC11775@elte.hu> (raw)
In-Reply-To: <47B1463E.1070809@linux.vnet.ibm.com>


* Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:

> While booting with the 2.6.25-rc1-git1 kernel on the powerbox the 
> softlockup is seen, with following trace.

> BUG: soft lockup - CPU#1 stuck for 61s! [insmod:377]
> TASK = c00000077cb2f0e0[377] 'insmod' THREAD: c00000077cb28000 CPU: 1
> NIP [c0000000001b172c] .radix_tree_gang_lookup+0xdc/0x1e4
> LR [c0000000001a6f00] .call_for_each_cic+0x50/0x10c
> Call Trace:
> [c00000077cb2bb20] [c0000000001a6f60] .call_for_each_cic+0xb0/0x10c (unreliable)
> [c00000077cb2bc60] [c00000000019ecd8] .exit_io_context+0xf0/0x110
> [c00000077cb2bcf0] [c00000000006254c] .do_exit+0x820/0x850
> [c00000077cb2bda0] [c000000000062648] .do_group_exit+0xcc/0xe8
> [c00000077cb2be30] [c00000000000872c] syscall_exit+0x0/0x40

this call_for_each_cic/radix_tree_gang_lookup locked up, and all other 
CPUs deadlocked in stopmachine, due to this one.

call_for_each_cic is in ./block/cfq-iosched.c uses RCU, but you've got 
classic-RCU:

  CONFIG_CLASSIC_RCU=y
  # CONFIG_PREEMPT_RCU is not set

so it's not related to the preempt-RCU changes either.

It is this part that locks up:

        do {
...
                nr = radix_tree_gang_lookup(&ioc->radix_root, (void **) cics,
                                                index, CIC_GANG_NR);
...
        } while (nr == CIC_GANG_NR);
...

it seems the radix tree will yield new entries again and again. Either 
it got corrupted, or some other CPU is filling it faster than we can 
deplete it [unlikely i think].

	Ingo

WARNING: multiple messages have this Message-ID (diff)
From: Ingo Molnar <mingo@elte.hu>
To: Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>
Cc: LKML <linux-kernel@vger.kernel.org>,
	linuxppc-dev@ozlabs.org, Dhaval Giani <dhaval@linux.vnet.ibm.com>,
	Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>,
	Andy Whitcroft <apw@shadowen.org>,
	Balbir Singh <balbir@linux.vnet.ibm.com>,
	Jens Axboe <jens.axboe@oracle.com>
Subject: Re: [BUG] 2.6.25-rc1-git1 softlockup while bootup on powerpc
Date: Tue, 12 Feb 2008 08:58:12 +0100	[thread overview]
Message-ID: <20080212075812.GC11775@elte.hu> (raw)
In-Reply-To: <47B1463E.1070809@linux.vnet.ibm.com>


* Kamalesh Babulal <kamalesh@linux.vnet.ibm.com> wrote:

> While booting with the 2.6.25-rc1-git1 kernel on the powerbox the 
> softlockup is seen, with following trace.

> BUG: soft lockup - CPU#1 stuck for 61s! [insmod:377]
> TASK = c00000077cb2f0e0[377] 'insmod' THREAD: c00000077cb28000 CPU: 1
> NIP [c0000000001b172c] .radix_tree_gang_lookup+0xdc/0x1e4
> LR [c0000000001a6f00] .call_for_each_cic+0x50/0x10c
> Call Trace:
> [c00000077cb2bb20] [c0000000001a6f60] .call_for_each_cic+0xb0/0x10c (unreliable)
> [c00000077cb2bc60] [c00000000019ecd8] .exit_io_context+0xf0/0x110
> [c00000077cb2bcf0] [c00000000006254c] .do_exit+0x820/0x850
> [c00000077cb2bda0] [c000000000062648] .do_group_exit+0xcc/0xe8
> [c00000077cb2be30] [c00000000000872c] syscall_exit+0x0/0x40

this call_for_each_cic/radix_tree_gang_lookup locked up, and all other 
CPUs deadlocked in stopmachine, due to this one.

call_for_each_cic is in ./block/cfq-iosched.c uses RCU, but you've got 
classic-RCU:

  CONFIG_CLASSIC_RCU=y
  # CONFIG_PREEMPT_RCU is not set

so it's not related to the preempt-RCU changes either.

It is this part that locks up:

        do {
...
                nr = radix_tree_gang_lookup(&ioc->radix_root, (void **) cics,
                                                index, CIC_GANG_NR);
...
        } while (nr == CIC_GANG_NR);
...

it seems the radix tree will yield new entries again and again. Either 
it got corrupted, or some other CPU is filling it faster than we can 
deplete it [unlikely i think].

	Ingo

  reply	other threads:[~2008-02-12  7:58 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-12  7:09 [BUG] 2.6.25-rc1-git1 softlockup while bootup on powerpc Kamalesh Babulal
2008-02-12  7:58 ` Ingo Molnar [this message]
2008-02-12  7:58   ` Ingo Molnar
2008-02-14  9:59   ` Kamalesh Babulal
2008-02-14  9:59     ` Kamalesh Babulal

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20080212075812.GC11775@elte.hu \
    --to=mingo@elte.hu \
    --cc=balbir@linux.vnet.ibm.com \
    --cc=dhaval@linux.vnet.ibm.com \
    --cc=jens.axboe@oracle.com \
    --cc=kamalesh@linux.vnet.ibm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linuxppc-dev@ozlabs.org \
    --cc=vatsa@linux.vnet.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.