Re: Fw: RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid:902, comm: md1_raid5

["Janos Haar" <janos.haar@netcenter.hu>]

Paul,

Thank you for your attention.
Yes, the PC makes 2-3 second "pauses" and drop this message again and again.
If i remove the RCU debugging, the message disappears, but the pauses still 
here, and makes 2-3 load on the idle system.
Can i do something?
You suggest to use PREEMPT? (This is a server.)

Thank you,
Janos Haar

----- Original Message ----- 
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: "Janos Haar" <djani22@netcenter.hu>
Cc: <linux-kernel@vger.kernel.org>
Sent: Thursday, May 21, 2009 4:50 AM
Subject: Re: Fw: RCU detected CPU 1 stall (t=4295904002/751 jiffies) 
Pid:902, comm: md1_raid5


> On Wed, May 20, 2009 at 11:46:07AM +0200, Janos Haar wrote:
>> Hello list,
>>
>> Somebody know, what is this?
>> May 17 23:12:13 gladiator-afth1 kernel: RCU detected CPU 1 stall
>> (t=4295904002/751 jiffies)
>> May 17 23:12:13 gladiator-afth1 kernel: Pid: 902, comm: md1_raid5 Not
>> tainted 2.6.28.10 #1
>> May 17 23:12:13 gladiator-afth1 kernel: Call Trace:
>> May 17 23:12:13 gladiator-afth1 kernel:  <IRQ>  [<ffffffff80265395>] ?
>> get_timestamp+0x9/0xf
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8026854f>]
>> __rcu_pending+0x64/0x1e2
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff80268703>]
>> rcu_pending+0x36/0x6f
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8023f546>]
>> update_process_times+0x37/0x5f
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff802504a7>]
>> tick_periodic+0x6e/0x7a
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff802504d4>]
>> tick_handle_periodic+0x21/0x65
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8021ad60>]
>> smp_apic_timer_interrupt+0x8f/0xad
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8020bf1b>]
>> apic_timer_interrupt+0x6b/0x70
>> May 17 23:12:13 gladiator-afth1 kernel:  <EOI>  [<ffffffff80820b8f>] ?
>> _spin_unlock_irqrestore+0x13/0x17
>
> One of the following functions is looping in the kernel.  If you are
> running with HZ=250, it has been looping for about three seconds.
> Interrupts are enabled, but preemption must be disabled (perhaps due
> to !CONFIG_PREEMPT).
>
> Thanx, Paul
>
>> May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff80728720>] ?
>> bitmap_daemon_work+0x142/0x3b0
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80724c58>] ?
>> md_check_recovery+0x1b/0x45b
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8071976c>] ?
>> raid5d+0x5d/0x503
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80726533>] ?
>> md_thread+0xd5/0xed
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80248e4a>] ?
>> autoremove_wake_function+0x0/0x38
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8072645e>] ?
>> md_thread+0x0/0xed
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80248d3b>] ?
>> kthread+0x49/0x76
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8020c169>] ?
>> child_rip+0xa/0x11
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80248cf2>] ?
>> kthread+0x0/0x76
>> May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8020c15f>] ?
>> child_rip+0x0/0x11
>> Neilbrown from the RAID list suggested to ask someone else...(The mail is
>> below.)Thanks,Janos Haar----- Original Message ----- From: "Janos Haar"
>> <janos.haar@netcenter.hu>
>> To: "Neil Brown" <neilb@suse.de>
>> Cc: <linux-raid@vger.kernel.org>
>> Sent: Tuesday, May 19, 2009 12:30 PM
>> Subject: Re: RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid: 
>> 902,
>> comm: md1_raid5
>>
>>
>>>
>>> ----- Original Message ----- From: "Neil Brown" <neilb@suse.de>
>>> To: "Janos Haar" <janos.haar@netcenter.hu>
>>> Cc: <linux-raid@vger.kernel.org>
>>> Sent: Tuesday, May 19, 2009 3:05 AM
>>> Subject: Re: RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid: 
>>> 902,
>>> comm: md1_raid5
>>>
>>>
>>>> On Tuesday May 19, janos.haar@netcenter.hu wrote:
>>>>> Hello list, Neil,
>>>>>
>>>>> Somebody can say something about this issue?
>>>>> I am not surprised, if it is hardware related, but this is on a brand
>>>>> new
>>>>> server, so i am looking for a solution... :-)
>>>>> May 17 23:12:13 gladiator-afth1 kernel: RCU detected CPU 1 stall
>>>>> (t=4295904002/751 jiffies)
>>>>
>>>> I have no idea what this means.
>>>> I've occasionally seen this sort of message in early boot then the
>>>> system continued to work perfectly so I figured it was an early-boot
>>>> glitch.  I suggest asking someone who understands RCU.
>>>>
>>>>>
>>>>> The entire log is here:
>>>>> http://download.netcenter.hu/bughunt/20090518/messages
>>>>>
>>>>> The system is on the md1, and working, but slowly.
>>>>
>>>> How slowly?  Is the slowness due to disk throughput?
>>>
>>> No no, this is a fresh and idle server.
>>> I have configured the disks, raid on another PC, and when it finished, i
>>> have copy up the known good, pre-installed sw pack with old 2.6.18.
>>> This pack is good, tested on many times, and this reports too this issue
>>> on this machine. (first)
>>> I have compiled the 2.6.28.10 on it, it takes about 6 hour! 8-/
>>> But the 2.6.28.10 reports this too.
>>>
>>> The slowness is not disk based, i think, on idle time if i move the
>>> selector line in mc, this stopps too for some seconds or i can't type in
>>> bash when this happens, and another one RCU message comes to the log...
>>> (It happens periodically, independently of i am doing something or not.)
>>>
>>> I am not sure, it is raid related or not, but the kernel reports only 
>>> the
>>> md1_raid5 pid, not another one.
>>> This is why i am asking here first.
>>>
>>> Thanks anyway. :-)
>>>
>>>> Have you tested the individual drives and compared that with the
>>>> array?
>>>
>>> This is a brand new hw, with 4x500GB samsung drive, wich reports no
>>> problem at all by smart.
>>>
>>>>
>>>>
>>>>> If i left the server for 1 day,  it will crash without a saved log.
>>>>
>>>> This is a concern!  It usually points to some sort of hardware
>>>> problem, but it is very hard to trace.
>>>> Is the power supply rated high enough to support all devices?
>>>
>>> I am using 550W good quality new PS, and the PC uses only 55-65W,
>>> measured. ;-)
>>> (1x core2duo, 4x hdd, nothing more interesting)
>>>
>>>> I cannot think of anything else to suggest .. except start swapping
>>>> components until the problem goes away...
>>>
>>> In this way, i need to start with the motherboard. 8-(
>>>
>>> Thanks a lot,
>>> Janos Haar
>>>
>>>>
>>>> NeilBrown
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" 
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/