RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid: 902, comm: md1_raid5

RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid: 902, comm: md1_raid5

[Janos Haar]

Hello list, Neil,

Somebody can say something about this issue?
I am not surprised, if it is hardware related, but this is on a brand new 
server, so i am looking for a solution... :-)
May 17 23:12:13 gladiator-afth1 kernel: RCU detected CPU 1 stall 
(t=4295904002/751 jiffies)
May 17 23:12:13 gladiator-afth1 kernel: Pid: 902, comm: md1_raid5 Not 
tainted 2.6.28.10 #1
May 17 23:12:13 gladiator-afth1 kernel: Call Trace:
May 17 23:12:13 gladiator-afth1 kernel:  <IRQ>  [<ffffffff80265395>] ? 
get_timestamp+0x9/0xf
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8026854f>] 
__rcu_pending+0x64/0x1e2
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff80268703>] 
rcu_pending+0x36/0x6f
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8023f546>] 
update_process_times+0x37/0x5f
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff802504a7>] 
tick_periodic+0x6e/0x7a
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff802504d4>] 
tick_handle_periodic+0x21/0x65
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8021ad60>] 
smp_apic_timer_interrupt+0x8f/0xad
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff8020bf1b>] 
apic_timer_interrupt+0x6b/0x70
May 17 23:12:13 gladiator-afth1 kernel:  <EOI>  [<ffffffff80820b8f>] ? 
_spin_unlock_irqrestore+0x13/0x17
May 17 23:12:13 gladiator-afth1 kernel:  [<ffffffff80728720>] ? 
bitmap_daemon_work+0x142/0x3b0
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80724c58>] ? 
md_check_recovery+0x1b/0x45b
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8071976c>] ? 
raid5d+0x5d/0x503
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80726533>] ? 
md_thread+0xd5/0xed
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80248e4a>] ? 
autoremove_wake_function+0x0/0x38
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8072645e>] ? 
md_thread+0x0/0xed
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80248d3b>] ? 
kthread+0x49/0x76
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8020c169>] ? 
child_rip+0xa/0x11
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff80248cf2>] ? 
kthread+0x0/0x76
May 17 23:12:18 gladiator-afth1 kernel:  [<ffffffff8020c15f>] ? 
child_rip+0x0/0x11

The entire log is here:
http://download.netcenter.hu/bughunt/20090518/messages

The system is on the md1, and working, but slowly.
If i left the server for 1 day,  it will crash without a saved log.

Memtest survived >48 hour.

Thanks,
Janos Haar

Re: RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid: 902, comm: md1_raid5

[Neil Brown]

On Tuesday May 19, janos.haar@netcenter.hu wrote:
> Hello list, Neil,
> 
> Somebody can say something about this issue?
> I am not surprised, if it is hardware related, but this is on a brand new 
> server, so i am looking for a solution... :-)
> May 17 23:12:13 gladiator-afth1 kernel: RCU detected CPU 1 stall 
> (t=4295904002/751 jiffies)

I have no idea what this means.
I've occasionally seen this sort of message in early boot then the
system continued to work perfectly so I figured it was an early-boot
glitch.  I suggest asking someone who understands RCU.

> 
> The entire log is here:
> http://download.netcenter.hu/bughunt/20090518/messages
> 
> The system is on the md1, and working, but slowly.

How slowly?  Is the slowness due to disk throughput?
Have you tested the individual drives and compared that with the
array?


> If i left the server for 1 day,  it will crash without a saved log.

This is a concern!  It usually points to some sort of hardware
problem, but it is very hard to trace.
Is the power supply rated high enough to support all devices?
I cannot think of anything else to suggest .. except start swapping
components until the problem goes away...

NeilBrown

Re: RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid: 902, comm: md1_raid5

[Janos Haar]

----- Original Message ----- 
From: "Neil Brown" <neilb@suse.de>
To: "Janos Haar" <janos.haar@netcenter.hu>
Cc: <linux-raid@vger.kernel.org>
Sent: Tuesday, May 19, 2009 3:05 AM
Subject: Re: RCU detected CPU 1 stall (t=4295904002/751 jiffies) Pid: 902, 
comm: md1_raid5


> On Tuesday May 19, janos.haar@netcenter.hu wrote:
>> Hello list, Neil,
>>
>> Somebody can say something about this issue?
>> I am not surprised, if it is hardware related, but this is on a brand new
>> server, so i am looking for a solution... :-)
>> May 17 23:12:13 gladiator-afth1 kernel: RCU detected CPU 1 stall
>> (t=4295904002/751 jiffies)
>
> I have no idea what this means.
> I've occasionally seen this sort of message in early boot then the
> system continued to work perfectly so I figured it was an early-boot
> glitch.  I suggest asking someone who understands RCU.
>
>>
>> The entire log is here:
>> http://download.netcenter.hu/bughunt/20090518/messages
>>
>> The system is on the md1, and working, but slowly.
>
> How slowly?  Is the slowness due to disk throughput?

No no, this is a fresh and idle server.
I have configured the disks, raid on another PC, and when it finished, i 
have copy up the known good, pre-installed sw pack with old 2.6.18.
This pack is good, tested on many times, and this reports too this issue on 
this machine. (first)
I have compiled the 2.6.28.10 on it, it takes about 6 hour! 8-/
But the 2.6.28.10 reports this too.

The slowness is not disk based, i think, on idle time if i move the selector 
line in mc, this stopps too for some seconds or i can't type in bash when 
this happens, and another one RCU message comes to the log...
(It happens periodically, independently of i am doing something or not.)

I am not sure, it is raid related or not, but the kernel reports only the 
md1_raid5 pid, not another one.
This is why i am asking here first.

Thanks anyway. :-)

> Have you tested the individual drives and compared that with the
> array?

This is a brand new hw, with 4x500GB samsung drive, wich reports no problem 
at all by smart.

>
>
>> If i left the server for 1 day,  it will crash without a saved log.
>
> This is a concern!  It usually points to some sort of hardware
> problem, but it is very hard to trace.
> Is the power supply rated high enough to support all devices?

I am using 550W good quality new PS, and the PC uses only 55-65W, measured. 
;-)
(1x core2duo, 4x hdd, nothing more interesting)

> I cannot think of anything else to suggest .. except start swapping
> components until the problem goes away...

In this way, i need to start with the motherboard. 8-(

Thanks a lot,
Janos Haar

>
> NeilBrown