2.6.33.5 rt23: machine lockup (nfs/autofs related?)

2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

We are having problems with 2.6.33.5+rt23, at least in our configuration
while accessing an nfs automounted directory. This causes a complete
machine lockup (press reset to exit as the only option). 

I simply use the Nautilus file manager (in Fedora 12) to navigate to an
autofs mounted directory and the process monitor goes to 100% on one
core (or maybe two), the mouse jerks a bit and the whole thing goes
catatonic almost immediately. 

I get this in any open terminal at the time of the crash:

--------
Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:------------[ cut here ]------------

Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:invalid opcode: 0000 [#1] PREEMPT SMP 

Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:last sysfs
file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:Process nautilus (pid: 2874, ti=f0204000 task=f17dd1f0
task.ti=f0204000)

Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:Stack:

Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:Call Trace:

Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43
0c 89 43 0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04
<0f> 0b eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 

Message from syslogd@localhost at Jul  8 10:13:54 ...
 kernel:EIP: [<c0792c0f>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP
0068:f0205cbc
--------

And that's it... nothing else in the logs. 

For now we are booting into the normal Fedora kernel (this is on Fedora
12) as this makes the rt kernel not usable in our setup. 

Let me know if there is anything else I can do to help debug this...
-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Thu, 2010-07-08 at 10:19 -0700, Fernando Lopez-Lezcano wrote:
> We are having problems with 2.6.33.5+rt23, at least in our configuration
> while accessing an nfs automounted directory. This causes a complete
> machine lockup (press reset to exit as the only option). 
> 
> I simply use the Nautilus file manager (in Fedora 12) to navigate to an
> autofs mounted directory and the process monitor goes to 100% on one
> core (or maybe two), the mouse jerks a bit and the whole thing goes
> catatonic almost immediately. 
> 
> I get this in any open terminal at the time of the crash:
> 
> --------
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:------------[ cut here ]------------
> 
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:invalid opcode: 0000 [#1] PREEMPT SMP 
> 
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:last sysfs
> file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> 
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:Process nautilus (pid: 2874, ti=f0204000 task=f17dd1f0
> task.ti=f0204000)
> 
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:Stack:
> 
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:Call Trace:
> 
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43
> 0c 89 43 0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04
> <0f> 0b eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
> 
> Message from syslogd@localhost at Jul  8 10:13:54 ...
>  kernel:EIP: [<c0792c0f>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP
> 0068:f0205cbc
> --------
> 
> And that's it... nothing else in the logs. 

Hrm. Not too much to go on there, but thanks for the report.


> For now we are booting into the normal Fedora kernel (this is on Fedora
> 12) as this makes the rt kernel not usable in our setup. 
> 
> Let me know if there is anything else I can do to help debug this...

Had you done any testing with earlier 2.6.33-rt kernels where this
didn't occur? If so what version?

thanks
-john

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Thu, 2010-07-08 at 15:33 -0700, john stultz wrote:
> On Thu, 2010-07-08 at 10:19 -0700, Fernando Lopez-Lezcano wrote:
> > We are having problems with 2.6.33.5+rt23, at least in our configuration
> > while accessing an nfs automounted directory. This causes a complete
> > machine lockup (press reset to exit as the only option). 
> > 
> > I simply use the Nautilus file manager (in Fedora 12) to navigate to an
> > autofs mounted directory and the process monitor goes to 100% on one
> > core (or maybe two), the mouse jerks a bit and the whole thing goes
> > catatonic almost immediately. 
> > 
> > I get this in any open terminal at the time of the crash:
> > 
> > --------
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:------------[ cut here ]------------
> > 
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:invalid opcode: 0000 [#1] PREEMPT SMP 
> > 
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:last sysfs
> > file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> > 
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:Process nautilus (pid: 2874, ti=f0204000 task=f17dd1f0
> > task.ti=f0204000)
> > 
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:Stack:
> > 
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:Call Trace:
> > 
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43
> > 0c 89 43 0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04
> > <0f> 0b eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
> > 
> > Message from syslogd@localhost at Jul  8 10:13:54 ...
> >  kernel:EIP: [<c0792c0f>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP
> > 0068:f0205cbc
> > --------
> > 
> > And that's it... nothing else in the logs. 
> 
> Hrm. Not too much to go on there, but thanks for the report.
> 
> 
> > For now we are booting into the normal Fedora kernel (this is on Fedora
> > 12) as this makes the rt kernel not usable in our setup. 
> > 
> > Let me know if there is anything else I can do to help debug this...
> 
> Had you done any testing with earlier 2.6.33-rt kernels where this
> didn't occur? If so what version?

I have been working with the whole series but my main usage case does
not use nfs/autofs (see next paragraphs). 

I have noticed that the problem does not appear to happen when I cd into
an nfs automounted directory directly. It appears to happen only when
listing the contents of a mount point (ie: when "/whatever/" is an
autofs mount point where several directories are mounted, not
necessarily from the same server). 

Before switching to Fedora 12 users were normally running 2.6.29 rt and
I had been running 2.6.31.x and 2.6.33.x rt, but I don't think it ever
happened to me personally (I'm always using the command line - this is
completely reproducible with nautilus). After the switch it started
happening almost immediately to regular users (using nautilus mostly). 

How could I try to get more debugging information?
-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Thu, 2010-07-08 at 15:44 -0700, Fernando Lopez-Lezcano wrote:
> On Thu, 2010-07-08 at 15:33 -0700, john stultz wrote:
> > On Thu, 2010-07-08 at 10:19 -0700, Fernando Lopez-Lezcano wrote:
> > > We are having problems with 2.6.33.5+rt23, at least in our configuration
> > > while accessing an nfs automounted directory. This causes a complete
> > > machine lockup (press reset to exit as the only option). 
> > > 
> > > I simply use the Nautilus file manager (in Fedora 12) to navigate to an
> > > autofs mounted directory and the process monitor goes to 100% on one
> > > core (or maybe two), the mouse jerks a bit and the whole thing goes
> > > catatonic almost immediately. 
> > > 
> > > I get this in any open terminal at the time of the crash:
> > > 
> > > --------
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:------------[ cut here ]------------
> > > 
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:invalid opcode: 0000 [#1] PREEMPT SMP 
> > > 
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:last sysfs
> > > file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> > > 
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:Process nautilus (pid: 2874, ti=f0204000 task=f17dd1f0
> > > task.ti=f0204000)
> > > 
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:Stack:
> > > 
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:Call Trace:
> > > 
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43
> > > 0c 89 43 0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04
> > > <0f> 0b eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
> > > 
> > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > >  kernel:EIP: [<c0792c0f>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP
> > > 0068:f0205cbc
> > > --------
> > > 
> > > And that's it... nothing else in the logs. 
> > 
> > Hrm. Not too much to go on there, but thanks for the report.
> > 
> > 
> > > For now we are booting into the normal Fedora kernel (this is on Fedora
> > > 12) as this makes the rt kernel not usable in our setup. 
> > > 
> > > Let me know if there is anything else I can do to help debug this...
> > 
> > Had you done any testing with earlier 2.6.33-rt kernels where this
> > didn't occur? If so what version?
> 
> I have been working with the whole series but my main usage case does
> not use nfs/autofs (see next paragraphs). 
> 
> I have noticed that the problem does not appear to happen when I cd into
> an nfs automounted directory directly. It appears to happen only when
> listing the contents of a mount point (ie: when "/whatever/" is an
> autofs mount point where several directories are mounted, not
> necessarily from the same server). 
> 
> Before switching to Fedora 12 users were normally running 2.6.29 rt and
> I had been running 2.6.31.x and 2.6.33.x rt, but I don't think it ever
> happened to me personally (I'm always using the command line - this is
> completely reproducible with nautilus). After the switch it started
> happening almost immediately to regular users (using nautilus mostly). 
> 
> How could I try to get more debugging information?

Any chance you have a serial port on the machine in question? If so its
likely any oops messages could be collected over that.

If not, you can look at the netconsole documentation
(Documentation/networking/netconsole.txt) to try to configure that to
collect any panic info.

thanks
-john


 

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Thu, 2010-07-08 at 16:00 -0700, john stultz wrote:
> On Thu, 2010-07-08 at 15:44 -0700, Fernando Lopez-Lezcano wrote:
> > On Thu, 2010-07-08 at 15:33 -0700, john stultz wrote:
> > > On Thu, 2010-07-08 at 10:19 -0700, Fernando Lopez-Lezcano wrote:
> > > > We are having problems with 2.6.33.5+rt23, at least in our configuration
> > > > while accessing an nfs automounted directory. This causes a complete
> > > > machine lockup (press reset to exit as the only option). 
> > > > 
> > > > I simply use the Nautilus file manager (in Fedora 12) to navigate to an
> > > > autofs mounted directory and the process monitor goes to 100% on one
> > > > core (or maybe two), the mouse jerks a bit and the whole thing goes
> > > > catatonic almost immediately. 
> > > > 
> > > > I get this in any open terminal at the time of the crash:
> > > > 
> > > > --------
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:------------[ cut here ]------------
> > > > 
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:invalid opcode: 0000 [#1] PREEMPT SMP 
> > > > 
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:last sysfs
> > > > file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> > > > 
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:Process nautilus (pid: 2874, ti=f0204000 task=f17dd1f0
> > > > task.ti=f0204000)
> > > > 
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:Stack:
> > > > 
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:Call Trace:
> > > > 
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43
> > > > 0c 89 43 0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04
> > > > <0f> 0b eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
> > > > 
> > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > >  kernel:EIP: [<c0792c0f>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP
> > > > 0068:f0205cbc
> > > > --------
> > > > 
> > > > And that's it... nothing else in the logs. 
> > > 
> > > Hrm. Not too much to go on there, but thanks for the report.
> > > 
> > > 
> > > > For now we are booting into the normal Fedora kernel (this is on Fedora
> > > > 12) as this makes the rt kernel not usable in our setup. 
> > > > 
> > > > Let me know if there is anything else I can do to help debug this...
> > > 
> > > Had you done any testing with earlier 2.6.33-rt kernels where this
> > > didn't occur? If so what version?
> > 
> > I have been working with the whole series but my main usage case does
> > not use nfs/autofs (see next paragraphs). 
> > 
> > I have noticed that the problem does not appear to happen when I cd into
> > an nfs automounted directory directly. It appears to happen only when
> > listing the contents of a mount point (ie: when "/whatever/" is an
> > autofs mount point where several directories are mounted, not
> > necessarily from the same server). 
> > 
> > Before switching to Fedora 12 users were normally running 2.6.29 rt and
> > I had been running 2.6.31.x and 2.6.33.x rt, but I don't think it ever
> > happened to me personally (I'm always using the command line - this is
> > completely reproducible with nautilus). After the switch it started
> > happening almost immediately to regular users (using nautilus mostly). 
> > 
> > How could I try to get more debugging information?
> 
> Any chance you have a serial port on the machine in question? If so its
> likely any oops messages could be collected over that.

Ok, I got one! (had to go buy a null modem cable, I thought I had one
but it has disappeared since the last time I did this :-)

Find it below... it keeps spewing stuff every once in a while. Hopefully
this will be enough. No response from the network or the keyboard or
mouse at this point, reset is the only way out. 

I can retest if somebody comes up with a patch...
Thanks.
-- Fernando

localhost login: ------------[ cut here ]------------
kernel BUG at kernel/rtmutex.c:808!
invalid opcode: 0000 [#1] PREEMPT SMP 
last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2872, comm: nautilus Not tainted
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0792f67>] EFLAGS: 00010046 CPU: 2
EIP is at rt_spin_lock_slowlock+0x43/0x1bb
EAX: f2ad2eb0 EBX: ef2a2da4 ECX: ef2a2da4 EDX: f2ad2eb0
ESI: ef2ededc EDI: ef2a893c EBP: f20e3d14 ESP: f20e3cbc
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000001
Process nautilus (pid: 2872, ti=f20e2000 task=f2ad2eb0 task.ti=f20e2000)
Stack:
 00000000 00000001 00000000 f20e3cd8 00000292 ef2a0ac4 00000000 f20e3d1c
<0> facf45e6 00000246 ef2a2da4 ef2a2d9c 00000000 f20e3cfc c0465523
ef2a0ac4
<0> 00000000 c07936b6 00000246 ef2a2da4 ef2ededc ef2a893c f20e3d20
c0465559
Call Trace:
 [<facf45e6>] ? nfs_refresh_inode_locked+0x79c/0xa1e [nfs]
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c0465559>] ? rt_spin_lock_fastlock.clone.1+0x5c/0x5f
 [<c07936a2>] ? rt_spin_lock+0x8/0xa
 [<c04e08d3>] ? d_materialise_unique+0xa9/0x29e
 [<facf55bc>] ? nfs_fhget+0x492/0x51d [nfs]
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<facf15a0>] ? nfs_do_filldir+0x27b/0x3a9 [nfs]
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<facf1d57>] ? nfs_readdir+0x689/0x707 [nfs]
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<fad020de>] ? nfs3_decode_dirent+0x0/0x10e [nfs]
 [<c04642a6>] ? do_futex+0x99/0x696
 [<c0793335>] ? rt_mutex_lock_killable+0x28/0x2c
 [<c04ddd56>] ? vfs_readdir+0x66/0x92
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<c04ddde5>] ? sys_getdents64+0x63/0xa5
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] ? sysenter_do_call+0x12/0x28
 [<c0790000>] ? timer_cpu_notify+0x11a/0x2bf
Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43 0c 89 43
0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04 <0f> 0b
eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
EIP: [<c0792f67>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP 0068:f20e3cbc
---[ end trace aebccf812d36404e ]---
note: nautilus[2872] exited with preempt_count 1
BUG: soft lockup - CPU#3 stuck for 61s! [gvfs-gdu-volume:2792]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2792, comm: gvfs-gdu-volume Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00200202 CPU: 3
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200292 EBX: c5f841ec ECX: 00000004 EDX: 00000020
ESI: 00000003 EDI: c5f84204 EBP: f23bfe1c ESP: f23bfe04
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gvfs-gdu-volume (pid: 2792, ti=f23be000 task=f2bf6330
task.ti=f23be000)
Stack:
 01000000 00000000 c042a1a2 f23be000 c042a1a2 000000b4 f23bfe2c c0465ce2
<0> 00000001 00000000 f23bfe40 c04442f5 000000d5 00000020 000000b4
f23bfe48
<0> c042a110 f23bfee4 c04b4784 00000000 00000020 000000b5 000000b6
000000b7
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b84b8>] ? handle_mm_fault+0xdc/0x92a
 [<c0455a51>] ? autoremove_wake_function+0x0/0x2f
 [<c07961ab>] ? do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] ? error_code+0x73/0x78
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b84b8>] handle_mm_fault+0xdc/0x92a
 [<c0455a51>] ? autoremove_wake_function+0x0/0x2f
 [<c07961ab>] do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] error_code+0x73/0x78
BUG: soft lockup - CPU#0 stuck for 61s! [gnome-session:2730]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2730, comm: gnome-session Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00200202 CPU: 0
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200292 EBX: c5e041ec ECX: 00000004 EDX: 00000020
ESI: 00000000 EDI: c5e04204 EBP: f2175e1c ESP: f2175e04
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gnome-session (pid: 2730, ti=f2174000 task=f5f603b0
task.ti=f2174000)
Stack:
 01000000 00000000 c042a1a2 f2174000 c042a1a2 000000d5 f2175e2c c0465ce2
<0> 00000001 00000000 f2175e40 c04442f5 000000f6 00000020 000000d5
f2175e48
<0> c042a110 f2175ee4 c04b4784 00000020 00000020 000000d6 000000d7
000000d8
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b84b8>] ? handle_mm_fault+0xdc/0x92a
 [<c07961ab>] ? do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] ? error_code+0x73/0x78
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b84b8>] handle_mm_fault+0xdc/0x92a
 [<c07961ab>] do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] error_code+0x73/0x78
BUG: soft lockup - CPU#2 stuck for 61s! [nautilus:2874]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2874, comm: nautilus Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00200202 CPU: 2
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200286 EBX: c5f041ec ECX: 00000004 EDX: 00000020
ESI: 00000002 EDI: c5f04204 EBP: f22a9d44 ESP: f22a9d2c
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process nautilus (pid: 2874, ti=f22a8000 task=f632e2f0 task.ti=f22a8000)
Stack:
 01000000 00000000 c042a1a2 f22a8000 c042a1a2 000000f6 f22a9d54 c0465ce2
<0> 00000001 00000000 f22a9d68 c04442f5 00000117 00000020 000000f6
f22a9d70
<0> c042a110 f22a9e0c c04b4784 00000020 00000020 000000f7 000000f8
000000f9
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b68ec>] ? unmap_vmas+0x27a/0x5ae
 [<c04ce6ac>] ? mem_cgroup_add_lru_list+0x56/0x64
 [<c04b6cbc>] ? zap_page_range+0x9c/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] ? sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] ? sysenter_do_call+0x12/0x28
 [<c0790000>] ? timer_cpu_notify+0x11a/0x2bf
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b68ec>] unmap_vmas+0x27a/0x5ae
 [<c04ce6ac>] ? mem_cgroup_add_lru_list+0x56/0x64
 [<c04b6cbc>] zap_page_range+0x9c/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] sysenter_do_call+0x12/0x28
 [<c0790000>] ? timer_cpu_notify+0x11a/0x2bf
BUG: soft lockup - CPU#3 stuck for 61s! [gvfs-gdu-volume:2792]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2792, comm: gvfs-gdu-volume Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00200202 CPU: 3
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200292 EBX: c5f841ec ECX: 00000004 EDX: 00000020
ESI: 00000003 EDI: c5f84204 EBP: f23bfe1c ESP: f23bfe04
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gvfs-gdu-volume (pid: 2792, ti=f23be000 task=f2bf6330
task.ti=f23be000)
Stack:
 01000000 00000000 c042a1a2 f23be000 c042a1a2 000000b4 f23bfe2c c0465ce2
<0> 00000001 00000000 f23bfe40 c04442f5 000000d5 00000020 000000b4
f23bfe48
<0> c042a110 f23bfee4 c04b4784 00000000 00000020 000000b5 000000b6
000000b7
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b84b8>] ? handle_mm_fault+0xdc/0x92a
 [<c0455a51>] ? autoremove_wake_function+0x0/0x2f
 [<c07961ab>] ? do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] ? error_code+0x73/0x78
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b84b8>] handle_mm_fault+0xdc/0x92a
 [<c0455a51>] ? autoremove_wake_function+0x0/0x2f
 [<c07961ab>] do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] error_code+0x73/0x78
BUG: soft lockup - CPU#0 stuck for 61s! [gnome-session:2730]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2730, comm: gnome-session Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cb2>] EFLAGS: 00200202 CPU: 0
EIP is at smp_call_function_many+0x16b/0x175
EAX: 00200292 EBX: c5e041ec ECX: 00000004 EDX: 00000020
ESI: 00000000 EDI: c5e04204 EBP: f2175e1c ESP: f2175e04
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gnome-session (pid: 2730, ti=f2174000 task=f5f603b0
task.ti=f2174000)
Stack:
 01000000 00000000 c042a1a2 f2174000 c042a1a2 000000d5 f2175e2c c0465ce2
<0> 00000001 00000000 f2175e40 c04442f5 000000f6 00000020 000000d5
f2175e48
<0> c042a110 f2175ee4 c04b4784 00000020 00000020 000000d6 000000d7
000000d8
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b84b8>] ? handle_mm_fault+0xdc/0x92a
 [<c07961ab>] ? do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] ? error_code+0x73/0x78
Code: c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae f0 89 f6 89
f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 f6 43 10 01 <75> f8
8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 e3 00 e0 ff 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b84b8>] handle_mm_fault+0xdc/0x92a
 [<c07961ab>] do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] error_code+0x73/0x78
BUG: soft lockup - CPU#2 stuck for 61s! [nautilus:2874]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2874, comm: nautilus Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cb2>] EFLAGS: 00200202 CPU: 2
EIP is at smp_call_function_many+0x16b/0x175
EAX: 00200286 EBX: c5f041ec ECX: 00000004 EDX: 00000020
ESI: 00000002 EDI: c5f04204 EBP: f22a9d44 ESP: f22a9d2c
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process nautilus (pid: 2874, ti=f22a8000 task=f632e2f0 task.ti=f22a8000)
Stack:
 01000000 00000000 c042a1a2 f22a8000 c042a1a2 000000f6 f22a9d54 c0465ce2
<0> 00000001 00000000 f22a9d68 c04442f5 00000117 00000020 000000f6
f22a9d70
<0> c042a110 f22a9e0c c04b4784 00000020 00000020 000000f7 000000f8
000000f9
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b68ec>] ? unmap_vmas+0x27a/0x5ae
 [<c04ce6ac>] ? mem_cgroup_add_lru_list+0x56/0x64
 [<c04b6cbc>] ? zap_page_range+0x9c/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] ? sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] ? sysenter_do_call+0x12/0x28
 [<c0790000>] ? timer_cpu_notify+0x11a/0x2bf
Code: c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae f0 89 f6 89
f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 f6 43 10 01 <75> f8
8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 e3 00 e0 ff 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b68ec>] unmap_vmas+0x27a/0x5ae
 [<c04ce6ac>] ? mem_cgroup_add_lru_list+0x56/0x64
 [<c04b6cbc>] zap_page_range+0x9c/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] sysenter_do_call+0x12/0x28
 [<c0790000>] ? timer_cpu_notify+0x11a/0x2bf
BUG: soft lockup - CPU#3 stuck for 61s! [gvfs-gdu-volume:2792]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2792, comm: gvfs-gdu-volume Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00200202 CPU: 3
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200292 EBX: c5f841ec ECX: 00000004 EDX: 00000020
ESI: 00000003 EDI: c5f84204 EBP: f23bfe1c ESP: f23bfe04
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gvfs-gdu-volume (pid: 2792, ti=f23be000 task=f2bf6330
task.ti=f23be000)
Stack:
 01000000 00000000 c042a1a2 f23be000 c042a1a2 000000b4 f23bfe2c c0465ce2
<0> 00000001 00000000 f23bfe40 c04442f5 000000d5 00000020 000000b4
f23bfe48
<0> c042a110 f23bfee4 c04b4784 00000000 00000020 000000b5 000000b6
000000b7
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b84b8>] ? handle_mm_fault+0xdc/0x92a
 [<c0455a51>] ? autoremove_wake_function+0x0/0x2f
 [<c07961ab>] ? do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] ? error_code+0x73/0x78
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b84b8>] handle_mm_fault+0xdc/0x92a
 [<c0455a51>] ? autoremove_wake_function+0x0/0x2f
 [<c07961ab>] do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] error_code+0x73/0x78
BUG: soft lockup - CPU#0 stuck for 61s! [gnome-session:2730]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2730, comm: gnome-session Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00200202 CPU: 0
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200292 EBX: c5e041ec ECX: 00000004 EDX: 00000020
ESI: 00000000 EDI: c5e04204 EBP: f2175e1c ESP: f2175e04
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gnome-session (pid: 2730, ti=f2174000 task=f5f603b0
task.ti=f2174000)
Stack:
 01000000 00000000 c042a1a2 f2174000 c042a1a2 000000d5 f2175e2c c0465ce2
<0> 00000001 00000000 f2175e40 c04442f5 000000f6 00000020 000000d5
f2175e48
<0> c042a110 f2175ee4 c04b4784 00000020 00000020 000000d6 000000d7
000000d8
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b84b8>] ? handle_mm_fault+0xdc/0x92a
 [<c07961ab>] ? do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] ? error_code+0x73/0x78
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b84b8>] handle_mm_fault+0xdc/0x92a
 [<c07961ab>] do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] error_code+0x73/0x78
BUG: soft lockup - CPU#2 stuck for 61s! [nautilus:2874]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_seq_dummy snd_mpu401_uart
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
snd_rawmidi snd_seq_device snd_timer snd i2c_i801 iTCO_wdt ohci1394
iTCO_vendor_support asus_atk0110 soundcore sky2 snd_page_alloc serio_raw
joydev sata_sil24 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core
[last unloaded: microcode]

Pid: 2874, comm: nautilus Tainted: G      D
2.6.33.6-147.rt23.1.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00200202 CPU: 2
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200286 EBX: c5f041ec ECX: 00000004 EDX: 00000020
ESI: 00000002 EDI: c5f04204 EBP: f22a9d44 ESP: f22a9d2c
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process nautilus (pid: 2874, ti=f22a8000 task=f632e2f0 task.ti=f22a8000)
Stack:
 01000000 00000000 c042a1a2 f22a8000 c042a1a2 000000f6 f22a9d54 c0465ce2
<0> 00000001 00000000 f22a9d68 c04442f5 00000117 00000020 000000f6
f22a9d70
<0> c042a110 f22a9e0c c04b4784 00000020 00000020 000000f7 000000f8
000000f9
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b68ec>] ? unmap_vmas+0x27a/0x5ae
 [<c04ce6ac>] ? mem_cgroup_add_lru_list+0x56/0x64
 [<c04b6cbc>] ? zap_page_range+0x9c/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] ? sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] ? sysenter_do_call+0x12/0x28
 [<c0790000>] ? timer_cpu_notify+0x11a/0x2bf
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b68ec>] unmap_vmas+0x27a/0x5ae
 [<c04ce6ac>] ? mem_cgroup_add_lru_list+0x56/0x64
 [<c04b6cbc>] zap_page_range+0x9c/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] sysenter_do_call+0x12/0x28
 [<c0790000>] ? timer_cpu_notify+0x11a/0x2bf

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> On Thu, 2010-07-08 at 16:00 -0700, john stultz wrote:
> > On Thu, 2010-07-08 at 15:44 -0700, Fernando Lopez-Lezcano wrote:
> > > On Thu, 2010-07-08 at 15:33 -0700, john stultz wrote:
> > > > On Thu, 2010-07-08 at 10:19 -0700, Fernando Lopez-Lezcano wrote:
> > > > > We are having problems with 2.6.33.5+rt23, at least in our configuration
> > > > > while accessing an nfs automounted directory. This causes a complete
> > > > > machine lockup (press reset to exit as the only option). 
> > > > > 
> > > > > I simply use the Nautilus file manager (in Fedora 12) to navigate to an
> > > > > autofs mounted directory and the process monitor goes to 100% on one
> > > > > core (or maybe two), the mouse jerks a bit and the whole thing goes
> > > > > catatonic almost immediately. 
> > > > > 
> > > > > I get this in any open terminal at the time of the crash:
> > > > > 
> > > > > --------
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:------------[ cut here ]------------
> > > > > 
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:invalid opcode: 0000 [#1] PREEMPT SMP 
> > > > > 
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:last sysfs
> > > > > file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
> > > > > 
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:Process nautilus (pid: 2874, ti=f0204000 task=f17dd1f0
> > > > > task.ti=f0204000)
> > > > > 
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:Stack:
> > > > > 
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:Call Trace:
> > > > > 
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43
> > > > > 0c 89 43 0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04
> > > > > <0f> 0b eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
> > > > > 
> > > > > Message from syslogd@localhost at Jul  8 10:13:54 ...
> > > > >  kernel:EIP: [<c0792c0f>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP
> > > > > 0068:f0205cbc
> > > > > --------
> > > > > 
> > > > > And that's it... nothing else in the logs. 
> > > > 
> > > > Hrm. Not too much to go on there, but thanks for the report.
> > > > 
> > > > 
> > > > > For now we are booting into the normal Fedora kernel (this is on Fedora
> > > > > 12) as this makes the rt kernel not usable in our setup. 
> > > > > 
> > > > > Let me know if there is anything else I can do to help debug this...
> > > > 
> > > > Had you done any testing with earlier 2.6.33-rt kernels where this
> > > > didn't occur? If so what version?
> > > 
> > > I have been working with the whole series but my main usage case does
> > > not use nfs/autofs (see next paragraphs). 
> > > 
> > > I have noticed that the problem does not appear to happen when I cd into
> > > an nfs automounted directory directly. It appears to happen only when
> > > listing the contents of a mount point (ie: when "/whatever/" is an
> > > autofs mount point where several directories are mounted, not
> > > necessarily from the same server). 
> > > 
> > > Before switching to Fedora 12 users were normally running 2.6.29 rt and
> > > I had been running 2.6.31.x and 2.6.33.x rt, but I don't think it ever
> > > happened to me personally (I'm always using the command line - this is
> > > completely reproducible with nautilus). After the switch it started
> > > happening almost immediately to regular users (using nautilus mostly). 
> > > 
> > > How could I try to get more debugging information?
> > 
> > Any chance you have a serial port on the machine in question? If so its
> > likely any oops messages could be collected over that.
> 
> No response from the network or the keyboard or
> mouse at this point, reset is the only way out. 

Not quite true, it does respond to the sysrq key (a sync command got an
immediate dump in the terminal). But the boot command does not reboot
the machine. 

-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> Ok, I got one! (had to go buy a null modem cable, I thought I had one
> but it has disappeared since the last time I did this :-)

Great! Sorry to make you go shopping! But this points out the problem
nicely.


> Find it below... it keeps spewing stuff every once in a while. Hopefully
> this will be enough. No response from the network or the keyboard or
> mouse at this point, reset is the only way out. 
> 
> I can retest if somebody comes up with a patch...
> Thanks.
> -- Fernando
> 
> localhost login: ------------[ cut here ]------------
> kernel BUG at kernel/rtmutex.c:808!

So that's a double lock deadlock.

> Call Trace:
>  [<facf45e6>] ? nfs_refresh_inode_locked+0x79c/0xa1e [nfs]
>  [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
>  [<c07936b6>] ? rt_spin_unlock+0x8/0xa
>  [<c0465559>] ? rt_spin_lock_fastlock.clone.1+0x5c/0x5f
>  [<c07936a2>] ? rt_spin_lock+0x8/0xa
>  [<c04e08d3>] ? d_materialise_unique+0xa9/0x29e
>  [<facf55bc>] ? nfs_fhget+0x492/0x51d [nfs]
>  [<c07936b6>] ? rt_spin_unlock+0x8/0xa
>  [<facf15a0>] ? nfs_do_filldir+0x27b/0x3a9 [nfs]
>  [<c04dda9c>] ? filldir64+0x0/0xcb


Looking at d_materialise_unique, I see:

	/* Is this an anonymous mountpoint that we could splice
	 * into our tree? */
	if (IS_ROOT(alias)) {
		spin_lock(&alias->d_lock);
		__d_materialise_dentry(dentry, alias);
		__d_drop(alias);
		goto found;
	}

The problem being that __d_materialise_dentry then tries to lock
alias->d_lock and we hang.

Not sure if the following is the right fix but it should avoid the
deadlock. Mind testing it to verify things work ok?

Nick: Any race possibility if something catches us between
__d_materialise_dentry and d_drop? Or should this be ok?


Signed-off-by: John Stultz <johnstul@us.ibm.com>

diff --git a/fs/dcache.c b/fs/dcache.c
index c9d21ae..d37f6f4 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -2159,8 +2159,8 @@ struct dentry *d_materialise_unique(struct dentry *dentry, struct inode *inode)
 			/* Is this an anonymous mountpoint that we could splice
 			 * into our tree? */
 			if (IS_ROOT(alias)) {
-				spin_lock(&alias->d_lock);
 				__d_materialise_dentry(dentry, alias);
+				spin_lock(&alias->d_lock);
 				__d_drop(alias);
 				goto found;
 			}

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Fri, 2010-07-09 at 12:54 -0700, john stultz wrote:
> On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> > Ok, I got one! (had to go buy a null modem cable, I thought I had one
> > but it has disappeared since the last time I did this :-)
> 
> Great! Sorry to make you go shopping! But this points out the problem
> nicely.

No problem at all, I was sure I had one... anyway, I still need it :-<

> > Find it below... it keeps spewing stuff every once in a while. Hopefully
> > this will be enough. No response from the network or the keyboard or
> > mouse at this point, reset is the only way out. 
> > 
> > I can retest if somebody comes up with a patch...

Sorry, it looks like the patch patched correctly and this is the new
kernel, but I still get a hang as before. 

-- Fernando


localhost ------------[ cut here ]------------
kernel BUG at kernel/rtmutex.c:808!
invalid opcode: 0000 [#1] PREEMPT SMP 
last sysfs
file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 2889, comm: nautilus Not tainted
2.6.33.6-147.rt23.2.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0792f67>] EFLAGS: 00210046 CPU: 0
EIP is at rt_spin_lock_slowlock+0x43/0x1bb
EAX: f2a9e170 EBX: efbb1804 ECX: efbb1804 EDX: f2a9e170
ESI: ef25e47c EDI: efbf197c EBP: f1579d14 ESP: f1579cbc
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000001
Process nautilus (pid: 2889, ti=f1578000 task=f2a9e170 task.ti=f1578000)
Stack:
 00000000 00000001 00000000 f1579cd8 00200292 ef263ddc 00000000 f1579d1c
<0> fad3c5e6 00200246 efbb1804 efbb17fc 00000000 f1579cfc c0465523
ef263ddc
<0> 00000000 c07936b6 00200246 efbb1804 ef25e47c efbf197c f1579d20
c0465559
Call Trace:
 [<fad3c5e6>] ? nfs_refresh_inode_locked+0x79c/0xa1e [nfs]
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c0465559>] ? rt_spin_lock_fastlock.clone.1+0x5c/0x5f
 [<c07936a2>] ? rt_spin_lock+0x8/0xa
 [<c04e08e9>] ? d_materialise_unique+0xbf/0x29e
 [<fad395a0>] ? nfs_do_filldir+0x27b/0x3a9 [nfs]
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<fad39d57>] ? nfs_readdir+0x689/0x707 [nfs]
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<fad4a0de>] ? nfs3_decode_dirent+0x0/0x10e [nfs]
 [<c04642a6>] ? do_futex+0x99/0x696
 [<c0793335>] ? rt_mutex_lock_killable+0x28/0x2c
 [<c04ddd56>] ? vfs_readdir+0x66/0x92
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<c04ddde5>] ? sys_getdents64+0x63/0xa5
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] ? sysenter_do_call+0x12/0x28
Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43 0c 89 43
0c 89 43 10 8b 43 14 64 8b 15 2c d1 a5 c0 83 e0 fc 39 c2 75 04 <0f> 0b
eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
EIP: [<c0792f67>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP 0068:f1579cbc
---[ end trace 54bac6c1fd35847f ]---
note: nautilus[2889] exited with preempt_count 1
BUG: scheduling while atomic: nautilus/0x00000001/2889, CPU#0
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]
Pid: 2889, comm: nautilus Tainted: G      D
2.6.33.6-147.rt23.2.fc12.ccrma.i686.rtPAE #1
Call Trace:
 [<c0436617>] __schedule_bug+0x54/0x59
 [<c0791d99>] __schedule+0x66/0x7bf
 [<c05b0742>] ? plist_add+0x60/0x66
 [<c0793928>] ? _raw_spin_unlock+0x1b/0x1d
 [<c0465008>] ? task_blocks_on_rt_mutex+0x102/0x147
 [<c07925d7>] schedule+0xe/0x1f
 [<c0792efa>] __rt_mutex_slowlock+0x9b/0xc5
 [<c0793286>] rt_mutex_slowlock+0xee/0x175
 [<c04654f1>] rt_mutex_fastlock.clone.0+0x29/0x35
 [<c079335e>] rt_mutex_lock+0x25/0x28
 [<c0465690>] rt_down_read+0x1e/0x24
 [<c04409de>] exit_mm+0x2b/0x111
 [<c044236e>] do_exit+0x1a8/0x5dc
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c043fc21>] ? kmsg_dump+0xde/0xe6
 [<c0794b98>] oops_end+0x99/0xa1
 [<c040af8d>] die+0x53/0x59
 [<c07944bb>] do_trap+0x89/0xa2
 [<c0409403>] ? do_invalid_op+0x0/0x80
 [<c0409479>] do_invalid_op+0x76/0x80
 [<c0792f67>] ? rt_spin_lock_slowlock+0x43/0x1bb
 [<fad3c5e6>] ? nfs_refresh_inode_locked+0x79c/0xa1e [nfs]
 [<fac48153>] ? rpc_put_task+0x6f/0x72 [sunrpc]
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c04c9083>] ? slab_irq_enable+0x4c/0x6d
 [<c0794267>] error_code+0x73/0x78
 [<c0792f67>] ? rt_spin_lock_slowlock+0x43/0x1bb
 [<fad3c5e6>] ? nfs_refresh_inode_locked+0x79c/0xa1e [nfs]
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c07936b6>] ? rt_spin_unlock+0x8/0xa
 [<c0465559>] rt_spin_lock_fastlock.clone.1+0x5c/0x5f
 [<c07936a2>] rt_spin_lock+0x8/0xa
 [<c04e08e9>] d_materialise_unique+0xbf/0x29e
 [<fad395a0>] nfs_do_filldir+0x27b/0x3a9 [nfs]
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<fad39d57>] nfs_readdir+0x689/0x707 [nfs]
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<fad4a0de>] ? nfs3_decode_dirent+0x0/0x10e [nfs]
 [<c04642a6>] ? do_futex+0x99/0x696
 [<c0793335>] ? rt_mutex_lock_killable+0x28/0x2c
 [<c04ddd56>] vfs_readdir+0x66/0x92
 [<c04dda9c>] ? filldir64+0x0/0xcb
 [<c04ddde5>] sys_getdents64+0x63/0xa5
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] sysenter_do_call+0x12/0x28
BUG: soft lockup - CPU#3 stuck for 61s! [nautilus:2892]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 2892, comm: nautilus Tainted: G      D
2.6.33.6-147.rt23.2.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c042a3e3>] EFLAGS: 00000202 CPU: 3
EIP is at native_flush_tlb_others+0x87/0xb0
EAX: 00000202 EBX: c0a82100 ECX: 00000800 EDX: fffff000
ESI: 000000c0 EDI: c0a8210c EBP: f236ded0 ESP: f236deb4
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process nautilus (pid: 2892, ti=f236c000 task=f63323b0 task.ti=f236c000)
Stack:
 f5afb280 f5afb404 00000003 c0a82108 f5afb280 f5afb404 b21ff000 f236dee0
<0> c042a2be f236df04 f236df04 f236deec c04b5a92 b2200000 f236df4c
c04b6cc9
<0> b2200000 f236df3c 00000000 00001000 f5afb280 00000000 00000400
00000000
Call Trace:
 [<c042a2be>] ? flush_tlb_mm+0x63/0x7d
 [<c04b5a92>] ? tlb_flush_mmu+0x1a/0x56
 [<c04b6cc9>] ? zap_page_range+0xa9/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] ? sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] ? sysenter_do_call+0x12/0x28
Code: 55 e8 89 7b 04 8d 7b 0c f7 d0 23 02 85 c0 89 43 0c 74 1d 8b 55 ec
89 f8 8b 0d 50 12 99 c0 81 c2 f0 00 00 00 ff 51 78 eb 02 f3 90 <83> 3f
00 75 f9 8b 45 f0 c7 86 40 20 a8 c0 00 00 00 00 c7 43 04 
Call Trace:
 [<c042a2be>] flush_tlb_mm+0x63/0x7d
 [<c04b5a92>] tlb_flush_mmu+0x1a/0x56
 [<c04b6cc9>] zap_page_range+0xa9/0xc7
 [<c0465523>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c04b4d42>] sys_madvise+0x34e/0x4e8
 [<c047ddff>] ? audit_syscall_entry+0x113/0x135
 [<c0408adf>] sysenter_do_call+0x12/0x28
BUG: soft lockup - CPU#0 stuck for 61s! [flush-8:0:1247]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 1247, comm: flush-8:0 Tainted: G      D
2.6.33.6-147.rt23.2.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cae>] EFLAGS: 00000202 CPU: 0
EIP is at smp_call_function_many+0x167/0x175
EAX: 00000296 EBX: c5e041ec ECX: 00000004 EDX: 00000020
ESI: 00000000 EDI: c5e04204 EBP: f2f9dc50 ESP: f2f9dc38
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process flush-8:0 (pid: 1247, ti=f2f9c000 task=f591aef0
task.ti=f2f9c000)
Stack:
 01000000 00000000 c042a1a2 f2f9c000 c042a1a2 000000f8 f2f9dc60 c0465ce2
<0> 00000001 00000000 f2f9dc74 c04442f5 00000119 00000020 000000f8
f2f9dc7c
<0> c042a110 f2f9dd18 c04b4784 00000020 00000020 000000f9 000000fa
000000fb
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04bd741>] ? page_check_address+0xa8/0x14b
 [<c04bd9cd>] ? page_mkclean+0xbf/0x192
 [<c04a82e3>] ? clear_page_dirty_for_io+0x47/0xa8
 [<c04a8fbd>] ? write_cache_pages+0x167/0x249
 [<c04a8089>] ? __writepage+0x0/0x23
 [<c04a90b8>] ? generic_writepages+0x19/0x23
 [<c04a90e0>] ? do_writepages+0x1e/0x24
 [<c04e9bbf>] ? writeback_single_inode+0xad/0x1b3
 [<c04ea22d>] ? writeback_inodes_wb+0x2c2/0x34c
 [<c04ea3ac>] ? wb_writeback+0xf5/0x180
 [<c04ea5ed>] ? wb_do_writeback+0x132/0x141
 [<c04ea624>] ? bdi_writeback_task+0x28/0x90
 [<c04b3864>] ? bdi_start_fn+0x0/0xb0
 [<c04b38bc>] ? bdi_start_fn+0x58/0xb0
 [<c04b3864>] ? bdi_start_fn+0x0/0xb0
 [<c04557ae>] ? kthread+0x5f/0x64
 [<c045574f>] ? kthread+0x0/0x64
 [<c040907e>] ? kernel_thread_helper+0x6/0x10
Code: 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00 0f ae
f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04bd741>] page_check_address+0xa8/0x14b
 [<c04bd9cd>] page_mkclean+0xbf/0x192
 [<c04a82e3>] clear_page_dirty_for_io+0x47/0xa8
 [<c04a8fbd>] write_cache_pages+0x167/0x249
 [<c04a8089>] ? __writepage+0x0/0x23
 [<c04a90b8>] generic_writepages+0x19/0x23
 [<c04a90e0>] do_writepages+0x1e/0x24
 [<c04e9bbf>] writeback_single_inode+0xad/0x1b3
 [<c04ea22d>] writeback_inodes_wb+0x2c2/0x34c
 [<c04ea3ac>] wb_writeback+0xf5/0x180
 [<c04ea5ed>] wb_do_writeback+0x132/0x141
 [<c04ea624>] bdi_writeback_task+0x28/0x90
 [<c04b3864>] ? bdi_start_fn+0x0/0xb0
 [<c04b38bc>] bdi_start_fn+0x58/0xb0
 [<c04b3864>] ? bdi_start_fn+0x0/0xb0
 [<c04557ae>] kthread+0x5f/0x64
 [<c045574f>] ? kthread+0x0/0x64
 [<c040907e>] kernel_thread_helper+0x6/0x10
BUG: soft lockup - CPU#1 stuck for 61s! [rsyslogd:1366]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm ac97_bus snd_i2c
snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer snd soundcore sky2
iTCO_wdt iTCO_vendor_support ohci1394 snd_page_alloc i2c_i801 serio_raw
joydev asus_atk0110 sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 1366, comm: rsyslogd Tainted: G      D
2.6.33.6-147.rt23.2.fc12.ccrma.i686.rtPAE #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0465cac>] EFLAGS: 00000202 CPU: 1
EIP is at smp_call_function_many+0x165/0x175
EAX: 00000292 EBX: c5e841ec ECX: 00000004 EDX: 00000020
ESI: 00000001 EDI: c5e84204 EBP: f63cde1c ESP: f63cde04
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process rsyslogd (pid: 1366, ti=f63cc000 task=f3c30d70 task.ti=f63cc000)
Stack:
 01000000 00000000 c042a1a2 f63cc000 c042a1a2 000000d7 f63cde2c c0465ce2
<0> 00000001 00000000 f63cde40 c04442f5 000000f8 00000020 000000d7
f63cde48
<0> c042a110 f63cdee4 c04b4784 00000000 00000020 000000d8 000000d9
000000da
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] ? smp_call_function+0x26/0x43
 [<c04442f5>] ? on_each_cpu+0x1b/0x4c
 [<c042a110>] ? flush_tlb_all+0x14/0x16
 [<c04b4784>] ? pkmap_get_free+0x146/0x188
 [<c04b47de>] ? pkmap_insert+0x18/0x123
 [<c04b491b>] ? kmap_high+0x32/0x39
 [<c042b1c2>] ? kmap+0x4b/0x59
 [<c04256ea>] ? kmap_atomic_func+0x16/0x18
 [<c04b84b8>] ? handle_mm_fault+0xdc/0x92a
 [<c07961ab>] ? do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] ? error_code+0x73/0x78
Code: 0b 89 1d c0 21 96 c0 89 c2 b8 c8 21 96 c0 89 59 04 e8 4f dc 32 00
0f ae f0 89 f6 89 f8 ff 15 b8 0f 99 c0 80 7d eb 00 75 04 eb 08 <f3> 90
f6 43 10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 
Call Trace:
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c042a1a2>] ? do_flush_tlb_all+0x0/0x37
 [<c0465ce2>] smp_call_function+0x26/0x43
 [<c04442f5>] on_each_cpu+0x1b/0x4c
 [<c042a110>] flush_tlb_all+0x14/0x16
 [<c04b4784>] pkmap_get_free+0x146/0x188
 [<c04b47de>] pkmap_insert+0x18/0x123
 [<c04b491b>] kmap_high+0x32/0x39
 [<c042b1c2>] kmap+0x4b/0x59
 [<c04256ea>] kmap_atomic_func+0x16/0x18
 [<c04b84b8>] handle_mm_fault+0xdc/0x92a
 [<c07961ab>] do_page_fault+0x29d/0x2b3
 [<c0795f0e>] ? do_page_fault+0x0/0x2b3
 [<c0794267>] error_code+0x73/0x78

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Fri, 2010-07-09 at 15:13 -0700, Fernando Lopez-Lezcano wrote:
> On Fri, 2010-07-09 at 12:54 -0700, john stultz wrote:
> > On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> > > Ok, I got one! (had to go buy a null modem cable, I thought I had one
> > > but it has disappeared since the last time I did this :-)
> > 
> > Great! Sorry to make you go shopping! But this points out the problem
> > nicely.
> 
> No problem at all, I was sure I had one... anyway, I still need it :-<
> 
> > > Find it below... it keeps spewing stuff every once in a while. Hopefully
> > > this will be enough. No response from the network or the keyboard or
> > > mouse at this point, reset is the only way out. 
> > > 
> > > I can retest if somebody comes up with a patch...
> 
> Sorry, it looks like the patch patched correctly and this is the new
> kernel, but I still get a hang as before. 

Huh. Exact same backtrace? Maybe there's more then one issue with that
function...

thanks for the testing!
-john

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

	On Fri, 2010-07-09 at 15:31 -0700, john stultz wrote:
> On Fri, 2010-07-09 at 15:13 -0700, Fernando Lopez-Lezcano wrote:
> > On Fri, 2010-07-09 at 12:54 -0700, john stultz wrote:
> > > On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> > > > Ok, I got one! (had to go buy a null modem cable, I thought I had one
> > > > but it has disappeared since the last time I did this :-)
> > > 
> > > Great! Sorry to make you go shopping! But this points out the problem
> > > nicely.
> > 
> > No problem at all, I was sure I had one... anyway, I still need it :-<
> > 
> > > > Find it below... it keeps spewing stuff every once in a while. Hopefully
> > > > this will be enough. No response from the network or the keyboard or
> > > > mouse at this point, reset is the only way out. 
> > > > 
> > > > I can retest if somebody comes up with a patch...
> > 
> > Sorry, it looks like the patch patched correctly and this is the new
> > kernel, but I still get a hang as before. 
> 
> Huh. Exact same backtrace? Maybe there's more then one issue with that
> function...

Hmmm, most likely I made some mistake - I'll double check and report
back. 

-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Fri, 2010-07-09 at 16:07 -0700, Fernando Lopez-Lezcano wrote:
> 	On Fri, 2010-07-09 at 15:31 -0700, john stultz wrote:
> > On Fri, 2010-07-09 at 15:13 -0700, Fernando Lopez-Lezcano wrote:
> > > On Fri, 2010-07-09 at 12:54 -0700, john stultz wrote:
> > > > On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> > > > > Ok, I got one! (had to go buy a null modem cable, I thought I had one
> > > > > but it has disappeared since the last time I did this :-)
> > > > 
> > > > Great! Sorry to make you go shopping! But this points out the problem
> > > > nicely.
> > > 
> > > No problem at all, I was sure I had one... anyway, I still need it :-<
> > > 
> > > > > Find it below... it keeps spewing stuff every once in a while. Hopefully
> > > > > this will be enough. No response from the network or the keyboard or
> > > > > mouse at this point, reset is the only way out. 
> > > > > 
> > > > > I can retest if somebody comes up with a patch...
> > > 
> > > Sorry, it looks like the patch patched correctly and this is the new
> > > kernel, but I still get a hang as before. 
> > 
> > Huh. Exact same backtrace? Maybe there's more then one issue with that
> > function...
> 
> Hmmm, most likely I made some mistake - I'll double check and report
> back. 

Argh, something's wrong with what I did, I'm looking at the source in
the build server and it appears to not be patched (but I saw it being
patched in the logs with no error? go figure...). 

-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Fri, 2010-07-09 at 15:13 -0700, Fernando Lopez-Lezcano wrote:
> On Fri, 2010-07-09 at 12:54 -0700, john stultz wrote:
> > On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> > > Ok, I got one! (had to go buy a null modem cable, I thought I had one
> > > but it has disappeared since the last time I did this :-)
> > 
> > Great! Sorry to make you go shopping! But this points out the problem
> > nicely.
> 
> No problem at all, I was sure I had one... anyway, I still need it :-<
> 
> > > Find it below... it keeps spewing stuff every once in a while. Hopefully
> > > this will be enough. No response from the network or the keyboard or
> > > mouse at this point, reset is the only way out. 
> > > 
> > > I can retest if somebody comes up with a patch...
> 
> Sorry, it looks like the patch patched correctly and this is the new
> kernel, but I still get a hang as before. 

So looking over it, I'm not easily seeing what else could be off.

So Lets see if we can cut some of the guess work out of this...

>  [<c04e08e9>] ? d_materialise_unique+0xbf/0x29e

I'm curious exactly where that is in d_materialise_unique. To find out,
can you find the vmlinux image in the base of the directory you built
the kernel you triggered this in?

Then run:
# gdb ./vmlinux

Once gdb loads:
(gdb) list *0xc04e08e9

That should point to exactly where in the function we are trying to
acquire a previously locked lock.

thanks
-john

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Fri, 2010-07-09 at 15:57 -0700, john stultz wrote:
> On Fri, 2010-07-09 at 15:13 -0700, Fernando Lopez-Lezcano wrote:
> > On Fri, 2010-07-09 at 12:54 -0700, john stultz wrote:
> > > On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> > > > Ok, I got one! (had to go buy a null modem cable, I thought I had one
> > > > but it has disappeared since the last time I did this :-)
> > > 
> > > Great! Sorry to make you go shopping! But this points out the problem
> > > nicely.
> > 
> > No problem at all, I was sure I had one... anyway, I still need it :-<
> > 
> > > > Find it below... it keeps spewing stuff every once in a while. Hopefully
> > > > this will be enough. No response from the network or the keyboard or
> > > > mouse at this point, reset is the only way out. 
> > > > 
> > > > I can retest if somebody comes up with a patch...
> > 
> > Sorry, it looks like the patch patched correctly and this is the new
> > kernel, but I still get a hang as before. 
> 
> So looking over it, I'm not easily seeing what else could be off.
> 
> So Lets see if we can cut some of the guess work out of this...
> 
> >  [<c04e08e9>] ? d_materialise_unique+0xbf/0x29e
> 
> I'm curious exactly where that is in d_materialise_unique. To find out,
> can you find the vmlinux image in the base of the directory you built
> the kernel you triggered this in?
> 
> Then run:
> # gdb ./vmlinux
> 
> Once gdb loads:
> (gdb) list *0xc04e08e9
> 
> That should point to exactly where in the function we are trying to
> acquire a previously locked lock.

I'll have to rebuild. I'm doing full rpm package builds or a build
server and the original build tree is gone. 

-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Fri, 2010-07-09 at 15:57 -0700, john stultz wrote:
> On Fri, 2010-07-09 at 15:13 -0700, Fernando Lopez-Lezcano wrote:
> > On Fri, 2010-07-09 at 12:54 -0700, john stultz wrote:
> > > On Fri, 2010-07-09 at 12:02 -0700, Fernando Lopez-Lezcano wrote:
> > > > Ok, I got one! (had to go buy a null modem cable, I thought I had one
> > > > but it has disappeared since the last time I did this :-)
> > > 
> > > Great! Sorry to make you go shopping! But this points out the problem
> > > nicely.
> > 
> > No problem at all, I was sure I had one... anyway, I still need it :-<
> > 
> > > > Find it below... it keeps spewing stuff every once in a while. Hopefully
> > > > this will be enough. No response from the network or the keyboard or
> > > > mouse at this point, reset is the only way out. 
> > > > 
> > > > I can retest if somebody comes up with a patch...
> > 
> > Sorry, it looks like the patch patched correctly and this is the new
> > kernel, but I still get a hang as before. 
> 
> So looking over it, I'm not easily seeing what else could be off.
> 
> So Lets see if we can cut some of the guess work out of this...
> 
> >  [<c04e08e9>] ? d_materialise_unique+0xbf/0x29e
> 
> I'm curious exactly where that is in d_materialise_unique. To find out,
> can you find the vmlinux image in the base of the directory you built
> the kernel you triggered this in?
> 
> Then run:
> # gdb ./vmlinux
> 
> Once gdb loads:
> (gdb) list *0xc04e08e9
> 
> That should point to exactly where in the function we are trying to
> acquire a previously locked lock.

Finally... I did a local build in my desktop machine so I now have
access to the full patched/compiled source tree. I confirmed that the
patch you sent is there (moving a spin_lock one line down). 

This is from a different kernel (non-PAE) so the exact address is
different from the previous report:

(gdb) list *0xc04d82dd
0xc04d82dd is in d_materialise_unique (fs/dcache.c:2100).
2095		spin_lock(&aparent->d_lock);
2096		spin_lock(&dparent->d_lock);
2097		spin_lock(&dentry->d_lock);
2098		spin_lock(&anon->d_lock);
2099	
2100		dentry->d_parent = (aparent == anon) ? dentry : aparent;
2101		list_del(&dentry->d_u.d_child);
2102		if (!IS_ROOT(dentry))
2103			list_add(&dentry->d_u.d_child, &dentry->d_parent->d_subdirs);
2104		else

See below for the full dump of the BUG through the serial console in
this particular occurrence. 

Let me know if there is anything else I could do. 
-- Fernando



------------[ cut here ]------------
kernel BUG at kernel/rtmutex.c:808!
invalid opcode: 0000 [#1] PREEMPT SMP 
last sysfs
file: /sys/devices/pci0000:00/0000:00:1c.5/0000:02:00.0/net/eth0/statistics/collisions
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_seq_dummy snd_cs8427 snd_ac97_codec iTCO_wdt snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_vendor_support ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi
snd_seq_device snd_timer i2c_i801 snd ohci1394 snd_page_alloc soundcore
asus_atk0110 joydev sky2 serio_raw sata_sil24 radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 2832, comm: nautilus Not tainted
2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c0780ab7>] EFLAGS: 00010046 CPU: 0
EIP is at rt_spin_lock_slowlock+0x43/0x1bb
EAX: f21522b0 EBX: f022f7a4 ECX: f022f7a4 EDX: f21522b0
ESI: efcf9f1c EDI: f022fd3c EBP: f23b7d14 ESP: f23b7cbc
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000001
Process nautilus (pid: 2832, ti=f23b6000 task=f21522b0 task.ti=f23b6000)
Stack:
 00000000 00000001 00000000 f23b7cd8 00000292 efcab7ac 00000000 f23b7d1c
<0> fab0f5e6 00000246 f022f7a4 f022f79c 00000000 f23b7cfc c045e4cb
efcab7ac
<0> 00000000 c0781206 00000246 f022f7a4 efcf9f1c f022fd3c f23b7d20
c045e501
Call Trace:
 [<fab0f5e6>] ? nfs_refresh_inode_locked+0x79c/0xa1e [nfs]
 [<c045e4cb>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c0781206>] ? rt_spin_unlock+0x8/0xa
 [<c045e501>] ? rt_spin_lock_fastlock.clone.1+0x5c/0x5f
 [<c07811f2>] ? rt_spin_lock+0x8/0xa
 [<c04d82dd>] ? d_materialise_unique+0xbf/0x29e
 [<fab0c5a0>] ? nfs_do_filldir+0x27b/0x3a9 [nfs]
 [<c04d5490>] ? filldir64+0x0/0xcb
 [<fab0cd57>] ? nfs_readdir+0x689/0x707 [nfs]
 [<c04d5490>] ? filldir64+0x0/0xcb
 [<c0424e61>] ? enqueue_task+0x70/0x80
 [<fab1d0de>] ? nfs3_decode_dirent+0x0/0x10e [nfs]
 [<c045d24e>] ? do_futex+0x99/0x696
 [<c0780e85>] ? rt_mutex_lock_killable+0x28/0x2c
 [<c04d574a>] ? vfs_readdir+0x66/0x92
 [<c04d5490>] ? filldir64+0x0/0xcb
 [<c04d57d9>] ? sys_getdents64+0x63/0xa5
 [<c0476d3f>] ? audit_syscall_entry+0x113/0x135
 [<c040321f>] ? sysenter_do_call+0x12/0x28
Code: 7b 08 00 89 45 b8 75 12 8d 43 04 89 43 04 89 43 08 8d 43 0c 89 43
0c 89 43 10 8b 43 14 64 8b 15 7c d4 a3 c0 83 e0 fc 39 c2 75 04 <0f> 0b
eb fe 8b 3a 81 ff 08 01 00 00 74 0a 83 ff 02 b8 04 00 00 
EIP: [<c0780ab7>] rt_spin_lock_slowlock+0x43/0x1bb SS:ESP 0068:f23b7cbc
---[ end trace 21aa2c1d673e60c7 ]---
note: nautilus[2832] exited with preempt_count 1
BUG: soft lockup - CPU#2 stuck for 61s! [gvfs-gdu-volume:2718]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_seq_dummy snd_cs8427 snd_ac97_codec iTCO_wdt snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_vendor_support ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi
snd_seq_device snd_timer i2c_i801 snd ohci1394 snd_page_alloc soundcore
asus_atk0110 joydev sky2 serio_raw sata_sil24 radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_seq_dummy snd_cs8427 snd_ac97_codec iTCO_wdt snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_vendor_support ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi
snd_seq_device snd_timer i2c_i801 snd ohci1394 snd_page_alloc soundcore
asus_atk0110 joydev sky2 serio_raw sata_sil24 radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 2718, comm: gvfs-gdu-volume Tainted: G      D
2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c045ec5a>] EFLAGS: 00200202 CPU: 2
EIP is at smp_call_function_many+0x16b/0x175
EAX: 00200296 EBX: c360353c ECX: 00000004 EDX: 00000020
ESI: 00000002 EDI: c3603554 EBP: f18c7c90 ESP: f18c7c78
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gvfs-gdu-volume (pid: 2718, ti=f18c6000 task=f214f0b0
task.ti=f18c6000)
Stack:
 014ad48f 00000000 c0423a7a f18c6000 c0423a7a 00000224 f18c7ca0 c045ec8a
<0> 00000001 00000000 f18c7cb4 c043d731 00000246 00000020 00000224
f18c7cbc
<0> c04239e8 f18c7d58 c04ad6ad 00000000 00000021 00000225 00000226
00000227
Call Trace:
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c045ec8a>] ? smp_call_function+0x26/0x43
 [<c043d731>] ? on_each_cpu+0x1b/0x4c
 [<c04239e8>] ? flush_tlb_all+0x14/0x16
 [<c04ad6ad>] ? pkmap_get_free+0x146/0x188
 [<c04ad702>] ? pkmap_insert+0x13/0x108
 [<c04ad823>] ? kmap_high+0x2c/0x33
 [<c0424605>] ? kmap+0x4b/0x5a
 [<c041fa66>] ? kmap_atomic_func+0x16/0x18
 [<c04b0a25>] ? handle_mm_fault+0xb4/0x7d1
 [<c056e902>] ? avc_has_perm+0x3c/0x46
 [<c0783ccb>] ? do_page_fault+0x29d/0x2b3
 [<c0783a2e>] ? do_page_fault+0x0/0x2b3
 [<c0781db7>] ? error_code+0x73/0x78
 [<c049bacb>] ? file_read_actor+0x28/0xda
 [<c049d3ca>] ? generic_file_aio_read+0x340/0x519
 [<c04c9896>] ? do_sync_read+0x87/0xc2
 [<c0569a84>] ? security_file_permission+0xf/0x11
 [<c04c9969>] ? rw_verify_area+0x98/0xbb
 [<c04c980f>] ? do_sync_read+0x0/0xc2
 [<c04c9eb8>] ? vfs_read+0x7d/0xdc
 [<c04d08d9>] ? path_put+0x15/0x18
 [<c04c9fab>] ? sys_read+0x3b/0x5d
 [<c040321f>] ? sysenter_do_call+0x12/0x28
Code: c0 89 c2 b8 88 61 94 c0 89 59 04 e8 f7 27 32 00 0f ae f0 89 f6 89
f8 ff 15 e8 4c 97 c0 80 7d eb 00 75 04 eb 08 f3 90 f6 43 10 01 <75> f8
8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 e3 00 e0 ff 
Call Trace:
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c045ec8a>] smp_call_function+0x26/0x43
 [<c043d731>] on_each_cpu+0x1b/0x4c
 [<c04239e8>] flush_tlb_all+0x14/0x16
 [<c04ad6ad>] pkmap_get_free+0x146/0x188
 [<c04ad702>] pkmap_insert+0x13/0x108
 [<c04ad823>] kmap_high+0x2c/0x33
 [<c0424605>] kmap+0x4b/0x5a
 [<c041fa66>] kmap_atomic_func+0x16/0x18
 [<c04b0a25>] handle_mm_fault+0xb4/0x7d1
 [<c056e902>] ? avc_has_perm+0x3c/0x46
 [<c0783ccb>] do_page_fault+0x29d/0x2b3
 [<c0783a2e>] ? do_page_fault+0x0/0x2b3
 [<c0781db7>] error_code+0x73/0x78
 [<c049bacb>] ? file_read_actor+0x28/0xda
 [<c049d3ca>] generic_file_aio_read+0x340/0x519
 [<c04c9896>] do_sync_read+0x87/0xc2
 [<c0569a84>] ? security_file_permission+0xf/0x11
 [<c04c9969>] ? rw_verify_area+0x98/0xbb
 [<c04c980f>] ? do_sync_read+0x0/0xc2
 [<c04c9eb8>] vfs_read+0x7d/0xdc
 [<c04d08d9>] ? path_put+0x15/0x18
 [<c04c9fab>] sys_read+0x3b/0x5d
 [<c040321f>] sysenter_do_call+0x12/0x28
BUG: soft lockup - CPU#3 stuck for 61s! [gnome-keyring-d:2595]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_seq_dummy snd_cs8427 snd_ac97_codec iTCO_wdt snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_vendor_support ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi
snd_seq_device snd_timer i2c_i801 snd ohci1394 snd_page_alloc soundcore
asus_atk0110 joydev sky2 serio_raw sata_sil24 radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_seq_dummy snd_cs8427 snd_ac97_codec iTCO_wdt snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_vendor_support ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi
snd_seq_device snd_timer i2c_i801 snd ohci1394 snd_page_alloc soundcore
asus_atk0110 joydev sky2 serio_raw sata_sil24 radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 2595, comm: gnome-keyring-d Tainted: G      D
2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c045ec56>] EFLAGS: 00200202 CPU: 3
EIP is at smp_call_function_many+0x167/0x175
EAX: 00200282 EBX: c370353c ECX: 00000004 EDX: 00000020
ESI: 00000003 EDI: c3703554 EBP: f23e5e34 ESP: f23e5e1c
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process gnome-keyring-d (pid: 2595, ti=f23e4000 task=f2240d30
task.ti=f23e4000)
Stack:
 014ad48f 00000000 c0423a7a f23e4000 c0423a7a 00000246 f23e5e44 c045ec8a
<0> 00000001 00000000 f23e5e58 c043d731 00000267 00000020 00000246
f23e5e60
<0> c04239e8 f23e5efc c04ad6ad 00000021 00000020 00000247 00000248
00000249
Call Trace:
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c045ec8a>] ? smp_call_function+0x26/0x43
 [<c043d731>] ? on_each_cpu+0x1b/0x4c
 [<c04239e8>] ? flush_tlb_all+0x14/0x16
 [<c04ad6ad>] ? pkmap_get_free+0x146/0x188
 [<c04ad702>] ? pkmap_insert+0x13/0x108
 [<c04ad823>] ? kmap_high+0x2c/0x33
 [<c0424605>] ? kmap+0x4b/0x5a
 [<c041fa66>] ? kmap_atomic_func+0x16/0x18
 [<c04b0a25>] ? handle_mm_fault+0xb4/0x7d1
 [<c044e9f9>] ? autoremove_wake_function+0x0/0x2f
 [<c0783ccb>] ? do_page_fault+0x29d/0x2b3
 [<c0783a2e>] ? do_page_fault+0x0/0x2b3
 [<c0781db7>] ? error_code+0x73/0x78
Code: 1d 80 61 94 c0 89 c2 b8 88 61 94 c0 89 59 04 e8 f7 27 32 00 0f ae
f0 89 f6 89 f8 ff 15 e8 4c 97 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c045ec8a>] smp_call_function+0x26/0x43
 [<c043d731>] on_each_cpu+0x1b/0x4c
 [<c04239e8>] flush_tlb_all+0x14/0x16
 [<c04ad6ad>] pkmap_get_free+0x146/0x188
 [<c04ad702>] pkmap_insert+0x13/0x108
 [<c04ad823>] kmap_high+0x2c/0x33
 [<c0424605>] kmap+0x4b/0x5a
 [<c041fa66>] kmap_atomic_func+0x16/0x18
 [<c04b0a25>] handle_mm_fault+0xb4/0x7d1
 [<c044e9f9>] ? autoremove_wake_function+0x0/0x2f
 [<c0783ccb>] do_page_fault+0x29d/0x2b3
 [<c0783a2e>] ? do_page_fault+0x0/0x2b3
 [<c0781db7>] error_code+0x73/0x78
BUG: soft lockup - CPU#0 stuck for 61s! [rsyslogd:1287]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_seq_dummy snd_cs8427 snd_ac97_codec iTCO_wdt snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_vendor_support ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi
snd_seq_device snd_timer i2c_i801 snd ohci1394 snd_page_alloc soundcore
asus_atk0110 joydev sky2 serio_raw sata_sil24 radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: microcode]
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_seq_dummy snd_cs8427 snd_ac97_codec iTCO_wdt snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm
iTCO_vendor_support ac97_bus snd_i2c snd_mpu401_uart snd_rawmidi
snd_seq_device snd_timer i2c_i801 snd ohci1394 snd_page_alloc soundcore
asus_atk0110 joydev sky2 serio_raw sata_sil24 radeon ttm drm_kms_helper
drm i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 1287, comm: rsyslogd Tainted: G      D
2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #1 P5K/EPU/P5K/EPU
EIP: 0060:[<c045ec56>] EFLAGS: 00000202 CPU: 0
EIP is at smp_call_function_many+0x167/0x175
EAX: 00000292 EBX: c340353c ECX: 00000004 EDX: 00000020
ESI: 00000000 EDI: c3403554 EBP: f4203cd0 ESP: f4203cb8
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00010003
Process rsyslogd (pid: 1287, ti=f4202000 task=f5ebc170 task.ti=f4202000)
Stack:
 014ad48f 00000000 c0423a7a f4202000 c0423a7a 00000267 f4203ce0 c045ec8a
<0> 00000001 00000000 f4203cf4 c043d731 00000288 00000020 00000267
f4203cfc
<0> c04239e8 f4203d98 c04ad6ad 00000020 00000020 00000268 00000269
0000026a
Call Trace:
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c045ec8a>] ? smp_call_function+0x26/0x43
 [<c043d731>] ? on_each_cpu+0x1b/0x4c
 [<c04239e8>] ? flush_tlb_all+0x14/0x16
 [<c04ad6ad>] ? pkmap_get_free+0x146/0x188
 [<c04ad702>] ? pkmap_insert+0x13/0x108
 [<c04ad823>] ? kmap_high+0x2c/0x33
 [<c0424605>] ? kmap+0x4b/0x5a
 [<c049b6ee>] ? iov_iter_copy_from_user_atomic+0x1d/0x63
 [<c049b80d>] ? generic_file_buffered_write+0xd9/0x1b3
 [<c049c861>] ? __generic_file_aio_write+0x256/0x287
 [<c049c8ef>] ? generic_file_aio_write+0x5d/0xa4
 [<c04c97d4>] ? do_sync_write+0x87/0xc2
 [<c0432002>] ? pick_next_task_fair+0xb3/0xbb
 [<c0569a84>] ? security_file_permission+0xf/0x11
 [<c04c9969>] ? rw_verify_area+0x98/0xbb
 [<c04c974d>] ? do_sync_write+0x0/0xc2
 [<c04c9d26>] ? vfs_write+0x80/0xdf
 [<c04c9e19>] ? sys_write+0x3b/0x5d
 [<c040321f>] ? sysenter_do_call+0x12/0x28
Code: 1d 80 61 94 c0 89 c2 b8 88 61 94 c0 89 59 04 e8 f7 27 32 00 0f ae
f0 89 f6 89 f8 ff 15 e8 4c 97 c0 80 7d eb 00 75 04 eb 08 f3 90 <f6> 43
10 01 75 f8 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 89 e3 81 
Call Trace:
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c0423a7a>] ? do_flush_tlb_all+0x0/0x37
 [<c045ec8a>] smp_call_function+0x26/0x43
 [<c043d731>] on_each_cpu+0x1b/0x4c
 [<c04239e8>] flush_tlb_all+0x14/0x16
 [<c04ad6ad>] pkmap_get_free+0x146/0x188
 [<c04ad702>] pkmap_insert+0x13/0x108
 [<c04ad823>] kmap_high+0x2c/0x33
 [<c0424605>] kmap+0x4b/0x5a
 [<c049b6ee>] iov_iter_copy_from_user_atomic+0x1d/0x63
 [<c049b80d>] generic_file_buffered_write+0xd9/0x1b3
 [<c049c861>] __generic_file_aio_write+0x256/0x287
 [<c049c8ef>] generic_file_aio_write+0x5d/0xa4
 [<c04c97d4>] do_sync_write+0x87/0xc2
 [<c0432002>] ? pick_next_task_fair+0xb3/0xbb
 [<c0569a84>] ? security_file_permission+0xf/0x11
 [<c04c9969>] ? rw_verify_area+0x98/0xbb
 [<c04c974d>] ? do_sync_write+0x0/0xc2
 [<c04c9d26>] vfs_write+0x80/0xdf
 [<c04c9e19>] sys_write+0x3b/0x5d
 [<c040321f>] sysenter_do_call+0x12/0x28

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Mon, 2010-07-12 at 16:37 -0700, Fernando Lopez-Lezcano wrote:
> On Fri, 2010-07-09 at 15:57 -0700, john stultz wrote:
> > So looking over it, I'm not easily seeing what else could be off.
> > 
> > So Lets see if we can cut some of the guess work out of this...
> > 
> > >  [<c04e08e9>] ? d_materialise_unique+0xbf/0x29e
> > 
> > I'm curious exactly where that is in d_materialise_unique. To find out,
> > can you find the vmlinux image in the base of the directory you built
> > the kernel you triggered this in?
> > 
> > Then run:
> > # gdb ./vmlinux
> > 
> > Once gdb loads:
> > (gdb) list *0xc04e08e9
> > 
> > That should point to exactly where in the function we are trying to
> > acquire a previously locked lock.
> 
> Finally... I did a local build in my desktop machine so I now have
> access to the full patched/compiled source tree. I confirmed that the
> patch you sent is there (moving a spin_lock one line down). 
> 
> This is from a different kernel (non-PAE) so the exact address is
> different from the previous report:
> 
> (gdb) list *0xc04d82dd
> 0xc04d82dd is in d_materialise_unique (fs/dcache.c:2100).
> 2095		spin_lock(&aparent->d_lock);
> 2096		spin_lock(&dparent->d_lock);
> 2097		spin_lock(&dentry->d_lock);
> 2098		spin_lock(&anon->d_lock);
> 2099	
> 2100		dentry->d_parent = (aparent == anon) ? dentry : aparent;
> 2101		list_del(&dentry->d_u.d_child);
> 2102		if (!IS_ROOT(dentry))
> 2103			list_add(&dentry->d_u.d_child, &dentry->d_parent->d_subdirs);
> 2104		else
> 
> See below for the full dump of the BUG through the serial console in
> this particular occurrence. 

Huh. I'm still baffled. Since we're blowing out on line 2098, the anon
pointer points to the alias pointer we passed in to 
__d_materialise_dentry(). So that means the anon dentry is already
locked, and we've moved the obviously wrong lock operation down so it
shouldn't be held.

Hrm. Ok.. I think the line 2100 above gives us a hint: (aparent == anon)
So if that were the case, we would have already locked aparent and that
would explain the blowup.

How does it do with the following change?

thanks
-john



diff --git a/fs/dcache.c b/fs/dcache.c
index c9d21ae..8d68504 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -2099,7 +2099,8 @@ static void __d_materialise_dentry(struct dentry *dentry, struct dentry *anon)
 	aparent = anon->d_parent;
 
 	/* XXX: hack */
-	spin_lock(&aparent->d_lock);
+	if (aparent != anon)
+		spin_lock(&aparent->d_lock);
 	spin_lock(&dparent->d_lock);
 	spin_lock(&dentry->d_lock);
 	spin_lock(&anon->d_lock);
@@ -2121,7 +2122,8 @@ static void __d_materialise_dentry(struct dentry *dentry, struct dentry *anon)
 	spin_unlock(&anon->d_lock);
 	spin_unlock(&dentry->d_lock);
 	spin_unlock(&dparent->d_lock);
-	spin_unlock(&aparent->d_lock);
+	if (aparent != anon)
+		spin_unlock(&aparent->d_lock);
 
 	anon->d_flags &= ~DCACHE_DISCONNECTED;
 }
@@ -2159,8 +2161,8 @@ struct dentry *d_materialise_unique(struct dentry *dentry, struct inode *inode)
 			/* Is this an anonymous mountpoint that we could splice
 			 * into our tree? */
 			if (IS_ROOT(alias)) {
-				spin_lock(&alias->d_lock);
 				__d_materialise_dentry(dentry, alias);
+				spin_lock(&alias->d_lock);
 				__d_drop(alias);
 				goto found;
 			}

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Mon, 2010-07-12 at 16:53 -0700, john stultz wrote:
> On Mon, 2010-07-12 at 16:37 -0700, Fernando Lopez-Lezcano wrote:
> > On Fri, 2010-07-09 at 15:57 -0700, john stultz wrote:
> > > So looking over it, I'm not easily seeing what else could be off.
> > > 
> > > So Lets see if we can cut some of the guess work out of this...
> > > 
> > > >  [<c04e08e9>] ? d_materialise_unique+0xbf/0x29e
> > > 
> > > I'm curious exactly where that is in d_materialise_unique. To find out,
> > > can you find the vmlinux image in the base of the directory you built
> > > the kernel you triggered this in?
> > > 
> > > Then run:
> > > # gdb ./vmlinux
> > > 
> > > Once gdb loads:
> > > (gdb) list *0xc04e08e9
> > > 
> > > That should point to exactly where in the function we are trying to
> > > acquire a previously locked lock.
> > 
> > Finally... I did a local build in my desktop machine so I now have
> > access to the full patched/compiled source tree. I confirmed that the
> > patch you sent is there (moving a spin_lock one line down). 
> > 
> > This is from a different kernel (non-PAE) so the exact address is
> > different from the previous report:
> > 
> > (gdb) list *0xc04d82dd
> > 0xc04d82dd is in d_materialise_unique (fs/dcache.c:2100).
> > 2095		spin_lock(&aparent->d_lock);
> > 2096		spin_lock(&dparent->d_lock);
> > 2097		spin_lock(&dentry->d_lock);
> > 2098		spin_lock(&anon->d_lock);
> > 2099	
> > 2100		dentry->d_parent = (aparent == anon) ? dentry : aparent;
> > 2101		list_del(&dentry->d_u.d_child);
> > 2102		if (!IS_ROOT(dentry))
> > 2103			list_add(&dentry->d_u.d_child, &dentry->d_parent->d_subdirs);
> > 2104		else
> > 
> > See below for the full dump of the BUG through the serial console in
> > this particular occurrence. 
> 
> Huh. I'm still baffled. Since we're blowing out on line 2098, the anon
> pointer points to the alias pointer we passed in to 
> __d_materialise_dentry(). So that means the anon dentry is already
> locked, and we've moved the obviously wrong lock operation down so it
> shouldn't be held.
> 
> Hrm. Ok.. I think the line 2100 above gives us a hint: (aparent == anon)
> So if that were the case, we would have already locked aparent and that
> would explain the blowup.
> 
> How does it do with the following change?

Ok, you are on to something. The machine did not crash hard!
But the serial console printed this:

--------
BUG: unable to handle kernel NULL pointer dereference at 0000008c
IP: [<c045e50a>] rt_spin_lock_fastunlock.clone.2+0x6/0x3e
*pde = 00000000 
Oops: 0000 [#1] PREEMPT SMP 
last sysfs
file: /sys/devices/pci0000:00/0000:00:1c.5/0000:02:00.0/net/eth0/statistics/collisions
Modules linked in: fuse autofs4 act_police cls_flow cls_fw cls_u32
sch_htb sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm
iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REDIRECT
ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah
ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre
nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss
xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log
xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_CONNMARK xt_connmark
xt_CLASSIFY ipt_LOG iptable_nat nf_nat iptable_mangle nfnetlink coretemp
hwmon_vid nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand
acpi_cpufreq ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables
ipv6 dm_multipath uinput snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_ac97_codec ac97_bus snd_i2c snd_mpu401_uart snd_seq_dummy
snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss
ohci1394 snd_pcm snd_rawmidi snd_seq_device sky2 snd_timer i2c_i801
asus_atk0110 snd iTCO_wdt iTCO_vendor_support joydev soundcore
snd_page_alloc serio_raw sata_sil24 radeon ttm drm_kms_helper drm
i2c_algo_bit i2c_core [last unloaded: microcode]

Pid: 2855, comm: nautilus Not tainted
2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #3 P5K/EPU/P5K/EPU
EIP: 0060:[<c045e50a>] EFLAGS: 00210246 CPU: 0
EIP is at rt_spin_lock_fastunlock.clone.2+0x6/0x3e
EAX: 00000078 EBX: ef45393c ECX: 00000000 EDX: 00000078
ESI: ef716edc EDI: 00000000 EBP: f1977c8c ESP: f1977c88
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000000
Process nautilus (pid: 2855, ti=f1976000 task=f2347130 task.ti=f1976000)
Stack:
 ef45393c f1977c94 c0781206 f1977cc8 c04d842e 00000000 ef703e54 faadd5bc
<0> f1977cdc 126bc87a ef703ddc 00000000 ef45393c ef716edc ef6f5494
faafdc6c
<0> f1977df8 faad9041 c3604b5c faafdc6c ef452bfc 00007e7f f5eb41e8
00000007
Call Trace:
 [<c0781206>] ? rt_spin_unlock+0x8/0xa
 [<c04d842e>] ? d_materialise_unique+0x210/0x2aa
 [<faadd5bc>] ? nfs_fhget+0x492/0x51d [nfs]
 [<faad9041>] ? nfs_lookup+0x124/0x17d [nfs]
 [<c045e4cb>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c0781206>] ? rt_spin_unlock+0x8/0xa
 [<c04c0a8b>] ? slab_irq_enable+0x4c/0x6d
 [<c045e4cb>] ? rt_spin_lock_fastlock.clone.1+0x26/0x5f
 [<c0781206>] ? rt_spin_unlock+0x8/0xa
 [<c04d86a1>] ? d_alloc+0x14d/0x156
 [<c04d0b6b>] ? do_lookup+0xc7/0x137
 [<c04d253a>] ? link_path_walk+0x367/0x48e
 [<c04d2998>] ? path_walk+0x3a/0x84
 [<c04d204b>] ? path_init+0x6e/0x10f
 [<c04d2b9a>] ? do_path_lookup+0x124/0x186
 [<c04d33e7>] ? user_path_at+0x39/0x5f
 [<c045d24e>] ? do_futex+0x99/0x696
 [<c04cce7b>] ? vfs_fstatat+0x2d/0x54
 [<c04ccee6>] ? vfs_lstat+0x16/0x18
 [<c04ccefc>] ? sys_lstat64+0x14/0x28
 [<c0781296>] ? _mutex_unlock+0x8/0xa
 [<c04d5765>] ? vfs_readdir+0x81/0x92
 [<c04d08d9>] ? path_put+0x15/0x18
 [<c0476d3f>] ? audit_syscall_entry+0x113/0x135
 [<c0476c17>] ? audit_syscall_exit+0xfa/0x10f
 [<c040321f>] ? sysenter_do_call+0x12/0x28
Code: 8d 74 26 00 f6 c4 02 74 19 64 8b 15 7c d4 a3 c0 31 c0 f0 0f b1 53
14 85 c0 74 07 89 d8 e8 73 25 32 00 5b 5d c3 55 89 c2 89 e5 53 <8b> 40
14 64 8b 0d 7c d4 a3 c0 83 e0 fc 39 c1 74 09 83 b9 f8 0c 
EIP: [<c045e50a>] rt_spin_lock_fastunlock.clone.2+0x6/0x3e SS:ESP
0068:f1977c88
CR2: 000000000000008c
---[ end trace ab297a18fc30cdc8 ]---

After that Nautilus is not usable on that autofs mount point (spins
forever). The rest of the system is still running (but I did not try too
many things). 

-- Fernando


> diff --git a/fs/dcache.c b/fs/dcache.c
> index c9d21ae..8d68504 100644
> --- a/fs/dcache.c
> +++ b/fs/dcache.c
> @@ -2099,7 +2099,8 @@ static void __d_materialise_dentry(struct dentry *dentry, struct dentry *anon)
>  	aparent = anon->d_parent;
>  
>  	/* XXX: hack */
> -	spin_lock(&aparent->d_lock);
> +	if (aparent != anon)
> +		spin_lock(&aparent->d_lock);
>  	spin_lock(&dparent->d_lock);
>  	spin_lock(&dentry->d_lock);
>  	spin_lock(&anon->d_lock);
> @@ -2121,7 +2122,8 @@ static void __d_materialise_dentry(struct dentry *dentry, struct dentry *anon)
>  	spin_unlock(&anon->d_lock);
>  	spin_unlock(&dentry->d_lock);
>  	spin_unlock(&dparent->d_lock);
> -	spin_unlock(&aparent->d_lock);
> +	if (aparent != anon)
> +		spin_unlock(&aparent->d_lock);
>  
>  	anon->d_flags &= ~DCACHE_DISCONNECTED;
>  }
> @@ -2159,8 +2161,8 @@ struct dentry *d_materialise_unique(struct dentry *dentry, struct inode *inode)
>  			/* Is this an anonymous mountpoint that we could splice
>  			 * into our tree? */
>  			if (IS_ROOT(alias)) {
> -				spin_lock(&alias->d_lock);
>  				__d_materialise_dentry(dentry, alias);
> +				spin_lock(&alias->d_lock);
>  				__d_drop(alias);
>  				goto found;
>  			}
> 

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Mon, 2010-07-12 at 18:10 -0700, Fernando Lopez-Lezcano wrote:
> On Mon, 2010-07-12 at 16:53 -0700, john stultz wrote:
> > 
> > Hrm. Ok.. I think the line 2100 above gives us a hint: (aparent == anon)
> > So if that were the case, we would have already locked aparent and that
> > would explain the blowup.
> > 
> > How does it do with the following change?
> 
> Ok, you are on to something. The machine did not crash hard!
> But the serial console printed this:

Sigh. Its never easy, is it? :)


> --------
> BUG: unable to handle kernel NULL pointer dereference at 0000008c
> IP: [<c045e50a>] rt_spin_lock_fastunlock.clone.2+0x6/0x3e
...
> Pid: 2855, comm: nautilus Not tainted
> 2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #3 P5K/EPU/P5K/EPU
> EIP: 0060:[<c045e50a>] EFLAGS: 00210246 CPU: 0
> EIP is at rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> EAX: 00000078 EBX: ef45393c ECX: 00000000 EDX: 00000078
> ESI: ef716edc EDI: 00000000 EBP: f1977c8c ESP: f1977c88
>  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000000
> Process nautilus (pid: 2855, ti=f1976000 task=f2347130 task.ti=f1976000)
> Stack:
>  ef45393c f1977c94 c0781206 f1977cc8 c04d842e 00000000 ef703e54 faadd5bc
> <0> f1977cdc 126bc87a ef703ddc 00000000 ef45393c ef716edc ef6f5494
> faafdc6c
> <0> f1977df8 faad9041 c3604b5c faafdc6c ef452bfc 00007e7f f5eb41e8
> 00000007
> Call Trace:
>  [<c0781206>] ? rt_spin_unlock+0x8/0xa
>  [<c04d842e>] ? d_materialise_unique+0x210/0x2aa

Can you gdb list *0xc04d842e ?

Thanks again for all the testing here! Its really appreciated!
-john

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Mon, 2010-07-12 at 18:40 -0700, john stultz wrote:
> On Mon, 2010-07-12 at 18:10 -0700, Fernando Lopez-Lezcano wrote:
> > On Mon, 2010-07-12 at 16:53 -0700, john stultz wrote:
> > > 
> > > Hrm. Ok.. I think the line 2100 above gives us a hint: (aparent == anon)
> > > So if that were the case, we would have already locked aparent and that
> > > would explain the blowup.
> > > 
> > > How does it do with the following change?
> > 
> > Ok, you are on to something. The machine did not crash hard!
> > But the serial console printed this:
> 
> Sigh. Its never easy, is it? :)

Hardly ever .... :-)
I have _read_ about stories of stuff being solved on the first try, ha.

> > --------
> > BUG: unable to handle kernel NULL pointer dereference at 0000008c
> > IP: [<c045e50a>] rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> ...
> > Pid: 2855, comm: nautilus Not tainted
> > 2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #3 P5K/EPU/P5K/EPU
> > EIP: 0060:[<c045e50a>] EFLAGS: 00210246 CPU: 0
> > EIP is at rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> > EAX: 00000078 EBX: ef45393c ECX: 00000000 EDX: 00000078
> > ESI: ef716edc EDI: 00000000 EBP: f1977c8c ESP: f1977c88
> >  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000000
> > Process nautilus (pid: 2855, ti=f1976000 task=f2347130 task.ti=f1976000)
> > Stack:
> >  ef45393c f1977c94 c0781206 f1977cc8 c04d842e 00000000 ef703e54 faadd5bc
> > <0> f1977cdc 126bc87a ef703ddc 00000000 ef45393c ef716edc ef6f5494
> > faafdc6c
> > <0> f1977df8 faad9041 c3604b5c faafdc6c ef452bfc 00007e7f f5eb41e8
> > 00000007
> > Call Trace:
> >  [<c0781206>] ? rt_spin_unlock+0x8/0xa
> >  [<c04d842e>] ? d_materialise_unique+0x210/0x2aa
> 
> Can you gdb list *0xc04d842e ?

(gdb) list *0xc04d842e
0xc04d842e is in d_materialise_unique (fs/dcache.c:2073).
2068	out_unalias:
2069		d_move_locked(alias, dentry);
2070		ret = alias;
2071	out_err:
2072		spin_unlock(&inode->i_lock);
2073		if (m2)
2074			mutex_unlock(m2);
2075		if (m1)
2076			mutex_unlock(m1);
2077		return ret;

> Thanks again for all the testing here! Its really appreciated!

No problem, not the first time (but it had been a while...)
-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Mon, 2010-07-12 at 20:06 -0700, Fernando Lopez-Lezcano wrote:
> > > --------
> > > BUG: unable to handle kernel NULL pointer dereference at 0000008c
> > > IP: [<c045e50a>] rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> > ...
> > > Pid: 2855, comm: nautilus Not tainted
> > > 2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #3 P5K/EPU/P5K/EPU
> > > EIP: 0060:[<c045e50a>] EFLAGS: 00210246 CPU: 0
> > > EIP is at rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> > > EAX: 00000078 EBX: ef45393c ECX: 00000000 EDX: 00000078
> > > ESI: ef716edc EDI: 00000000 EBP: f1977c8c ESP: f1977c88
> > >  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000000
> > > Process nautilus (pid: 2855, ti=f1976000 task=f2347130 task.ti=f1976000)
> > > Stack:
> > >  ef45393c f1977c94 c0781206 f1977cc8 c04d842e 00000000 ef703e54 faadd5bc
> > > <0> f1977cdc 126bc87a ef703ddc 00000000 ef45393c ef716edc ef6f5494
> > > faafdc6c
> > > <0> f1977df8 faad9041 c3604b5c faafdc6c ef452bfc 00007e7f f5eb41e8
> > > 00000007
> > > Call Trace:
> > >  [<c0781206>] ? rt_spin_unlock+0x8/0xa
> > >  [<c04d842e>] ? d_materialise_unique+0x210/0x2aa
> > 
> > Can you gdb list *0xc04d842e ?
> 
> (gdb) list *0xc04d842e
> 0xc04d842e is in d_materialise_unique (fs/dcache.c:2073).
> 2068	out_unalias:
> 2069		d_move_locked(alias, dentry);
> 2070		ret = alias;
> 2071	out_err:
> 2072		spin_unlock(&inode->i_lock);
> 2073		if (m2)
> 2074			mutex_unlock(m2);
> 2075		if (m1)
> 2076			mutex_unlock(m1);
> 2077		return ret;
> 
> > Thanks again for all the testing here! Its really appreciated!

I just tried building 2.6.33.6 rt26 (saw it this morning on the site)
and it does not exhibit this problem. Woohoo!

Thanks again!
-- Fernando

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[john stultz]

On Wed, 2010-07-14 at 14:32 -0700, Fernando Lopez-Lezcano wrote:
> On Mon, 2010-07-12 at 20:06 -0700, Fernando Lopez-Lezcano wrote:
> > > > --------
> > > > BUG: unable to handle kernel NULL pointer dereference at 0000008c
> > > > IP: [<c045e50a>] rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> > > ...
> > > > Pid: 2855, comm: nautilus Not tainted
> > > > 2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #3 P5K/EPU/P5K/EPU
> > > > EIP: 0060:[<c045e50a>] EFLAGS: 00210246 CPU: 0
> > > > EIP is at rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> > > > EAX: 00000078 EBX: ef45393c ECX: 00000000 EDX: 00000078
> > > > ESI: ef716edc EDI: 00000000 EBP: f1977c8c ESP: f1977c88
> > > >  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000000
> > > > Process nautilus (pid: 2855, ti=f1976000 task=f2347130 task.ti=f1976000)
> > > > Stack:
> > > >  ef45393c f1977c94 c0781206 f1977cc8 c04d842e 00000000 ef703e54 faadd5bc
> > > > <0> f1977cdc 126bc87a ef703ddc 00000000 ef45393c ef716edc ef6f5494
> > > > faafdc6c
> > > > <0> f1977df8 faad9041 c3604b5c faafdc6c ef452bfc 00007e7f f5eb41e8
> > > > 00000007
> > > > Call Trace:
> > > >  [<c0781206>] ? rt_spin_unlock+0x8/0xa
> > > >  [<c04d842e>] ? d_materialise_unique+0x210/0x2aa
> > > 
> > > Can you gdb list *0xc04d842e ?
> > 
> > (gdb) list *0xc04d842e
> > 0xc04d842e is in d_materialise_unique (fs/dcache.c:2073).
> > 2068	out_unalias:
> > 2069		d_move_locked(alias, dentry);
> > 2070		ret = alias;
> > 2071	out_err:
> > 2072		spin_unlock(&inode->i_lock);
> > 2073		if (m2)
> > 2074			mutex_unlock(m2);
> > 2075		if (m1)
> > 2076			mutex_unlock(m1);
> > 2077		return ret;
> > 
> > > Thanks again for all the testing here! Its really appreciated!
> 
> I just tried building 2.6.33.6 rt26 (saw it this morning on the site)
> and it does not exhibit this problem. Woohoo!

Yea. The vfs-scalability patches were dropped. Sort of a bummer, but I'm
glad that fixes it for you.

We may run into the issue again when the patches can get beaten back
into shape, but I'll try to hunt down the issue from your log output
prior to that.

Thanks again for the testing and great feedback!
-john

Re: 2.6.33.5 rt23: machine lockup (nfs/autofs related?)

[Fernando Lopez-Lezcano]

On Wed, 2010-07-14 at 14:36 -0700, john stultz wrote:
> On Wed, 2010-07-14 at 14:32 -0700, Fernando Lopez-Lezcano wrote:
> > On Mon, 2010-07-12 at 20:06 -0700, Fernando Lopez-Lezcano wrote:
> > > > > --------
> > > > > BUG: unable to handle kernel NULL pointer dereference at 0000008c
> > > > > IP: [<c045e50a>] rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> > > > ...
> > > > > Pid: 2855, comm: nautilus Not tainted
> > > > > 2.6.33.6-147.rt23.3.fc12.ccrma.i686.rt #3 P5K/EPU/P5K/EPU
> > > > > EIP: 0060:[<c045e50a>] EFLAGS: 00210246 CPU: 0
> > > > > EIP is at rt_spin_lock_fastunlock.clone.2+0x6/0x3e
> > > > > EAX: 00000078 EBX: ef45393c ECX: 00000000 EDX: 00000078
> > > > > ESI: ef716edc EDI: 00000000 EBP: f1977c8c ESP: f1977c88
> > > > >  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 preempt:00000000
> > > > > Process nautilus (pid: 2855, ti=f1976000 task=f2347130 task.ti=f1976000)
> > > > > Stack:
> > > > >  ef45393c f1977c94 c0781206 f1977cc8 c04d842e 00000000 ef703e54 faadd5bc
> > > > > <0> f1977cdc 126bc87a ef703ddc 00000000 ef45393c ef716edc ef6f5494
> > > > > faafdc6c
> > > > > <0> f1977df8 faad9041 c3604b5c faafdc6c ef452bfc 00007e7f f5eb41e8
> > > > > 00000007
> > > > > Call Trace:
> > > > >  [<c0781206>] ? rt_spin_unlock+0x8/0xa
> > > > >  [<c04d842e>] ? d_materialise_unique+0x210/0x2aa
> > > > 
> > > > Can you gdb list *0xc04d842e ?
> > > 
> > > (gdb) list *0xc04d842e
> > > 0xc04d842e is in d_materialise_unique (fs/dcache.c:2073).
> > > 2068	out_unalias:
> > > 2069		d_move_locked(alias, dentry);
> > > 2070		ret = alias;
> > > 2071	out_err:
> > > 2072		spin_unlock(&inode->i_lock);
> > > 2073		if (m2)
> > > 2074			mutex_unlock(m2);
> > > 2075		if (m1)
> > > 2076			mutex_unlock(m1);
> > > 2077		return ret;
> > > 
> > > > Thanks again for all the testing here! Its really appreciated!
> > 
> > I just tried building 2.6.33.6 rt26 (saw it this morning on the site)
> > and it does not exhibit this problem. Woohoo!
> 
> Yea. The vfs-scalability patches were dropped. Sort of a bummer, but I'm
> glad that fixes it for you.
> 
> We may run into the issue again when the patches can get beaten back
> into shape, but I'll try to hunt down the issue from your log output
> prior to that.

Ok, let me know if you need me to test again. I have the serial cable
now! :-)

> Thanks again for the testing and great feedback!

Sure.
-- Fernando