reiserfs deadlock

All of lore.kernel.org
 help / color / mirror / Atom feed

* reiserfs deadlock
@ 2010-02-03 19:03 Alexander Beregalov
  2010-02-03 19:08 ` Alexander Beregalov
  0 siblings, 1 reply; 10+ messages in thread
From: Alexander Beregalov @ 2010-02-03 19:03 UTC (permalink / raw)
  To: Frederic Weisbecker, Linux Kernel Mailing List

Hi Frederic

I do not have previous messages and do not know how to reproduce it.
Kernel was 2.6.33-rc5-00237-g9a3cbe3

INFO: task svscan:1395 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D f73094f0  5248  1395      1 0x00000000
 f6565da0 00000046 00000007 f73094f0 c134e5fd f7309770 f73094f0 c104e4b4
 f6411c84 f6411c84 00000246 f73094f0 f6565de4 c134e58a 00000000 00000002
 00000000 c1110e38 f6411cac c1110e38 f6411cc0 f6335d18 f706bf14 f73094f0
Call Trace:
 [<c134e5fd>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4b4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e58a>] mutex_lock_nested+0x12a/0x2b0
 [<c1110e38>] ? reiserfs_write_lock+0x28/0x40
 [<c1110e38>] ? reiserfs_write_lock+0x28/0x40
 [<c1110e38>] reiserfs_write_lock+0x28/0x40
 [<c10f059a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c75>] ? sched_clock_cpu+0x95/0x110
 [<c104ca2b>] ? trace_hardirqs_off+0xb/0x10
 [<c109fe10>] ? filldir+0x0/0xd0
 [<c1042c75>] ? sched_clock_cpu+0x95/0x110
 [<c10502d9>] ? __lock_acquire+0x509/0xa70
 [<c104e1f2>] ? mark_held_locks+0x62/0x90
 [<c134e361>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4b4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e36b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0022>] ? vfs_readdir+0x52/0xa0
 [<c10a0022>] ? vfs_readdir+0x52/0xa0
 [<c109fe10>] ? filldir+0x0/0xd0
 [<c10f0c77>] reiserfs_readdir+0x17/0x20
 [<c10a0055>] vfs_readdir+0x85/0xa0
 [<c10a0188>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1395:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0022>]
vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110e38>]
reiserfs_write_lock+0x28/0x40

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-03 19:03 reiserfs deadlock Alexander Beregalov
@ 2010-02-03 19:08 ` Alexander Beregalov
  2010-02-03 20:29   ` Frederic Weisbecker
  0 siblings, 1 reply; 10+ messages in thread
From: Alexander Beregalov @ 2010-02-03 19:08 UTC (permalink / raw)
  To: Frederic Weisbecker, Linux Kernel Mailing List

On 3 February 2010 22:03, Alexander Beregalov <a.beregalov@gmail.com> wrote:
> Hi Frederic
>
> I do not have previous messages and do not know how to reproduce it.
> Kernel was 2.6.33-rc5-00237-g9a3cbe3
>

Hm, I have the same after reboot.

Do you need me to do anything before I try to fsck ?


INFO: task metalog:1561 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
metalog       D f66768b0  5692  1561      1 0x00000000
 f6463d3c 00000046 00000007 f66768b0 c134e72d f6676b30 f66768b0 c104e4c4
 f7313c84 f7313c84 00000246 f66768b0 f6463d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313cac c1110ed8 f7313cc0 f3a19dc4 f3817d18 f66768b0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by metalog/1561:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>]
generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>]
reiserfs_write_lock_once+0x28/0x50

INFO: task svscan:1603 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D f66da9e0  5248  1603      1 0x00000000
 f3a19da0 00000046 00000007 f66da9e0 c134e72d f66dac60 f66da9e0 c104e4c4
 f7313c84 f7313c84 00000246 f66da9e0 f3a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313cac c1110f28 f7313cc0 f3b5ddc4 f6463d60 f66da9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1603:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>]
vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40

INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D a741d0d1  5700  1740      2 0x00000000
 f6757a78 00000046 00000007 a741d0d1 00000010 f38d2c60 f38d29e0 c104e4c4
 f7313c84 f7313c84 00000246 f38d29e0 f6757abc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313cac c1110f28 f7313cc0 f3817d18 f7313cac f38d29e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bc52>] ? __bfs+0x102/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40

INFO: task agetty:1867 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f380d3c0  5444  1867      1 0x00000000
 f3805db4 00000046 00000007 f380d3c0 c134e72d f380d640 f380d3c0 c104e4c4
 f6dce118 f6dce118 00000246 f380d3c0 f3805df8 c134e6ba 00000000 00000002
 00000000 c109a4bb f6dce140 c109a4bb f6dce154 f6dce140 f3813dd8 f380d3c0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c109a4bb>] ? do_lookup+0xbb/0x1c0
 [<c109a4bb>] ? do_lookup+0xbb/0x1c0
 [<c109a4bb>] do_lookup+0xbb/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c1042d1e>] ? cpu_clock+0x4e/0x60
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c10a739a>] ? alloc_fd+0xba/0xf0
 [<c11d6545>] ? strncpy_from_user+0x35/0x60
 [<c109060f>] do_sys_open+0x4f/0x110
 [<c13526a5>] ? do_page_fault+0x155/0x370
 [<c1090739>] sys_open+0x29/0x40
 [<c1002b10>] sysenter_do_call+0x12/0x36
1 lock held by agetty/1867:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>]
do_lookup+0xbb/0x1c0

and so on...

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-03 19:08 ` Alexander Beregalov
@ 2010-02-03 20:29   ` Frederic Weisbecker
  2010-02-03 22:43     ` Alexander Beregalov
  0 siblings, 1 reply; 10+ messages in thread
From: Frederic Weisbecker @ 2010-02-03 20:29 UTC (permalink / raw)
  To: Alexander Beregalov; +Cc: Linux Kernel Mailing List

On Wed, Feb 03, 2010 at 10:08:57PM +0300, Alexander Beregalov wrote:
> On 3 February 2010 22:03, Alexander Beregalov <a.beregalov@gmail.com> wrote:
> > Hi Frederic
> >
> > I do not have previous messages and do not know how to reproduce it.
> > Kernel was 2.6.33-rc5-00237-g9a3cbe3
> >
> 
> Hm, I have the same after reboot.
> 
> Do you need me to do anything before I try to fsck ?

Yeah. Rebooting again makes your kernel soft lockup?

Usually such softlockup happens because we have a lock
inversion, in which case you should have a lockdep report
before the softlockup.

Otherwise this can also happen when we wait for an event
that needs the lock to complete but
that can not happen because we already have the lock.

Task A hold reiserfs lock and wait for event 1
Task B wants to complete event 1 but it need the reisers lock
for that => deadlock.

This can usually be found in a softlockup report: lots of
tasks are blocked on reiserfs_write_lock/mutex_lock
except one, and this one is important as it is probably
the waiter: the task that holds the lock and that is waiting
for another event (that in turn needs the lock to complete).

Having more reports could probably help us:

echo 100 > /proc/sys/kernel/hung_task_warnings

Hopefully you can still reproduce it :-s

Thanks a lot!

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-03 20:29   ` Frederic Weisbecker
@ 2010-02-03 22:43     ` Alexander Beregalov
  2010-02-03 22:52       ` Frederic Weisbecker
  0 siblings, 1 reply; 10+ messages in thread
From: Alexander Beregalov @ 2010-02-03 22:43 UTC (permalink / raw)
  To: Frederic Weisbecker; +Cc: Linux Kernel Mailing List

On 3 February 2010 23:29, Frederic Weisbecker <fweisbec@gmail.com> wrote:
> On Wed, Feb 03, 2010 at 10:08:57PM +0300, Alexander Beregalov wrote:
>> On 3 February 2010 22:03, Alexander Beregalov <a.beregalov@gmail.com> wrote:
>> > Hi Frederic
>> >
>> > I do not have previous messages and do not know how to reproduce it.
>> > Kernel was 2.6.33-rc5-00237-g9a3cbe3
>> >
>>
>> Hm, I have the same after reboot.
>>
>> Do you need me to do anything before I try to fsck ?
>
>
> Yeah. Rebooting again makes your kernel soft lockup?
Yes, reboot does not help. I even can't login, agetty and sshd are frozen.

INFO: task sshd:1863 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D 6f60ec44  6576  1863   1810 0x00000000
 f633dd78 00000046 ffffffff 6f60ec44 0000000f f7306b30 f73068b0 00000000
 f7306d84 7fffffff 00000000 f633de70 f633dde8 c134da45 00000000 f633dd8c
 c104ca3b 00000000 7fffffff 0000000f 6f618f50 f73068b0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1863.

No locks - what does it mean?

>
> Usually such softlockup happens because we have a lock
> inversion, in which case you should have a lockdep report
> before the softlockup.

No, I do not have it. 120 seconds after boot I see these messages on
the console,
no lockdep reports (lockdep is enabled).

>
> Otherwise this can also happen when we wait for an event
> that needs the lock to complete but
> that can not happen because we already have the lock.
>
> Task A hold reiserfs lock and wait for event 1
> Task B wants to complete event 1 but it need the reisers lock
> for that => deadlock.
>
> This can usually be found in a softlockup report: lots of
> tasks are blocked on reiserfs_write_lock/mutex_lock
> except one, and this one is important as it is probably
> the waiter: the task that holds the lock and that is waiting
> for another event (that in turn needs the lock to complete).
>
> Having more reports could probably help us:
>
> echo 100 > /proc/sys/kernel/hung_task_warnings

Ok, I will modify rc scripts to do it, as I can't login.

>
> Hopefully you can still reproduce it :-s
>
> Thanks a lot!
>
>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-03 22:43     ` Alexander Beregalov
@ 2010-02-03 22:52       ` Frederic Weisbecker
  2010-02-05  1:46         ` Alexander Beregalov
  0 siblings, 1 reply; 10+ messages in thread
From: Frederic Weisbecker @ 2010-02-03 22:52 UTC (permalink / raw)
  To: Alexander Beregalov; +Cc: Linux Kernel Mailing List

On Thu, Feb 04, 2010 at 01:43:53AM +0300, Alexander Beregalov wrote:
> On 3 February 2010 23:29, Frederic Weisbecker <fweisbec@gmail.com> wrote:
> > On Wed, Feb 03, 2010 at 10:08:57PM +0300, Alexander Beregalov wrote:
> >> On 3 February 2010 22:03, Alexander Beregalov <a.beregalov@gmail.com> wrote:
> >> > Hi Frederic
> >> >
> >> > I do not have previous messages and do not know how to reproduce it.
> >> > Kernel was 2.6.33-rc5-00237-g9a3cbe3
> >> >
> >>
> >> Hm, I have the same after reboot.
> >>
> >> Do you need me to do anything before I try to fsck ?
> >
> >
> > Yeah. Rebooting again makes your kernel soft lockup?
> Yes, reboot does not help. I even can't login, agetty and sshd are frozen.
> 
> INFO: task sshd:1863 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> sshd          D 6f60ec44  6576  1863   1810 0x00000000
>  f633dd78 00000046 ffffffff 6f60ec44 0000000f f7306b30 f73068b0 00000000
>  f7306d84 7fffffff 00000000 f633de70 f633dde8 c134da45 00000000 f633dd8c
>  c104ca3b 00000000 7fffffff 0000000f 6f618f50 f73068b0 00000000 00000000
> Call Trace:
>  [<c134da45>] schedule_timeout+0x125/0x1b0
>  [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
>  [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
>  [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
>  [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
>  [<c134d7d0>] wait_for_common+0xd0/0x130
>  [<c1024850>] ? default_wake_function+0x0/0x10
>  [<c134d8c2>] wait_for_completion+0x12/0x20
>  [<c1039709>] call_usermodehelper_exec+0x89/0xb0
>  [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
>  [<c134d730>] ? wait_for_common+0x30/0x130
>  [<c10398e2>] __request_module+0xa2/0xf0
>  [<c10a6136>] ? new_inode+0x76/0x80
>  [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
>  [<c12cc89f>] __sock_create+0x18f/0x1f0
>  [<c107b22a>] ? might_fault+0x4a/0xa0
>  [<c12cc967>] sock_create+0x37/0x40
>  [<c12ccb1e>] sys_socket+0x3e/0x70
>  [<c12ccbb0>] sys_socketcall+0x60/0x270
>  [<c1002b43>] ? sysenter_exit+0xf/0x18
>  [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
>  [<c1002b10>] sysenter_do_call+0x12/0x36
> no locks held by sshd/1863.
> 
> No locks - what does it mean?



This is the call_usermodehelper_exec path, so probably
the kernel tries to ask userspace to load a module, but
since the filesystem is locked up, this can't happen.



> >
> > Usually such softlockup happens because we have a lock
> > inversion, in which case you should have a lockdep report
> > before the softlockup.
> 
> No, I do not have it. 120 seconds after boot I see these messages on
> the console,
> no lockdep reports (lockdep is enabled).



So this is probably this event waited thing.



> >
> > Otherwise this can also happen when we wait for an event
> > that needs the lock to complete but
> > that can not happen because we already have the lock.
> >
> > Task A hold reiserfs lock and wait for event 1
> > Task B wants to complete event 1 but it need the reisers lock
> > for that => deadlock.
> >
> > This can usually be found in a softlockup report: lots of
> > tasks are blocked on reiserfs_write_lock/mutex_lock
> > except one, and this one is important as it is probably
> > the waiter: the task that holds the lock and that is waiting
> > for another event (that in turn needs the lock to complete).
> >
> > Having more reports could probably help us:
> >
> > echo 100 > /proc/sys/kernel/hung_task_warnings
> 
> Ok, I will modify rc scripts to do it, as I can't login.


Thanks!


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-03 22:52       ` Frederic Weisbecker
@ 2010-02-05  1:46         ` Alexander Beregalov
  2010-02-05  3:59           ` Frederic Weisbecker
  0 siblings, 1 reply; 10+ messages in thread
From: Alexander Beregalov @ 2010-02-05  1:46 UTC (permalink / raw)
  To: Frederic Weisbecker; +Cc: Linux Kernel Mailing List

Hi Frederic

Does it help?

2 locks held by metalog/1558:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>]
generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>]
reiserfs_write_lock_once+0x28/0x50

2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>]
vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40

3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>]
prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>]
do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>]
reiserfs_write_lock_once+0x28/0x50

2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>]
vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40

2 locks held by fcron/1872:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109d903>]
do_filp_open+0x173/0x9f0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>]
reiserfs_write_lock_once+0x28/0x50

3 locks held by ddclient/1588:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c1091417>]
do_truncate+0x57/0x90
 #1:  (&sb->s_type->i_alloc_sem_key#3){+.+...}, at: [<c10a65e6>]
notify_change+0x1d6/0x330
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>]
reiserfs_write_lock_once+0x28/0x50

2 locks held by multilog/1615:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>]
generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>]
reiserfs_write_lock_once+0x28/0x50

2 locks held by nfsd/1734:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40
2 locks held by nfsd/1735:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>]
reiserfs_write_lock+0x28/0x40

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-05  1:46         ` Alexander Beregalov
@ 2010-02-05  3:59           ` Frederic Weisbecker
  2010-02-05  9:37             ` Alexander Beregalov
  0 siblings, 1 reply; 10+ messages in thread
From: Frederic Weisbecker @ 2010-02-05  3:59 UTC (permalink / raw)
  To: Alexander Beregalov; +Cc: Linux Kernel Mailing List

On Fri, Feb 05, 2010 at 04:46:38AM +0300, Alexander Beregalov wrote:
> Hi Frederic
> 
> Does it help?

Nope :-S

I need the full stracktraces. Plus the fact a thread holds
the reiserfs lock doesn't help much in this regard. This
is a drawback of lockdep: "holding" a lock in such lock dump
means either: the task holds the lock _or_ the task requests the
lock and block on it.

The only way to determine if a task really holds the reiserfs
lock is to look at the stack trace and see if it blocks on
reiserfs_write_lock_*() or something else. If this is something
else, then it means it really holds it, and there are fair
chances this is exactly the stacktrace we want: the one that
raised the softlockup.

Thanks!

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-05  3:59           ` Frederic Weisbecker
@ 2010-02-05  9:37             ` Alexander Beregalov
  2010-02-06 10:29               ` Frederic Weisbecker
  2010-02-11 18:35               ` [PATCH] reiserfs: Fix softlockup while waiting on an inode Frederic Weisbecker
  0 siblings, 2 replies; 10+ messages in thread
From: Alexander Beregalov @ 2010-02-05  9:37 UTC (permalink / raw)
  To: Frederic Weisbecker; +Cc: Linux Kernel Mailing List

On Fri, Feb 05, 2010 at 04:59:57AM +0100, Frederic Weisbecker wrote:
> On Fri, Feb 05, 2010 at 04:46:38AM +0300, Alexander Beregalov wrote:
> > Hi Frederic
> > 
> > Does it help?
> 
> 
> Nope :-S
> 
> I need the full stracktraces. Plus the fact a thread holds
> the reiserfs lock doesn't help much in this regard. This
> is a drawback of lockdep: "holding" a lock in such lock dump
> means either: the task holds the lock _or_ the task requests the
> lock and block on it.
> 
> The only way to determine if a task really holds the reiserfs
> lock is to look at the stack trace and see if it blocks on
> reiserfs_write_lock_*() or something else. If this is something
> else, then it means it really holds it, and there are fair
> chances this is exactly the stacktrace we want: the one that
> raised the softlockup.
> 
> Thanks!


INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1736 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290cabe3  5576  1736      2 0x00000000
 f679fcb8 00000046 00000007 290cabe3 00000027 f6200280 f6200000 c104e4c4
 f7313a54 f7313a54 00000246 f6200000 f679fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6087cdc f628fcdc f6200000
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 215298ab  5912  1737      2 0x00000000
 f6087cb8 00000046 00000007 215298ab 00000035 f6201770 f62014f0 c104e4c4
 f7313a54 f7313a54 00000246 f62014f0 f6087cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6187cdc f679fcdc f62014f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1738 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 21535e50  6560  1738      2 0x00000000
 f6187cb8 00000046 00000007 21535e50 00000035 f6202c60 f62029e0 c104e4c4
 f7313a54 f7313a54 00000246 f62029e0 f6187cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6427d60 f6087cdc f62029e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290bda91  5944  1740      2 0x00000000
 f628fcb8 00000046 00000007 290bda91 00000027 f6205640 f62053c0 c104e4c4
 f7313a54 f7313a54 00000246 f62053c0 f628fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f679fcdc f6017cf8 f62053c0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30 f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task metalog:1558 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
metalog       D 4f4322f1  5448  1558      1 0x00000000
 f6427d3c 00000046 00000007 4f4322f1 00000039 f61cc150 f61cbed0 c104e4c4
 f7313a54 f7313a54 00000246 f61cbed0 f6427d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f674bcdc f6187cdc f61cbed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by metalog/1558:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>] generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1734 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D f66d68b0  6560  1734      2 0x00000000
 f677bcb8 00000046 00000007 f66d68b0 c134e72d f66d6b30 f66d68b0 c104e4c4
 f7313a54 f7313a54 00000246 f66d68b0 f677bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6463da8 f674bcdc f66d68b0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1734:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1735 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 11df5955  6940  1735      2 0x00000000
 f674bcb8 00000046 00000007 11df5955 00000051 f676ac60 f676a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f676a9e0 f674bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f677bcdc f6427d60 f676a9e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1735:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1736 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290cabe3  5576  1736      2 0x00000000
 f679fcb8 00000046 00000007 290cabe3 00000027 f6200280 f6200000 c104e4c4
 f7313a54 f7313a54 00000246 f6200000 f679fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6087cdc f628fcdc f6200000
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object0/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 215298ab  5912  1737      2 0x00000000
 f6087cb8 00000046 00000007 215298ab 00000035 f6201770 f62014f0 c104e4c4
 f7313a54 f7313a54 00000246 f62014f0 f6087cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6187cdc f679fcdc f62014f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1738 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 21535e50  6560  1738      2 0x00000000
 f6187cb8 00000046 00000007 21535e50 00000035 f6202c60 f62029e0 c104e4c4
 f7313a54 f7313a54 00000246 f62029e0 f6187cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6427d60 f6087cdc f62029e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290bda91  5944  1740      2 0x00000000
 f628fcb8 00000046 00000007 290bda91 00000027 f6205640 f62053c0 c104e4c4
 f7313a54 f7313a54 00000246 f62053c0 f628fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f679fcdc f6017cf8 f62053c0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30 f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task metalog:1558 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
metalog       D 4f4322f1  5448  1558      1 0x00000000
 f6427d3c 00000046 00000007 4f4322f1 00000039 f61cc150 f61cbed0 c104e4c4
 f7313a54 f7313a54 00000246 f61cbed0 f6427d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f674bcdc f6187cdc f61cbed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by metalog/1558:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>] generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1734 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D f66d68b0  6560  1734      2 0x00000000
 f677bcb8 00000046 00000007 f66d68b0 c134e72d f66d6b30 f66d68b0 c104e4c4
 f7313a54 f7313a54 00000246 f66d68b0 f677bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6463da8 f674bcdc f66d68b0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1734:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1735 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 11df5955  6940  1735      2 0x00000000
 f674bcb8 00000046 00000007 11df5955 00000051 f676ac60 f676a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f676a9e0 f674bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f677bcdc f6427d60 f676a9e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1735:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1736 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290cabe3  5576  1736      2 0x00000000
 f679fcb8 00000046 00000007 290cabe3 00000027 f6200280 f6200000 c104e4c4
 f7313a54 f7313a54 00000246 f6200000 f679fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6087cdc f628fcdc f6200000
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 215298ab  5912  1737      2 0x00000000
 f6087cb8 00000046 00000007 215298ab 00000035 f6201770 f62014f0 c104e4c4
 f7313a54 f7313a54 00000246 f62014f0 f6087cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6187cdc f679fcdc f62014f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1738 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 21535e50  6560  1738      2 0x00000000
 f6187cb8 00000046 00000007 21535e50 00000035 f6202c60 f62029e0 c104e4c4
 f7313a54 f7313a54 00000246 f62029e0 f6187cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6427d60 f6087cdc f62029e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290bda91  5944  1740      2 0x00000000
 f628fcb8 00000046 00000007 290bda91 00000027 f6205640 f62053c0 c104e4c4
 f7313a54 f7313a54 00000246 f62053c0 f628fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f679fcdc f6017cf8 f62053c0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30 f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task fcron:1872 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fcron         D f62d8000  7144  1872   1636 0x00000000
 f6463d84 00000046 00000007 f62d8000 c134e72d f62d8280 f62d8000 c104e4c4
 f7313a54 f7313a54 00000246 f62d8000 f6463dc8 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f7313a7c f677bcdc f62d8000
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a674>] __lookup_hash+0xb4/0x110
 [<c109a6fd>] lookup_hash+0x2d/0x40
 [<c109d90b>] do_filp_open+0x17b/0x9f0
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c10a739a>] ? alloc_fd+0xba/0xf0
 [<c11d6545>] ? strncpy_from_user+0x35/0x60
 [<c109060f>] do_sys_open+0x4f/0x110
 [<c13526a5>] ? do_page_fault+0x155/0x370
 [<c1090739>] sys_open+0x29/0x40
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by fcron/1872:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109d903>] do_filp_open+0x173/0x9f0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task metalog:1558 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
metalog       D 4f4322f1  5448  1558      1 0x00000000
 f6427d3c 00000046 00000007 4f4322f1 00000039 f61cc150 f61cbed0 c104e4c4
 f7313a54 f7313a54 00000246 f61cbed0 f6427d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f674bcdc f6187cdc f61cbed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by metalog/1558:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>] generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1734 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D f66d68b0  6560  1734      2 0x00000000
 f677bcb8 00000046 00000007 f66d68b0 c134e72d f66d6b30 f66d68b0 c104e4c4
 f7313a54 f7313a54 00000246 f66d68b0 f677bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6463da8 f674bcdc f66d68b0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/00
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1734:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1735 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 11df5955  6940  1735      2 0x00000000
 f674bcb8 00000046 00000007 11df5955 00000051 f676ac60 f676a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f676a9e0 f674bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f677bcdc f6427d60 f676a9e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x60x1c
2 locks held by nfsd/1735:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1736 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290cabe3  5576  1736      2 0x00000000
 f679fcb8 00000046 00000007 290cabe3 00000027 f6200280 f6200000 c104e4c4
 f7313a54 f7313a54 00000246 f6200000 f679fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6087cdc f628fcdc f6200000
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 215298ab  5912  1737      2 0x00000000
 f6087cb8 00000046 00000007 215298ab 00000035 f6201770 f62014f0 c104e4c4
 f7313a54 f7313a54 00000246 f62014f0 f6087cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6187cdc f679fcdc f62014f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1738 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 21535e50  6560  1738      2 0x00000000
 f6187cb8 00000046 00000007 21535e50 00000035 f6202c60 f62029e0 c104e4c4
 f7313a54 f7313a54 00000246 f62029e0 f6187cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6427d60 f6087cdc f62029e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290bda91  5944  1740      2 0x00000000
 f628fcb8 00000046 00000007 290bda91 00000027 f6205640 f62053c0 c104e4c4
 f7313a54 f7313a54 00000246 f62053c0 f628fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f679fcdc f6017cf8 f62053c0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30 f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task fcron:1872 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fcron         D f62d8000  7144  1872   1636 0x00000000
 f6463d84 00000046 00000007 f62d8000 c134e72d f62d8280 f62d8000 c104e4c4
 f7313a54 f7313a54 00000246 f62d8000 f6463dc8 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f7313a7c f677bcdc f62d8000
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a674>] __lookup_hash+0xb4/0x110
 [<c109a6fd>] lookup_hash+0x2d/0x40
 [<c109d90b>] do_filp_open+0x17b/0x9f0
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c10a739a>] ? alloc_fd+0xba/0xf0
 [<c11d6545>] ? strncpy_from_user+0x35/0x60
 [<c109060f>] do_sys_open+0x4f/0x110
 [<c13526a5>] ? do_page_fault+0x155/0x370
 [<c1090739>] sys_open+0x29/0x40
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by fcron/1872:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109d903>] do_filp_open+0x173/0x9f0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task metalog:1558 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
metalog       D 4f4322f1  5448  1558      1 0x00000000
 f6427d3c 00000046 00000007 4f4322f1 00000039 f61cc150 f61cbed0 c104e4c4
 f7313a54 f7313a54 00000246 f61cbed0 f6427d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f674bcdc f6187cdc f61cbed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by metalog/1558:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>] generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1734 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D f66d68b0  6560  1734      2 0x00000000
 f677bcb8 00000046 00000007 f66d68b0 c134e72d f66d6b30 f66d68b0 c104e4c4
 f7313a54 f7313a54 00000246 f66d68b0 f677bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6463da8 f674bcdc f66d68b0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1734:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1735 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 11df5955  6940  1735      2 0x00000000
 f674bcb8 00000046 00000007 11df5955 00000051 f676ac60 f676a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f676a9e0 f674bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f677bcdc f6427d60 f676a9e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1735:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1736 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290cabe3  5576  1736      2 0x00000000
 f679fcb8 00000046 00000007 290cabe3 00000027 f6200280 f6200000 c104e4c4
 f7313a54 f7313a54 00000246 f6200000 f679fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6087cdc f628fcdc f6200000
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 215298ab  5912  1737      2 0x00000000
 f6087cb8 00000046 00000007 215298ab 00000035 f6201770 f62014f0 c104e4c4
 f7313a54 f7313a54 00000246 f62014f0 f6087cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6187cdc f679fcdc f62014f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1738 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 21535e50  6560  1738      2 0x00000000
 f6187cb8 00000046 00000007 21535e50 00000035 f6202c60 f62029e0 c104e4c4
 f7313a54 f7313a54 00000246 f62029e0 f6187cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6427d60 f6087cdc f62029e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290bda91  5944  1740      2 0x00000000
 f628fcb8 00000046 00000007 290bda91 00000027 f6205640 f62053c0 c104e4c4
 f7313a54 f7313a54 00000246 f62053c0 f628fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f679fcdc f6017cf8 f62053c0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30 f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task fcron:1872 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fcron         D f62d8000  7144  1872   1636 0x00000000
 f6463d84 00000046 00000007 f62d8000 c134e72d f62d8280 f62d8000 c104e4c4
 f7313a54 f7313a54 00000246 f62d8000 f6463dc8 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f7313a7c f677bcdc f62d8000
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a674>] __lookup_hash+0xb4/0x110
 [<c109a6fd>] lookup_hash+0x2d/0x40
 [<c109d90b>] do_filp_open+0x17b/0x9f0
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c10a739a>] ? alloc_fd+0xba/0xf0
 [<c11d6545>] ? strncpy_from_user+0x35/0x60
 [<c109060f>] do_sys_open+0x4f/0x110
 [<c13526a5>] ? do_page_fault+0x155/0x370
 [<c1090739>] sys_open+0x29/0x40
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by fcron/1872:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109d903>] do_filp_open+0x173/0x9f0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task metalog:1558 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
metalog       D 4f4322f1  5448  1558      1 0x00000000
 f6427d3c 00000046 00000007 4f4322f1 00000039 f61cc150 f61cbed0 c104e4c4
 f7313a54 f7313a54 00000246 f61cbed0 f6427d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f674bcdc f6187cdc f61cbed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by metalog/1558:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>] generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1734 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D f66d68b0  6560  1734      2 0x00000000
 f677bcb8 00000046 00000007 f66d68b0 c134e72d f66d6b30 f66d68b0 c104e4c4
 f7313a54 f7313a54 00000246 f66d68b0 f677bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6463da8 f674bcdc f66d68b0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1734:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1735 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 11df5955  6940  1735      2 0x00000000
 f674bcb8 00000046 00000007 11df5955 00000051 f676ac60 f676a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f676a9e0 f674bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f677bcdc f6427d60 f676a9e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1735:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1736 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290cabe3  5576  1736      2 0x00000000
 f679fcb8 00000046 00000007 290cabe3 00000027 f6200280 f6200000 c104e4c4
 f7313a54 f7313a54 00000246 f6200000 f679fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6087cdc f628fcdc f6200000
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 215298ab  5912  1737      2 0x00000000
 f6087cb8 00000046 00000007 215298ab 00000035 f6201770 f62014f0 c104e4c4
 f7313a54 f7313a54 00000246 f62014f0 f6087cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6187cdc f679fcdc f62014f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1738 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 21535e50  6560  1738      2 0x00000000
 f6187cb8 00000046 00000007 21535e50 00000035 f6202c60 f62029e0 c104e4c4
 f7313a54 f7313a54 00000246 f62029e0 f6187cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6427d60 f6087cdc f62029e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290bda91  5944  1740      2 0x00000000
 f628fcb8 00000046 00000007 290bda91 00000027 f6205640 f62053c0 c104e4c4
 f7313a54 f7313a54 00000246 f62053c0 f628fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f679fcdc f6017cf8 f62053c0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30 f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task fcron:1872 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fcron         D f62d8000  7144  1872   1636 0x00000000
 f6463d84 00000046 00000007 f62d8000 c134e72d f62d8280 f62d8000 c104e4c4
 f7313a54 f7313a54 00000246 f62d8000 f6463dc8 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f6473d60 f677bcdc f62d8000
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a674>] __lookup_hash+0xb4/0x110
 [<c109a6fd>] lookup_hash+0x2d/0x40
 [<c109d90b>] do_filp_open+0x17b/0x9f0
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c10a739a>] ? alloc_fd+0xba/0xf0
 [<c11d6545>] ? strncpy_from_user+0x35/0x60
 [<c109060f>] do_sys_open+0x4f/0x110
 [<c13526a5>] ? do_page_fault+0x155/0x370
 [<c1090739>] sys_open+0x29/0x40
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by fcron/1872:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109d903>] do_filp_open+0x173/0x9f0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task metalog:1558 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
metalog       D 4f4322f1  5448  1558      1 0x00000000
 f6427d3c 00000046 00000007 4f4322f1 00000039 f61cc150 f61cbed0 c104e4c4
 f7313a54 f7313a54 00000246 f61cbed0 f6427d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f674bcdc f6187cdc f61cbed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by metalog/1558:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>] generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task ddclient:1588 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ddclient      D f7303ed0  5180  1588      1 0x00000000
 f6103da0 00200046 00000007 f7303ed0 c134e72d f7304150 f7303ed0 c104e4c4
 f7313a54 f7313a54 00200246 f7303ed0 f6103de4 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f7313a7c f6473d60 f7303ed0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10eb0b3>] reiserfs_setattr+0x23/0x290
 [<c134eb15>] ? down_write+0x85/0x90
 [<c10a65e6>] ? notify_change+0x1d6/0x330
 [<c10a64f7>] notify_change+0xe7/0x330
 [<c1091417>] ? do_truncate+0x57/0x90
 [<c1091417>] ? do_truncate+0x57/0x90
 [<c1091421>] do_truncate+0x61/0x90
 [<c109e0bf>] do_filp_open+0x92f/0x9f0
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c10a739a>] ? alloc_fd+0xba/0xf0
 [<c11d6545>] ? strncpy_from_user+0x35/0x60
 [<c109060f>] do_sys_open+0x4f/0x110
 [<c13526a5>] ? do_page_fault+0x155/0x370
 [<c1090739>] sys_open+0x29/0x40
 [<c1002b10>] sysenter_do_call+0x12/0x36
3 locks held by ddclient/1588:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c1091417>] do_truncate+0x57/0x90
 #1:  (&sb->s_type->i_alloc_sem_key#3){+.+...}, at: [<c10a65e6>] notify_change+0x1d6/0x330
 #2:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task svscan:1600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan        D bea5aec3  5248  1600      1 0x00000000
 f67abda0 00000046 00000007 bea5aec3 0000000b f676eb30 f676e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f676e8b0 f67abde4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f5a19dc4 f614fa9c f676e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by svscan/1600:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task multilog:1615 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
multilog      D 2b1ba5ab  5044  1615   1605 0x00000000
 f6473d3c 00000046 00000007 2b1ba5ab 000000db f66d1770 f66d14f0 c104e4c4
 f7313a54 f7313a54 00000246 f66d14f0 f6473d80 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f6103dc4 f6463da8 f66d14f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10f7152>] reiserfs_dirty_inode+0x22/0xc0
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c10ade0c>] __mark_inode_dirty+0x2c/0x130
 [<c10a48e2>] file_update_time+0xa2/0x110
 [<c106a199>] __generic_file_aio_write+0x1c9/0x4f0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c106a514>] generic_file_aio_write+0x54/0xb0
 [<c109218c>] do_sync_write+0x9c/0xd0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f01d8>] reiserfs_file_write+0x88/0xa0
 [<c1092b5a>] vfs_write+0x9a/0x160
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c10f0150>] ? reiserfs_file_write+0x0/0xa0
 [<c1092ccd>] sys_write+0x3d/0x70
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by multilog/1615:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c106a501>] generic_file_aio_write+0x41/0xb0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task nfsd:1734 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D f66d68b0  6560  1734      2 0x00000000
 f677bcb8 00000046 00000007 f66d68b0 c134e72d f66d6b30 f66d68b0 c104e4c4
 f7313a54 f7313a54 00000246 f66d68b0 f677bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6463da8 f674bcdc f66d68b0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1734:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1735 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 11df5955  6940  1735      2 0x00000000
 f674bcb8 00000046 00000007 11df5955 00000051 f676ac60 f676a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f676a9e0 f674bcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f677bcdc f6427d60 f676a9e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1735:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1736 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290cabe3  5576  1736      2 0x00000000
 f679fcb8 00000046 00000007 290cabe3 00000027 f6200280 f6200000 c104e4c4
 f7313a54 f7313a54 00000246 f6200000 f679fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6087cdc f628fcdc f6200000
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1736:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 215298ab  5912  1737      2 0x00000000
 f6087cb8 00000046 00000007 215298ab 00000035 f6201770 f62014f0 c104e4c4
 f7313a54 f7313a54 00000246 f62014f0 f6087cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6187cdc f679fcdc f62014f0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1737:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1738 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 21535e50  6560  1738      2 0x00000000
 f6187cb8 00000046 00000007 21535e50 00000035 f6202c60 f62029e0 c104e4c4
 f7313a54 f7313a54 00000246 f62029e0 f6187cfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6427d60 f6087cdc f62029e0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1738:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 39af5827  5700  1739      2 0x00000000
 f614fa78 00000046 00000007 39af5827 0000000b f6204150 f6203ed0 c104e4c4
 f7313a54 f7313a54 00000246 f6203ed0 f614fabc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f67abdc4 f7313a7c f6203ed0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c1102687>] search_by_key+0x247/0x21b0
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c104b5a0>] ? usage_match+0x0/0x20
 [<c105891b>] ? __module_text_address+0xb/0x50
 [<c1058968>] ? is_module_text_address+0x8/0x10
 [<c103b257>] ? __kernel_text_address+0x47/0x70
 [<c10051a1>] ? print_context_stack+0x41/0xb0
 [<c10042ff>] ? dump_trace+0x7f/0xd0
 [<c104bb6a>] ? __bfs+0x1a/0x200
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c10ebfdf>] reiserfs_read_locked_inode+0x5f/0x5a0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec5af>] reiserfs_iget+0x8f/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1739:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 290bda91  5944  1740      2 0x00000000
 f628fcb8 00000046 00000007 290bda91 00000027 f6205640 f62053c0 c104e4c4
 f7313a54 f7313a54 00000246 f62053c0 f628fcfc c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f679fcdc f6017cf8 f62053c0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10ec5ef>] reiserfs_get_dentry+0x1f/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c105f75b>] ? __call_rcu+0x3b/0x70
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1740:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task nfsd:1741 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D 38f62cfa  6636  1741      2 0x00000000
 f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
 f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
 c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
Call Trace:
 [<c10a4838>] inode_wait+0x8/0x10
 [<c134dc75>] __wait_on_bit+0x45/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
 [<c10a4830>] ? inode_wait+0x0/0x10
 [<c103d530>] ? wake_bit_function+0x0/0x50
 [<c10a51dc>] ifind+0x8c/0xb0
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10a5f90>] iget5_locked+0x40/0x170
 [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
 [<c10ec554>] reiserfs_iget+0x34/0xb0
 [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
 [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
 [<c1128b85>] exportfs_decode_fh+0x35/0x200
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c133fbc0>] ? cache_check+0x30/0x330
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108cab4>] ? slab_pad_check+0x34/0x120
 [<c11305fa>] ? exp_get_by_name+0x4a/0x70
 [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
 [<c108ce74>] ? check_object+0xe4/0x200
 [<c108d470>] ? init_object+0x40/0x70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c1042de2>] ? prepare_creds+0x22/0x50
 [<c112c3a7>] fh_verify+0x2f7/0x580
 [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
 [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
 [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
 [<c1129055>] nfsd_dispatch+0x95/0x200
 [<c133697a>] svc_process+0x40a/0x730
 [<c1129654>] nfsd+0xa4/0x130
 [<c11295b0>] ? nfsd+0x0/0x130
 [<c103d1fc>] kthread+0x6c/0x80
 [<c103d190>] ? kthread+0x0/0x80
 [<c100303a>] kernel_thread_helper+0x6/0x1c
2 locks held by nfsd/1741:
 #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40
INFO: task agetty:1868 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
agetty        D f601a9e0  5444  1868      1 0x00000000
 f6017cd4 00000046 00000007 f601a9e0 c134e72d f601ac60 f601a9e0 c104e4c4
 f7313a54 f7313a54 00000246 f601a9e0 f6017d18 c134e6ba 00000000 00000002
 00000000 c1110ed8 f7313a7c c1110ed8 f7313a90 f628fcdc f5a19dc4 f601a9e0
Call Trace:
 [<c134e72d>] ? mutex_lock_nested+0x19d/0x2b0
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] ? reiserfs_write_lock_once+0x28/0x50
 [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
 [<c10e87ab>] reiserfs_lookup+0x5b/0x130
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c109a563>] do_lookup+0x163/0x1c0
 [<c1099eda>] ? generic_permission+0x1a/0xa0
 [<c109c06e>] link_path_walk+0x3be/0xc10
 [<c109ca23>] path_walk+0x43/0xa0
 [<c109cb51>] do_path_lookup+0x51/0x60
 [<c109dc8a>] do_filp_open+0x4fa/0x9f0
 [<c1097229>] ? check_unsafe_exec+0xd9/0x110
 [<c1097c4f>] open_exec+0x2f/0x100
 [<c1097179>] ? check_unsafe_exec+0x29/0x110
 [<c109847b>] do_execve+0xab/0x250
 [<c1008b0d>] sys_execve+0x2d/0x60
 [<c1002b86>] ptregs_execve+0x12/0x18
 [<c1002b10>] ? sysenter_do_call+0x12/0x36
3 locks held by agetty/1868:
 #0:  (&p->cred_guard_mutex){+.+.+.}, at: [<c10972f4>] prepare_bprm_creds+0x24/0x60
 #1:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c109a4bb>] do_lookup+0xbb/0x1c0
 #:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110ed8>] reiserfs_write_lock_once+0x28/0x50
INFO: task sshd:1869 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D be338e6a  6576  1869   1816 0x00000000
 f5a15d78 00000046 ffffffff be338e6a 0000000f f601c150 f601bed0 00000000
 f601c3a4 7fffffff 00000000 f5a15e70 f5a15de8 c134da45 00000000 f5a15d8c
 c104ca3b 00000000 7fffffff 0000000f be3432d8 f601bed0 00000000 00000000
Call Trace:
 [<c134da45>] schedule_timeout+0x125/0x1b0
 [<c104ca3b>] ? trace_hardirqs_off+0xb/0x10
 [<c1350152>] ? _raw_spin_unlock_irq+0x22/0x30
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
 [<c134d7d0>] wait_for_common+0xd0/0x130
 [<c1024850>] ? default_wake_function+0x0/0x10
 [<c134d8c2>] wait_for_completion+0x12/0x20
 [<c1039709>] call_usermodehelper_exec+0x89/0xb0
 [<c1039471>] ? call_usermodehelper_setup+0x71/0xb0
 [<c134d730>] ? wait_for_common+0x30/0x130
 [<c10398e2>] __request_module+0xa2/0xf0
 [<c10a6136>] ? new_inode+0x76/0x80
 [<c13501cd>] ? _raw_spin_unlock+0x1d/0x20
 [<c12cc89f>] __sock_create+0x18f/0x1f0
 [<c107b22a>] ? might_fault+0x4a/0xa0
 [<c12cc967>] sock_create+0x37/0x40
 [<c12ccb1e>] sys_socket+0x3e/0x70
 [<c12ccbb0>] sys_socketcall+0x60/0x270
 [<c1002b43>] ? sysenter_exit+0xf/0x18
 [<c11d5eb4>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c1002b10>] sysenter_do_call+0x12/0x36
no locks held by sshd/1869.
INFO: task modprobe:1871 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D be576d77  6740  1871   1870 0x00000000
 f5a19da0 00000046 00000007 be576d77 0000000f f601eb30 f601e8b0 c104e4c4
 f7313a54 f7313a54 00000246 f601e8b0 f5a19de4 c134e6ba 00000000 00000002
 00000000 c1110f28 f7313a7c c1110f28 f7313a90 f6017cf8 f67abdc4 f601e8b0
Call Trace:
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e6ba>] mutex_lock_nested+0x12a/0x2b0
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] ? reiserfs_write_lock+0x28/0x40
 [<c1110f28>] reiserfs_write_lock+0x28/0x40
 [<c10f068a>] reiserfs_readdir_dentry+0x5a/0x720
 [<c1042975>] ? T.324+0x15/0x1b0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c1042c55>] ? sched_clock_cpu+0x95/0x110
 [<c1050609>] ? __lock_acquire+0x509/0xa70
 [<c104e202>] ? mark_held_locks+0x62/0x90
 [<c134e491>] ? mutex_lock_killable_nested+0x201/0x300
 [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
 [<c134e49b>] ? mutex_lock_killable_nested+0x20b/0x300
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c10a0112>] ? vfs_readdir+0x52/0xa0
 [<c109ff00>] ? filldir+0x0/0xd0
 [<c10f0d67>] reiserfs_readdir+0x17/0x20
 [<c10a0145>] vfs_readdir+0x85/0xa0
 [<c10a0278>] sys_getdents+0x68/0xc0
 [<c1002b10>] sysenter_do_call+0x12/0x36
2 locks held by modprobe/1871:
 #0:  (&sb->s_type->i_mutex_key#5){+.+.+.}, at: [<c10a0112>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: reiserfs deadlock
  2010-02-05  9:37             ` Alexander Beregalov
@ 2010-02-06 10:29               ` Frederic Weisbecker
  2010-02-11 18:35               ` [PATCH] reiserfs: Fix softlockup while waiting on an inode Frederic Weisbecker
  1 sibling, 0 replies; 10+ messages in thread
From: Frederic Weisbecker @ 2010-02-06 10:29 UTC (permalink / raw)
  To: Alexander Beregalov; +Cc: Linux Kernel Mailing List

On Fri, Feb 05, 2010 at 12:37:52PM +0300, Alexander Beregalov wrote:
> INFO: task nfsd:1741 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> nfsd          D 38f62cfa  6636  1741      2 0x00000000
>  f62d3c50 00000046 f62d3c30 38f62cfa 0000000b f6206b30 f62068b0 f62d3c90
>  f62d3c58 f62d3c88 00000000 f62d3c90 f62d3c58 c10a4838 f62d3c74 c134dc75
>  c10a4830 c27efab8 c27efab8 f6fbbf9c f62d3ca4 f62d3cb0 c134dd00 00000002
> Call Trace:
>  [<c10a4838>] inode_wait+0x8/0x10
>  [<c134dc75>] __wait_on_bit+0x45/0x70
>  [<c10a4830>] ? inode_wait+0x0/0x10
>  [<c134dd00>] out_of_line_wait_on_bit+0x60/0x70
>  [<c10a4830>] ? inode_wait+0x0/0x10
>  [<c103d530>] ? wake_bit_function+0x0/0x50
>  [<c10a51dc>] ifind+0x8c/0xb0
>  [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
>  [<c10a5f90>] iget5_locked+0x40/0x170
>  [<c10ea460>] ? reiserfs_find_actor+0x0/0x30
>  [<c10ec554>] reiserfs_iget+0x34/0xb0
>  [<c10ea440>] ? reiserfs_init_locked_inode+0x0/0x20
>  [<c10ec5f9>] reiserfs_get_dentry+0x29/0x70
>  [<c1042c55>] ? sched_clock_cpu+0x95/0x110
>  [<c10ec6cf>] reiserfs_fh_to_dentry+0x3f/0xb0
>  [<c1128b85>] exportfs_decode_fh+0x35/0x200
>  [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
>  [<c133fbc0>] ? cache_check+0x30/0x330
>  [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
>  [<c108cab4>] ? slab_pad_check+0x34/0x120
>  [<c11305fa>] ? exp_get_by_name+0x4a/0x70
>  [<c134018c>] ? sunrpc_cache_lookup+0x5c/0x140
>  [<c108ce74>] ? check_object+0xe4/0x200
>  [<c108d470>] ? init_object+0x40/0x70
>  [<c104e202>] ? mark_held_locks+0x62/0x90
>  [<c108ed55>] ? kmem_cache_alloc+0xa5/0xf0
>  [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
>  [<c104e51b>] ? trace_hardirqs_on+0xb/0x10
>  [<c1042de2>] ? prepare_creds+0x22/0x50
>  [<c1042de2>] ? prepare_creds+0x22/0x50
>  [<c112c3a7>] fh_verify+0x2f7/0x580
>  [<c112bf50>] ? nfsd_acceptable+0x0/0xf0
>  [<c102ea21>] ? local_bh_enable_ip+0x61/0xc0
>  [<c104e4c4>] ? trace_hardirqs_on_caller+0x124/0x170
>  [<c13422ab>] ? svc_xprt_enqueue+0x7b/0x240
>  [<c1135245>] nfsd3_proc_getattr+0x55/0xb0
>  [<c1129055>] nfsd_dispatch+0x95/0x200
>  [<c133697a>] svc_process+0x40a/0x730
>  [<c1129654>] nfsd+0xa4/0x130
>  [<c11295b0>] ? nfsd+0x0/0x130
>  [<c103d1fc>] kthread+0x6c/0x80
>  [<c103d190>] ? kthread+0x0/0x80
>  [<c100303a>] kernel_thread_helper+0x6/0x1c
> 2 locks held by nfsd/1741:
>  #0:  (hash_sem){.+.+.+}, at: [<c1131a2d>] exp_readlock+0xd/0x10
>  #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c1110f28>] reiserfs_write_lock+0x28/0x40


Yes! That must be the culprit. We are waiting for the inode to be
un-dirtied, but it can't since the writeback will need the reiserfs lock,
which we hold already.

Fine, I'll fix this, thanks a lot again for your report Alexander!


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH] reiserfs: Fix softlockup while waiting on an inode
  2010-02-05  9:37             ` Alexander Beregalov
  2010-02-06 10:29               ` Frederic Weisbecker
@ 2010-02-11 18:35               ` Frederic Weisbecker
  1 sibling, 0 replies; 10+ messages in thread
From: Frederic Weisbecker @ 2010-02-11 18:35 UTC (permalink / raw)
  To: Alexander Beregalov
  Cc: LKML, Frederic Weisbecker, Christian Kujau, Chris Mason

When we wait for an inode through reiserfs_iget(), we hold
the reiserfs lock. And waiting for an inode may imply waiting
for its writeback. But the inode writeback path may also require
the reiserfs lock, which leads to a deadlock.

We just need to release the reiserfs lock from reiserfs_iget()
to fix this.

Reported-by: Alexander Beregalov <a.beregalov@gmail.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Christian Kujau <lists@nerdbynature.de>
Cc: Chris Mason <chris.mason@oracle.com>
---
 fs/reiserfs/inode.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/fs/reiserfs/inode.c b/fs/reiserfs/inode.c
index 9087b10..2df0f5c 100644
--- a/fs/reiserfs/inode.c
+++ b/fs/reiserfs/inode.c
@@ -1497,9 +1497,11 @@ struct inode *reiserfs_iget(struct super_block *s, const struct cpu_key *key)
 
 	args.objectid = key->on_disk_key.k_objectid;
 	args.dirid = key->on_disk_key.k_dir_id;
+	reiserfs_write_unlock(s);
 	inode = iget5_locked(s, key->on_disk_key.k_objectid,
 			     reiserfs_find_actor, reiserfs_init_locked_inode,
 			     (void *)(&args));
+	reiserfs_write_lock(s);
 	if (!inode)
 		return ERR_PTR(-ENOMEM);
 
-- 
1.6.2.3


^ permalink raw reply related	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2010-02-11 18:35 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-02-03 19:03 reiserfs deadlock Alexander Beregalov
2010-02-03 19:08 ` Alexander Beregalov
2010-02-03 20:29   ` Frederic Weisbecker
2010-02-03 22:43     ` Alexander Beregalov
2010-02-03 22:52       ` Frederic Weisbecker
2010-02-05  1:46         ` Alexander Beregalov
2010-02-05  3:59           ` Frederic Weisbecker
2010-02-05  9:37             ` Alexander Beregalov
2010-02-06 10:29               ` Frederic Weisbecker
2010-02-11 18:35               ` [PATCH] reiserfs: Fix softlockup while waiting on an inode Frederic Weisbecker

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.