Re: Soft lockup problem

public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed

* Re: Soft lockup problem
       [not found] <CAOgv-ziomS-+ML0Xb4oSoWTjzkG8BBBQcMjXwn7rjQV2pFtQ6Q@mail.gmail.com>
@ 2012-02-06 22:51 ` Jan Kara
  2012-02-07 16:35   ` Gerard Saraber
  0 siblings, 1 reply; 5+ messages in thread
From: Jan Kara @ 2012-02-06 22:51 UTC (permalink / raw)
  To: Gerard Saraber; +Cc: linux-kernel, xfs

On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
> Greetings everyone,
> I've been having a bit of a problem since upgrading to the linux 3.x
> series, I have a machine that we're using as a NAS that runs various
> rsync processes (mostly at night), lately after a day or two, I will
> come in in the morning to a load average of 49, but the machine not
> really doing anything, when trying to run 'dstat' the command just
> hung with no output at all. there were no errors in the logs, or even
> anything that would vaguely point at anything I could work with.
> So needing to get the machine back to work I attempted to reboot it
> "shutdown -r now" on console... it gives a nice message saying it's
> going to reboot, but nothing ever happens.. the only way to reboot it
> is by using ctrl + alt + sysrq + b. after which the machine reboots
> and the raid array comes back clean.
> 
> I'm not sure how to troubleshoot this, any pointers would be appreciated.
> 
> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
> options in the kernel debugging section:
> detect hard/soft lockups and detect hung tasks, maybe it'll give me
> something more to go on.
> 
> Some details about the machine:
> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
> together into a 23TB XFS filesystem.
> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
  Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Soft lockup problem
  2012-02-06 22:51 ` Soft lockup problem Jan Kara
@ 2012-02-07 16:35   ` Gerard Saraber
  2012-02-07 16:54     ` Jan Kara
  0 siblings, 1 reply; 5+ messages in thread
From: Gerard Saraber @ 2012-02-07 16:35 UTC (permalink / raw)
  To: Jan Kara; +Cc: linux-kernel, xfs

On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <jack@suse.cz> wrote:
> On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
>> Greetings everyone,
>> I've been having a bit of a problem since upgrading to the linux 3.x
>> series, I have a machine that we're using as a NAS that runs various
>> rsync processes (mostly at night), lately after a day or two, I will
>> come in in the morning to a load average of 49, but the machine not
>> really doing anything, when trying to run 'dstat' the command just
>> hung with no output at all. there were no errors in the logs, or even
>> anything that would vaguely point at anything I could work with.
>> So needing to get the machine back to work I attempted to reboot it
>> "shutdown -r now" on console... it gives a nice message saying it's
>> going to reboot, but nothing ever happens.. the only way to reboot it
>> is by using ctrl + alt + sysrq + b. after which the machine reboots
>> and the raid array comes back clean.
>>
>> I'm not sure how to troubleshoot this, any pointers would be appreciated.
>>
>> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
>> options in the kernel debugging section:
>> detect hard/soft lockups and detect hung tasks, maybe it'll give me
>> something more to go on.
>>
>> Some details about the machine:
>> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
>> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
>> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
>> together into a 23TB XFS filesystem.
>> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
>  Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
> Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
>
>                                                                Honza
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR

Thanks for the quick reply,
the machine is running good at the moment so I'm not sure if the
output helps, but here it is:
[I'll also be sure to grab this log the next time it locks]

-Gerard


Feb 07 10:32:04 [kernel] [88365.922121] SysRq : Show Blocked State
Feb 07 10:32:04 [kernel] [88365.922141]   task
PC stack   pid father
Feb 07 10:32:04 [kernel] [88365.922170] flush-253:0     D
0000000000000000  1320  2941      2 0x00000000
Feb 07 10:32:04 [kernel] [88365.922177]  ffff8801ada63570
0000000000000046 ffffffff8189813a ffff8801b554a800
Feb 07 10:32:04 [kernel] [88365.922182]  ffff880100000000
00000000001d2880 ffff8801ad8d8000 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922188]  ffff8801ada63fd8
ffff8801ada62000 00000000001d2880 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922193] Call Trace:
Feb 07 10:32:04 [kernel] [88365.922203]  [<ffffffff8189813a>] ?
__schedule+0x70a/0x920
Feb 07 10:32:04 [kernel] [88365.922207]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 07 10:32:04 [kernel] [88365.922211]  [<ffffffff816bc640>]
get_active_stripe+0x2f0/0x660
Feb 07 10:32:04 [kernel] [88365.922217]  [<ffffffff810c0000>] ?
sysfs_show_available_clocksources+0x10/0xf0
Feb 07 10:32:04 [kernel] [88365.922222]  [<ffffffff81088560>] ?
try_to_wake_up+0x2d0/0x2d0
Feb 07 10:32:04 [kernel] [88365.922226]  [<ffffffff816c39b6>]
make_request+0x196/0x430
Feb 07 10:32:04 [kernel] [88365.922231]  [<ffffffff810b4e30>] ?
wake_up_bit+0x40/0x40
Feb 07 10:32:04 [kernel] [88365.922236]  [<ffffffff816da26b>] ?
dm_request+0x13b/0x230
Feb 07 10:32:04 [kernel] [88365.922239]  [<ffffffff816cf1a4>]
md_make_request+0x174/0x2b0
Feb 07 10:32:04 [kernel] [88365.922243]  [<ffffffff816cf098>] ?
md_make_request+0x68/0x2b0
Feb 07 10:32:04 [kernel] [88365.922247]  [<ffffffff810b9d5e>] ?
up_read+0x1e/0x40
Feb 07 10:32:04 [kernel] [88365.922250]  [<ffffffff816da26b>] ?
dm_request+0x13b/0x230
Feb 07 10:32:04 [kernel] [88365.922253]  [<ffffffff816da15e>] ?
dm_request+0x2e/0x230
Feb 07 10:32:04 [kernel] [88365.922259]  [<ffffffff8145c722>]
generic_make_request+0xc2/0x100
Feb 07 10:32:04 [kernel] [88365.922262]  [<ffffffff8145d4d5>]
submit_bio+0x75/0xf0
Feb 07 10:32:04 [kernel] [88365.922267]  [<ffffffff811ac88b>] ?
__mark_inode_dirty+0x3b/0x220
Feb 07 10:32:04 [kernel] [88365.922272]  [<ffffffff8133f512>]
xfs_submit_ioend_bio.clone.12+0x52/0x80
Feb 07 10:32:04 [kernel] [88365.922275]  [<ffffffff8133f62e>]
xfs_submit_ioend+0xee/0x110
Feb 07 10:32:04 [kernel] [88365.922278]  [<ffffffff8133f89c>]
xfs_vm_writepage+0x24c/0x520
Feb 07 10:32:04 [kernel] [88365.922284]  [<ffffffff81136115>]
__writepage+0x15/0x50
Feb 07 10:32:04 [kernel] [88365.922287]  [<ffffffff81136749>]
write_cache_pages+0x219/0x4c0
Feb 07 10:32:04 [kernel] [88365.922291]  [<ffffffff81136100>] ?
set_page_dirty+0x70/0x70
Feb 07 10:32:04 [kernel] [88365.922295]  [<ffffffff81136a3f>]
generic_writepages+0x4f/0x70
Feb 07 10:32:04 [kernel] [88365.922298]  [<ffffffff8133e4e8>]
xfs_vm_writepages+0x58/0x70
Feb 07 10:32:04 [kernel] [88365.922302]  [<ffffffff8113822f>]
do_writepages+0x1f/0x40
Feb 07 10:32:04 [kernel] [88365.922306]  [<ffffffff811ad5c9>]
writeback_single_inode+0x189/0x400
Feb 07 10:32:04 [kernel] [88365.922309]  [<ffffffff811adc28>]
writeback_sb_inodes+0x1a8/0x270
Feb 07 10:32:04 [kernel] [88365.922313]  [<ffffffff811add86>]
__writeback_inodes_wb+0x96/0xc0
Feb 07 10:32:04 [kernel] [88365.922316]  [<ffffffff811adf93>]
wb_writeback+0x1e3/0x2c0
Feb 07 10:32:04 [kernel] [88365.922319]  [<ffffffff811370e5>] ?
determine_dirtyable_memory+0x15/0x30
Feb 07 10:32:04 [kernel] [88365.922323]  [<ffffffff81137192>] ?
global_dirty_limits+0x32/0x160
Feb 07 10:32:04 [kernel] [88365.922326]  [<ffffffff811aee3e>]
wb_do_writeback+0x11e/0x210
Feb 07 10:32:04 [kernel] [88365.922330]  [<ffffffff811aefda>]
bdi_writeback_thread+0xaa/0x290
Feb 07 10:32:04 [kernel] [88365.922333]  [<ffffffff811aef30>] ?
wb_do_writeback+0x210/0x210
Feb 07 10:32:04 [kernel] [88365.922337]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
Feb 07 10:32:04 [kernel] [88365.922342]  [<ffffffff8189e1b4>]
kernel_thread_helper+0x4/0x10
Feb 07 10:32:04 [kernel] [88365.922346]  [<ffffffff8189b8dd>] ?
retint_restore_args+0xe/0xe
Feb 07 10:32:04 [kernel] [88365.922350]  [<ffffffff810b4820>] ?
__init_kthread_worker+0x70/0x70
Feb 07 10:32:04 [kernel] [88365.922353]  [<ffffffff8189e1b0>] ?
gs_change+0xb/0xb
Feb 07 10:32:04 [kernel] [88365.922357] ssh             D
0000000000000000  4080 23086  23078 0x00000000
Feb 07 10:32:04 [kernel] [88365.922363]  ffff8801492d79a8
0000000000000046 ffffffff8189813a 0000000000000006
Feb 07 10:32:04 [kernel] [88365.922368]  0000000000000000
00000000001d2880 ffff8801b7902680 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922373]  ffff8801492d7fd8
ffff8801492d6000 00000000001d2880 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922379] Call Trace:
Feb 07 10:32:04 [kernel] [88365.922382]  [<ffffffff8189813a>] ?
__schedule+0x70a/0x920
Feb 07 10:32:04 [kernel] [88365.922385]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 07 10:32:04 [kernel] [88365.922389]  [<ffffffff8189891e>]
schedule_timeout+0x18e/0x2e0
Feb 07 10:32:04 [kernel] [88365.922393]  [<ffffffff810a0a10>] ?
cascade+0xa0/0xa0
Feb 07 10:32:04 [kernel] [88365.922397]  [<ffffffff810be378>] ?
ktime_get_ts+0xa8/0xe0
Feb 07 10:32:04 [kernel] [88365.922401]  [<ffffffff8189871a>]
io_schedule_timeout+0x9a/0xf0
Feb 07 10:32:04 [kernel] [88365.922404]  [<ffffffff811372ec>] ?
bdi_dirty_limit+0x2c/0xc0
Feb 07 10:32:04 [kernel] [88365.922408]  [<ffffffff811379ea>]
balance_dirty_pages_ratelimited_nr+0x2ba/0x770
Feb 07 10:32:04 [kernel] [88365.922417]  [<ffffffff81348114>] ?
xfs_iunlock+0x74/0xf0
Feb 07 10:32:04 [kernel] [88365.922422]  [<ffffffff8112c349>]
generic_file_buffered_write+0x1b9/0x280
Feb 07 10:32:04 [kernel] [88365.922426]  [<ffffffff81345900>]
xfs_file_buffered_aio_write+0xf0/0x1a0
Feb 07 10:32:04 [kernel] [88365.922430]  [<ffffffff8114e7ab>] ?
might_fault+0x3b/0x90
Feb 07 10:32:04 [kernel] [88365.922434]  [<ffffffff81345b62>]
xfs_file_aio_write+0x1b2/0x2f0
Feb 07 10:32:04 [kernel] [88365.922439]  [<ffffffff81198677>] ?
core_sys_select+0x47/0x390
Feb 07 10:32:04 [kernel] [88365.922442]  [<ffffffff8114e7ab>] ?
might_fault+0x3b/0x90
Feb 07 10:32:04 [kernel] [88365.922446]  [<ffffffff8118432a>]
do_sync_write+0xda/0x120
Feb 07 10:32:04 [kernel] [88365.922452]  [<ffffffff8143f167>] ?
security_file_permission+0x27/0xb0
Feb 07 10:32:04 [kernel] [88365.922455]  [<ffffffff81184c46>]
vfs_write+0xc6/0x190
Feb 07 10:32:04 [kernel] [88365.922458]  [<ffffffff81184f7f>]
sys_write+0x4f/0xa0
Feb 07 10:32:04 [kernel] [88365.922462]  [<ffffffff8189bffb>]
system_call_fastpath+0x16/0x1b
Feb 07 10:32:04 [kernel] [88365.922465] ssh             D
0000000000000000  4080 23087  23078 0x00000000
Feb 07 10:32:04 [kernel] [88365.922471]  ffff88010e2159a8
0000000000000046 ffffffff8189813a 0000000000000006
Feb 07 10:32:04 [kernel] [88365.922476]  0000000000000000
00000000001d2880 ffff8801b32d2680 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922482]  ffff88010e215fd8
ffff88010e214000 00000000001d2880 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922487] Call Trace:
Feb 07 10:32:04 [kernel] [88365.922490]  [<ffffffff8189813a>] ?
__schedule+0x70a/0x920
Feb 07 10:32:04 [kernel] [88365.922493]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 07 10:32:04 [kernel] [88365.922497]  [<ffffffff8189891e>]
schedule_timeout+0x18e/0x2e0
Feb 07 10:32:04 [kernel] [88365.922500]  [<ffffffff810a0a10>] ?
cascade+0xa0/0xa0
Feb 07 10:32:04 [kernel] [88365.922504]  [<ffffffff810be378>] ?
ktime_get_ts+0xa8/0xe0
Feb 07 10:32:04 [kernel] [88365.922507]  [<ffffffff8189871a>]
io_schedule_timeout+0x9a/0xf0
Feb 07 10:32:04 [kernel] [88365.922511]  [<ffffffff811372ec>] ?
bdi_dirty_limit+0x2c/0xc0
Feb 07 10:32:04 [kernel] [88365.922515]  [<ffffffff811379ea>]
balance_dirty_pages_ratelimited_nr+0x2ba/0x770
Feb 07 10:32:04 [kernel] [88365.922518]  [<ffffffff811ac88b>] ?
__mark_inode_dirty+0x3b/0x220
Feb 07 10:32:04 [kernel] [88365.922522]  [<ffffffff81348114>] ?
xfs_iunlock+0x74/0xf0
Feb 07 10:32:04 [kernel] [88365.922526]  [<ffffffff8112c349>]
generic_file_buffered_write+0x1b9/0x280
Feb 07 10:32:04 [kernel] [88365.922537]  [<ffffffff81345b62>]
xfs_file_aio_write+0x1b2/0x2f0
Feb 07 10:32:04 [kernel] [88365.922541]  [<ffffffff81198677>] ?
core_sys_select+0x47/0x390
Feb 07 10:32:04 [kernel] [88365.922544]  [<ffffffff8114e7ab>] ?
might_fault+0x3b/0x90
Feb 07 10:32:04 [kernel] [88365.922547]  [<ffffffff8118432a>]
do_sync_write+0xda/0x120
Feb 07 10:32:04 [kernel] [88365.922551]  [<ffffffff8143f167>] ?
security_file_permission+0x27/0xb0
Feb 07 10:32:04 [kernel] [88365.922555]  [<ffffffff81184c46>]
vfs_write+0xc6/0x190
Feb 07 10:32:04 [kernel] [88365.922558]  [<ffffffff81184f7f>]
sys_write+0x4f/0xa0
Feb 07 10:32:04 [kernel] [88365.922562]  [<ffffffff8189bffb>]
system_call_fastpath+0x16/0x1b
Feb 07 10:32:04 [kernel] [88365.922568] Sched Debug Version: v0.10, 3.2.4 #2
Feb 07 10:32:04 [kernel] [88365.922571] ktime
         : 88365922.568795
Feb 07 10:32:04 [kernel] [88365.922574] sched_clk
         : 88191370.131237
Feb 07 10:32:04 [kernel] [88365.922576] cpu_clk
         : 88365922.567625
Feb 07 10:32:04 [kernel] [88365.922578] jiffies
         : 4383033218
Feb 07 10:32:04 [kernel] [88365.922580] sched_clock_stable
         : 0
Feb 07 10:32:04 [kernel] [88365.922582]
Feb 07 10:32:04 [kernel] [88365.922583] sysctl_sched
Feb 07 10:32:04 [kernel] [88365.922585]   .sysctl_sched_latency
            : 18.000000
Feb 07 10:32:04 [kernel] [88365.922588]
.sysctl_sched_min_granularity            : 2.250000
Feb 07 10:32:04 [kernel] [88365.922590]
.sysctl_sched_wakeup_granularity         : 3.000000
Feb 07 10:32:04 [kernel] [88365.922592]
.sysctl_sched_child_runs_first           : 0
Feb 07 10:32:04 [kernel] [88365.922594]   .sysctl_sched_features
            : 24119
Feb 07 10:32:04 [kernel] [88365.922597]
.sysctl_sched_tunable_scaling            : 1 (logaritmic)
Feb 07 10:32:04 [kernel] [88365.922600]
Feb 07 10:32:04 [kernel] [88365.922600] cpu#0, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.922602]   .nr_running                    : 2
Feb 07 10:32:04 [kernel] [88365.922605]   .load                          : 2048
Feb 07 10:32:04 [kernel] [88365.922607]   .nr_switches
  : 47384343
Feb 07 10:32:04 [kernel] [88365.922609]   .nr_load_updates
  : 27680552
Feb 07 10:32:04 [kernel] [88365.922611]   .nr_uninterruptible            : 5780
Feb 07 10:32:04 [kernel] [88365.922613]   .next_balance
  : 4383.033233
Feb 07 10:32:04 [kernel] [88365.922615]   .curr->pid                     : 26616
Feb 07 10:32:04 [kernel] [88365.922620]   .cpu_load[0]                   : 2048
Feb 07 10:32:04 [kernel] [88365.922622]   .cpu_load[1]                   : 1024
Feb 07 10:32:04 [kernel] [88365.922623]   .cpu_load[2]                   : 512
Feb 07 10:32:04 [kernel] [88365.922625]   .cpu_load[3]                   : 256
Feb 07 10:32:04 [kernel] [88365.922627]   .cpu_load[4]                   : 128
Feb 07 10:32:04 [kernel] [88365.922629]   .yld_count                     : 0
Feb 07 10:32:04 [kernel] [88365.922631]   .sched_switch                  : 0
Feb 07 10:32:04 [kernel] [88365.922633]   .sched_count
  : 47523458
Feb 07 10:32:04 [kernel] [88365.922635]   .sched_goidle
  : 22579080
Feb 07 10:32:04 [kernel] [88365.922637]   .avg_idle
  : 888834
Feb 07 10:32:04 [kernel] [88365.922639]   .ttwu_count
  : 25633221
Feb 07 10:32:04 [kernel] [88365.922641]   .ttwu_local
  : 19078353
Feb 07 10:32:04 [kernel] [88365.922645]
Feb 07 10:32:04 [kernel] [88365.922646] cfs_rq[0]:/autogroup-861
Feb 07 10:32:04 [kernel] [88365.922648]   .exec_clock
  : 13.544484
Feb 07 10:32:04 [kernel] [88365.922652]   .MIN_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922654]   .min_vruntime
  : 6.378969
Feb 07 10:32:04 [kernel] [88365.922656]   .max_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922658]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922660]   .spread0
  : -3347531.515424
Feb 07 10:32:04 [kernel] [88365.922663]   .nr_spread_over                : 0
Feb 07 10:32:04 [kernel] [88365.922664]   .nr_running                    : 1
Feb 07 10:32:04 [kernel] [88365.922666]   .load                          : 1024
Feb 07 10:32:04 [kernel] [88365.922668]   .load_avg
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922671]   .load_period
  : 9.999999
Feb 07 10:32:04 [kernel] [88365.922673]   .load_contrib                  : 0
Feb 07 10:32:04 [kernel] [88365.922674]   .load_tg                       : 0
Feb 07 10:32:04 [kernel] [88365.922677]   .se->exec_start
  : 88365922.005056
Feb 07 10:32:04 [kernel] [88365.922679]   .se->vruntime
  : 3347529.024236
Feb 07 10:32:04 [kernel] [88365.922681]   .se->sum_exec_runtime
  : 13.544484
Feb 07 10:32:04 [kernel] [88365.922683]   .se->statistics.wait_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922686]   .se->statistics.sleep_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922688]   .se->statistics.block_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922690]   .se->statistics.sleep_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922692]   .se->statistics.block_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922694]   .se->statistics.exec_max
  : 1.397658
Feb 07 10:32:04 [kernel] [88365.922698]   .se->statistics.wait_max
  : 0.037956
Feb 07 10:32:04 [kernel] [88365.922700]   .se->statistics.wait_sum
  : 0.405641
Feb 07 10:32:04 [kernel] [88365.922702]   .se->statistics.wait_count     : 43
Feb 07 10:32:04 [kernel] [88365.922704]   .se->load.weight               : 1024
Feb 07 10:32:04 [kernel] [88365.922707]
Feb 07 10:32:04 [kernel] [88365.922707] cfs_rq[0]:/
Feb 07 10:32:04 [kernel] [88365.922709]   .exec_clock
  : 3857099.252074
Feb 07 10:32:04 [kernel] [88365.922712]   .MIN_vruntime
  : 3347528.894393
Feb 07 10:32:04 [kernel] [88365.922714]   .min_vruntime
  : 3347537.894393
Feb 07 10:32:04 [kernel] [88365.922716]   .max_vruntime
  : 3347528.894393
Feb 07 10:32:04 [kernel] [88365.922719]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922721]   .spread0
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922723]   .nr_spread_over                : 278
Feb 07 10:32:04 [kernel] [88365.922725]   .nr_running                    : 2
Feb 07 10:32:04 [kernel] [88365.922727]   .load                          : 2048
Feb 07 10:32:04 [kernel] [88365.922729]   .load_avg
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922731]   .load_period
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922733]   .load_contrib                  : 0
Feb 07 10:32:04 [kernel] [88365.922734]   .load_tg                       : 0
Feb 07 10:32:04 [kernel] [88365.922737]
Feb 07 10:32:04 [kernel] [88365.922737] rt_rq[0]:
Feb 07 10:32:04 [kernel] [88365.922739]   .rt_nr_running                 : 0
Feb 07 10:32:04 [kernel] [88365.922741]   .rt_throttled                  : 0
Feb 07 10:32:04 [kernel] [88365.922743]   .rt_time
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922745]   .rt_runtime
  : 950.000000
Feb 07 10:32:04 [kernel] [88365.922747]
Feb 07 10:32:04 [kernel] [88365.922748] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.922749]             task   PID
tree-key  switches  prio     exec-runtime         sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.922750]
----------------------------------------------------------------------------------------------------------
Feb 07 10:32:04 [kernel] [88365.922759]      kworker/0:1 25833
3347528.894393     19293   120   3347528.894393      1314.935421
2056117.209075 /
Feb 07 10:32:04 [kernel] [88365.922767] R           bash 26616
5.470536        54   120         5.470536        18.185136
37388.063060 /autogroup-861
Feb 07 10:32:04 [kernel] [88365.922777]
Feb 07 10:32:04 [kernel] [88365.922777] cpu#1, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.922779]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.922781]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.922783]   .nr_switches
  : 48208480
Feb 07 10:32:04 [kernel] [88365.922785]   .nr_load_updates
  : 28422632
Feb 07 10:32:04 [kernel] [88365.922788]   .nr_uninterruptible            : 6243
Feb 07 10:32:04 [kernel] [88365.922790]   .next_balance
  : 4383.033438
Feb 07 10:32:04 [kernel] [88365.922792]   .curr->pid                     : 0
Feb 07 10:32:04 [kernel] [88365.922794]   .clock
  : 88365922.005609
Feb 07 10:32:04 [kernel] [88365.922796]   .cpu_load[0]                   : 0
Feb 07 10:32:04 [kernel] [88365.922798]   .cpu_load[1]                   : 30
Feb 07 10:32:04 [kernel] [88365.922800]   .cpu_load[2]                   : 78
Feb 07 10:32:04 [kernel] [88365.922802]   .cpu_load[3]                   : 74
Feb 07 10:32:04 [kernel] [88365.922803]   .cpu_load[4]                   : 62
Feb 07 10:32:04 [kernel] [88365.922805]   .yld_count                     : 2
Feb 07 10:32:04 [kernel] [88365.922807]   .sched_switch                  : 0
Feb 07 10:32:04 [kernel] [88365.922809]   .sched_count
  : 48351933
Feb 07 10:32:04 [kernel] [88365.922811]   .sched_goidle
  : 22985997
Feb 07 10:32:04 [kernel] [88365.922813]   .avg_idle
  : 988263
Feb 07 10:32:04 [kernel] [88365.922815]   .ttwu_count
  : 26052282
Feb 07 10:32:04 [kernel] [88365.922817]   .ttwu_local
  : 19528683
Feb 07 10:32:04 [kernel] [88365.922820]
Feb 07 10:32:04 [kernel] [88365.922821] cfs_rq[1]:/autogroup-776
Feb 07 10:32:04 [kernel] [88365.922823]   .exec_clock
  : 358297.296203
Feb 07 10:32:04 [kernel] [88365.922826]   .MIN_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922828]   .min_vruntime
  : 299834.428916
Feb 07 10:32:04 [kernel] [88365.922831]   .max_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922833]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922835]   .spread0
  : -3047703.465477
Feb 07 10:32:04 [kernel] [88365.922837]   .nr_spread_over                : 322
Feb 07 10:32:04 [kernel] [88365.922839]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.922841]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.922843]   .load_avg
  : 943.694796
Feb 07 10:32:04 [kernel] [88365.922845]   .load_period
  : 8.490465
Feb 07 10:32:04 [kernel] [88365.922849]   .load_tg                       : 214
Feb 07 10:32:04 [kernel] [88365.922854]   .se->vruntime
  : 3442136.706757
Feb 07 10:32:04 [kernel] [88365.922856]   .se->sum_exec_runtime
  : 358335.275642
Feb 07 10:32:04 [kernel] [88365.922858]   .se->statistics.wait_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922860]   .se->statistics.sleep_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922862]   .se->statistics.block_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922864]   .se->statistics.sleep_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922866]   .se->statistics.block_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922869]   .se->statistics.exec_max
  : 6.979434
Feb 07 10:32:04 [kernel] [88365.922871]   .se->statistics.slice_max
  : 8.909837
Feb 07 10:32:04 [kernel] [88365.922873]   .se->statistics.wait_max
  : 335.735845
Feb 07 10:32:04 [kernel] [88365.922875]   .se->statistics.wait_sum
  : 4606.806398
Feb 07 10:32:04 [kernel] [88365.922877]   .se->statistics.wait_count
  : 318111
Feb 07 10:32:04 [kernel] [88365.922879]   .se->load.weight               : 2
Feb 07 10:32:04 [kernel] [88365.922881]
Feb 07 10:32:04 [kernel] [88365.922882] cfs_rq[1]:/
Feb 07 10:32:04 [kernel] [88365.922883]   .exec_clock
  : 3999784.031074
Feb 07 10:32:04 [kernel] [88365.922886]   .MIN_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922888]   .min_vruntime
  : 3442142.654334
Feb 07 10:32:04 [kernel] [88365.922891]   .max_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922893]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922895]   .spread0
  : 94604.759941
Feb 07 10:32:04 [kernel] [88365.922897]   .nr_spread_over                : 267
Feb 07 10:32:04 [kernel] [88365.922899]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.922901]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.922903]   .load_avg
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922905]   .load_period
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922907]   .load_contrib                  : 0
Feb 07 10:32:04 [kernel] [88365.922909]   .load_tg                       : 0
Feb 07 10:32:04 [kernel] [88365.922911]
Feb 07 10:32:04 [kernel] [88365.922911] rt_rq[1]:
Feb 07 10:32:04 [kernel] [88365.922913]   .rt_nr_running                 : 0
Feb 07 10:32:04 [kernel] [88365.922915]   .rt_throttled                  : 0
Feb 07 10:32:04 [kernel] [88365.922917]   .rt_time
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922919]   .rt_runtime
  : 950.000000
Feb 07 10:32:04 [kernel] [88365.922921]
Feb 07 10:32:04 [kernel] [88365.922922] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.922922]             task   PID
tree-key  switches  prio     exec-runtime         sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.922924]
----------------------------------------------------------------------------------------------------------
Feb 07 10:32:04 [kernel] [88365.922932]
Feb 07 10:32:04 [kernel] [88365.922932] cpu#2, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.922934]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.922936]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.922938]   .nr_switches
  : 46035661
Feb 07 10:32:04 [kernel] [88365.922940]   .nr_load_updates
  : 27665281
Feb 07 10:32:04 [kernel] [88365.922942]   .nr_uninterruptible            : -5740
Feb 07 10:32:04 [kernel] [88365.922944]   .next_balance
  : 4383.033403
Feb 07 10:32:04 [kernel] [88365.922946]   .curr->pid                     : 0
Feb 07 10:32:04 [kernel] [88365.922949]   .clock
  : 88365922.005806
Feb 07 10:32:04 [kernel] [88365.922951]   .cpu_load[0]                   : 0
Feb 07 10:32:04 [kernel] [88365.922953]   .cpu_load[1]                   : 256
Feb 07 10:32:04 [kernel] [88365.922954]   .cpu_load[2]                   : 197
Feb 07 10:32:04 [kernel] [88365.922956]   .cpu_load[3]                   : 134
Feb 07 10:32:04 [kernel] [88365.922958]   .cpu_load[4]                   : 94
Feb 07 10:32:04 [kernel] [88365.922960]   .yld_count                     : 1
Feb 07 10:32:04 [kernel] [88365.922962]   .sched_switch                  : 0
Feb 07 10:32:04 [kernel] [88365.922964]   .sched_count
  : 46171251
Feb 07 10:32:04 [kernel] [88365.922966]   .sched_goidle
  : 22169750
Feb 07 10:32:04 [kernel] [88365.922968]   .avg_idle
  : 674493
Feb 07 10:32:04 [kernel] [88365.922970]   .ttwu_count
  : 22246832
Feb 07 10:32:04 [kernel] [88365.922972]   .ttwu_local
  : 17360455
Feb 07 10:32:04 [kernel] [88365.922975]
Feb 07 10:32:04 [kernel] [88365.922976] cfs_rq[2]:/autogroup-860
Feb 07 10:32:04 [kernel] [88365.922978]   .exec_clock
  : 11.396092
Feb 07 10:32:04 [kernel] [88365.922981]   .MIN_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922983]   .min_vruntime
  : 9.488006
Feb 07 10:32:04 [kernel] [88365.922985]   .max_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.922987]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.922989]   .spread0
  : -3347528.406387
Feb 07 10:32:04 [kernel] [88365.922991]   .nr_spread_over                : 0
Feb 07 10:32:04 [kernel] [88365.922993]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.923002]   .load_contrib                  : 1021
Feb 07 10:32:04 [kernel] [88365.923004]   .load_tg                       : 1021
Feb 07 10:32:04 [kernel] [88365.923004]   .se->exec_start
  : 88365921.858706
Feb 07 10:32:04 [kernel] [88365.923004]   .se->vruntime
  : 2777211.488892
Feb 07 10:32:04 [kernel] [88365.923004]   .se->sum_exec_runtime
  : 11.396092
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.sleep_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.block_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.sleep_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.block_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.exec_max
  : 0.999893
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.slice_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_max
  : 0.051092
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_sum
  : 0.064620
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_count     : 36
Feb 07 10:32:04 [kernel] [88365.923004]   .se->load.weight               : 2
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cfs_rq[2]:/autogroup-776
Feb 07 10:32:04 [kernel] [88365.923004]   .exec_clock
  : 342105.615121
Feb 07 10:32:04 [kernel] [88365.923004]   .MIN_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.923004]   .min_vruntime
  : 289931.479153
Feb 07 10:32:04 [kernel] [88365.923004]   .max_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.923004]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .spread0
  : -3057606.415240
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_spread_over                : 307
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load_avg
  : 525.511404
Feb 07 10:32:04 [kernel] [88365.923004]   .load_period
  : 5.090393
Feb 07 10:32:04 [kernel] [88365.923004]   .load_contrib                  : 103
Feb 07 10:32:04 [kernel] [88365.923004]   .load_tg                       : 214
Feb 07 10:32:04 [kernel] [88365.923004]   .se->exec_start
  : 88365920.891042
Feb 07 10:32:04 [kernel] [88365.923004]   .se->vruntime
  : 2777220.245050
Feb 07 10:32:04 [kernel] [88365.923004]   .se->sum_exec_runtime
  : 342139.770710
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.sleep_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.block_start
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.sleep_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.block_max
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.exec_max
  : 6.074755
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.slice_max
  : 15.625920
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_max
  : 16.153034
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_sum
  : 4251.654869
Feb 07 10:32:04 [kernel] [88365.923004]   .se->statistics.wait_count
  : 308775
Feb 07 10:32:04 [kernel] [88365.923004]   .se->load.weight               : 2
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cfs_rq[2]:/
Feb 07 10:32:04 [kernel] [88365.923004]   .exec_clock
  : 3362268.845416
Feb 07 10:32:04 [kernel] [88365.923004]   .MIN_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.923004]   .min_vruntime
  : 2777220.368423
Feb 07 10:32:04 [kernel] [88365.923004]   .max_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.923004]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .spread0
  : -570317.525970
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_spread_over                : 500
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load_avg
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .load_period
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .load_contrib                  : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load_tg                       : 0
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] rt_rq[2]:
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_nr_running                 : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_throttled                  : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_time
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_runtime
  : 950.000000
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.923004]             task   PID
tree-key  switches  prio     exec-runtime         sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cpu#3, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_switches
  : 45074548
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_load_updates
  : 27257305
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_uninterruptible            : -6280
Feb 07 10:32:04 [kernel] [88365.923004]   .next_balance
  : 4383.033407
Feb 07 10:32:04 [kernel] [88365.923004]   .curr->pid                     : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .clock
  : 88365923.004961
Feb 07 10:32:04 [kernel] [88365.923004]   .cpu_load[0]                   : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .cpu_load[1]                   : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .cpu_load[2]                   : 2
Feb 07 10:32:04 [kernel] [88365.923004]   .cpu_load[3]                   : 31
Feb 07 10:32:04 [kernel] [88365.923004]   .cpu_load[4]                   : 71
Feb 07 10:32:04 [kernel] [88365.923004]   .yld_count                     : 1
Feb 07 10:32:04 [kernel] [88365.923004]   .sched_switch                  : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .sched_count
  : 45213014
Feb 07 10:32:04 [kernel] [88365.923004]   .sched_goidle
  : 21712500
Feb 07 10:32:04 [kernel] [88365.923004]   .avg_idle
  : 1000000
Feb 07 10:32:04 [kernel] [88365.923004]   .ttwu_count
  : 21742662
Feb 07 10:32:04 [kernel] [88365.923004]   .ttwu_local
  : 16876989
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cfs_rq[3]:/
Feb 07 10:32:04 [kernel] [88365.923004]   .exec_clock
  : 3311584.271942
Feb 07 10:32:04 [kernel] [88365.923004]   .MIN_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.923004]   .min_vruntime
  : 2796366.160205
Feb 07 10:32:04 [kernel] [88365.923004]   .max_vruntime
  : 0.000001
Feb 07 10:32:04 [kernel] [88365.923004]   .spread
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .spread0
  : -551171.734188
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_spread_over                : 1027
Feb 07 10:32:04 [kernel] [88365.923004]   .nr_running                    : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load                          : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load_avg
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .load_period
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .load_contrib                  : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .load_tg                       : 0
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] rt_rq[3]:
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_nr_running                 : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_throttled                  : 0
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_time
  : 0.000000
Feb 07 10:32:04 [kernel] [88365.923004]   .rt_runtime
  : 950.000000
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.923004]             task   PID
tree-key  switches  prio     exec-runtime         sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.923004]
----------------------------------------------------------------------------------------------------------
Feb 07 10:32:04 [kernel] [88365.923004]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Soft lockup problem
  2012-02-07 16:35   ` Gerard Saraber
@ 2012-02-07 16:54     ` Jan Kara
  2012-02-27 14:38       ` Gerard Saraber
  0 siblings, 1 reply; 5+ messages in thread
From: Jan Kara @ 2012-02-07 16:54 UTC (permalink / raw)
  To: Gerard Saraber; +Cc: Jan Kara, linux-kernel, xfs

On Tue 07-02-12 10:35:37, Gerard Saraber wrote:
> On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <jack@suse.cz> wrote:
> > On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
> >> Greetings everyone,
> >> I've been having a bit of a problem since upgrading to the linux 3.x
> >> series, I have a machine that we're using as a NAS that runs various
> >> rsync processes (mostly at night), lately after a day or two, I will
> >> come in in the morning to a load average of 49, but the machine not
> >> really doing anything, when trying to run 'dstat' the command just
> >> hung with no output at all. there were no errors in the logs, or even
> >> anything that would vaguely point at anything I could work with.
> >> So needing to get the machine back to work I attempted to reboot it
> >> "shutdown -r now" on console... it gives a nice message saying it's
> >> going to reboot, but nothing ever happens.. the only way to reboot it
> >> is by using ctrl + alt + sysrq + b. after which the machine reboots
> >> and the raid array comes back clean.
> >>
> >> I'm not sure how to troubleshoot this, any pointers would be appreciated.
> >>
> >> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
> >> options in the kernel debugging section:
> >> detect hard/soft lockups and detect hung tasks, maybe it'll give me
> >> something more to go on.
> >>
> >> Some details about the machine:
> >> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
> >> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> >> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
> >> together into a 23TB XFS filesystem.
> >> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
> >  Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
> > Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
> >
> >                                                                Honza
> > --
> > Jan Kara <jack@suse.cz>
> > SUSE Labs, CR
> 
> Thanks for the quick reply,
> the machine is running good at the moment so I'm not sure if the
> output helps, but here it is:
> [I'll also be sure to grab this log the next time it locks]
  Yeah. Sorry, I was not clear but I meant you should grab the traces when
the machine locks up again...
								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Soft lockup problem
  2012-02-07 16:54     ` Jan Kara
@ 2012-02-27 14:38       ` Gerard Saraber
       [not found]         ` <CAOgv-zjF3YSLuGu0p0xkCp5ZfWu2H1Xg8aQ7xcpCQqWOdwh1aA@mail.gmail.com>
  0 siblings, 1 reply; 5+ messages in thread
From: Gerard Saraber @ 2012-02-27 14:38 UTC (permalink / raw)
  To: Jan Kara; +Cc: linux-kernel, xfs

[-- Attachment #1: Type: text/plain, Size: 4129 bytes --]

Hi everyone,
It finally did it again, it took longer then I expected, it also
locked itself up so bad that I couldn't get into it to hit
ctrl+alt+sysrq+w ..
I had turned on the debugging feature that automatically logs the hung
tasks, and I've attached the log below, I hope it's helpful.

I was running 3.2.4 from kernel.org on a 4 core Xeon machine:
model name	: Intel(R) Xeon(R) CPU            5140  @ 2.33GHz

6GB Ram

2x Intel 80003ES2LAN Gigabit Ethernet Controllers bonded together
2 LSI SAS controllers:
08:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
0a:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
2008 [Falcon] (rev 03)

16  drives in a mix of 2 and 3TB, in 3 raid5 arrays and combined
together with LVM
/dev/mapper/pool-main   23T   12T   11T  52%
for a 23TB volume formatted with XFS.

The root partition is ext4 on an older SATA drive, the reason I bring
this up is that when I hit (on a whim) ctrl+sysrq+J that is supposed
to unfreeze frozen filesystems, the console started dumping lots of
messages about attempting to unfreeze /dev/sda3 [my root partition] so
maybe there's a problem with my sda drive.
But I get no i/o or other errors in my logs at all. I monitor all
drives with smartd to head off any drive failures before they happen
and it seems to think sda is fine.

Hopefully my attached log helps.
I appreciate any input, also please call me an idiot if I'm missing
something obvious.

-Gerard Saraber


On Tue, Feb 7, 2012 at 10:54 AM, Jan Kara <jack@suse.cz> wrote:
> On Tue 07-02-12 10:35:37, Gerard Saraber wrote:
>> On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <jack@suse.cz> wrote:
>> > On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
>> >> Greetings everyone,
>> >> I've been having a bit of a problem since upgrading to the linux 3.x
>> >> series, I have a machine that we're using as a NAS that runs various
>> >> rsync processes (mostly at night), lately after a day or two, I will
>> >> come in in the morning to a load average of 49, but the machine not
>> >> really doing anything, when trying to run 'dstat' the command just
>> >> hung with no output at all. there were no errors in the logs, or even
>> >> anything that would vaguely point at anything I could work with.
>> >> So needing to get the machine back to work I attempted to reboot it
>> >> "shutdown -r now" on console... it gives a nice message saying it's
>> >> going to reboot, but nothing ever happens.. the only way to reboot it
>> >> is by using ctrl + alt + sysrq + b. after which the machine reboots
>> >> and the raid array comes back clean.
>> >>
>> >> I'm not sure how to troubleshoot this, any pointers would be appreciated.
>> >>
>> >> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
>> >> options in the kernel debugging section:
>> >> detect hard/soft lockups and detect hung tasks, maybe it'll give me
>> >> something more to go on.
>> >>
>> >> Some details about the machine:
>> >> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
>> >> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
>> >> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
>> >> together into a 23TB XFS filesystem.
>> >> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
>> >  Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
>> > Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
>> >
>> >                                                                Honza
>> > --
>> > Jan Kara <jack@suse.cz>
>> > SUSE Labs, CR
>>
>> Thanks for the quick reply,
>> the machine is running good at the moment so I'm not sure if the
>> output helps, but here it is:
>> [I'll also be sure to grab this log the next time it locks]
>  Yeah. Sorry, I was not clear but I meant you should grab the traces when
> the machine locks up again...
>                                                                Honza
>
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR

[-- Attachment #2: hlog.txt --]
[-- Type: text/plain, Size: 17943 bytes --]

Feb 26 09:41:19 [kernel] [1726920.709038] INFO: task kswapd0:590 blocked for more than 120 seconds.
Feb 26 09:41:19 [kernel] [1726920.709042] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 26 09:41:19 [kernel] [1726920.709045] kswapd0         D 0000000000000000  1800   590      2 0x00000000
Feb 26 09:41:19 [kernel] [1726920.709052]  ffff8801b626b9b0 0000000000000046 00000000001d3300 0000000000000000
Feb 26 09:41:19 [kernel] [1726920.709058]  ffff8801b626b940 00000000001d2880 ffff8801b6262680 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709064]  ffff8801b626bfd8 ffff8801b626a000 00000000001d2880 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709069] Call Trace:
Feb 26 09:41:19 [kernel] [1726920.709079]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
Feb 26 09:41:19 [kernel] [1726920.709083]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 26 09:41:19 [kernel] [1726920.709087]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
Feb 26 09:41:19 [kernel] [1726920.709092]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709096]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709100]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709106]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
Feb 26 09:41:19 [kernel] [1726920.709111]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
Feb 26 09:41:19 [kernel] [1726920.709115]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
Feb 26 09:41:19 [kernel] [1726920.709119]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
Feb 26 09:41:19 [kernel] [1726920.709122]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
Feb 26 09:41:19 [kernel] [1726920.709127]  [<ffffffff8139be30>] ? xfs_ail_push_all+0x80/0x90
Feb 26 09:41:19 [kernel] [1726920.709132]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
Feb 26 09:41:19 [kernel] [1726920.709136]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
Feb 26 09:41:19 [kernel] [1726920.709139]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
Feb 26 09:41:19 [kernel] [1726920.709144]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
Feb 26 09:41:19 [kernel] [1726920.709149]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709153]  [<ffffffff81141fda>] kswapd+0x70a/0xa60
Feb 26 09:41:19 [kernel] [1726920.709158]  [<ffffffff810b4e30>] ? wake_up_bit+0x40/0x40
Feb 26 09:41:19 [kernel] [1726920.709161]  [<ffffffff811418d0>] ? try_to_free_pages+0x110/0x110
Feb 26 09:41:19 [kernel] [1726920.709165]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
Feb 26 09:41:19 [kernel] [1726920.709173]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
Feb 26 09:41:19 [kernel] [1726920.709177]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
Feb 26 09:41:19 [kernel] [1726920.709180]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
Feb 26 09:41:19 [kernel] [1726920.709183] 3 locks held by kswapd0/590:
Feb 26 09:41:19 [kernel] [1726920.709185]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff8113ea14>] shrink_slab+0x34/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709192]  #1:  (&type->s_umount_key#31){.+.+.+}, at: [<ffffffff8118807f>] grab_super_passive+0x4f/0xc0
Feb 26 09:41:19 [kernel] [1726920.709199]  #2:  (&pag->pag_ici_reclaim_lock){+.+.-.}, at: [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709211] INFO: task gkrellmd:2341 blocked for more than 120 seconds.
Feb 26 09:41:19 [kernel] [1726920.709214] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 26 09:41:19 [kernel] [1726920.709216] gkrellmd        D 0000000000000000  1552  2341      1 0x00000000
Feb 26 09:41:19 [kernel] [1726920.709221]  ffff8801b4b5d6f8 0000000000000046 ffffffff8189813a 0000000000000000
Feb 26 09:41:19 [kernel] [1726920.709226]  ffff880100000000 00000000001d2880 ffff8801b5ac9340 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709232]  ffff8801b4b5dfd8 ffff8801b4b5c000 00000000001d2880 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709237] Call Trace:
Feb 26 09:41:19 [kernel] [1726920.709240]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
Feb 26 09:41:19 [kernel] [1726920.709244]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
Feb 26 09:41:19 [kernel] [1726920.709247]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 26 09:41:19 [kernel] [1726920.709251]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
Feb 26 09:41:19 [kernel] [1726920.709254]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709258]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709262]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709265]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
Feb 26 09:41:19 [kernel] [1726920.709269]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
Feb 26 09:41:19 [kernel] [1726920.709273]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
Feb 26 09:41:19 [kernel] [1726920.709304]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
Feb 26 09:41:19 [kernel] [1726920.709312]  [<ffffffff8139be30>] ? xfs_ail_push_all+0x80/0x90
Feb 26 09:41:19 [kernel] [1726920.709316]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
Feb 26 09:41:19 [kernel] [1726920.709319]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
Feb 26 09:41:19 [kernel] [1726920.709323]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
Feb 26 09:41:19 [kernel] [1726920.709327]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
Feb 26 09:41:19 [kernel] [1726920.709330]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709334]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
Feb 26 09:41:19 [kernel] [1726920.709337]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
Feb 26 09:41:19 [kernel] [1726920.709342]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
Feb 26 09:41:19 [kernel] [1726920.709347]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
Feb 26 09:41:19 [kernel] [1726920.709352]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
Feb 26 09:41:19 [kernel] [1726920.709356]  [<ffffffff81174335>] new_slab+0x265/0x300
Feb 26 09:41:19 [kernel] [1726920.709359]  [<ffffffff8189911c>] ? mutex_lock_nested+0x28c/0x330
Feb 26 09:41:19 [kernel] [1726920.709363]  [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
Feb 26 09:41:19 [kernel] [1726920.709367]  [<ffffffff8118fb46>] ? getname_flags+0x36/0x270
Feb 26 09:41:19 [kernel] [1726920.709371]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
Feb 26 09:41:19 [kernel] [1726920.709374]  [<ffffffff8118fb46>] ? getname_flags+0x36/0x270
Feb 26 09:41:19 [kernel] [1726920.709378]  [<ffffffff8117891b>] kmem_cache_alloc+0xdb/0x120
Feb 26 09:41:19 [kernel] [1726920.709381]  [<ffffffff8118fb46>] getname_flags+0x36/0x270
Feb 26 09:41:19 [kernel] [1726920.709384]  [<ffffffff8118fd8d>] getname+0xd/0x10
Feb 26 09:41:19 [kernel] [1726920.709388]  [<ffffffff81183fa8>] do_sys_open+0xc8/0x1d0
Feb 26 09:41:19 [kernel] [1726920.709391]  [<ffffffff811840cb>] sys_open+0x1b/0x20
Feb 26 09:41:19 [kernel] [1726920.709395]  [<ffffffff8189bffb>] system_call_fastpath+0x16/0x1b
Feb 26 09:41:19 [kernel] [1726920.709397] 3 locks held by gkrellmd/2341:
Feb 26 09:41:19 [kernel] [1726920.709399]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff8113ea14>] shrink_slab+0x34/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709405]  #1:  (&type->s_umount_key#31){.+.+.+}, at: [<ffffffff8118807f>] grab_super_passive+0x4f/0xc0
Feb 26 09:41:19 [kernel] [1726920.709412]  #2:  (&pag->pag_ici_reclaim_lock){+.+.-.}, at: [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709423] INFO: task nfsd:2821 blocked for more than 120 seconds.
Feb 26 09:41:19 [kernel] [1726920.709425] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 26 09:41:19 [kernel] [1726920.709428] nfsd            D 0000000000000000  1160  2821      2 0x00000000
Feb 26 09:41:19 [kernel] [1726920.709432]  ffff8801ae9577a0 0000000000000046 ffffffff8189813a 0000000000000000
Feb 26 09:41:19 [kernel] [1726920.709438]  ffff880100000000 00000000001d2880 ffff8801b7ae4d00 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709443]  ffff8801ae957fd8 ffff8801ae956000 00000000001d2880 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709448] Call Trace:
Feb 26 09:41:19 [kernel] [1726920.709452]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
Feb 26 09:41:19 [kernel] [1726920.709455]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
Feb 26 09:41:19 [kernel] [1726920.709459]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 26 09:41:19 [kernel] [1726920.709462]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
Feb 26 09:41:19 [kernel] [1726920.709466]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709469]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709473]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709477]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
Feb 26 09:41:19 [kernel] [1726920.709481]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
Feb 26 09:41:19 [kernel] [1726920.709484]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
Feb 26 09:41:19 [kernel] [1726920.709488]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
Feb 26 09:41:19 [kernel] [1726920.709491]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
Feb 26 09:41:19 [kernel] [1726920.709495]  [<ffffffff8139be30>] ? xfs_ail_push_all+0x80/0x90
Feb 26 09:41:19 [kernel] [1726920.709499]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
Feb 26 09:41:19 [kernel] [1726920.709502]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
Feb 26 09:41:19 [kernel] [1726920.709506]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
Feb 26 09:41:19 [kernel] [1726920.709509]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
Feb 26 09:41:19 [kernel] [1726920.709513]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709517]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
Feb 26 09:41:19 [kernel] [1726920.709520]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
Feb 26 09:41:19 [kernel] [1726920.709524]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
Feb 26 09:41:19 [kernel] [1726920.709534]  [<ffffffff8184b07c>] svc_recv+0xec/0x900
Feb 26 09:41:19 [kernel] [1726920.709539]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709543]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
Feb 26 09:41:19 [kernel] [1726920.709547]  [<ffffffff812d8d6d>] nfsd+0x9d/0x150
Feb 26 09:41:19 [kernel] [1726920.709550]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
Feb 26 09:41:19 [kernel] [1726920.709553]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
Feb 26 09:41:19 [kernel] [1726920.709557]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
Feb 26 09:41:19 [kernel] [1726920.709561]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
Feb 26 09:41:19 [kernel] [1726920.709565]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
Feb 26 09:41:19 [kernel] [1726920.709568]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
Feb 26 09:41:19 [kernel] [1726920.709570] 3 locks held by nfsd/2821:
Feb 26 09:41:19 [kernel] [1726920.709572]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff8113ea14>] shrink_slab+0x34/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709578]  #1:  (&type->s_umount_key#31){.+.+.+}, at: [<ffffffff8118807f>] grab_super_passive+0x4f/0xc0
Feb 26 09:41:19 [kernel] [1726920.709585]  #2:  (&pag->pag_ici_reclaim_lock){+.+.-.}, at: [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709592] INFO: task nfsd:2822 blocked for more than 120 seconds.
Feb 26 09:41:19 [kernel] [1726920.709594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 26 09:41:19 [kernel] [1726920.709596] nfsd            D 0000000000000000  1224  2822      2 0x00000000
Feb 26 09:41:19 [kernel] [1726920.709601]  ffff8801aea517a0 0000000000000046 00000000001d3300 0000000000000002
Feb 26 09:41:19 [kernel] [1726920.709607]  ffff8801aea51730 00000000001d2880 ffff8801b5acb9c0 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709612]  ffff8801aea51fd8 ffff8801aea50000 00000000001d2880 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709617] Call Trace:
Feb 26 09:41:19 [kernel] [1726920.709621]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
Feb 26 09:41:19 [kernel] [1726920.709624]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 26 09:41:19 [kernel] [1726920.709627]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
Feb 26 09:41:19 [kernel] [1726920.709631]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709634]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709638]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
Feb 26 09:41:19 [kernel] [1726920.709642]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
Feb 26 09:41:19 [kernel] [1726920.709646]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
Feb 26 09:41:19 [kernel] [1726920.709649]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
Feb 26 09:41:19 [kernel] [1726920.709653]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
Feb 26 09:41:19 [kernel] [1726920.709656]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
Feb 26 09:41:19 [kernel] [1726920.709660]  [<ffffffff8139be30>] ? xfs_ail_push_all+0x80/0x90
Feb 26 09:41:19 [kernel] [1726920.709664]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
Feb 26 09:41:19 [kernel] [1726920.709667]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
Feb 26 09:41:19 [kernel] [1726920.709671]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
Feb 26 09:41:19 [kernel] [1726920.709674]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
Feb 26 09:41:19 [kernel] [1726920.709677]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709681]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
Feb 26 09:41:19 [kernel] [1726920.709685]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
Feb 26 09:41:19 [kernel] [1726920.709689]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
Feb 26 09:41:19 [kernel] [1726920.709694]  [<ffffffff8183e36d>] ? svc_tcp_recvfrom+0x50d/0x760
Feb 26 09:41:19 [kernel] [1726920.709698]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
Feb 26 09:41:19 [kernel] [1726920.709702]  [<ffffffff8184b07c>] svc_recv+0xec/0x900
Feb 26 09:41:19 [kernel] [1726920.709705]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709708]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
Feb 26 09:41:19 [kernel] [1726920.709712]  [<ffffffff812d8d6d>] nfsd+0x9d/0x150
Feb 26 09:41:19 [kernel] [1726920.709715]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
Feb 26 09:41:19 [kernel] [1726920.709718]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
Feb 26 09:41:19 [kernel] [1726920.709722]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
Feb 26 09:41:19 [kernel] [1726920.709726]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
Feb 26 09:41:19 [kernel] [1726920.709729]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
Feb 26 09:41:19 [kernel] [1726920.709732]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
Feb 26 09:41:19 [kernel] [1726920.709735] 3 locks held by nfsd/2822:
Feb 26 09:41:19 [kernel] [1726920.709737]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff8113ea14>] shrink_slab+0x34/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709757] INFO: task nfsd:2824 blocked for more than 120 seconds.
Feb 26 09:41:19 [kernel] [1726920.709759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 26 09:41:19 [kernel] [1726920.709761] nfsd            D 0000000000000000  1208  2824      2 0x00000000
Feb 26 09:41:19 [kernel] [1726920.709766]  ffff8801ae065d60 0000000000000046 00000000001d3300 0000000000000000
Feb 26 09:41:19 [kernel] [1726920.709771]  ffff8801ae065cf0 00000000001d2880 ffff8801b54aa680 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709777]  ffff8801ae065fd8 ffff8801ae064000 00000000001d2880 00000000001d2880
Feb 26 09:41:19 [kernel] [1726920.709782] Call Trace:
Feb 26 09:41:19 [kernel] [1726920.709785]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
Feb 26 09:41:19 [kernel] [1726920.709789]  [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 26 09:41:19 [kernel] [1726920.709792]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
Feb 26 09:41:19 [kernel] [1726920.709796]  [<ffffffff8184ad79>] ? svc_send+0x59/0xf0
Feb 26 09:41:19 [kernel] [1726920.709799]  [<ffffffff8184ad79>] ? svc_send+0x59/0xf0
Feb 26 09:41:19 [kernel] [1726920.709803]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
Feb 26 09:41:19 [kernel] [1726920.709806]  [<ffffffff8184ad79>] svc_send+0x59/0xf0
Feb 26 09:41:19 [kernel] [1726920.709809]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
Feb 26 09:41:19 [kernel] [1726920.709813]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
Feb 26 09:41:19 [kernel] [1726920.709816]  [<ffffffff8183cce0>] svc_process+0x120/0x150
Feb 26 09:41:19 [kernel] [1726920.709819]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
Feb 26 09:41:19 [kernel] [1726920.709823]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
Feb 26 09:41:19 [kernel] [1726920.709826]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
Feb 26 09:41:19 [kernel] [1726920.709830]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
Feb 26 09:41:19 [kernel] [1726920.709833]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
Feb 26 09:41:19 [kernel] [1726920.709837]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
Feb 26 09:41:19 [kernel] [1726920.709841]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
Feb 26 09:41:19 [kernel] [1726920.709843] 1 lock held by nfsd/2824:


[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 5+ messages in thread

[parent not found: <CAOgv-zjF3YSLuGu0p0xkCp5ZfWu2H1Xg8aQ7xcpCQqWOdwh1aA@mail.gmail.com>]

* Re: Soft lockup problem
       [not found]         ` <CAOgv-zjF3YSLuGu0p0xkCp5ZfWu2H1Xg8aQ7xcpCQqWOdwh1aA@mail.gmail.com>
@ 2012-02-29 19:14           ` Jan Kara
  0 siblings, 0 replies; 5+ messages in thread
From: Jan Kara @ 2012-02-29 19:14 UTC (permalink / raw)
  To: Gerard Saraber; +Cc: linux-kernel, xfs

  Hi,

On Wed 29-02-12 12:20:10, Gerard Saraber wrote:
> I have another update, after rebooting two days ago,
> it's now running :
> Linux xenbox 3.2.5 #1 SMP Tue Feb 7 13:51:06 CST 2012 x86_64 Intel(R)
> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> 
> I noticed high load on the machine, and another dump in the logs about
> stuck processes, so I ran ctrl+alt+sysrq + w and got this log
> [attached below]
> 
> I hope finally getting the right log will be helpful!
  Thanks for the traces. I was looking at them and I don't see the
immediate problem. There is a task (kswapd0) reclaiming XFS inodes and
waiting for IO to complete. Quite some other tasks are waiting in
xfs_reclaim_inodes_ag() for reclaim mutex. Since some of them are called
from NFS code (e.g. nfs4_process_open1()), that seems to be a reason why
other tasks are blocked in nfs4_lock_state().

So all in all it seems like some problem in how mm does reclaim or we are
inefficient doing inode reclaim in XFS or for some reason the IO is too
slow. As a result we are low on memory and things go bad.  Do you know with
which kernel version the problem started to appear?

								Honza


> On Mon, Feb 27, 2012 at 8:38 AM, Gerard Saraber <gsaraber@gmail.com> wrote:
> > Hi everyone,
> > It finally did it again, it took longer then I expected, it also
> > locked itself up so bad that I couldn't get into it to hit
> > ctrl+alt+sysrq+w ..
> > I had turned on the debugging feature that automatically logs the hung
> > tasks, and I've attached the log below, I hope it's helpful.
> >
> > I was running 3.2.4 from kernel.org on a 4 core Xeon machine:
> > model name      : Intel(R) Xeon(R) CPU            5140  @ 2.33GHz
> >
> > 6GB Ram
> >
> > 2x Intel 80003ES2LAN Gigabit Ethernet Controllers bonded together
> > 2 LSI SAS controllers:
> > 08:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
> > PCI-Express Fusion-MPT SAS (rev 08)
> > 0a:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> > 2008 [Falcon] (rev 03)
> >
> > 16  drives in a mix of 2 and 3TB, in 3 raid5 arrays and combined
> > together with LVM
> > /dev/mapper/pool-main   23T   12T   11T  52%
> > for a 23TB volume formatted with XFS.
> >
> > The root partition is ext4 on an older SATA drive, the reason I bring
> > this up is that when I hit (on a whim) ctrl+sysrq+J that is supposed
> > to unfreeze frozen filesystems, the console started dumping lots of
> > messages about attempting to unfreeze /dev/sda3 [my root partition] so
> > maybe there's a problem with my sda drive.
> > But I get no i/o or other errors in my logs at all. I monitor all
> > drives with smartd to head off any drive failures before they happen
> > and it seems to think sda is fine.
> >
> > Hopefully my attached log helps.
> > I appreciate any input, also please call me an idiot if I'm missing
> > something obvious.
> >
> > -Gerard Saraber
> >
> >
> > On Tue, Feb 7, 2012 at 10:54 AM, Jan Kara <jack@suse.cz> wrote:
> >> On Tue 07-02-12 10:35:37, Gerard Saraber wrote:
> >>> On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <jack@suse.cz> wrote:
> >>> > On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
> >>> >> Greetings everyone,
> >>> >> I've been having a bit of a problem since upgrading to the linux 3.x
> >>> >> series, I have a machine that we're using as a NAS that runs various
> >>> >> rsync processes (mostly at night), lately after a day or two, I will
> >>> >> come in in the morning to a load average of 49, but the machine not
> >>> >> really doing anything, when trying to run 'dstat' the command just
> >>> >> hung with no output at all. there were no errors in the logs, or even
> >>> >> anything that would vaguely point at anything I could work with.
> >>> >> So needing to get the machine back to work I attempted to reboot it
> >>> >> "shutdown -r now" on console... it gives a nice message saying it's
> >>> >> going to reboot, but nothing ever happens.. the only way to reboot it
> >>> >> is by using ctrl + alt + sysrq + b. after which the machine reboots
> >>> >> and the raid array comes back clean.
> >>> >>
> >>> >> I'm not sure how to troubleshoot this, any pointers would be appreciated.
> >>> >>
> >>> >> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
> >>> >> options in the kernel debugging section:
> >>> >> detect hard/soft lockups and detect hung tasks, maybe it'll give me
> >>> >> something more to go on.
> >>> >>
> >>> >> Some details about the machine:
> >>> >> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
> >>> >> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> >>> >> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
> >>> >> together into a 23TB XFS filesystem.
> >>> >> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
> >>> >  Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
> >>> > Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
> >>> >
> >>> >                                                                Honza
> >>> > --
> >>> > Jan Kara <jack@suse.cz>
> >>> > SUSE Labs, CR
> >>>
> >>> Thanks for the quick reply,
> >>> the machine is running good at the moment so I'm not sure if the
> >>> output helps, but here it is:
> >>> [I'll also be sure to grab this log the next time it locks]
> >>  Yeah. Sorry, I was not clear but I meant you should grab the traces when
> >> the machine locks up again...
> >>                                                                Honza
> >>
> >> --
> >> Jan Kara <jack@suse.cz>
> >> SUSE Labs, CR

> Feb 29 12:05:38 [kernel] [186732.828746] SysRq : Show Blocked State
> Feb 29 12:05:38 [kernel] [186732.828829]   task                        PC stack   pid father
> Feb 29 12:05:38 [kernel] [186732.828833] kthreadd        D 0000000000000000  2560     2      0 0x00000000
> Feb 29 12:05:38 [kernel] [186732.828839]  ffff8801b78836c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.828845]  ffff880100000000 00000000001d2880 ffff8801b7879340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.828850]  ffff8801b7883fd8 ffff8801b7882000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.828856] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.828865]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.828869]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.828873]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.828876]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.828881]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.828885]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.828888]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.828894]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.828900]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.828904]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.828907]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.828911]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.828916]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.828920]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.828924]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.828928]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.828933]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.828937]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.828941]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.828945]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.828949]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.828959]  [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.828963]  [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.828967]  [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.828971]  [<ffffffff8107d350>] ? finish_task_switch+0x80/0x110
> Feb 29 12:05:38 [kernel] [186732.828974]  [<ffffffff810bb325>] ? sched_clock_local+0x25/0x90
> Feb 29 12:05:38 [kernel] [186732.828979]  [<ffffffff8104a54c>] kernel_thread+0x6c/0x70
> Feb 29 12:05:38 [kernel] [186732.828984]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.828988]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.828991]  [<ffffffff810b4cc2>] kthreadd+0x122/0x160
> Feb 29 12:05:38 [kernel] [186732.828995]  [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.828998]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829002]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829006]  [<ffffffff810b4ba0>] ? tsk_fork_get_node+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829009]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829015] kswapd0         D 0000000000000000  2248   590      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829020]  ffff8801b70872c0 0000000000000046 ffffffff8189813a 0000000000000006
> Feb 29 12:05:38 [kernel] [186732.829025]  0000000000000000 00000000001d2880 ffff8801b7a1b9c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829030]  ffff8801b7087fd8 ffff8801b7086000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829035] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829039]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829042]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829048]  [<ffffffff816cb13d>] md_flush_request+0x7d/0x140
> Feb 29 12:05:38 [kernel] [186732.829052]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829056]  [<ffffffff816c3c28>] make_request+0x428/0x430
> Feb 29 12:05:38 [kernel] [186732.829059]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829063]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829066]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829070]  [<ffffffff816cf14f>] ? md_make_request+0x13f/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829074]  [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829080]  [<ffffffff816cf078>] ? md_make_request+0x68/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829084]  [<ffffffff810b9d5e>] ? up_read+0x1e/0x40
> Feb 29 12:05:38 [kernel] [186732.829087]  [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829091]  [<ffffffff816da13e>] ? dm_request+0x2e/0x230
> Feb 29 12:05:38 [kernel] [186732.829095]  [<ffffffff8145c722>] generic_make_request+0xc2/0x100
> Feb 29 12:05:38 [kernel] [186732.829099]  [<ffffffff8145d4d5>] submit_bio+0x75/0xf0
> Feb 29 12:05:38 [kernel] [186732.829104]  [<ffffffff81341a82>] _xfs_buf_ioapply+0x162/0x1d0
> Feb 29 12:05:38 [kernel] [186732.829108]  [<ffffffff81394111>] ? xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829111]  [<ffffffff81341bca>] xfs_buf_iorequest+0x4a/0xa0
> Feb 29 12:05:38 [kernel] [186732.829115]  [<ffffffff81394111>] xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829118]  [<ffffffff81395d9e>] xlog_sync+0x23e/0x3d0
> Feb 29 12:05:38 [kernel] [186732.829122]  [<ffffffff81396793>] xlog_state_release_iclog+0xe3/0xf0
> Feb 29 12:05:38 [kernel] [186732.829125]  [<ffffffff81397258>] xlog_write+0x618/0x780
> Feb 29 12:05:38 [kernel] [186732.829129]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829132]  [<ffffffff81398830>] ? xlog_cil_push+0x1c0/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829136]  [<ffffffff813988a9>] xlog_cil_push+0x239/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829139]  [<ffffffff81399184>] xlog_cil_force_lsn+0x114/0x120
> Feb 29 12:05:38 [kernel] [186732.829143]  [<ffffffff81397871>] _xfs_log_force_lsn+0x51/0x2f0
> Feb 29 12:05:38 [kernel] [186732.829147]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829151]  [<ffffffff81386675>] ? xfs_iunpin_wait+0x45/0xd0
> Feb 29 12:05:38 [kernel] [186732.829155]  [<ffffffff81397b23>] xfs_log_force_lsn+0x13/0x40
> Feb 29 12:05:38 [kernel] [186732.829158]  [<ffffffff8189b68b>] ? _raw_spin_unlock_irqrestore+0x3b/0x70
> Feb 29 12:05:38 [kernel] [186732.829162]  [<ffffffff813846f5>] xfs_iunpin_nowait+0x35/0x70
> Feb 29 12:05:38 [kernel] [186732.829165]  [<ffffffff81386675>] xfs_iunpin_wait+0x45/0xd0
> Feb 29 12:05:38 [kernel] [186732.829170]  [<ffffffff8107cab6>] ? try_wait_for_completion+0x46/0x60
> Feb 29 12:05:38 [kernel] [186732.829173]  [<ffffffff81352274>] xfs_reclaim_inode+0xb4/0x2c0
> Feb 29 12:05:38 [kernel] [186732.829177]  [<ffffffff81352fdf>] xfs_reclaim_inodes_ag+0x23f/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829180]  [<ffffffff81352e78>] ? xfs_reclaim_inodes_ag+0xd8/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829188]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829192]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829195]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829199]  [<ffffffff81141fda>] kswapd+0x70a/0xa60
> Feb 29 12:05:38 [kernel] [186732.829203]  [<ffffffff810b4e30>] ? wake_up_bit+0x40/0x40
> Feb 29 12:05:38 [kernel] [186732.829206]  [<ffffffff811418d0>] ? try_to_free_pages+0x110/0x110
> Feb 29 12:05:38 [kernel] [186732.829210]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829214]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829217]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829221]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829224]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829232] exim            D 0000000000000000  2816  2718      1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829237]  ffff8801afb837c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829242]  ffff880100000000 00000000001d2880 ffff8801af161340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829248]  ffff8801afb83fd8 ffff8801afb82000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829253] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829256]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829259]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829263]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829266]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829270]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829273]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829277]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829280]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829284]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829288]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829291]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829295]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829302]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829313]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829316]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829320]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829324]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829328]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829332]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829336]  [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829339]  [<ffffffff8189b68b>] ? _raw_spin_unlock_irqrestore+0x3b/0x70
> Feb 29 12:05:38 [kernel] [186732.829343]  [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829347]  [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829350]  [<ffffffff81095b9c>] ? do_wait+0x12c/0x250
> Feb 29 12:05:38 [kernel] [186732.829354]  [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829358]  [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829361]  [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829365]  [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829368] gkrellmd        D 0000000000000000  1872  2732      1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829373]  ffff8801afafd7c0 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829378]  ffff880100000000 00000000001d2880 ffff8801b1996040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829383]  ffff8801afafdfd8 ffff8801afafc000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829388] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829391]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829395]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829398]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829402]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829405]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829412]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829416]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829420]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829423]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829427]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829430]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829434]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829438]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829441]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829445]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829448]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829452]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829455]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829459]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829463]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829467]  [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829471]  [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829476]  [<ffffffff8119c185>] ? dput+0x145/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829479]  [<ffffffff8114e7f4>] ? might_fault+0x84/0x90
> Feb 29 12:05:38 [kernel] [186732.829482]  [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829486]  [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829490]  [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829493]  [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829497]  [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829500]  [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829503] smbd            D 0000000000000000  2792  2878      1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829508]  ffff8801b57037c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829513]  ffff880100000000 00000000001d2880 ffff8801b7a6cd00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829524] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829527]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829531]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829534]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829548]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829552]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829555]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829559]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829563]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b9d5e>] ? up_read+0x1e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8ca>] ? retint_swapgs+0xe/0x13
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] sshd            D 0000000000000000  2920  2919      1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801afbb57c0 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801b7946040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801afbb5fd8 ffff8801afbb4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] cron            D 0000000000000000  2504  2952      1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801afaf77c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801b7ab8000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801afaf7fd8 ffff8801afaf6000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8119c185>] ? dput+0x145/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8114e7f4>] ? might_fault+0x84/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] xfsaild/dm-0    D 0000000000000000  3800  3130      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880185593b40 0000000000000046 ffffffff8189813a ffffffff810c7b3d
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801b1b34d00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880185593fd8 ffff880185592000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139864c>] xlog_wait+0x6c/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189acdd>] ? _raw_spin_lock+0x5d/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81399184>] xlog_cil_force_lsn+0x114/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb325>] ? sched_clock_local+0x25/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810a1900>] ? try_to_del_timer_sync+0xe0/0xe0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813975a6>] _xfs_log_force+0x66/0x2a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139b710>] ? xfsaild+0x110/0x700
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813977f3>] xfs_log_force+0x13/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139b73d>] xfsaild+0x13d/0x700
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1656  3221      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801b0b81c40 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801b7a6b9c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801b0b81fd8 ffff8801b0b80000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1480  3222      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801b0afbc40 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164900000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801b0afbfd8 ffff8801b0afa000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801648a5c40 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164901340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801648a5fd8 ffff8801648a4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1544  3224      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88018579f4a0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164902680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88018579ffd8 ffff88018579e000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff814773c8>] ? idr_pre_get+0x68/0x90
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8117891b>] kmem_cache_alloc+0xdb/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff814773c8>] idr_pre_get+0x68/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f506a>] nfs4_alloc_stid+0x1a/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f8e26>] nfsd4_process_open1+0x176/0x3d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2fa>] nfsd4_open+0xca/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1400  3225      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164699c40 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801649039c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  2120  3226      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801647b17a0 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164904d00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801647b1fd8 ffff8801647b0000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8184b05c>] svc_recv+0xec/0x900
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d6d>] nfsd+0x9d/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1880  3227      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164cb37a0 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164906040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164cb3fd8 ffff880164cb2000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1768  3228      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164dadc40 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164db0000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164dadfd8 ffff880164dac000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1928  3229      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164ecf940 0000000000000046 ffffffff8189813a ffffffff810c7b3d
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164db1340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164ecffd8 ffff880164ece000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8172154d>] __lock_sock+0x6d/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4e30>] ? wake_up_bit+0x40/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189aff2>] ? _raw_spin_lock_bh+0x62/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81721639>] lock_sock_nested+0x39/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178b640>] tcp_recvmsg+0x30/0xb00
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff817230cf>] ? sock_update_classid+0x5f/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff817b0208>] inet_recvmsg+0xc8/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81723110>] ? sock_update_classid+0xa0/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8171d518>] sock_recvmsg+0xf8/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810a18a5>] ? try_to_del_timer_sync+0x85/0xe0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8171d58f>] kernel_recvmsg+0x3f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183e1ae>] svc_tcp_recvfrom+0x36e/0x760
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8184b256>] ? svc_recv+0x2e6/0x900
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d6d>] nfsd+0x9d/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1480  3230      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164fc9c40 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164db2680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880164fc9fd8 ffff880164fc8000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1704  3231      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801650cb2d0 0000000000000046 00000000001d3300 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801650cb260 00000000001d2880 ffff880164db39c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801650cbfd8 ffff8801650ca000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81179c33>] __kmalloc_node_track_caller+0x83/0x190
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811770bf>] ? kmem_cache_alloc_node+0x2f/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81726965>] __alloc_skb+0x75/0x170
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178976c>] sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178a9f2>] tcp_sendpage+0x3f2/0x740
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff817230cf>] ? sock_update_classid+0x5f/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff817b03ee>] inet_sendpage+0xce/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff817b0320>] ? inet_sendmsg+0xf0/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81723110>] ? sock_update_classid+0xa0/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8171bee3>] kernel_sendpage+0x53/0x80
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183f2e2>] svc_send_common+0x62/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183f42e>] svc_sendto+0x5e/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899108>] ? mutex_lock_nested+0x278/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8184ad59>] ? svc_send+0x59/0xf0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183f604>] svc_tcp_sendto+0x34/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8184adab>] svc_send+0xab/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183ccc0>] svc_process+0x120/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1848  3232      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801651c57a0 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff880164db4d00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801651c5fd8 ffff8801651c4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183e34d>] ? svc_tcp_recvfrom+0x50d/0x760
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8184b05c>] svc_recv+0xec/0x900
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d6d>] nfsd+0x9d/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1304  3233      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801652e5c40 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801652e5fd8 ffff8801652e4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1848  3234      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801653dfc40 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801653e0000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801653dffd8 ffff8801653de000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  2024  3235      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801654e9c40 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801653e1340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801654e9fd8 ffff8801654e8000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd            D 0000000000000000  1304  3236      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8801656032d0 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880100000000 00000000001d2880 ffff8801653e2680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880165603fd8 ffff880165602000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81179c33>] __kmalloc_node_track_caller+0x83/0x190
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811770bf>] ? kmem_cache_alloc_node+0x2f/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81726965>] __alloc_skb+0x75/0x170
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178976c>] sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8178a9f2>] tcp_sendpage+0x3f2/0x740
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff817b03ee>] inet_sendpage+0xce/0x130
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff817b0320>] ? inet_sendmsg+0xf0/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81723110>] ? sock_update_classid+0xa0/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8171bee3>] kernel_sendpage+0x53/0x80
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183f2e2>] svc_send_common+0x62/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183f42e>] svc_sendto+0x5e/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189911c>] ? mutex_lock_nested+0x28c/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899108>] ? mutex_lock_nested+0x278/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8184ad59>] ? svc_send+0x59/0xf0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183f604>] svc_tcp_sendto+0x34/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8184adab>] svc_send+0xab/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8183ccc0>] svc_process+0x120/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/u:4     D 0000000000000000  4520 23447      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8800471c3af0 0000000000000046 ffff8800471c3a70 ffffffff810bb325
> Feb 29 12:05:38 [kernel] [186732.829564]  00000000001d3300 00000000001d2880 ffff8801b5af2680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb325>] ? sched_clock_local+0x25/0x90
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898985>] schedule_timeout+0x1f5/0x2e0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b63b>] ? _raw_spin_unlock_irq+0x2b/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff818978e0>] wait_for_common+0xc0/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81897a28>] wait_for_completion+0x18/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81897846>] ? wait_for_common+0x26/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af11b>] create_worker+0xbb/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af31d>] manage_workers.clone.20+0x11d/0x240
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af6c8>] worker_thread+0x288/0x340
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af440>] ? manage_workers.clone.20+0x240/0x240
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/u:2     D 0000000000000000  4520 25347      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8800a9a05c50 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff8800a9a05fd8 ffff8800a9a04000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f88c5>] laundromat_main+0x25/0x410
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ada09>] process_one_work+0x1a9/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ad99a>] ? process_one_work+0x13a/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff812f88a0>] ? nfsd4_exchange_id+0x440/0x440
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af5a2>] worker_thread+0x162/0x340
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af440>] ? manage_workers.clone.20+0x240/0x240
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/3:3     D 0000000000000000  4264 27581      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88000ec6d640 0000000000000046 ffffffff8189813a 0000000000000006
> Feb 29 12:05:38 [kernel] [186732.829564]  0000000000000001 00000000001d2880 ffff8801b26ab9c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88000ec6dfd8 ffff88000ec6c000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816cb13d>] md_flush_request+0x7d/0x140
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816cf14f>] ? md_make_request+0x13f/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816cf184>] md_make_request+0x174/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816cf078>] ? md_make_request+0x68/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b9d5e>] ? up_read+0x1e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816da13e>] ? dm_request+0x2e/0x230
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8145c722>] generic_make_request+0xc2/0x100
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8145d4d5>] submit_bio+0x75/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81341a82>] _xfs_buf_ioapply+0x162/0x1d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81394111>] ? xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81341bca>] xfs_buf_iorequest+0x4a/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81394111>] xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81395d9e>] xlog_sync+0x23e/0x3d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81397258>] xlog_write+0x618/0x780
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81398830>] ? xlog_cil_push+0x1c0/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813988a9>] xlog_cil_push+0x239/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81399184>] xlog_cil_force_lsn+0x114/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81397871>] _xfs_log_force_lsn+0x51/0x2f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81393d3b>] xfs_trans_commit+0x28b/0x2a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134782d>] xfs_fs_log_dummy+0x5d/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81351e14>] xfs_sync_worker+0x84/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ad99a>] ? process_one_work+0x13a/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81351d90>] ? xfs_sync_inode_attr+0xf0/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af5a2>] worker_thread+0x162/0x340
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af440>] ? manage_workers.clone.20+0x240/0x240
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/0:1     D 0000000000000000  1928 27768      2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88005ee6d460 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880000000000 00000000001d2880 ffff8801653e6040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88005ee6dfd8 ffff88005ee6c000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810d633d>] ? __module_text_address+0xd/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8112f3a0>] ? mempool_alloc_slab+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8112f3a0>] ? mempool_alloc_slab+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8117891b>] kmem_cache_alloc+0xdb/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811082c2>] ? __delayacct_blkio_end+0x32/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8112f3a0>] mempool_alloc_slab+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4e30>] ? wake_up_bit+0x40/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811b9919>] bio_alloc_bioset+0x39/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816ceef9>] ? submit_flushes+0xd9/0x1f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816cb2d5>] bio_alloc_mddev+0x35/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816cef2e>] submit_flushes+0x10e/0x1f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816ceef9>] ? submit_flushes+0xd9/0x1f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ada09>] process_one_work+0x1a9/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810ad99a>] ? process_one_work+0x13a/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff816cee20>] ? md_do_sync+0xcd0/0xcd0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810af5a2>] worker_thread+0x162/0x340
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] rsync           D 0000000000000000  1976 28091  28084 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88000bcc7478 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880000000001 00000000001d2880 ffff880165682680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88000bcc7fd8 ffff88000bcc6000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8112cc27>] __page_cache_alloc+0x87/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113871a>] __do_page_cache_readahead+0x13a/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113869e>] ? __do_page_cache_readahead+0xbe/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81138a2c>] ra_submit+0x1c/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81138b3e>] ondemand_readahead+0x10e/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81138b70>] ? ondemand_readahead+0x140/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81138daf>] page_cache_sync_readahead+0x2f/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8112e774>] generic_file_aio_read+0x444/0x740
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81344fc9>] xfs_file_aio_read+0x179/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8118444a>] do_sync_read+0xda/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811c1898>] ? fsnotify+0x1d8/0x320
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811c1745>] ? fsnotify+0x85/0x320
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8143f1d6>] ? security_file_permission+0x96/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81897e1c>] ? __schedule+0x3ec/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81184edf>] sys_read+0x4f/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189bffb>] system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] run-crons       D 0000000000000000  2904 28235  28234 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88005199b7c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff880000000000 00000000001d2880 ffff880165680000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564]  ffff88005199bfd8 ffff88005199a000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810cd87f>] ? lock_release_non_nested+0x37f/0x3c0
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
>                 - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564]  [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] Sched Debug Version: v0.10, 3.2.5 #1
> Feb 29 12:05:38 [kernel] [186732.829564] ktime                                   : 186732833.030196
> Feb 29 12:05:38 [kernel] [186732.829564] sched_clk                               : 186363810.286830
> Feb 29 12:05:38 [kernel] [186732.829564] cpu_clk                                 : 186732829.564698
> Feb 29 12:05:38 [kernel] [186732.829564] jiffies                                 : 4481400126
> Feb 29 12:05:38 [kernel] [186732.829564] sched_clock_stable                      : 0
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] sysctl_sched
> Feb 29 12:05:38 [kernel] [186732.829564]   .sysctl_sched_latency                    : 18.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .sysctl_sched_min_granularity            : 2.250000
> Feb 29 12:05:38 [kernel] [186732.829564]   .sysctl_sched_wakeup_granularity         : 3.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .sysctl_sched_child_runs_first           : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .sysctl_sched_features                   : 24119
> Feb 29 12:05:38 [kernel] [186732.829564]   .sysctl_sched_tunable_scaling            : 1 (logaritmic)
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#0, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_switches                   : 127626917
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_load_updates               : 64695581
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_uninterruptible            : -5185
> Feb 29 12:05:38 [kernel] [186732.829564]   .clock                         : 186732809.005246
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[0]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[1]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[2]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[3]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[4]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .yld_count                     : 2
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_switch                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_goidle                  : 59908314
> Feb 29 12:05:38 [kernel] [186732.829564]   .avg_idle                      : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_count                    : 70228802
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_local                    : 49835264
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[0]:/
> Feb 29 12:05:38 [kernel] [186732.829564]   .exec_clock                    : 12099620.925352
> Feb 29 12:05:38 [kernel] [186732.829564]   .MIN_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .min_vruntime                  : 10865130.739847
> Feb 29 12:05:38 [kernel] [186732.829564]   .max_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread                        : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread0                       : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_spread_over                : 633
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_avg                      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_period                   : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_contrib                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_tg                       : 0
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[0]:
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_nr_running                 : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_throttled                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_time                       : 0.010808
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_runtime                    : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#1, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_switches                   : 128426315
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_load_updates               : 65499669
> Feb 29 12:05:38 [kernel] [186732.829564]   .next_balance                  : 4481.400004
> Feb 29 12:05:38 [kernel] [186732.829564]   .curr->pid                     : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .clock                         : 186732704.045991
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[0]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[1]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[2]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[3]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[4]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .yld_count                     : 3
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_switch                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_count                   : 128816327
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_goidle                  : 60378764
> Feb 29 12:05:38 [kernel] [186732.829564]   .avg_idle                      : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_count                    : 70202501
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_local                    : 50152935
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[1]:/
> Feb 29 12:05:38 [kernel] [186732.829564]   .exec_clock                    : 11942938.971418
> Feb 29 12:05:38 [kernel] [186732.829564]   .MIN_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .min_vruntime                  : 10965585.421468
> Feb 29 12:05:38 [kernel] [186732.829564]   .max_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread                        : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread0                       : 100454.681621
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_spread_over                : 637
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_avg                      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_contrib                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_tg                       : 0
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[1]:
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_nr_running                 : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_throttled                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_time                       : 0.011839
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_runtime                    : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#2, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_switches                   : 124877810
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_load_updates               : 63257405
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_uninterruptible            : 4945
> Feb 29 12:05:38 [kernel] [186732.829564]   .next_balance                  : 4481.400130
> Feb 29 12:05:38 [kernel] [186732.829564]   .curr->pid                     : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .clock                         : 186732830.026053
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[0]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[1]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[2]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[3]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[4]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .yld_count                     : 1
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_switch                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_count                   : 125239550
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_goidle                  : 59393188
> Feb 29 12:05:38 [kernel] [186732.829564]   .avg_idle                      : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_count                    : 60314669
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_local                    : 44799719
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[2]:/autogroup-41
> Feb 29 12:05:38 [kernel] [186732.829564]   .exec_clock                    : 1550.967553
> Feb 29 12:05:38 [kernel] [186732.829564]   .MIN_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .min_vruntime                  : 1549.918977
> Feb 29 12:05:38 [kernel] [186732.829564]   .max_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread                        : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread0                       : -10863580.820870
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_avg                      : 5119.999488
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_period                   : 6.468593
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_contrib                  : 791
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_tg                       : 791
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->exec_start                : 186732828.548911
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->vruntime                  : 8360658.005921
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->sum_exec_runtime          : 1552.122095
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_start     : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.sleep_start    : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.block_start    : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.sleep_max      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.block_max      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.exec_max       : 3.133477
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.slice_max      : 1.465148
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_max       : 3.850534
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_sum       : 133.221461
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_count     : 16329
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->load.weight               : 2
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[2]:/
> Feb 29 12:05:38 [kernel] [186732.829564]   .exec_clock                    : 9867308.118613
> Feb 29 12:05:38 [kernel] [186732.829564]   .MIN_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .min_vruntime                  : 8360666.433800
> Feb 29 12:05:38 [kernel] [186732.829564]   .max_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_spread_over                : 745
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_avg                      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_period                   : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_contrib                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_tg                       : 0
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[2]:
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_nr_running                 : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_throttled                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_time                       : 0.007240
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_runtime                    : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#3, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 2
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 2048
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_switches                   : 124026375
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_load_updates               : 62682931
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_uninterruptible            : 3772
> Feb 29 12:05:38 [kernel] [186732.829564]   .next_balance                  : 4481.400098
> Feb 29 12:05:38 [kernel] [186732.829564]   .curr->pid                     : 2991
> Feb 29 12:05:38 [kernel] [186732.829564]   .clock                         : 186732828.596971
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[0]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[1]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[2]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[3]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .cpu_load[4]                   : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .yld_count                     : 4
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_switch                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_count                   : 124393065
> Feb 29 12:05:38 [kernel] [186732.829564]   .sched_goidle                  : 58899951
> Feb 29 12:05:38 [kernel] [186732.829564]   .avg_idle                      : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_count                    : 59877024
> Feb 29 12:05:38 [kernel] [186732.829564]   .ttwu_local                    : 44537104
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[3]:/autogroup-42
> Feb 29 12:05:38 [kernel] [186732.829564]   .MIN_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .min_vruntime                  : 626168.958817
> Feb 29 12:05:38 [kernel] [186732.829564]   .max_vruntime                  : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread                        : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread0                       : -10238961.781030
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_spread_over                : 23016
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 1
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 1024
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_avg                      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_period                   : 9.999999
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_contrib                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_tg                       : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->exec_start                : 186732828.596971
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->vruntime                  : 8279374.470424
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->sum_exec_runtime          : 263285.952949
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_start     : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.sleep_start    : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.block_start    : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.sleep_max      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.block_max      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.exec_max       : 13.400424
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.slice_max      : 46.505344
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_max       : 16.575587
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_sum       : 2448.726587
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->statistics.wait_count     : 242426
> Feb 29 12:05:38 [kernel] [186732.829564]   .se->load.weight               : 1024
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564]   .exec_clock                    : 9783020.902642
> Feb 29 12:05:38 [kernel] [186732.829564]   .MIN_vruntime                  : 8279372.216698
> Feb 29 12:05:38 [kernel] [186732.829564]   .min_vruntime                  : 8279380.963316
> Feb 29 12:05:38 [kernel] [186732.829564]   .max_vruntime                  : 8279372.216698
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread                        : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .spread0                       : -2585749.776531
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_spread_over                : 1511
> Feb 29 12:05:38 [kernel] [186732.829564]   .nr_running                    : 2
> Feb 29 12:05:38 [kernel] [186732.829564]   .load                          : 2048
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_avg                      : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_period                   : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_contrib                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .load_tg                       : 0
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[3]:
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_nr_running                 : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_throttled                  : 0
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_time                       : 0.109158
> Feb 29 12:05:38 [kernel] [186732.829564]   .rt_runtime                    : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564] 
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564] R           bash  2991    626164.399889      1092   120    626164.399889       164.832603 186663838.488864 /autogroup-42
> Feb 29 12:05:38 [kernel] [186732.829564]      kworker/3:1 27652   8279372.216698     31913   120   8279372.216698      2529.050613   7144046.892045 /
> Feb 29 12:05:38 [kernel] [186732.829564] 
> 

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2012-02-29 19:14 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <CAOgv-ziomS-+ML0Xb4oSoWTjzkG8BBBQcMjXwn7rjQV2pFtQ6Q@mail.gmail.com>
2012-02-06 22:51 ` Soft lockup problem Jan Kara
2012-02-07 16:35   ` Gerard Saraber
2012-02-07 16:54     ` Jan Kara
2012-02-27 14:38       ` Gerard Saraber
     [not found]         ` <CAOgv-zjF3YSLuGu0p0xkCp5ZfWu2H1Xg8aQ7xcpCQqWOdwh1aA@mail.gmail.com>
2012-02-29 19:14           ` Jan Kara

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox