* 3.11 and 3.12rc continual hard lock ups
@ 2013-10-23 19:26 Sean Clarke
2013-10-23 20:05 ` Duncan
2013-10-24 8:47 ` Chris Mason
0 siblings, 2 replies; 5+ messages in thread
From: Sean Clarke @ 2013-10-23 19:26 UTC (permalink / raw)
To: linux-btrfs
Hi,
I have an Intel Core i7 based fileserver with 18TB BTRFS in a 6x
3TB RAID 1+0 configuration. The system was working fine running Ubuntu
13.04 (kernel 3.11.0-12-generic). The system was upgraded to Ubuntu
13.10 (kernel 3.11) and began to lock up daily, sometimes every couple
of hours. Previously it never crashed and was only taken down for
maintenance so a bug was files on launchpad
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1237794).
I have updated the system with mainline kernels "3.12.0-999-generic
#201310090426" and "3.12.0-999-generic #201310170405" and still the
lockups occur.
Absolutely nothing is logged in the syslog, it is a complete freeze. I
could get the system to hang by attempting to delete a 95GB tar file
from a client (over NFS), it would lock up every time. At other times
it just locks up (overnight for example).
I have had a top session running a couple of times when it locks up
and btrfs-transaction and btrfs-flush_del are using 3 to 5 cores at 50
to 100% (see top extract below).
top - 12:06:37 up 1:51, 1 user, load average: 3.33, 0.87, 0.33
Tasks: 226 total, 5 running, 221 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 48.5 sy, 0.0 ni, 50.9 id, 0.1 wa, 0.0 hi, 0.5 si, 0.0 st
KiB Mem: 12295372 total, 12138296 used, 157076 free, 312 buffers
KiB Swap: 7831536 total, 56 used, 7831480 free, 24644 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1005 root 20 0 0 0 0 R 100. 1 0.0 0:50.16
btrfs-transacti
2462 root 20 0 0 0 0 R 75.8 0.0 0:06.66
btrfs-flush_del
2459 root 20 0 0 0 0 S 72.5 0.0 0:10.18
btrfs-flush_del
2463 root 20 0 0 0 0 S 70.5 0.0 0:14.41
btrfs-flush_del
2457 root 20 0 0 0 0 R 38.6 0.0 0:28.35
btrfs-flush_del
2458 root 20 0 0 0 0 S 17.0 0.0 0:39.31
btrfs-flush_del
1959 root 20 0 0 0 0 R 9.6 0.0 0:31.68
btrfs-flush_del
100 root 20 0 0 0 0 S 7.6 0.0 0:00.31 kswap
I have looked in bugzilla and can't relate this to anything already
reported, has anyone seen something similar to this?
Regards
Sean
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 3.11 and 3.12rc continual hard lock ups
2013-10-23 19:26 3.11 and 3.12rc continual hard lock ups Sean Clarke
@ 2013-10-23 20:05 ` Duncan
2013-10-24 8:47 ` Chris Mason
1 sibling, 0 replies; 5+ messages in thread
From: Duncan @ 2013-10-23 20:05 UTC (permalink / raw)
To: linux-btrfs
Sean Clarke posted on Wed, 23 Oct 2013 20:26:15 +0100 as excerpted:
> I have looked in bugzilla and can't relate this to anything already
> reported, has anyone seen something similar to this?
I don't do NFS so didn't pay attention to the details, but look here on-
list as IIRC there was a very similar NFS-on-btrfs-related bug reported
recently. I /believe/ there's a fix in the pipeline, hopefully pre-3.12
release, but it's only in the last couple days so probably isn't in
Linus' tree yet.
Do look tho, as I neither remember the details nor am 100% positive that
what I think I recall is correct, but hopefully this at least lets you
know to look.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 3.11 and 3.12rc continual hard lock ups
2013-10-23 19:26 3.11 and 3.12rc continual hard lock ups Sean Clarke
2013-10-23 20:05 ` Duncan
@ 2013-10-24 8:47 ` Chris Mason
2013-10-27 9:06 ` Sean Clarke
1 sibling, 1 reply; 5+ messages in thread
From: Chris Mason @ 2013-10-24 8:47 UTC (permalink / raw)
To: Sean Clarke, linux-btrfs
Quoting Sean Clarke (2013-10-23 15:26:15)
> Hi,
> I have an Intel Core i7 based fileserver with 18TB BTRFS in a 6x
> 3TB RAID 1+0 configuration. The system was working fine running Ubuntu
> 13.04 (kernel 3.11.0-12-generic). The system was upgraded to Ubuntu
> 13.10 (kernel 3.11) and began to lock up daily, sometimes every couple
> of hours. Previously it never crashed and was only taken down for
> maintenance so a bug was files on launchpad
> (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1237794).
Does your kernel config include the softlockup detector? If you could
please do sysrq-w and sysrq-t while it is locked, it'll help us track it
down.
It will be much easier if you have a serial console or network console,
and you've increased your kernel logging level to the highest value
(dmesg -n7)
-chris
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 3.11 and 3.12rc continual hard lock ups
2013-10-24 8:47 ` Chris Mason
@ 2013-10-27 9:06 ` Sean Clarke
2013-10-27 21:08 ` Sean Clarke
0 siblings, 1 reply; 5+ messages in thread
From: Sean Clarke @ 2013-10-27 9:06 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-btrfs
Hi Chris,
I set dmesg -n7 and didn't see anything extra logged (I had to use
sudo to enable it. I am not familiar with sysrq-w and sysrq-t, however
you ask "while it is locked", this is a "real" hard lock. everything
is unresponsive and the machine even fails to respond to ping request.
It happened again over night and again the only clues where the
btrfs-transaction and btrfs-flush_del going ballistic.
Regards
Sean Clarke
---------------------------------------------
SEC Consulting Limited
Phone: +44 (0)23 8040 5599
Website: http://www.sec-consulting.co.uk
Email: sean.clarke@sec-consulting.co.uk
On Thu, Oct 24, 2013 at 9:47 AM, Chris Mason <chris.mason@fusionio.com> wrote:
> Quoting Sean Clarke (2013-10-23 15:26:15)
>> Hi,
>> I have an Intel Core i7 based fileserver with 18TB BTRFS in a 6x
>> 3TB RAID 1+0 configuration. The system was working fine running Ubuntu
>> 13.04 (kernel 3.11.0-12-generic). The system was upgraded to Ubuntu
>> 13.10 (kernel 3.11) and began to lock up daily, sometimes every couple
>> of hours. Previously it never crashed and was only taken down for
>> maintenance so a bug was files on launchpad
>> (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1237794).
>
> Does your kernel config include the softlockup detector? If you could
> please do sysrq-w and sysrq-t while it is locked, it'll help us track it
> down.
>
> It will be much easier if you have a serial console or network console,
> and you've increased your kernel logging level to the highest value
> (dmesg -n7)
>
> -chris
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: 3.11 and 3.12rc continual hard lock ups
2013-10-27 9:06 ` Sean Clarke
@ 2013-10-27 21:08 ` Sean Clarke
0 siblings, 0 replies; 5+ messages in thread
From: Sean Clarke @ 2013-10-27 21:08 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-btrfs
OK, hard another crash this afternoon - I had "dmesg -n7" set and this
time (unusually) I had something in the logs:
Oct 27 14:48:15 enterprise kernel: [23752.263442] INFO: task nfsd:1537
blocked for more than 120 seconds.
Oct 27 14:48:15 enterprise kernel: [23752.263452] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:48:21 enterprise kernel: [23752.263460] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:48:21 enterprise kernel: [23752.263469] nfsd D
0000000000000000 0 1537 2 0x00000000
Oct 27 14:48:21 enterprise kernel: [23752.263484] ffff8800ba29db38
0000000000000046 ffff880036bcc1a0 ffff88003602b800
Oct 27 14:48:21 enterprise kernel: [23752.263503] ffff8800ba29dfd8
ffff8800ba29dfd8 ffff8800ba29dfd8 00000000000144c0
Oct 27 14:48:21 enterprise kernel: [23752.263513] ffff880329daaf60
ffff8800ba1a17b0 ffff8800ba29db48 ffff88032b893430
Oct 27 14:48:21 enterprise kernel: [23752.263523] Call Trace:
Oct 27 14:48:21 enterprise kernel: [23752.263531]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:48:21 enterprise kernel: [23752.263557]
[<ffffffffa01642f1>] wait_current_trans.isra.33+0xc1/0x120 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.263566]
[<ffffffff8108cc40>] ? add_wait_queue+0x60/0x60
Oct 27 14:48:21 enterprise kernel: [23752.263580]
[<ffffffffa016615e>] start_transaction.part.35+0x2ee/0x510 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.263594]
[<ffffffffa01663a9>] start_transaction+0x29/0x30 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.263607]
[<ffffffffa01665fb>] btrfs_start_transaction+0x1b/0x20 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.263622]
[<ffffffffa01744b6>] btrfs_create+0x46/0x220 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.263628]
[<ffffffff812fb7fc>] ? security_inode_permission+0x1c/0x30
Oct 27 14:48:21 enterprise kernel: [23752.263634]
[<ffffffff811cc0b5>] vfs_create+0xb5/0x120
Oct 27 14:48:21 enterprise kernel: [23752.263643]
[<ffffffffa065941c>] do_nfsd_create+0x50c/0x5f0 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263653]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263660]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263673]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.263685]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.263692]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263699]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263704]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:48:21 enterprise kernel: [23752.263709]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.263715]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.263720]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.263725] INFO: task nfsd:1539
blocked for more than 120 seconds.
Oct 27 14:48:21 enterprise kernel: [23752.263729] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:48:21 enterprise kernel: [23752.263733] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:48:21 enterprise kernel: [23752.263737] nfsd D
0000000000000000 0 1539 2 0x00000000
Oct 27 14:48:21 enterprise kernel: [23752.263744] ffff88032d783c38
0000000000000046 ffff88032d783be8 ffffffff81094340
Oct 27 14:48:21 enterprise kernel: [23752.263753] ffff88032d783fd8
ffff88032d783fd8 ffff88032d783fd8 00000000000144c0
Oct 27 14:48:21 enterprise kernel: [23752.263764] ffff88032e5ddec0
ffff8800ba1a4710 ffff88032d783c18 ffff8802f2573408
Oct 27 14:48:21 enterprise kernel: [23752.263773] Call Trace:
Oct 27 14:48:21 enterprise kernel: [23752.263778]
[<ffffffff81094340>] ? set_groups+0x40/0x60
Oct 27 14:48:21 enterprise kernel: [23752.263783]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:48:21 enterprise kernel: [23752.263788]
[<ffffffff8174509e>] schedule_preempt_disabled+0xe/0x10
Oct 27 14:48:21 enterprise kernel: [23752.263794]
[<ffffffff81743024>] __mutex_lock_slowpath+0x114/0x1b0
Oct 27 14:48:21 enterprise kernel: [23752.263800]
[<ffffffff817430e3>] mutex_lock+0x23/0x40
Oct 27 14:48:21 enterprise kernel: [23752.263807]
[<ffffffffa0659088>] do_nfsd_create+0x178/0x5f0 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263816]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263824]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263835]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.263846]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.263853]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263860]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.263865]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:48:21 enterprise kernel: [23752.263870]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.263875]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.263880]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.263884] INFO: task nfsd:1540
blocked for more than 120 seconds.
Oct 27 14:48:21 enterprise kernel: [23752.263888] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:48:21 enterprise kernel: [23752.263892] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:48:21 enterprise kernel: [23752.263896] nfsd D
0000000000000000 0 1540 2 0x00000000
Oct 27 14:48:21 enterprise kernel: [23752.263903] ffff88032c35db38
0000000000000046 ffff880036bcc1a0 ffff88003602d800
Oct 27 14:48:21 enterprise kernel: [23752.263912] ffff88032c35dfd8
ffff88032c35dfd8 ffff88032c35dfd8 00000000000144c0
Oct 27 14:48:21 enterprise kernel: [23752.263920] ffff88032e5ddec0
ffff8800ba1a5ec0 ffff88032c35db48 ffff88032b893430
Oct 27 14:48:21 enterprise kernel: [23752.263929] Call Trace:
Oct 27 14:48:21 enterprise kernel: [23752.264156]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:48:21 enterprise kernel: [23752.264620]
[<ffffffffa01642f1>] wait_current_trans.isra.33+0xc1/0x120 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.265110]
[<ffffffff8108cc40>] ? add_wait_queue+0x60/0x60
Oct 27 14:48:21 enterprise kernel: [23752.265601]
[<ffffffffa016615e>] start_transaction.part.35+0x2ee/0x510 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.266081]
[<ffffffffa01663a9>] start_transaction+0x29/0x30 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.266556]
[<ffffffffa01665fb>] btrfs_start_transaction+0x1b/0x20 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.267031]
[<ffffffffa01744b6>] btrfs_create+0x46/0x220 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.267541]
[<ffffffff812fb7fc>] ? security_inode_permission+0x1c/0x30
Oct 27 14:48:21 enterprise kernel: [23752.267949]
[<ffffffff811cc0b5>] vfs_create+0xb5/0x120
Oct 27 14:48:21 enterprise kernel: [23752.268420]
[<ffffffffa065941c>] do_nfsd_create+0x50c/0x5f0 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.268883]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.269398]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.269868]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.270329]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.270782]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.271234]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.271698]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:48:21 enterprise kernel: [23752.272099]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.272557]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.273071]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.273465] INFO: task nfsd:1542
blocked for more than 120 seconds.
Oct 27 14:48:21 enterprise kernel: [23752.273934] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:48:21 enterprise kernel: [23752.274439] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:48:21 enterprise kernel: [23752.274903] nfsd D
0000000000000000 0 1542 2 0x00000000
Oct 27 14:48:21 enterprise kernel: [23752.275386] ffff88032bf51b38
0000000000000046 ffff880036bcc1a0 ffff88032af0d800
Oct 27 14:48:21 enterprise kernel: [23752.275815] ffff88032bf51fd8
ffff88032bf51fd8 ffff88032bf51fd8 00000000000144c0
Oct 27 14:48:21 enterprise kernel: [23752.276296] ffffffff81c144a0
ffff8800baab4710 ffff88032bf51b48 ffff88032b893430
Oct 27 14:48:21 enterprise kernel: [23752.276779] Call Trace:
Oct 27 14:48:21 enterprise kernel: [23752.277252]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:48:21 enterprise kernel: [23752.277740]
[<ffffffffa01642f1>] wait_current_trans.isra.33+0xc1/0x120 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.278214]
[<ffffffff8108cc40>] ? add_wait_queue+0x60/0x60
Oct 27 14:48:21 enterprise kernel: [23752.278704]
[<ffffffffa016615e>] start_transaction.part.35+0x2ee/0x510 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.279239]
[<ffffffffa01663a9>] start_transaction+0x29/0x30 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.279680]
[<ffffffffa01665fb>] btrfs_start_transaction+0x1b/0x20 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.280162]
[<ffffffffa01744b6>] btrfs_create+0x46/0x220 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.280628]
[<ffffffff812fb7fc>] ? security_inode_permission+0x1c/0x30
Oct 27 14:48:21 enterprise kernel: [23752.281101]
[<ffffffff811cc0b5>] vfs_create+0xb5/0x120
Oct 27 14:48:21 enterprise kernel: [23752.281571]
[<ffffffffa065941c>] do_nfsd_create+0x50c/0x5f0 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.282051]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.282520]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.282997]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.283518]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.283949]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.284421]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.284891]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:48:21 enterprise kernel: [23752.285363]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.285836]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.286310]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.286782] INFO: task nfsd:1543
blocked for more than 120 seconds.
Oct 27 14:48:21 enterprise kernel: [23752.287311] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:48:21 enterprise kernel: [23752.287747] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:48:21 enterprise kernel: [23752.288235] nfsd D
0000000000000000 0 1543 2 0x00000000
Oct 27 14:48:21 enterprise kernel: [23752.288731] ffff88032b46bb38
0000000000000046 ffff880036bcc1a0 ffff88032cced800
Oct 27 14:48:21 enterprise kernel: [23752.289235] ffff88032b46bfd8
ffff88032b46bfd8 ffff88032b46bfd8 00000000000144c0
Oct 27 14:48:21 enterprise kernel: [23752.289740] ffff88032e5ddec0
ffff8800baab5ec0 ffff88032b46bb48 ffff88032b893430
Oct 27 14:48:21 enterprise kernel: [23752.290247] Call Trace:
Oct 27 14:48:21 enterprise kernel: [23752.290740]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:48:21 enterprise kernel: [23752.291288]
[<ffffffffa01642f1>] wait_current_trans.isra.33+0xc1/0x120 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.291724]
[<ffffffff8108cc40>] ? add_wait_queue+0x60/0x60
Oct 27 14:48:21 enterprise kernel: [23752.292220]
[<ffffffffa016615e>] start_transaction.part.35+0x2ee/0x510 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.292712]
[<ffffffffa01663a9>] start_transaction+0x29/0x30 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.293200]
[<ffffffffa01665fb>] btrfs_start_transaction+0x1b/0x20 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.293695]
[<ffffffffa01744b6>] btrfs_create+0x46/0x220 [btrfs]
Oct 27 14:48:21 enterprise kernel: [23752.294173]
[<ffffffff812fb7fc>] ? security_inode_permission+0x1c/0x30
Oct 27 14:48:21 enterprise kernel: [23752.294660]
[<ffffffff811cc0b5>] vfs_create+0xb5/0x120
Oct 27 14:48:21 enterprise kernel: [23752.295148]
[<ffffffffa065941c>] do_nfsd_create+0x50c/0x5f0 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.295652]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.296113]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.296603]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.297086]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:48:21 enterprise kernel: [23752.297564]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.298046]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:48:21 enterprise kernel: [23752.298522]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:48:21 enterprise kernel: [23752.299002]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.299498]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:48:21 enterprise kernel: [23752.299957]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.257224] INFO: task nfsd:1536
blocked for more than 120 seconds.
Oct 27 14:50:15 enterprise kernel: [23872.257489] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:50:15 enterprise kernel: [23872.257716] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:50:15 enterprise kernel: [23872.258178] nfsd D
0000000000000000 0 1536 2 0x00000000
Oct 27 14:50:15 enterprise kernel: [23872.258683] ffff8800ba19bc38
0000000000000046 ffff8800ba19bbe8 ffffffff81094340
Oct 27 14:50:15 enterprise kernel: [23872.259190] ffff8800ba19bfd8
ffff8800ba19bfd8 ffff8800ba19bfd8 00000000000144c0
Oct 27 14:50:15 enterprise kernel: [23872.259699] ffff88032e08af60
ffff8800ba1a0000 ffff8800ba19bc18 ffff8802f2573408
Oct 27 14:50:15 enterprise kernel: [23872.260201] Call Trace:
Oct 27 14:50:15 enterprise kernel: [23872.260703]
[<ffffffff81094340>] ? set_groups+0x40/0x60
Oct 27 14:50:15 enterprise kernel: [23872.261198]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:50:15 enterprise kernel: [23872.261680]
[<ffffffff8174509e>] schedule_preempt_disabled+0xe/0x10
Oct 27 14:50:15 enterprise kernel: [23872.262169]
[<ffffffff81743024>] __mutex_lock_slowpath+0x114/0x1b0
Oct 27 14:50:15 enterprise kernel: [23872.262655]
[<ffffffff817430e3>] mutex_lock+0x23/0x40
Oct 27 14:50:15 enterprise kernel: [23872.263138]
[<ffffffffa0659088>] do_nfsd_create+0x178/0x5f0 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.263616]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.264095]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.264579]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.265145]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.265526]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.265996]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.266463]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:50:15 enterprise kernel: [23872.266929]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.267395]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.267862]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.268326] INFO: task nfsd:1537
blocked for more than 120 seconds.
Oct 27 14:50:15 enterprise kernel: [23872.268793] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:50:15 enterprise kernel: [23872.269377] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:50:15 enterprise kernel: [23872.269746] nfsd D
0000000000000000 0 1537 2 0x00000000
Oct 27 14:50:15 enterprise kernel: [23872.270233] ffff8800ba29db38
0000000000000046 ffff880036bcc1a0 ffff88003602b800
Oct 27 14:50:15 enterprise kernel: [23872.270731] ffff8800ba29dfd8
ffff8800ba29dfd8 ffff8800ba29dfd8 00000000000144c0
Oct 27 14:50:15 enterprise kernel: [23872.271226] ffff880329daaf60
ffff8800ba1a17b0 ffff8800ba29db48 ffff88032b893430
Oct 27 14:50:15 enterprise kernel: [23872.271722] Call Trace:
Oct 27 14:50:15 enterprise kernel: [23872.272206]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:50:15 enterprise kernel: [23872.272706]
[<ffffffffa01642f1>] wait_current_trans.isra.33+0xc1/0x120 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.273293]
[<ffffffff8108cc40>] ? add_wait_queue+0x60/0x60
Oct 27 14:50:15 enterprise kernel: [23872.273683]
[<ffffffffa016615e>] start_transaction.part.35+0x2ee/0x510 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.274178]
[<ffffffffa01663a9>] start_transaction+0x29/0x30 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.274670]
[<ffffffffa01665fb>] btrfs_start_transaction+0x1b/0x20 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.275158]
[<ffffffffa01744b6>] btrfs_create+0x46/0x220 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.275632]
[<ffffffff812fb7fc>] ? security_inode_permission+0x1c/0x30
Oct 27 14:50:15 enterprise kernel: [23872.276105]
[<ffffffff811cc0b5>] vfs_create+0xb5/0x120
Oct 27 14:50:15 enterprise kernel: [23872.276585]
[<ffffffffa065941c>] do_nfsd_create+0x50c/0x5f0 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.277120]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.277545]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.278021]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.278498]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.278970]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.279435]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.279902]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:50:15 enterprise kernel: [23872.280365]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.280834]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.281410]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.281763] INFO: task nfsd:1538
blocked for more than 120 seconds.
Oct 27 14:50:15 enterprise kernel: [23872.282230] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:50:15 enterprise kernel: [23872.282697] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:50:15 enterprise kernel: [23872.283175] nfsd D
0000000000000000 0 1538 2 0x00000000
Oct 27 14:50:15 enterprise kernel: [23872.283659] ffff8800ba397c38
0000000000000046 ffff8800ba397be8 ffffffff81094340
Oct 27 14:50:15 enterprise kernel: [23872.284152] ffff8800ba397fd8
ffff8800ba397fd8 ffff8800ba397fd8 00000000000144c0
Oct 27 14:50:15 enterprise kernel: [23872.284648] ffff88032a21dec0
ffff8800ba1a2f60 ffff8800ba397c18 ffff8802f2573408
Oct 27 14:50:15 enterprise kernel: [23872.285242] Call Trace:
Oct 27 14:50:15 enterprise kernel: [23872.285621]
[<ffffffff81094340>] ? set_groups+0x40/0x60
Oct 27 14:50:15 enterprise kernel: [23872.286102]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:50:15 enterprise kernel: [23872.286585]
[<ffffffff8174509e>] schedule_preempt_disabled+0xe/0x10
Oct 27 14:50:15 enterprise kernel: [23872.287073]
[<ffffffff81743024>] __mutex_lock_slowpath+0x114/0x1b0
Oct 27 14:50:15 enterprise kernel: [23872.287559]
[<ffffffff817430e3>] mutex_lock+0x23/0x40
Oct 27 14:50:15 enterprise kernel: [23872.288037]
[<ffffffffa0659088>] do_nfsd_create+0x178/0x5f0 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.288512]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.288982]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.289563]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.289918]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.290383]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.290848]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.291307]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:50:15 enterprise kernel: [23872.291769]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.292234]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.292699]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.293270] INFO: task nfsd:1539
blocked for more than 120 seconds.
Oct 27 14:50:15 enterprise kernel: [23872.293631] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:50:15 enterprise kernel: [23872.294100] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:50:15 enterprise kernel: [23872.294577] nfsd D
0000000000000000 0 1539 2 0x00000000
Oct 27 14:50:15 enterprise kernel: [23872.295063] ffff88032d783c38
0000000000000046 ffff88032d783be8 ffffffff81094340
Oct 27 14:50:15 enterprise kernel: [23872.295557] ffff88032d783fd8
ffff88032d783fd8 ffff88032d783fd8 00000000000144c0
Oct 27 14:50:15 enterprise kernel: [23872.296048] ffff88032e5ddec0
ffff8800ba1a4710 ffff88032d783c18 ffff8802f2573408
Oct 27 14:50:15 enterprise kernel: [23872.296537] Call Trace:
Oct 27 14:50:15 enterprise kernel: [23872.297018]
[<ffffffff81094340>] ? set_groups+0x40/0x60
Oct 27 14:50:15 enterprise kernel: [23872.297615]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:50:15 enterprise kernel: [23872.297988]
[<ffffffff8174509e>] schedule_preempt_disabled+0xe/0x10
Oct 27 14:50:15 enterprise kernel: [23872.298476]
[<ffffffff81743024>] __mutex_lock_slowpath+0x114/0x1b0
Oct 27 14:50:15 enterprise kernel: [23872.298960]
[<ffffffff817430e3>] mutex_lock+0x23/0x40
Oct 27 14:50:15 enterprise kernel: [23872.299441]
[<ffffffffa0659088>] do_nfsd_create+0x178/0x5f0 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.299916]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.300385]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.300858]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.301437]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.301792]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.302256]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.302717]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:50:15 enterprise kernel: [23872.303178]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.303635]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.304092]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.304549] INFO: task nfsd:1540
blocked for more than 120 seconds.
Oct 27 14:50:15 enterprise kernel: [23872.305006] Not tainted
3.12.0-999-generic #201310210405
Oct 27 14:50:15 enterprise kernel: [23872.305563] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 27 14:50:15 enterprise kernel: [23872.305938] nfsd D
0000000000000000 0 1540 2 0x00000000
Oct 27 14:50:15 enterprise kernel: [23872.306415] ffff88032c35db38
0000000000000046 ffff880036bcc1a0 ffff88003602d800
Oct 27 14:50:15 enterprise kernel: [23872.306899] ffff88032c35dfd8
ffff88032c35dfd8 ffff88032c35dfd8 00000000000144c0
Oct 27 14:50:15 enterprise kernel: [23872.307380] ffff88032e5ddec0
ffff8800ba1a5ec0 ffff88032c35db48 ffff88032b893430
Oct 27 14:50:15 enterprise kernel: [23872.307865] Call Trace:
Oct 27 14:50:15 enterprise kernel: [23872.308340]
[<ffffffff81744d79>] schedule+0x29/0x70
Oct 27 14:50:15 enterprise kernel: [23872.308823]
[<ffffffffa01642f1>] wait_current_trans.isra.33+0xc1/0x120 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.309383]
[<ffffffff8108cc40>] ? add_wait_queue+0x60/0x60
Oct 27 14:50:15 enterprise kernel: [23872.309783]
[<ffffffffa016615e>] start_transaction.part.35+0x2ee/0x510 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.310266]
[<ffffffffa01663a9>] start_transaction+0x29/0x30 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.310754]
[<ffffffffa01665fb>] btrfs_start_transaction+0x1b/0x20 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.311245]
[<ffffffffa01744b6>] btrfs_create+0x46/0x220 [btrfs]
Oct 27 14:50:15 enterprise kernel: [23872.311722]
[<ffffffff812fb7fc>] ? security_inode_permission+0x1c/0x30
Oct 27 14:50:15 enterprise kernel: [23872.312205]
[<ffffffff811cc0b5>] vfs_create+0xb5/0x120
Oct 27 14:50:15 enterprise kernel: [23872.312694]
[<ffffffffa065941c>] do_nfsd_create+0x50c/0x5f0 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.313259]
[<ffffffffa06608ad>] nfsd3_proc_create+0x16d/0x250 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.313648]
[<ffffffffa0651d65>] nfsd_dispatch+0xe5/0x230 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.314117]
[<ffffffffa05dbf55>] svc_process_common+0x345/0x680 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.314584]
[<ffffffffa05dc5e3>] svc_process+0x103/0x160 [sunrpc]
Oct 27 14:50:15 enterprise kernel: [23872.315052]
[<ffffffffa06518cf>] nfsd+0xbf/0x130 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.315519]
[<ffffffffa0651810>] ? nfsd_destroy+0x80/0x80 [nfsd]
Oct 27 14:50:15 enterprise kernel: [23872.315983]
[<ffffffff8108c0e0>] kthread+0xc0/0xd0
Oct 27 14:50:15 enterprise kernel: [23872.316446]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.316913]
[<ffffffff8174f6fc>] ret_from_fork+0x7c/0xb0
Oct 27 14:50:15 enterprise kernel: [23872.317470]
[<ffffffff8108c020>] ? flush_kthread_worker+0xb0/0xb0
Oct 27 15:19:10 enterprise CRON[4310]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Oct 27 16:07:32 enterprise kernel: [28472.130147] rpcbind invoked
oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Oct 27 16:07:32 enterprise kernel: [28472.130376] rpcbind cpuset=/
mems_allowed=0
Oct 27 16:07:32 enterprise kernel: [28472.130593] CPU: 0 PID: 891
Comm: rpcbind Not tainted 3.12.0-999-generic #201310210405
Oct 27 16:07:32 enterprise kernel: [28472.131065] Hardware name:
System manufacturer System Product Name/P6T SE, BIOS 0805
02/24/2010
Oct 27 16:07:32 enterprise kernel: [28472.131559] 0000000000000000
ffff8800363c9378 ffffffff81739f5f 0000000000000007
Oct 27 16:07:32 enterprise kernel: [28472.132066] ffff88032cc0c710
ffff8800363c93c8 ffffffff8172f2f3 ffff880000000000
Oct 27 16:07:32 enterprise kernel: [28472.132573] 000200da8137e978
ffff88032dcd97b0 ffff88032a110000 0000000000000000
Oct 27 16:07:32 enterprise kernel: [28472.133072] Call Trace:
Oct 27 16:07:32 enterprise kernel: [28472.133572]
[<ffffffff81739f5f>] dump_stack+0x46/0x58
Oct 27 16:07:32 enterprise kernel: [28472.134165]
[<ffffffff8172f2f3>] dump_header+0x7e/0xbd
Oct 27 16:07:32 enterprise kernel: [28472.134564]
[<ffffffff8172f389>] oom_kill_process.part.6+0x57/0x2d4
Oct 27 16:07:32 enterprise kernel: [28472.135068]
[<ffffffff81151eed>] oom_kill_process+0x4d/0x50
Oct 27 16:07:32 enterprise kernel: [28472.135565]
[<ffffffff81152225>] out_of_memory+0x145/0x1d0
Oct 27 16:07:32 enterprise kernel: [28472.136056]
[<ffffffff811580f9>] __alloc_pages_nodemask+0xa19/0xa30
Oct 27 16:07:32 enterprise kernel: [28472.136541]
[<ffffffff8119adf3>] alloc_pages_vma+0xa3/0x150
Oct 27 16:07:32 enterprise kernel: [28472.137023]
[<ffffffff8118cf7b>] read_swap_cache_async+0x10b/0x190
Oct 27 16:07:32 enterprise kernel: [28472.137502]
[<ffffffff8118d09e>] swapin_readahead+0x9e/0xf0
Oct 27 16:07:32 enterprise kernel: [28472.138030]
[<ffffffff81177df5>] do_swap_page.isra.49+0x125/0x600
Nothing - complete hard lockup, no ping responses etc. until 19:30
when the system was restarted.
Regards
Sean Clarke
---------------------------------------------
SEC Consulting Limited
Phone: +44 (0)23 8040 5599
Website: http://www.sec-consulting.co.uk
Email: sean.clarke@sec-consulting.co.uk
On Sun, Oct 27, 2013 at 9:06 AM, Sean Clarke
<sean.clarke@sec-consulting.co.uk> wrote:
> Hi Chris,
> I set dmesg -n7 and didn't see anything extra logged (I had to use
> sudo to enable it. I am not familiar with sysrq-w and sysrq-t, however
> you ask "while it is locked", this is a "real" hard lock. everything
> is unresponsive and the machine even fails to respond to ping request.
> It happened again over night and again the only clues where the
> btrfs-transaction and btrfs-flush_del going ballistic.
>
> Regards
> Sean Clarke
> ---------------------------------------------
> SEC Consulting Limited
> Phone: +44 (0)23 8040 5599
> Website: http://www.sec-consulting.co.uk
> Email: sean.clarke@sec-consulting.co.uk
>
>
> On Thu, Oct 24, 2013 at 9:47 AM, Chris Mason <chris.mason@fusionio.com> wrote:
>> Quoting Sean Clarke (2013-10-23 15:26:15)
>>> Hi,
>>> I have an Intel Core i7 based fileserver with 18TB BTRFS in a 6x
>>> 3TB RAID 1+0 configuration. The system was working fine running Ubuntu
>>> 13.04 (kernel 3.11.0-12-generic). The system was upgraded to Ubuntu
>>> 13.10 (kernel 3.11) and began to lock up daily, sometimes every couple
>>> of hours. Previously it never crashed and was only taken down for
>>> maintenance so a bug was files on launchpad
>>> (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1237794).
>>
>> Does your kernel config include the softlockup detector? If you could
>> please do sysrq-w and sysrq-t while it is locked, it'll help us track it
>> down.
>>
>> It will be much easier if you have a serial console or network console,
>> and you've increased your kernel logging level to the highest value
>> (dmesg -n7)
>>
>> -chris
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2013-10-27 21:08 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-23 19:26 3.11 and 3.12rc continual hard lock ups Sean Clarke
2013-10-23 20:05 ` Duncan
2013-10-24 8:47 ` Chris Mason
2013-10-27 9:06 ` Sean Clarke
2013-10-27 21:08 ` Sean Clarke
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).