* Can you help me to investiage a soft lockup?
[not found] <1211918248.8179166.1412774638265.JavaMail.zimbra@enovance.com>
@ 2014-10-08 14:14 ` victor stinner
2014-10-08 21:23 ` Dave Chinner
0 siblings, 1 reply; 3+ messages in thread
From: victor stinner @ 2014-10-08 14:14 UTC (permalink / raw)
To: xfs
[-- Attachment #1: Type: text/plain, Size: 33133 bytes --]
Hello,
I'm working on OpenStack, and we hit a bug on Swift (distributed storage). The Linux kernel 3.2 logged many "soft lockup" messages which looks to be related to XFS: see kernel messages at the end of this email (it's only an extract of first messages, there are more later).
I'm trying to find with google if the bug is known or not. "soft lockup" looks to be a "common" bug on Linux, but it's hard (for me) to check if existing bug reports are similar to our issue or not.
And how can I check if it's an hardware issue or a Linux kernel (XFS?) issue?
The following issues look similiar:
- http://oss.sgi.com/archives/xfs/2013-06/msg00865.html : Centos, Linux kernel 2.6.32
This email mentions two bugzilla issues, probably the following ones, but I'm not allowed to see them :-/
https://bugzilla.redhat.com/show_bug.cgi?id=896224
https://bugzilla.redhat.com/show_bug.cgi?id=921961
- https://retrace.fedoraproject.org/faf/reports/98448/ : Fedora 18, Linux kernel 3.8
- https://bugs.centos.org/view.php?id=6287 : Linux 2.6.32
- https://lkml.org/lkml/2013/10/16/263 : RC version of Linux (3.12.0-rc1), maybe not reliable
Some info about the server:
- Dell PowerEdge R720xd/0X6H47
- Ubuntu 12.04
- Linux kernel 3.2.0-51-generic
- Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz : 32 cores
- 2 NUMA nodes
There are 26 disks, HDD and SSD. All partitions used to store Swift objects are formatted with XFS, only a few system partitions are formatted as ext4 (and ext3 for /boot). There are 2 disk controllers: internal and a PCI Express controller.
Victor
Linux kernel messages:
BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 28
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293
RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960
RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8
RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac
R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50
R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8
FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00)
Stack:
ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548
ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000
00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 30
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff88041116bc20 EFLAGS: 00000297
RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231
RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78
RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000
R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88
FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00)
Stack:
ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4
ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe
ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 0
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293
RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003
RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78
RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b
FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00)
Stack:
ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b
ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500
ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
BUG: soft lockup - CPU#7 stuck for 23s! [swift-object-se:4340]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 7
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4340, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8803362a1498 EFLAGS: 00000297
RAX: 0000000000007c1c RBX: 0000000700000006 RCX: ffff8804186a7ce8
RDX: 0000000000007c1d RSI: ffff8804112c9700 RDI: ffff88082420fea8
RBP: ffff8803362a1498 R08: ffff8803362a15d0 R09: ffff8803362a15dc
R10: 000000000000121c R11: 0000000000000006 R12: ffffffffa01a6587
R13: ffff8803362a1428 R14: ffff8803362a1544 R15: 0000000000000000
FS: 00007f3db1ffb700(0000) GS:ffff88083fc60000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000002955f6000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4340, threadinfo ffff8803362a0000, task ffff8804112c9700)
Stack:
ffff8803362a14a8 ffffffff816606be ffff8803362a1548 ffffffffa01cc548
ffff8803362a15dc ffff8803362a15d0 ffff8803362a15c0 000013248103ec59
00000000000144c0 ffff8804112c9700 0000000000000286 ffff880556253d78
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff810918a2>] ? up+0x32/0x50
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs]
[<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs]
[<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs]
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs]
[<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs]
[<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs]
[<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs]
[<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs]
[<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs]
[<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs]
[<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs]
[<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs]
[<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs]
[<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs]
[<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90
[<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0
[<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10
[<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0
[<ffffffff8119e026>] setxattr+0xc6/0x120
[<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0
[<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0
[<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff810918a2>] ? up+0x32/0x50
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs]
[<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs]
[<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs]
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs]
[<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs]
[<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs]
[<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs]
[<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs]
[<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs]
[<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs]
[<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs]
[<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs]
[<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs]
[<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs]
[<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90
[<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0
[<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10
[<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0
[<ffffffff8119e026>] setxattr+0xc6/0x120
[<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0
[<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0
[<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#9 stuck for 23s! [swift-object-se:4362]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 9
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4362, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8803df863b68 EFLAGS: 00000293
RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff880418632708
RDX: 0000000000007c1f RSI: ffff880110e99700 RDI: ffff88082420fea8
RBP: ffff8803df863b68 R08: ffff8803df863ca0 R09: ffff8803df863cac
R10: 00000000000021c4 R11: 000000000000000a R12: ffff8803df863b50
R13: ffff88082420fe00 R14: ffff880808307a40 R15: ffff8803df863ae8
FS: 00007f3d6f7fe700(0000) GS:ffff88083fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4362, threadinfo ffff8803df862000, task ffff880110e99700)
Stack:
ffff8803df863b78 ffffffff816606be ffff8803df863c18 ffffffffa01cc548
ffff8803df863cac ffff8803df863ca0 ffff8803df863c90 000022e4244506c0
00000000000144c0 ffff880110e99700 ffff8803df863df5 000000000000000e
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 28
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293
RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960
RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8
RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac
R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50
R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8
FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00)
Stack:
ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548
ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000
00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 30
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff88041116bc20 EFLAGS: 00000297
RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231
RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78
RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000
R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88
FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00)
Stack:
ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4
ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe
ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 0
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc2>] [<ffffffff8103ebc2>] __ticket_spin_lock+0x22/0x30
RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293
RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003
RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78
RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b
FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00)
Stack:
ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b
ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500
ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
[-- Attachment #2: soft_lockup.txt --]
[-- Type: text/plain, Size: 32044 bytes --]
BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 28
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293
RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960
RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8
RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac
R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50
R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8
FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00)
Stack:
ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548
ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000
00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 30
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff88041116bc20 EFLAGS: 00000297
RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231
RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78
RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000
R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88
FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00)
Stack:
ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4
ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe
ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 0
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293
RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003
RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78
RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b
FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00)
Stack:
ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b
ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500
ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
BUG: soft lockup - CPU#7 stuck for 23s! [swift-object-se:4340]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 7
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4340, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8803362a1498 EFLAGS: 00000297
RAX: 0000000000007c1c RBX: 0000000700000006 RCX: ffff8804186a7ce8
RDX: 0000000000007c1d RSI: ffff8804112c9700 RDI: ffff88082420fea8
RBP: ffff8803362a1498 R08: ffff8803362a15d0 R09: ffff8803362a15dc
R10: 000000000000121c R11: 0000000000000006 R12: ffffffffa01a6587
R13: ffff8803362a1428 R14: ffff8803362a1544 R15: 0000000000000000
FS: 00007f3db1ffb700(0000) GS:ffff88083fc60000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000002955f6000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4340, threadinfo ffff8803362a0000, task ffff8804112c9700)
Stack:
ffff8803362a14a8 ffffffff816606be ffff8803362a1548 ffffffffa01cc548
ffff8803362a15dc ffff8803362a15d0 ffff8803362a15c0 000013248103ec59
00000000000144c0 ffff8804112c9700 0000000000000286 ffff880556253d78
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff810918a2>] ? up+0x32/0x50
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs]
[<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs]
[<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs]
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs]
[<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs]
[<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs]
[<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs]
[<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs]
[<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs]
[<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs]
[<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs]
[<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs]
[<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs]
[<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs]
[<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90
[<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0
[<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10
[<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0
[<ffffffff8119e026>] setxattr+0xc6/0x120
[<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0
[<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0
[<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff810918a2>] ? up+0x32/0x50
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs]
[<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs]
[<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs]
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs]
[<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs]
[<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs]
[<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs]
[<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs]
[<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs]
[<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs]
[<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs]
[<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs]
[<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs]
[<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs]
[<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90
[<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0
[<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10
[<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0
[<ffffffff8119e026>] setxattr+0xc6/0x120
[<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0
[<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0
[<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#9 stuck for 23s! [swift-object-se:4362]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 9
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4362, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8803df863b68 EFLAGS: 00000293
RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff880418632708
RDX: 0000000000007c1f RSI: ffff880110e99700 RDI: ffff88082420fea8
RBP: ffff8803df863b68 R08: ffff8803df863ca0 R09: ffff8803df863cac
R10: 00000000000021c4 R11: 000000000000000a R12: ffff8803df863b50
R13: ffff88082420fe00 R14: ffff880808307a40 R15: ffff8803df863ae8
FS: 00007f3d6f7fe700(0000) GS:ffff88083fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4362, threadinfo ffff8803df862000, task ffff880110e99700)
Stack:
ffff8803df863b78 ffffffff816606be ffff8803df863c18 ffffffffa01cc548
ffff8803df863cac ffff8803df863ca0 ffff8803df863c90 000022e4244506c0
00000000000144c0 ffff880110e99700 ffff8803df863df5 000000000000000e
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 28
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293
RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960
RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8
RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac
R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50
R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8
FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00)
Stack:
ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548
ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000
00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs]
[<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140
[<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs]
[<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs]
[<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs]
[<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs]
[<ffffffff8119e031>] ? setxattr+0xd1/0x120
[<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs]
[<ffffffff8165f986>] ? down_read+0x16/0x2b
[<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs]
[<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs]
[<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs]
[<ffffffff811a8c56>] do_fsync+0x56/0x80
[<ffffffff811a8f80>] sys_fsync+0x10/0x20
[<ffffffff81668d02>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 30
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30
RSP: 0018:ffff88041116bc20 EFLAGS: 00000297
RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231
RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78
RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000
R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88
FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00)
Stack:
ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4
ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe
ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs]
[<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs]
[<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs]
[<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs]
[<ffffffff810916d6>] ? down_trylock+0x36/0x50
[<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs]
[<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs]
[<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs]
[<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs]
[<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs]
[<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs]
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580]
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
CPU 0
Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si]
Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47
RIP: 0010:[<ffffffff8103ebc2>] [<ffffffff8103ebc2>] __ticket_spin_lock+0x22/0x30
RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293
RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003
RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78
RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b
FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00)
Stack:
ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b
ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500
ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89
Call Trace:
[<ffffffff816606be>] _raw_spin_lock+0xe/0x20
[<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs]
[<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs]
[<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs]
[<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs]
[<ffffffff81085e37>] process_one_work+0x127/0x470
[<ffffffff81086f44>] worker_thread+0x164/0x370
[<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130
[<ffffffff8108b78c>] kthread+0x8c/0xa0
[<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
[<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
[<ffffffff8166ae70>] ? gs_change+0x13/0x13
[-- Attachment #3: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 3+ messages in thread