* [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] @ 2014-06-16 6:35 Tsutomu Itoh 2014-06-16 17:56 ` Chris Mason 0 siblings, 1 reply; 13+ messages in thread From: Tsutomu Itoh @ 2014-06-16 6:35 UTC (permalink / raw) To: Linux Btrfs I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. ================ [ 1121.983975] BTRFS: device fsid 8cc641f9-563a-4d62-9157-9503156ce62c devid 1 transid 4 /dev/sdc5 [ 1121.987994] BTRFS info (device sdc5): disk space caching is enabled [ 1121.987999] BTRFS: flagging fs with big metadata feature [ 1121.996312] BTRFS: creating UUID tree [ 1122.050762] BTRFS info (device sdc5): qgroup scan completed [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] [ 1147.949658] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1147.949695] CPU: 0 PID: 5189 Comm: kworker/u25:4 Not tainted 3.16.0-rc1 #1 [ 1147.949696] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1147.949717] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1147.949718] task: ffff88007408b840 ti: ffff8800740e4000 task.ti: ffff8800740e4000 [ 1147.949720] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1147.949725] RSP: 0018:ffff8800740e7b08 EFLAGS: 00000202 [ 1147.949726] RAX: 00000000000000ee RBX: 0000000000000012 RCX: 0000000000000001 [ 1147.949727] RDX: 00000000000000ef RSI: 0000000000000004 RDI: ffff88003db221b0 [ 1147.949728] RBP: ffff8800740e7b08 R08: 0000000000000001 R09: ffff880071486b58 [ 1147.949729] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800715d1900 [ 1147.949730] R13: 0000000000000000 R14: 0000000000000003 R15: 0000000000000002 [ 1147.949732] FS: 0000000000000000(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000 [ 1147.949733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1147.949734] CR2: 00000036c97b9490 CR3: 0000000001c0e000 CR4: 00000000000007f0 [ 1147.949735] Stack: [ 1147.949736] ffff8800740e7b18 ffffffff8166bcbc ffff8800740e7b38 ffffffffa070ffe4 [ 1147.949738] 0000000000000001 ffffffffffffffe8 ffff8800740e7b58 ffffffffa06b0c2a [ 1147.949740] ffff8800719de360 0000000000000000 ffff8800740e7bf8 ffffffffa06b5e32 [ 1147.949742] Call Trace: [ 1147.949748] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1147.949760] [<ffffffffa070ffe4>] btrfs_clear_lock_blocking_rw+0x44/0x150 [btrfs] [ 1147.949767] [<ffffffffa06b0c2a>] btrfs_clear_path_blocking+0x3a/0x80 [btrfs] [ 1147.949774] [<ffffffffa06b5e32>] btrfs_search_slot+0x702/0xa10 [btrfs] [ 1147.949783] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1147.949792] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1147.949798] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1147.949809] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1147.949819] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1147.949830] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1147.949841] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1147.949844] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1147.949846] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1147.949848] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1147.949851] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1147.949853] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.949855] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1147.949857] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.949858] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-16 6:35 [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] Tsutomu Itoh @ 2014-06-16 17:56 ` Chris Mason 2014-06-16 23:28 ` Tsutomu Itoh 0 siblings, 1 reply; 13+ messages in thread From: Chris Mason @ 2014-06-16 17:56 UTC (permalink / raw) To: Tsutomu Itoh, Linux Btrfs On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: > I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. > Did we recover, or was it stuck forever? -chris ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-16 17:56 ` Chris Mason @ 2014-06-16 23:28 ` Tsutomu Itoh 2014-06-16 23:52 ` Chris Mason 0 siblings, 1 reply; 13+ messages in thread From: Tsutomu Itoh @ 2014-06-16 23:28 UTC (permalink / raw) To: Chris Mason; +Cc: Linux Btrfs Hi Chris, On 2014/06/17 2:56, Chris Mason wrote: > On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >> > > Did we recover, or was it stuck forever? The following messages are repeatedly output. And stuck forever. [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] Thanks, Tsutomu ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-16 23:28 ` Tsutomu Itoh @ 2014-06-16 23:52 ` Chris Mason 2014-06-16 23:57 ` Tsutomu Itoh 0 siblings, 1 reply; 13+ messages in thread From: Chris Mason @ 2014-06-16 23:52 UTC (permalink / raw) To: Tsutomu Itoh; +Cc: Linux Btrfs On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: > Hi Chris, > > On 2014/06/17 2:56, Chris Mason wrote: >> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>> >> >> Did we recover, or was it stuck forever? > > The following messages are repeatedly output. > And stuck forever. > > [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] > [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] > [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] > [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] > [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] Can you please capture a stack trace from all the cpus? -chris ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-16 23:52 ` Chris Mason @ 2014-06-16 23:57 ` Tsutomu Itoh 2014-06-17 0:47 ` Chris Mason 2014-06-17 1:11 ` Chris Mason 0 siblings, 2 replies; 13+ messages in thread From: Tsutomu Itoh @ 2014-06-16 23:57 UTC (permalink / raw) To: Chris Mason; +Cc: Linux Btrfs On 2014/06/17 8:52, Chris Mason wrote: > On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >> Hi Chris, >> >> On 2014/06/17 2:56, Chris Mason wrote: >>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>> >>> >>> Did we recover, or was it stuck forever? >> >> The following messages are repeatedly output. >> And stuck forever. >> >> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] > > Can you please capture a stack trace from all the cpus? > crash> log ... [ 1121.996312] BTRFS: creating UUID tree [ 1122.050762] BTRFS info (device sdc5): qgroup scan completed [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] [ 1147.949658] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1147.949695] CPU: 0 PID: 5189 Comm: kworker/u25:4 Not tainted 3.16.0-rc1 #1 [ 1147.949696] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1147.949717] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1147.949718] task: ffff88007408b840 ti: ffff8800740e4000 task.ti: ffff8800740e4000 [ 1147.949720] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1147.949725] RSP: 0018:ffff8800740e7b08 EFLAGS: 00000202 [ 1147.949726] RAX: 00000000000000ee RBX: 0000000000000012 RCX: 0000000000000001 [ 1147.949727] RDX: 00000000000000ef RSI: 0000000000000004 RDI: ffff88003db221b0 [ 1147.949728] RBP: ffff8800740e7b08 R08: 0000000000000001 R09: ffff880071486b58 [ 1147.949729] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800715d1900 [ 1147.949730] R13: 0000000000000000 R14: 0000000000000003 R15: 0000000000000002 [ 1147.949732] FS: 0000000000000000(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000 [ 1147.949733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1147.949734] CR2: 00000036c97b9490 CR3: 0000000001c0e000 CR4: 00000000000007f0 [ 1147.949735] Stack: [ 1147.949736] ffff8800740e7b18 ffffffff8166bcbc ffff8800740e7b38 ffffffffa070ffe4 [ 1147.949738] 0000000000000001 ffffffffffffffe8 ffff8800740e7b58 ffffffffa06b0c2a [ 1147.949740] ffff8800719de360 0000000000000000 ffff8800740e7bf8 ffffffffa06b5e32 [ 1147.949742] Call Trace: [ 1147.949748] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1147.949760] [<ffffffffa070ffe4>] btrfs_clear_lock_blocking_rw+0x44/0x150 [btrfs] [ 1147.949767] [<ffffffffa06b0c2a>] btrfs_clear_path_blocking+0x3a/0x80 [btrfs] [ 1147.949774] [<ffffffffa06b5e32>] btrfs_search_slot+0x702/0xa10 [btrfs] [ 1147.949783] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1147.949792] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1147.949798] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1147.949809] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1147.949819] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1147.949830] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1147.949841] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1147.949844] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1147.949846] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1147.949848] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1147.949851] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1147.949853] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.949855] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1147.949857] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.949858] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1147.949877] Kernel panic - not syncing: softlockup: hung tasks [ 1147.956981] CPU: 0 PID: 5189 Comm: kworker/u25:4 Not tainted 3.16.0-rc1 #1 [ 1147.965228] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] [ 1147.967194] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1147.967195] CPU: 3 PID: 5194 Comm: kworker/u25:9 Not tainted 3.16.0-rc1 #1 [ 1147.967196] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1147.967209] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1147.967210] task: ffff88007408aee0 ti: ffff88007ba88000 task.ti: ffff88007ba88000 [ 1147.967212] RIP: 0010:[<ffffffff810ba7c2>] [<ffffffff810ba7c2>] queue_write_lock_slowpath+0x62/0x90 [ 1147.967213] RSP: 0018:ffff88007ba8bae0 EFLAGS: 00000206 [ 1147.967214] RAX: 0000000000000101 RBX: 0000000000000000 RCX: 0000000000000101 [ 1147.967214] RDX: 00000000000000ff RSI: 0000000000000202 RDI: ffff88003db221b0 [ 1147.967215] RBP: ffff88007ba8bae0 R08: ffff88007ba8bb18 R09: 0000000000000001 [ 1147.967216] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88007dcd2e80 [ 1147.967216] R13: 0000000000000000 R14: ffff88007408b3d0 R15: 0000000300000000 [ 1147.967217] FS: 0000000000000000(0000) GS:ffff88007dcc0000(0000) knlGS:0000000000000000 [ 1147.967218] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1147.967218] CR2: 000000000136e000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1147.967219] Stack: [ 1147.967220] ffff88007ba8baf0 ffffffff8166bc51 ffff88007ba8bb50 ffffffffa0710519 [ 1147.967221] 0000000000000000 ffff88007408aee0 ffffffff810b4840 ffff88007ba8bb18 [ 1147.967222] ffff88007ba8bb18 ffff88003db22140 ffff880072435000 ffff880000000000 [ 1147.967223] Call Trace: [ 1147.967226] [<ffffffff8166bc51>] _raw_write_lock+0x21/0x30 [ 1147.967237] [<ffffffffa0710519>] btrfs_tree_lock+0xc9/0x1d0 [btrfs] [ 1147.967240] [<ffffffff810b4840>] ? abort_exclusive_wait+0xb0/0xb0 [ 1147.967246] [<ffffffffa06b0e3b>] btrfs_lock_root_node+0x3b/0x50 [btrfs] [ 1147.967252] [<ffffffffa06b6029>] btrfs_search_slot+0x8f9/0xa10 [btrfs] [ 1147.967258] [<ffffffffa06b0ba9>] ? btrfs_set_path_blocking+0x39/0x80 [btrfs] [ 1147.967265] [<ffffffffa06b7970>] btrfs_insert_empty_items+0x70/0xc0 [btrfs] [ 1147.967273] [<ffffffffa06cd3b9>] btrfs_csum_file_blocks+0x2d9/0x680 [btrfs] [ 1147.967279] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1147.967289] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1147.967298] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1147.967300] [<ffffffff810a702f>] ? set_next_entity+0x5f/0x80 [ 1147.967310] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1147.967321] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1147.967322] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1147.967323] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1147.967325] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1147.967326] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1147.967328] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.967329] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1147.967331] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.967342] Code: 2e 0f 1f 84 00 00 00 00 00 f3 90 8b 17 84 d2 75 f8 89 d1 89 d0 83 c9 01 f0 0f b1 0f 39 d0 75 e9 ba ff 00 00 00 eb 04 66 90 f3 90 <8b> 07 83 f8 01 75 f7 f0 0f b1 17 83 f8 01 75 ee eb bc f3 90 0f [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] [ 1147.979190] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1147.979191] CPU: 4 PID: 5200 Comm: kworker/u25:15 Not tainted 3.16.0-rc1 #1 [ 1147.979192] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1147.979204] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1147.979205] task: ffff88007bc4b840 ti: ffff8800755a0000 task.ti: ffff8800755a0000 [ 1147.979207] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1147.979208] RSP: 0018:ffff8800755a3ac8 EFLAGS: 00000212 [ 1147.979209] RAX: 00000000000000ee RBX: 0000000000000000 RCX: 000000000000000d [ 1147.979209] RDX: 00000000000000f6 RSI: 000000000000000d RDI: ffff88003db221b0 [ 1147.979210] RBP: ffff8800755a3ac8 R08: ffff88003db22174 R09: 0000000000000000 [ 1147.979210] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1147.979211] R13: 0000000000000000 R14: ffff88007b68f000 R15: ffffffffa06b60a1 [ 1147.979212] FS: 0000000000000000(0000) GS:ffff88007dd00000(0000) knlGS:0000000000000000 [ 1147.979213] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1147.979213] CR2: 00000000014b6000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1147.979214] Stack: [ 1147.979215] ffff8800755a3ad8 ffffffff8166bcbc ffff8800755a3b38 ffffffffa0710140 [ 1147.979216] 0000000000000084 0000000000000035 0000000000000000 ffff8800755a3b58 [ 1147.979217] ffffffffa06ae81f ffff88003db22140 ffff880072435000 ffff880000000000 [ 1147.979218] Call Trace: [ 1147.979220] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1147.979231] [<ffffffffa0710140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1147.979237] [<ffffffffa06ae81f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1147.979243] [<ffffffffa06b0deb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1147.979249] [<ffffffffa06b5c3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1147.979260] [<ffffffffa0720ee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1147.979268] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1147.979277] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1147.979283] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1147.979292] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1147.979302] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1147.979304] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0 [ 1147.979314] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1147.979325] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1147.979327] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1147.979328] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1147.979329] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1147.979331] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1147.979332] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.979334] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1147.979335] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.979346] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] [ 1147.991187] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1147.991188] CPU: 5 PID: 5192 Comm: kworker/u25:7 Not tainted 3.16.0-rc1 #1 [ 1147.991189] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1147.991201] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1147.991202] task: ffff8800740892c0 ti: ffff880036238000 task.ti: ffff880036238000 [ 1147.991204] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1147.991205] RSP: 0018:ffff88003623bac8 EFLAGS: 00000216 [ 1147.991206] RAX: 00000000000000ee RBX: 0000000000000000 RCX: 0000000000000008 [ 1147.991206] RDX: 00000000000000f1 RSI: 0000000000000008 RDI: ffff88003db221b0 [ 1147.991207] RBP: ffff88003623bac8 R08: ffff88003db22174 R09: 0000000000000000 [ 1147.991208] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1147.991208] R13: 0000000000000000 R14: ffff880037e1f000 R15: ffffffffa06b60a1 [ 1147.991209] FS: 0000000000000000(0000) GS:ffff88007dd40000(0000) knlGS:0000000000000000 [ 1147.991210] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1147.991210] CR2: 00000036c97b9490 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1147.991211] Stack: [ 1147.991212] ffff88003623bad8 ffffffff8166bcbc ffff88003623bb38 ffffffffa0710140 [ 1147.991213] 0000000000000084 0000000000000035 0000000000000000 ffff88003623bb58 [ 1147.991214] ffffffffa06ae81f ffff88003db22140 ffff880072435000 ffff880000000000 [ 1147.991215] Call Trace: [ 1147.991217] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1147.991228] [<ffffffffa0710140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1147.991234] [<ffffffffa06ae81f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1147.991240] [<ffffffffa06b0deb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1147.991246] [<ffffffffa06b5c3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1147.991257] [<ffffffffa0720ee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1147.991265] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1147.991274] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1147.991280] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1147.991289] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1147.991299] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1147.991301] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0 [ 1147.991310] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1147.991321] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1147.991323] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1147.991324] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1147.991325] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1147.991327] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1147.991328] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.991330] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1147.991331] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1147.991342] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] [ 1148.064174] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1148.064176] CPU: 6 PID: 3182 Comm: kworker/u26:3 Not tainted 3.16.0-rc1 #1 [ 1148.064176] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1148.064192] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1148.064193] task: ffff880132520960 ti: ffff8801385b0000 task.ti: ffff8801385b0000 [ 1148.064197] RIP: 0010:[<ffffffff810ba85c>] [<ffffffff810ba85c>] queue_read_lock_slowpath+0x6c/0x90 [ 1148.064198] RSP: 0018:ffff8801385b3ac8 EFLAGS: 00000212 [ 1148.064198] RAX: 00000000000000ee RBX: 0000000000000000 RCX: 000000000000000c [ 1148.064199] RDX: 00000000000000f5 RSI: 000000000000000c RDI: ffff88003db221b0 [ 1148.064199] RBP: ffff8801385b3ac8 R08: ffff88003db22174 R09: 0000000000000000 [ 1148.064200] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1148.064200] R13: 0000000000000000 R14: ffff8800b7e1e000 R15: ffffffffa06b60a1 [ 1148.064201] FS: 0000000000000000(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000 [ 1148.064202] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1148.064203] CR2: 00000036cac4b8d8 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1148.064203] Stack: [ 1148.064204] ffff8801385b3ad8 ffffffff8166bcbc ffff8801385b3b38 ffffffffa0710140 [ 1148.064205] 0000000000000085 0000000000000035 0000000000000000 ffff8801385b3b58 [ 1148.064206] ffffffffa06ae81f ffff88003db22140 ffff880072435000 ffff880000000000 [ 1148.064207] Call Trace: [ 1148.064211] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1148.064221] [<ffffffffa0710140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1148.064226] [<ffffffffa06ae81f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1148.064231] [<ffffffffa06b0deb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1148.064236] [<ffffffffa06b5c3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1148.064245] [<ffffffffa0720ee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1148.064252] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1148.064259] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1148.064264] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1148.064272] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1148.064280] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1148.064283] [<ffffffff810a702f>] ? set_next_entity+0x5f/0x80 [ 1148.064291] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1148.064300] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1148.064302] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1148.064303] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1148.064304] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1148.064305] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1148.064306] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.064308] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1148.064309] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.064318] Code: 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 0f b6 47 04 <38> c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c ff 74 f8 5d [ 1148.076150] BUG: soft lockup - CPU#7 stuck for 23s! [kworker/u26:8:30979] [ 1148.076165] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1148.076166] CPU: 7 PID: 30979 Comm: kworker/u26:8 Not tainted 3.16.0-rc1 #1 [ 1148.076167] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1148.076177] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1148.076178] task: ffff88013c030000 ti: ffff88013465c000 task.ti: ffff88013465c000 [ 1148.076180] RIP: 0010:[<ffffffff810ba85c>] [<ffffffff810ba85c>] queue_read_lock_slowpath+0x6c/0x90 [ 1148.076181] RSP: 0018:ffff88013465fac8 EFLAGS: 00000216 [ 1148.076181] RAX: 00000000000000ee RBX: 0000000000000000 RCX: 000000000000000b [ 1148.076182] RDX: 00000000000000f4 RSI: 000000000000000b RDI: ffff88003db221b0 [ 1148.076182] RBP: ffff88013465fac8 R08: ffff88003db22174 R09: 0000000000000000 [ 1148.076183] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1148.076183] R13: 0000000000000000 R14: ffff8800b4ba1000 R15: ffffffffa06b60a1 [ 1148.076184] FS: 0000000000000000(0000) GS:ffff88013fc40000(0000) knlGS:0000000000000000 [ 1148.076185] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1148.076185] CR2: 0000000000d36000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1148.076186] Stack: [ 1148.076187] ffff88013465fad8 ffffffff8166bcbc ffff88013465fb38 ffffffffa0710140 [ 1148.076188] 0000000000000088 0000000000000035 0000000000000000 ffff88013465fb58 [ 1148.076189] ffffffffa06ae81f ffff88003db22140 ffff880072435000 ffff880000000000 [ 1148.076189] Call Trace: [ 1148.076191] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1148.076200] [<ffffffffa0710140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1148.076205] [<ffffffffa06ae81f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1148.076210] [<ffffffffa06b0deb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1148.076215] [<ffffffffa06b5c3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1148.076224] [<ffffffffa0720ee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1148.076231] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1148.076238] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1148.076243] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1148.076251] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1148.076259] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1148.076268] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1148.076277] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1148.076278] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1148.076279] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1148.076280] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1148.076281] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1148.076283] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.076284] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1148.076285] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.076295] Code: 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 0f b6 47 04 <38> c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c ff 74 f8 5d [ 1148.088148] BUG: soft lockup - CPU#8 stuck for 23s! [kworker/u26:4:3183] [ 1148.088162] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1148.088164] CPU: 8 PID: 3183 Comm: kworker/u26:4 Not tainted 3.16.0-rc1 #1 [ 1148.088164] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1148.088175] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1148.088175] task: ffff8801325212c0 ti: ffff88013bdf0000 task.ti: ffff88013bdf0000 [ 1148.088178] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1148.088178] RSP: 0018:ffff88013bdf3ac8 EFLAGS: 00000212 [ 1148.088179] RAX: 00000000000000ee RBX: 0000000000000000 RCX: 0000000000000007 [ 1148.088180] RDX: 00000000000000f0 RSI: 0000000000000007 RDI: ffff88003db221b0 [ 1148.088180] RBP: ffff88013bdf3ac8 R08: ffff88003db22174 R09: 0000000000000000 [ 1148.088181] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1148.088181] R13: 0000000000000000 R14: ffff8800b4ba1000 R15: ffffffffa06b60a1 [ 1148.088182] FS: 0000000000000000(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000 [ 1148.088183] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1148.088183] CR2: 0000000000cb9000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1148.088184] Stack: [ 1148.088185] ffff88013bdf3ad8 ffffffff8166bcbc ffff88013bdf3b38 ffffffffa0710140 [ 1148.088186] 0000000000000087 0000000000000035 0000000000000000 ffff88013bdf3b58 [ 1148.088187] ffffffffa06ae81f ffff88003db22140 ffff880072435000 ffff880000000000 [ 1148.088187] Call Trace: [ 1148.088189] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1148.088199] [<ffffffffa0710140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1148.088203] [<ffffffffa06ae81f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1148.088208] [<ffffffffa06b0deb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1148.088214] [<ffffffffa06b5c3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1148.088223] [<ffffffffa0720ee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1148.088229] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1148.088236] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1148.088241] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1148.088249] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1148.088257] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1148.088266] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1148.088275] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1148.088276] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1148.088277] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1148.088278] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1148.088280] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1148.088281] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.088282] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1148.088284] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.088293] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1148.100145] BUG: soft lockup - CPU#9 stuck for 23s! [kworker/u26:16:23157] [ 1148.100160] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1148.100161] CPU: 9 PID: 23157 Comm: kworker/u26:16 Not tainted 3.16.0-rc1 #1 [ 1148.100162] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1148.100172] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1148.100173] task: ffff8800b795e720 ti: ffff8800b44fc000 task.ti: ffff8800b44fc000 [ 1148.100175] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1148.100176] RSP: 0018:ffff8800b44ffac8 EFLAGS: 00000212 [ 1148.100176] RAX: 00000000000000ee RBX: ffff88007d84bf00 RCX: 0000000000000009 [ 1148.100177] RDX: 00000000000000f2 RSI: 0000000000000009 RDI: ffff88003db221b0 [ 1148.100177] RBP: ffff8800b44ffac8 R08: ffff88003db22174 R09: 0000000000000000 [ 1148.100178] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800b795e720 [ 1148.100178] R13: ffffffff8119da74 R14: ffff8800b44ffa68 R15: ffff8800b795e720 [ 1148.100179] FS: 0000000000000000(0000) GS:ffff88013fcc0000(0000) knlGS:0000000000000000 [ 1148.100180] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1148.100180] CR2: 000000000243b000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1148.100180] Stack: [ 1148.100182] ffff8800b44ffad8 ffffffff8166bcbc ffff8800b44ffb38 ffffffffa0710140 [ 1148.100182] ffffffff00000000 ffffe8ff00000000 ffffffffa0720e54 0000005000000002 [ 1148.100183] 0000000000000296 ffff88003db22140 ffff880072435000 ffff880000000000 [ 1148.100184] Call Trace: [ 1148.100186] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1148.100195] [<ffffffffa0710140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1148.100204] [<ffffffffa0720e54>] ? btrfs_add_delayed_data_ref+0x84/0x180 [btrfs] [ 1148.100209] [<ffffffffa06b0deb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1148.100215] [<ffffffffa06b5c3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1148.100224] [<ffffffffa0720ee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1148.100230] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1148.100237] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1148.100242] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1148.100250] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1148.100258] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1148.100260] [<ffffffff810a702f>] ? set_next_entity+0x5f/0x80 [ 1148.100268] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1148.100277] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1148.100278] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1148.100279] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1148.100280] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1148.100282] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1148.100283] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.100284] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1148.100285] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.100295] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1148.113142] BUG: soft lockup - CPU#10 stuck for 23s! [kworker/u26:5:13041] [ 1148.113156] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1148.113158] CPU: 10 PID: 13041 Comm: kworker/u26:5 Not tainted 3.16.0-rc1 #1 [ 1148.113158] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1148.113169] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1148.113169] task: ffff8800ba7bcb00 ti: ffff8800b58a8000 task.ti: ffff8800b58a8000 [ 1148.113172] RIP: 0010:[<ffffffff810ba832>] [<ffffffff810ba832>] queue_read_lock_slowpath+0x42/0x90 [ 1148.113172] RSP: 0018:ffff8800b58abae0 EFLAGS: 00000286 [ 1148.113173] RAX: 00000000000000ff RBX: ffff8800b58abb80 RCX: 0000000000000001 [ 1148.113174] RDX: 0000000000000015 RSI: 00000000b98af000 RDI: ffff8800715d1970 [ 1148.113174] RBP: ffff8800b58abae0 R08: 0000000000000001 R09: ffff880071486b58 [ 1148.113174] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa06d2f56 [ 1148.113175] R13: ffff8800b58aba78 R14: 0000000000000001 R15: 0000000000000012 [ 1148.113176] FS: 0000000000000000(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000 [ 1148.113176] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1148.113177] CR2: 0000000001945000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1148.113177] Stack: [ 1148.113178] ffff8800b58abaf0 ffffffff8166bcbc ffff8800b58abb10 ffffffffa0710221 [ 1148.113179] ffff8800b658f750 0000000000000000 ffff8800b58abbb0 ffffffffa06b5c95 [ 1148.113180] ffff8800b658f750 0000000000000000 0000000000000000 ffff8800b58abb58 [ 1148.113180] Call Trace: [ 1148.113182] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1148.113192] [<ffffffffa0710221>] btrfs_try_tree_read_lock+0x31/0x60 [btrfs] [ 1148.113197] [<ffffffffa06b5c95>] btrfs_search_slot+0x565/0xa10 [btrfs] [ 1148.113203] [<ffffffffa06b8eb8>] btrfs_next_old_leaf+0x218/0x4b0 [btrfs] [ 1148.113209] [<ffffffffa06b9160>] btrfs_next_leaf+0x10/0x20 [btrfs] [ 1148.113215] [<ffffffffa06cd44b>] btrfs_csum_file_blocks+0x36b/0x680 [btrfs] [ 1148.113220] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1148.113228] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1148.113237] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1148.113245] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1148.113254] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1148.113255] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1148.113256] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1148.113257] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1148.113258] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1148.113260] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.113261] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1148.113263] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.113272] Code: 5a f0 81 2f 00 01 00 00 b8 00 01 00 00 f0 66 0f c1 47 04 0f b6 d4 38 c2 75 37 8b 07 84 c0 74 11 66 0f 1f 84 00 00 00 00 00 f3 90 <8b> 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f [ 1148.125139] BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u26:6:30977] [ 1148.125154] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix [ 1148.125156] CPU: 11 PID: 30977 Comm: kworker/u26:6 Not tainted 3.16.0-rc1 #1 [ 1148.125156] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1148.125167] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1148.125167] task: ffff88013c032ee0 ti: ffff8801362fc000 task.ti: ffff8801362fc000 [ 1148.125170] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1148.125170] RSP: 0018:ffff8801362ffac8 EFLAGS: 00000216 [ 1148.125171] RAX: 00000000000000ee RBX: 0000000000000000 RCX: 000000000000000a [ 1148.125171] RDX: 00000000000000f3 RSI: 000000000000000a RDI: ffff88003db221b0 [ 1148.125172] RBP: ffff8801362ffac8 R08: ffff88003db22174 R09: 0000000000000000 [ 1148.125172] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1148.125173] R13: 0000000000000000 R14: ffff8800b593e000 R15: ffffffffa06b60a1 [ 1148.125173] FS: 0000000000000000(0000) GS:ffff88013fd40000(0000) knlGS:0000000000000000 [ 1148.125174] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1148.125175] CR2: 000000000131e000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1148.125175] Stack: [ 1148.125176] ffff8801362ffad8 ffffffff8166bcbc ffff8801362ffb38 ffffffffa0710140 [ 1148.125177] 0000000000000087 0000000000000035 0000000000000000 ffff8801362ffb58 [ 1148.125178] ffffffffa06ae81f ffff88003db22140 ffff880072435000 ffff880000000000 [ 1148.125178] Call Trace: [ 1148.125180] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1148.125190] [<ffffffffa0710140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1148.125194] [<ffffffffa06ae81f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1148.125199] [<ffffffffa06b0deb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1148.125205] [<ffffffffa06b5c3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1148.125214] [<ffffffffa0720ee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1148.125220] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1148.125227] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1148.125232] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1148.125240] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1148.125249] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1148.125251] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0 [ 1148.125259] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1148.125268] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1148.125269] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1148.125270] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1148.125271] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1148.125272] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1148.125274] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.125275] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1148.125276] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1148.125285] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1151.904945] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1151.912879] 0000000000000416 ffff88007dc03e28 ffffffff81664f20 ffffffff81a1008f [ 1151.922093] ffff88007dc03ea0 ffffffff81660ab5 0000000000000008 ffff88007dc03eb0 [ 1151.931277] ffff88007dc03e50 00000000000000c5 0000000000000046 0000000000000007 [ 1151.940446] Call Trace: [ 1151.944023] <IRQ> [<ffffffff81664f20>] dump_stack+0x45/0x56 [ 1151.951312] [<ffffffff81660ab5>] panic+0xc8/0x1fc [ 1151.957504] [<ffffffff81110605>] watchdog_timer_fn+0x165/0x170 [ 1151.964952] [<ffffffff810949d5>] __run_hrtimer+0x75/0x1d0 [ 1151.971894] [<ffffffff811104a0>] ? watchdog_cleanup+0x10/0x10 [ 1151.979219] [<ffffffff81094dc7>] hrtimer_interrupt+0xf7/0x240 [ 1151.986525] [<ffffffff810328f7>] local_apic_timer_interrupt+0x37/0x60 [ 1151.994615] [<ffffffff8166e7cf>] smp_apic_timer_interrupt+0x3f/0x60 [ 1152.002514] [<ffffffff8166d16d>] apic_timer_interrupt+0x6d/0x80 [ 1152.010020] <EOI> [<ffffffff810ba858>] ? queue_read_lock_slowpath+0x68/0x90 [ 1152.018830] [<ffffffffa0710519>] ? btrfs_tree_lock+0xc9/0x1d0 [btrfs] [ 1152.026939] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1152.033890] [<ffffffffa070ffe4>] btrfs_clear_lock_blocking_rw+0x44/0x150 [btrfs] [ 1152.043070] [<ffffffffa06b0c2a>] btrfs_clear_path_blocking+0x3a/0x80 [btrfs] [ 1152.051856] [<ffffffffa06b5e32>] btrfs_search_slot+0x702/0xa10 [btrfs] [ 1152.060056] [<ffffffffa06cbe2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1152.068148] [<ffffffffa06cd19c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1152.076719] [<ffffffffa06b0d46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1152.084709] [<ffffffffa06db1e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1152.093472] [<ffffffffa06e1410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1152.102235] [<ffffffffa06e1785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1152.110222] [<ffffffffa0706b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1152.118392] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1152.125686] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1152.132686] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1152.139972] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1152.146182] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1152.153754] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1152.160547] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 crash> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-16 23:57 ` Tsutomu Itoh @ 2014-06-17 0:47 ` Chris Mason 2014-06-17 2:10 ` Tsutomu Itoh 2014-06-17 1:11 ` Chris Mason 1 sibling, 1 reply; 13+ messages in thread From: Chris Mason @ 2014-06-17 0:47 UTC (permalink / raw) To: Tsutomu Itoh; +Cc: Linux Btrfs On 06/16/2014 07:57 PM, Tsutomu Itoh wrote: > On 2014/06/17 8:52, Chris Mason wrote: >> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >>> Hi Chris, >>> >>> On 2014/06/17 2:56, Chris Mason wrote: >>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>>> >>>> >>>> Did we recover, or was it stuck forever? >>> >>> The following messages are repeatedly output. >>> And stuck forever. >>> >>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] >> >> Can you please capture a stack trace from all the cpus? Very strange, please try to reproduce again, I'll dig through things here. -chris ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-17 0:47 ` Chris Mason @ 2014-06-17 2:10 ` Tsutomu Itoh 2014-06-17 3:06 ` Tsutomu Itoh 0 siblings, 1 reply; 13+ messages in thread From: Tsutomu Itoh @ 2014-06-17 2:10 UTC (permalink / raw) To: Chris Mason; +Cc: Linux Btrfs On 2014/06/17 9:47, Chris Mason wrote: > On 06/16/2014 07:57 PM, Tsutomu Itoh wrote: >> On 2014/06/17 8:52, Chris Mason wrote: >>> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >>>> Hi Chris, >>>> >>>> On 2014/06/17 2:56, Chris Mason wrote: >>>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>>>> >>>>> >>>>> Did we recover, or was it stuck forever? >>>> >>>> The following messages are repeatedly output. >>>> And stuck forever. >>>> >>>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >>>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >>>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >>>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >>>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] >>> >>> Can you please capture a stack trace from all the cpus? > > Very strange, please try to reproduce again, I'll dig through things here. I can reproduce it easily in my environment. Thanks, Tsutomu ========== Message from syslogd@luna at Jun 17 11:06:33 ... kernel:[ 560.076282] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u25:1:193] Message from syslogd@luna at Jun 17 11:06:33 ... kernel:[ 560.076283] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u25:3:3174] Message from syslogd@luna at Jun 17 11:06:33 ... kernel:[ 560.101276] BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u25:4:3175] Message from syslogd@luna at Jun 17 11:06:33 ... kernel:[ 560.113274] BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u25:6:3177] Message from syslogd@luna at Jun 17 11:06:33 ... kernel:[ 560.125271] BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u25:5:3176] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-17 2:10 ` Tsutomu Itoh @ 2014-06-17 3:06 ` Tsutomu Itoh 2014-06-17 4:48 ` Chris Mason 2014-06-17 4:52 ` Tsutomu Itoh 0 siblings, 2 replies; 13+ messages in thread From: Tsutomu Itoh @ 2014-06-17 3:06 UTC (permalink / raw) To: Chris Mason; +Cc: Linux Btrfs On 2014/06/17 11:10, Tsutomu Itoh wrote: > On 2014/06/17 9:47, Chris Mason wrote: >> On 06/16/2014 07:57 PM, Tsutomu Itoh wrote: >>> On 2014/06/17 8:52, Chris Mason wrote: >>>> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >>>>> Hi Chris, >>>>> >>>>> On 2014/06/17 2:56, Chris Mason wrote: >>>>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>>>>> >>>>>> >>>>>> Did we recover, or was it stuck forever? >>>>> >>>>> The following messages are repeatedly output. >>>>> And stuck forever. >>>>> >>>>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >>>>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >>>>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >>>>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >>>>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] >>>> >>>> Can you please capture a stack trace from all the cpus? >> >> Very strange, please try to reproduce again, I'll dig through things here. > > I can reproduce it easily in my environment. This is my reproducer. =================== #! /bin/bash export TEST_DIR=/test5 export TEST_DEV=/dev/sdb4 TEST_DEV2=/dev/sdc6 export SCRATCH_MNT=/test6 export SCRATCH_DEV=/dev/sdc5 export MKFS_OPTIONS="" export MOUNT_OPTIONS="" umount ${TEST_DIR} ${SCRATCH_MNT} mkfs.btrfs -f ${TEST_DEV} ${TEST_DEV2} for i in $(seq 1 10) do ./check 'btrfs/042' done umount ${TEST_DIR} ${SCRATCH_MNT} =================== Stack trace is here. ================== [ 1631.835052] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u25:1:193] [ 1631.835053] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u25:3:3174] [ 1631.835074] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1631.835076] CPU: 1 PID: 3174 Comm: kworker/u25:3 Not tainted 3.16.0-rc1 #1 [ 1631.835076] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1631.835094] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1631.835094] task: ffff880073d60000 ti: ffff88007188c000 task.ti: ffff88007188c000 [ 1631.835097] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1631.835098] RSP: 0018:ffff88007188fac8 EFLAGS: 00000206 [ 1631.835098] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 000000000000000a [ 1631.835099] RDX: 0000000000000039 RSI: 000000000000000a RDI: ffff8801356755b0 [ 1631.835099] RBP: ffff88007188fac8 R08: ffff880135675574 R09: 0000000000000000 [ 1631.835100] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1631.835100] R13: 0000000000000000 R14: ffff8800766d3000 R15: ffffffffa05800a1 [ 1631.835101] FS: 0000000000000000(0000) GS:ffff88007dc40000(0000) knlGS:0000000000000000 [ 1631.835101] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1631.835102] CR2: 0000000001858000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1631.835102] Stack: [ 1631.835103] ffff88007188fad8 ffffffff8166bcbc ffff88007188fb38 ffffffffa05da140 [ 1631.835104] 0000000000000069 0000000000000035 0000000000000000 ffff88007188fb58 [ 1631.835105] ffffffffa057881f ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1631.835105] Call Trace: [ 1631.835109] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1631.835119] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1631.835124] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1631.835129] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1631.835134] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1631.835143] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1631.835150] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1631.835157] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1631.835162] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1631.835170] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1631.835178] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1631.835187] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1631.835196] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1631.835198] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1631.835199] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1631.835200] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1631.835201] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1631.835203] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.835204] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1631.835205] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.835215] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1631.849920] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1631.849942] CPU: 0 PID: 193 Comm: kworker/u25:1 Not tainted 3.16.0-rc1 #1 [ 1631.849943] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1631.849953] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1631.849954] task: ffff880036a40960 ti: ffff88003695c000 task.ti: ffff88003695c000 [ 1631.849955] RIP: 0010:[<ffffffff810ba7c2>] [<ffffffff810ba7c2>] queue_write_lock_slowpath+0x62/0x90 [ 1631.849957] RSP: 0018:ffff88003695fae0 EFLAGS: 00000206 [ 1631.849958] RAX: 0000000000000101 RBX: 00000000000000cf RCX: 0000000000000101 [ 1631.849959] RDX: 00000000000000ff RSI: 0000000000000283 RDI: ffff8801356755b0 [ 1631.849960] RBP: ffff88003695fae0 R08: ffff88003695fb18 R09: 0000000000000000 [ 1631.849961] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffffa059cf56 [ 1631.849962] R13: ffff88003695fa78 R14: 0000000000000001 R15: 0000000000000012 [ 1631.849963] FS: 0000000000000000(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000 [ 1631.849964] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1631.849965] CR2: 000000000173e000 CR3: 0000000001c0e000 CR4: 00000000000007f0 [ 1631.849966] Stack: [ 1631.849967] ffff88003695faf0 ffffffff8166bc51 ffff88003695fb50 ffffffffa05da519 [ 1631.849968] 0000000000000000 ffff880036a40960 ffffffff810b4840 ffff88003695fb18 [ 1631.849970] ffff88003695fb18 ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1631.849971] Call Trace: [ 1631.849974] [<ffffffff8166bc51>] _raw_write_lock+0x21/0x30 [ 1631.849984] [<ffffffffa05da519>] btrfs_tree_lock+0xc9/0x1d0 [btrfs] [ 1631.849986] [<ffffffff810b4840>] ? abort_exclusive_wait+0xb0/0xb0 [ 1631.849992] [<ffffffffa057ae3b>] btrfs_lock_root_node+0x3b/0x50 [btrfs] [ 1631.849998] [<ffffffffa0580029>] btrfs_search_slot+0x8f9/0xa10 [btrfs] [ 1631.850003] [<ffffffffa057aba9>] ? btrfs_set_path_blocking+0x39/0x80 [btrfs] [ 1631.850009] [<ffffffffa0581970>] btrfs_insert_empty_items+0x70/0xc0 [btrfs] [ 1631.850016] [<ffffffffa05973b9>] btrfs_csum_file_blocks+0x2d9/0x680 [btrfs] [ 1631.850022] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1631.850030] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1631.850039] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1631.850048] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1631.850057] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1631.850059] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1631.850060] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1631.850062] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1631.850064] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1631.850066] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.850067] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1631.850069] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.850070] Code: 2e 0f 1f 84 00 00 00 00 00 f3 90 8b 17 84 d2 75 f8 89 d1 89 d0 83 c9 01 f0 0f b1 0f 39 d0 75 e9 ba ff 00 00 00 eb 04 66 90 f3 90 <8b> 07 83 f8 01 75 f7 f0 0f b1 17 83 f8 01 75 ee eb bc f3 90 0f [ 1631.860046] BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u25:4:3175] [ 1631.868742] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1631.868763] CPU: 3 PID: 3175 Comm: kworker/u25:4 Not tainted 3.16.0-rc1 #1 [ 1631.868764] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1631.868774] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1631.868775] task: ffff880073d60960 ti: ffff88007bde0000 task.ti: ffff88007bde0000 [ 1631.868776] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1631.868778] RSP: 0018:ffff88007bde3ac8 EFLAGS: 00000206 [ 1631.868779] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000007 [ 1631.868780] RDX: 0000000000000036 RSI: 0000000000000007 RDI: ffff8801356755b0 [ 1631.868781] RBP: ffff88007bde3ac8 R08: ffff880135675574 R09: 0000000000000000 [ 1631.868782] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1631.868783] R13: 0000000000000000 R14: ffff88007385f000 R15: ffffffffa05800a1 [ 1631.868784] FS: 0000000000000000(0000) GS:ffff88007dcc0000(0000) knlGS:0000000000000000 [ 1631.868785] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1631.868786] CR2: 0000000000d6e000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1631.868787] Stack: [ 1631.868788] ffff88007bde3ad8 ffffffff8166bcbc ffff88007bde3b38 ffffffffa05da140 [ 1631.868789] 0000000000000068 0000000000000035 0000000000000000 ffff88007bde3b58 [ 1631.868791] ffffffffa057881f ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1631.868792] Call Trace: [ 1631.868795] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1631.868804] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1631.868810] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1631.868815] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1631.868821] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1631.868830] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1631.868837] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1631.868845] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1631.868850] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1631.868859] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1631.868867] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1631.868869] [<ffffffff810a702f>] ? set_next_entity+0x5f/0x80 [ 1631.868878] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1631.868887] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1631.868889] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1631.868891] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1631.868892] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1631.868894] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1631.868896] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.868898] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1631.868899] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.868900] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1631.872043] BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u25:6:3177] [ 1631.880892] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1631.880913] CPU: 4 PID: 3177 Comm: kworker/u25:6 Not tainted 3.16.0-rc1 #1 [ 1631.880914] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1631.880924] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1631.880926] task: ffff880073d61c20 ti: ffff880073d70000 task.ti: ffff880073d70000 [ 1631.880927] RIP: 0010:[<ffffffff810ba85c>] [<ffffffff810ba85c>] queue_read_lock_slowpath+0x6c/0x90 [ 1631.880929] RSP: 0018:ffff880073d73ac8 EFLAGS: 00000202 [ 1631.880930] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 000000000000000c [ 1631.880931] RDX: 000000000000003b RSI: 000000000000000c RDI: ffff8801356755b0 [ 1631.880932] RBP: ffff880073d73ac8 R08: ffff880135675574 R09: 0000000000000000 [ 1631.880933] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1631.880934] R13: 0000000000000000 R14: ffff8800718af000 R15: ffffffffa05800a1 [ 1631.880935] FS: 0000000000000000(0000) GS:ffff88007dd00000(0000) knlGS:0000000000000000 [ 1631.880936] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1631.880937] CR2: 0000000000c98000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1631.880938] Stack: [ 1631.880938] ffff880073d73ad8 ffffffff8166bcbc ffff880073d73b38 ffffffffa05da140 [ 1631.880940] 0000000000000068 0000000000000035 0000000000000000 ffff880073d73b58 [ 1631.880941] ffffffffa057881f ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1631.880943] Call Trace: [ 1631.880945] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1631.880955] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1631.880960] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1631.880966] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1631.880972] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1631.880981] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1631.880988] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1631.880996] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1631.881001] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1631.881010] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1631.881018] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1631.881021] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0 [ 1631.881029] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1631.881039] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1631.881040] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1631.881042] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1631.881043] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1631.881045] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1631.881047] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.881049] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1631.881051] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.881051] Code: 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 0f b6 47 04 <38> c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c ff 74 f8 5d [ 1631.884041] BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u25:5:3176] [ 1631.893041] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1631.893063] CPU: 5 PID: 3176 Comm: kworker/u25:5 Not tainted 3.16.0-rc1 #1 [ 1631.893064] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1631.893074] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1631.893075] task: ffff880073d612c0 ti: ffff880073d5c000 task.ti: ffff880073d5c000 [ 1631.893076] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1631.893078] RSP: 0018:ffff880073d5fac8 EFLAGS: 00000206 [ 1631.893079] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000009 [ 1631.893080] RDX: 0000000000000038 RSI: 0000000000000009 RDI: ffff8801356755b0 [ 1631.893081] RBP: ffff880073d5fac8 R08: ffff880135675574 R09: 0000000000000000 [ 1631.893082] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1631.893083] R13: 0000000000001000 R14: ffff88007c65a000 R15: ffffffffa05800a1 [ 1631.893084] FS: 0000000000000000(0000) GS:ffff88007dd40000(0000) knlGS:0000000000000000 [ 1631.893085] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1631.893086] CR2: 0000000001c95000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1631.893087] Stack: [ 1631.893088] ffff880073d5fad8 ffffffff8166bcbc ffff880073d5fb38 ffffffffa05da140 [ 1631.893089] 00000000000000cc 00000000000000a0 0000000000000000 ffff880073d5fb58 [ 1631.893091] ffffffffa057881f ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1631.893092] Call Trace: [ 1631.893095] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1631.893104] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1631.893110] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1631.893115] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1631.893121] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1631.893130] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1631.893138] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1631.893145] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1631.893151] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1631.893159] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1631.893168] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1631.893169] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0 [ 1631.893178] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1631.893188] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1631.893189] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1631.893191] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1631.893192] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1631.893194] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1631.893196] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.893198] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1631.893199] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.893200] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1631.969022] BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u26:6:3182] [ 1631.976495] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1631.976516] CPU: 7 PID: 3182 Comm: kworker/u26:6 Not tainted 3.16.0-rc1 #1 [ 1631.976517] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1631.976527] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1631.976528] task: ffff8800ba65e720 ti: ffff880134714000 task.ti: ffff880134714000 [ 1631.976529] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1631.976531] RSP: 0018:ffff880134717ac8 EFLAGS: 00000202 [ 1631.976532] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000008 [ 1631.976533] RDX: 0000000000000037 RSI: 0000000000000008 RDI: ffff8801356755b0 [ 1631.976534] RBP: ffff880134717ac8 R08: ffff880135675574 R09: 0000000000000000 [ 1631.976535] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1631.976536] R13: 0000000000001000 R14: ffff880075cd5000 R15: ffffffffa05800a1 [ 1631.976537] FS: 0000000000000000(0000) GS:ffff88013fc40000(0000) knlGS:0000000000000000 [ 1631.976538] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1631.976539] CR2: 0000000000715000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1631.976540] Stack: [ 1631.976540] ffff880134717ad8 ffffffff8166bcbc ffff880134717b38 ffffffffa05da140 [ 1631.976542] 00000000000000ce 00000000000000a0 0000000000000000 ffff880134717b58 [ 1631.976544] ffffffffa057881f ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1631.976545] Call Trace: [ 1631.976547] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1631.976557] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1631.976562] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1631.976568] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1631.976573] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1631.976583] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1631.976590] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1631.976598] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1631.976603] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1631.976611] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1631.976620] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1631.976622] [<ffffffff810a702f>] ? set_next_entity+0x5f/0x80 [ 1631.976630] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1631.976640] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1631.976641] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1631.976643] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1631.976644] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1631.976646] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1631.976648] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.976650] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1631.976651] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.976652] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1631.981019] BUG: soft lockup - CPU#8 stuck for 22s! [kworker/u26:3:3179] [ 1631.989015] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1631.989036] CPU: 8 PID: 3179 Comm: kworker/u26:3 Not tainted 3.16.0-rc1 #1 [ 1631.989037] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1631.989047] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1631.989048] task: ffff8800b7b13840 ti: ffff880137774000 task.ti: ffff880137774000 [ 1631.989049] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1631.989051] RSP: 0018:ffff880137777b08 EFLAGS: 00000202 [ 1631.989052] RAX: 0000000000000033 RBX: 0000000000000012 RCX: 0000000000000001 [ 1631.989053] RDX: 0000000000000035 RSI: 0000000000000004 RDI: ffff8801356755b0 [ 1631.989054] RBP: ffff880137777b08 R08: 0000000000000001 R09: ffff8801357a0490 [ 1631.989055] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880135675680 [ 1631.989056] R13: 0000000000000000 R14: 0000000000000003 R15: 0000000000000002 [ 1631.989057] FS: 0000000000000000(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000 [ 1631.989058] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1631.989059] CR2: 0000000001945000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1631.989060] Stack: [ 1631.989060] ffff880137777b18 ffffffff8166bcbc ffff880137777b38 ffffffffa05d9fe4 [ 1631.989062] 0000000000000001 ffffffffffffffe8 ffff880137777b58 ffffffffa057ac2a [ 1631.989063] ffff8801357bee10 0000000000000000 ffff880137777bf8 ffffffffa057fe32 [ 1631.989065] Call Trace: [ 1631.989067] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1631.989077] [<ffffffffa05d9fe4>] btrfs_clear_lock_blocking_rw+0x44/0x150 [btrfs] [ 1631.989082] [<ffffffffa057ac2a>] btrfs_clear_path_blocking+0x3a/0x80 [btrfs] [ 1631.989088] [<ffffffffa057fe32>] btrfs_search_slot+0x702/0xa10 [btrfs] [ 1631.989095] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1631.989102] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1631.989108] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1631.989116] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1631.989125] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1631.989127] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0 [ 1631.989135] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1631.989145] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1631.989147] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1631.989148] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1631.989150] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1631.989151] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1631.989153] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.989155] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1631.989157] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1631.989158] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1631.994016] BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u26:0:3178] [ 1632.002726] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1632.002747] CPU: 9 PID: 3178 Comm: kworker/u26:0 Not tainted 3.16.0-rc1 #1 [ 1632.002748] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1632.002758] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1632.002759] task: ffff8800b7b15dc0 ti: ffff88013397c000 task.ti: ffff88013397c000 [ 1632.002760] RIP: 0010:[<ffffffff810ba7a4>] [<ffffffff810ba7a4>] queue_write_lock_slowpath+0x44/0x90 [ 1632.002762] RSP: 0018:ffff88013397fb30 EFLAGS: 00000286 [ 1632.002763] RAX: 00000000000000ff RBX: ffff880135675680 RCX: 0000000000000001 [ 1632.002764] RDX: 00000000000000ff RSI: 0000000137fb4000 RDI: ffff8801356756f0 [ 1632.002765] RBP: ffff88013397fb30 R08: 0000000000000001 R09: ffff8801357a0490 [ 1632.002766] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000086 [ 1632.002767] R13: ffffffffa059cf56 R14: ffff88013397fac0 R15: 0000000000000001 [ 1632.002768] FS: 0000000000000000(0000) GS:ffff88013fcc0000(0000) knlGS:0000000000000000 [ 1632.002769] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1632.002770] CR2: 00000036c97b9490 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1632.002771] Stack: [ 1632.002771] ffff88013397fb40 ffffffff8166bc51 ffff88013397fb58 ffffffffa05da279 [ 1632.002773] ffff880135765750 ffff88013397fbf8 ffffffffa057fb4c 0000000000000001 [ 1632.002774] ffff88013a9df320 ffff88013a9df1e0 ffff88013397fbc0 ffff88013a9df1e0 [ 1632.002776] Call Trace: [ 1632.002778] [<ffffffff8166bc51>] _raw_write_lock+0x21/0x30 [ 1632.002788] [<ffffffffa05da279>] btrfs_try_tree_write_lock+0x29/0x80 [btrfs] [ 1632.002793] [<ffffffffa057fb4c>] btrfs_search_slot+0x41c/0xa10 [btrfs] [ 1632.002801] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1632.002808] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1632.002814] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1632.002822] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1632.002831] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1632.002839] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1632.002849] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1632.002851] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1632.002852] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1632.002854] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1632.002855] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1632.002857] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1632.002859] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1632.002861] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1632.002862] Code: 75 5b 8b 07 85 c0 75 21 ba ff 00 00 00 f0 0f b1 17 85 c0 75 14 66 90 80 47 04 01 5d c3 66 2e 0f 1f 84 00 00 00 00 00 f3 90 8b 17 <84> d2 75 f8 89 d1 89 d0 83 c9 01 f0 0f b1 0f 39 d0 75 e9 ba ff [ 1632.006013] BUG: soft lockup - CPU#10 stuck for 22s! [kworker/u26:1:88] [ 1632.014778] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1632.014799] CPU: 10 PID: 88 Comm: kworker/u26:1 Not tainted 3.16.0-rc1 #1 [ 1632.014800] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1632.014810] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1632.014811] task: ffff88013bb68000 ti: ffff88013bb5c000 task.ti: ffff88013bb5c000 [ 1632.014812] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1632.014814] RSP: 0018:ffff88013bb5fac8 EFLAGS: 00000202 [ 1632.014815] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 000000000000000b [ 1632.014816] RDX: 000000000000003a RSI: 000000000000000b RDI: ffff8801356755b0 [ 1632.014817] RBP: ffff88013bb5fac8 R08: ffff880135675574 R09: 0000000000000000 [ 1632.014818] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000001000 [ 1632.014819] R13: 0000000000000000 R14: ffff88013b9bd000 R15: ffffffffa05800a1 [ 1632.014820] FS: 0000000000000000(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000 [ 1632.014821] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1632.014822] CR2: 0000000000715000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1632.014823] Stack: [ 1632.014824] ffff88013bb5fad8 ffffffff8166bcbc ffff88013bb5fb38 ffffffffa05da140 [ 1632.014825] 0000000000000069 0000000000000035 0000000000000000 ffff88013bb5fb58 [ 1632.014827] ffffffffa057881f ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1632.014828] Call Trace: [ 1632.014830] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1632.014840] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1632.014845] [<ffffffffa057881f>] ? leaf_space_used+0xcf/0x110 [btrfs] [ 1632.014851] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1632.014856] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1632.014866] [<ffffffffa05eaee4>] ? btrfs_add_delayed_data_ref+0x114/0x180 [btrfs] [ 1632.014873] [<ffffffffa0595e2e>] btrfs_lookup_csum+0x5e/0x180 [btrfs] [ 1632.014880] [<ffffffffa059719c>] btrfs_csum_file_blocks+0xbc/0x680 [btrfs] [ 1632.014886] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1632.014894] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1632.014903] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1632.014905] [<ffffffff8109ea79>] ? ttwu_do_wakeup+0x19/0xc0 [ 1632.014913] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1632.014923] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1632.014925] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1632.014926] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1632.014928] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1632.014929] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1632.014931] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1632.014933] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1632.014935] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1632.014936] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c [ 1632.018011] BUG: soft lockup - CPU#11 stuck for 22s! [kworker/u26:7:3183] [ 1632.027027] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd xor raid6_pq igb microcode ptp pps_core dca iTCO_wdt auth_rpcgss shpchp iTCO_vendor_support nfs_acl i7core_edac pcspkr edac_core lpc_ich i2c_i801 mfd_core acpi_power_meter tpm_infineon tpm_tis tpm ipmi_si lockd ipmi_msghandler acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ata_generic pata_acpi megaraid_sas i2c_core ata_piix [ 1632.027048] CPU: 11 PID: 3183 Comm: kworker/u26:7 Not tainted 3.16.0-rc1 #1 [ 1632.027049] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 [ 1632.027059] Workqueue: btrfs-endio-write normal_work_helper [btrfs] [ 1632.027061] task: ffff8800ba65b840 ti: ffff88013a714000 task.ti: ffff88013a714000 [ 1632.027061] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 [ 1632.027064] RSP: 0018:ffff88013a717a80 EFLAGS: 00000202 [ 1632.027065] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000006 [ 1632.027066] RDX: 0000000000000034 RSI: 0000000000000006 RDI: ffff8801356755b0 [ 1632.027066] RBP: ffff88013a717a80 R08: ffff880135675574 R09: 0000000000000000 [ 1632.027067] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 1632.027068] R13: ffff8801355f0870 R14: ffffffff00000000 R15: ffff8801355f0870 [ 1632.027069] FS: 0000000000000000(0000) GS:ffff88013fd40000(0000) knlGS:0000000000000000 [ 1632.027070] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1632.027071] CR2: 0000000000b55000 CR3: 0000000001c0e000 CR4: 00000000000007e0 [ 1632.027072] Stack: [ 1632.027073] ffff88013a717a90 ffffffff8166bcbc ffff88013a717af0 ffffffffa05da140 [ 1632.027074] ffffffff810b4178 00000000000000cf ffff88013a717b48 ffffffffa057d07d [ 1632.027076] ffff88013a717bc0 ffff880135675540 ffff8800b7117800 ffff880000000000 [ 1632.027077] Call Trace: [ 1632.027080] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 [ 1632.027090] [<ffffffffa05da140>] btrfs_tree_read_lock+0x50/0x100 [btrfs] [ 1632.027091] [<ffffffff810b4178>] ? __wake_up_common+0x58/0x90 [ 1632.027097] [<ffffffffa057d07d>] ? generic_bin_search.constprop.35+0x8d/0x1a0 [btrfs] [ 1632.027103] [<ffffffffa057adeb>] btrfs_read_lock_root_node+0x3b/0x50 [btrfs] [ 1632.027108] [<ffffffffa057fc3e>] btrfs_search_slot+0x50e/0xa10 [btrfs] [ 1632.027118] [<ffffffffa05c38ef>] ? free_extent_buffer+0x4f/0xb0 [btrfs] [ 1632.027124] [<ffffffffa0582eb8>] btrfs_next_old_leaf+0x218/0x4b0 [btrfs] [ 1632.027130] [<ffffffffa0583160>] btrfs_next_leaf+0x10/0x20 [btrfs] [ 1632.027137] [<ffffffffa059744b>] btrfs_csum_file_blocks+0x36b/0x680 [btrfs] [ 1632.027143] [<ffffffffa057ad46>] ? btrfs_free_path+0x26/0x30 [btrfs] [ 1632.027151] [<ffffffffa05a51e0>] add_pending_csums.isra.42+0x50/0x80 [btrfs] [ 1632.027160] [<ffffffffa05ab410>] btrfs_finish_ordered_io+0x470/0x570 [btrfs] [ 1632.027168] [<ffffffffa05ab785>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1632.027178] [<ffffffffa05d0b3c>] normal_work_helper+0xbc/0x290 [btrfs] [ 1632.027180] [<ffffffff8108ab95>] process_one_work+0x175/0x430 [ 1632.027181] [<ffffffff8108b23b>] worker_thread+0x11b/0x520 [ 1632.027183] [<ffffffff8108b120>] ? rescuer_thread+0x2d0/0x2d0 [ 1632.027184] [<ffffffff81091712>] kthread+0xd2/0xf0 [ 1632.027186] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1632.027188] [<ffffffff8166c23c>] ret_from_fork+0x7c/0xb0 [ 1632.027190] [<ffffffff81091640>] ? insert_kthread_work+0x40/0x40 [ 1632.027191] Code: 00 00 00 f3 90 8b 07 84 c0 75 f8 b8 00 01 00 00 f0 0f c1 07 3c ff 75 0b 0f 1f 00 f3 90 8b 07 3c ff 74 f8 80 47 04 01 5d c3 f3 90 <0f> b6 47 04 38 c2 75 f6 eb bf 8b 07 3c ff 75 ec f3 90 8b 07 3c > > Thanks, > Tsutomu > > ========== > Message from syslogd@luna at Jun 17 11:06:33 ... > kernel:[ 560.076282] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u25:1:193] > > Message from syslogd@luna at Jun 17 11:06:33 ... > kernel:[ 560.076283] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u25:3:3174] > > Message from syslogd@luna at Jun 17 11:06:33 ... > kernel:[ 560.101276] BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u25:4:3175] > > Message from syslogd@luna at Jun 17 11:06:33 ... > kernel:[ 560.113274] BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u25:6:3177] > > Message from syslogd@luna at Jun 17 11:06:33 ... > kernel:[ 560.125271] BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u25:5:3176] > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-17 3:06 ` Tsutomu Itoh @ 2014-06-17 4:48 ` Chris Mason 2014-07-09 17:14 ` James Cloos 2014-06-17 4:52 ` Tsutomu Itoh 1 sibling, 1 reply; 13+ messages in thread From: Chris Mason @ 2014-06-17 4:48 UTC (permalink / raw) To: Tsutomu Itoh; +Cc: Linux Btrfs On 06/16/2014 11:06 PM, Tsutomu Itoh wrote: > On 2014/06/17 11:10, Tsutomu Itoh wrote: >> On 2014/06/17 9:47, Chris Mason wrote: >>> On 06/16/2014 07:57 PM, Tsutomu Itoh wrote: >>>> On 2014/06/17 8:52, Chris Mason wrote: >>>>> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >>>>>> Hi Chris, >>>>>> >>>>>> On 2014/06/17 2:56, Chris Mason wrote: >>>>>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>>>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>>>>>> >>>>>>> >>>>>>> Did we recover, or was it stuck forever? >>>>>> >>>>>> The following messages are repeatedly output. >>>>>> And stuck forever. >>>>>> >>>>>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >>>>>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >>>>>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >>>>>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >>>>>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] >>>>> >>>>> Can you please capture a stack trace from all the cpus? >>> >>> Very strange, please try to reproduce again, I'll dig through things here. >> >> I can reproduce it easily in my environment. > > This is my reproducer. > Great, I was able to trigger it here, but only with lockdep disabled. When called from this part of btrfs_search_slot, the code for btrfs_set_path_blocking goes through an extra step to set the lock on the extent buffer we just found blocking before it is put into the path. But this is only done with lockdep on. With lockdep off, we assume the lock ordering inside the tree is protecting us and that we don't need to worry about blocks that are not in the path yet. Something is breaking this rule, probably in the quota code. I'll try to nail down what is going on. -chris ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-17 4:48 ` Chris Mason @ 2014-07-09 17:14 ` James Cloos 0 siblings, 0 replies; 13+ messages in thread From: James Cloos @ 2014-07-09 17:14 UTC (permalink / raw) To: Chris Mason; +Cc: Tsutomu Itoh, Linux Btrfs It looks like I just hit this on rc4. (I compiled the earlier rc versions, but hadn't rebooted until yesterday.) I don't think I see anything in the git log since rc1 which is relevant, but just in case one of those tried to fix this, its still there. The stuck processes are: ]btrfs-transacti] btrfs_wait_and_free_delalloc_work rsync writeback_inodes_sb_nr rsync reserve_metadata_bytes git-merge sleep_on_page_killable /bin/sync sync_inodes_sb (The two rsync processes are separate; each to its one directory.) The fs with those processes is mounted with noatime,compress. Interestingly, I also use btrfs /var/log, also with noatime,compress, but that one is fine. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 0x997A9F17ED7DAEA6 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-17 3:06 ` Tsutomu Itoh 2014-06-17 4:48 ` Chris Mason @ 2014-06-17 4:52 ` Tsutomu Itoh 1 sibling, 0 replies; 13+ messages in thread From: Tsutomu Itoh @ 2014-06-17 4:52 UTC (permalink / raw) To: Chris Mason; +Cc: Linux Btrfs On 2014/06/17 12:06, Tsutomu Itoh wrote: > On 2014/06/17 11:10, Tsutomu Itoh wrote: >> On 2014/06/17 9:47, Chris Mason wrote: >>> On 06/16/2014 07:57 PM, Tsutomu Itoh wrote: >>>> On 2014/06/17 8:52, Chris Mason wrote: >>>>> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >>>>>> Hi Chris, >>>>>> >>>>>> On 2014/06/17 2:56, Chris Mason wrote: >>>>>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>>>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>>>>>> >>>>>>> >>>>>>> Did we recover, or was it stuck forever? >>>>>> >>>>>> The following messages are repeatedly output. >>>>>> And stuck forever. >>>>>> >>>>>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >>>>>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >>>>>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >>>>>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >>>>>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] >>>>> >>>>> Can you please capture a stack trace from all the cpus? >>> >>> Very strange, please try to reproduce again, I'll dig through things here. >> >> I can reproduce it easily in my environment. > 3.16-rc1 is slower than 3.15. And, 3.16-rc1 sometimes hang up. =============== - 3.15 WARNING! - Btrfs v3.14.2-50-gea6501b IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 adding device /dev/sdc6 id 2 fs created label (null) on /dev/sdb4 nodesize 16384 leafsize 16384 sectorsize 4096 size 65.19GiB Btrfs v3.14.2-50-gea6501b FSTYP -- btrfs PLATFORM -- Linux/x86_64 luna 3.15.0 MKFS_OPTIONS -- /dev/sdc5 MOUNT_OPTIONS -- /dev/sdc5 /test6 btrfs/042 2s ... 1s Ran: btrfs/042 Passed all 1 tests FSTYP -- btrfs PLATFORM -- Linux/x86_64 luna 3.15.0 MKFS_OPTIONS -- /dev/sdc5 MOUNT_OPTIONS -- /dev/sdc5 /test6 btrfs/042 1s ... 2s Ran: btrfs/042 Passed all 1 tests ... ... --------------- - 3.16-rc1 WARNING! - Btrfs v3.14.2-50-gea6501b IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 adding device /dev/sdc6 id 2 fs created label (null) on /dev/sdb4 nodesize 16384 leafsize 16384 sectorsize 4096 size 65.19GiB Btrfs v3.14.2-50-gea6501b FSTYP -- btrfs PLATFORM -- Linux/x86_64 luna 3.16.0-rc1 MKFS_OPTIONS -- /dev/sdc5 MOUNT_OPTIONS -- /dev/sdc5 /test6 btrfs/042 2s ... 6s Ran: btrfs/042 Passed all 1 tests FSTYP -- btrfs PLATFORM -- Linux/x86_64 luna 3.16.0-rc1 MKFS_OPTIONS -- /dev/sdc5 MOUNT_OPTIONS -- /dev/sdc5 /test6 btrfs/042 6s ... [ 160.167530] BUG: soft lockup - CPU#1 ... =============== ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-16 23:57 ` Tsutomu Itoh 2014-06-17 0:47 ` Chris Mason @ 2014-06-17 1:11 ` Chris Mason 2014-06-17 1:36 ` Tsutomu Itoh 1 sibling, 1 reply; 13+ messages in thread From: Chris Mason @ 2014-06-17 1:11 UTC (permalink / raw) To: Tsutomu Itoh; +Cc: Linux Btrfs On 06/16/2014 07:57 PM, Tsutomu Itoh wrote: > On 2014/06/17 8:52, Chris Mason wrote: >> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >>> Hi Chris, >>> >>> On 2014/06/17 2:56, Chris Mason wrote: >>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>>> >>>> >>>> Did we recover, or was it stuck forever? >>> >>> The following messages are repeatedly output. >>> And stuck forever. >>> >>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] >> >> Can you please capture a stack trace from all the cpus? >> > > crash> log > ... > [ 1121.996312] BTRFS: creating UUID tree > [ 1122.050762] BTRFS info (device sdc5): qgroup scan completed > [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] > [ 1147.949658] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix > [ 1147.949695] CPU: 0 PID: 5189 Comm: kworker/u25:4 Not tainted 3.16.0-rc1 #1 > [ 1147.949696] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 > [ 1147.949717] Workqueue: btrfs-endio-write normal_work_helper [btrfs] > [ 1147.949718] task: ffff88007408b840 ti: ffff8800740e4000 task.ti: ffff8800740e4000 > [ 1147.949720] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 > [ 1147.949725] RSP: 0018:ffff8800740e7b08 EFLAGS: 00000202 > [ 1147.949726] RAX: 00000000000000ee RBX: 0000000000000012 RCX: 0000000000000001 > [ 1147.949727] RDX: 00000000000000ef RSI: 0000000000000004 RDI: ffff88003db221b0 > [ 1147.949728] RBP: ffff8800740e7b08 R08: 0000000000000001 R09: ffff880071486b58 > [ 1147.949729] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800715d1900 > [ 1147.949730] R13: 0000000000000000 R14: 0000000000000003 R15: 0000000000000002 > [ 1147.949732] FS: 0000000000000000(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000 > [ 1147.949733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 1147.949734] CR2: 00000036c97b9490 CR3: 0000000001c0e000 CR4: 00000000000007f0 > [ 1147.949735] Stack: > [ 1147.949736] ffff8800740e7b18 ffffffff8166bcbc ffff8800740e7b38 ffffffffa070ffe4 > [ 1147.949738] 0000000000000001 ffffffffffffffe8 ffff8800740e7b58 ffffffffa06b0c2a > [ 1147.949740] ffff8800719de360 0000000000000000 ffff8800740e7bf8 ffffffffa06b5e32 > [ 1147.949742] Call Trace: > [ 1147.949748] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 > [ 1147.949760] [<ffffffffa070ffe4>] btrfs_clear_lock_blocking_rw+0x44/0x150 [btrfs] > [ 1147.949767] [<ffffffffa06b0c2a>] btrfs_clear_path_blocking+0x3a/0x80 [btrfs] > [ 1147.949774] [<ffffffffa06b5e32>] btrfs_search_slot+0x702/0xa10 [btrfs] Also, which line of ctree.c is this call? Thanks! -chris ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] 2014-06-17 1:11 ` Chris Mason @ 2014-06-17 1:36 ` Tsutomu Itoh 0 siblings, 0 replies; 13+ messages in thread From: Tsutomu Itoh @ 2014-06-17 1:36 UTC (permalink / raw) To: Chris Mason; +Cc: Linux Btrfs On 2014/06/17 10:11, Chris Mason wrote: > > > On 06/16/2014 07:57 PM, Tsutomu Itoh wrote: >> On 2014/06/17 8:52, Chris Mason wrote: >>> On 06/16/2014 07:28 PM, Tsutomu Itoh wrote: >>>> Hi Chris, >>>> >>>> On 2014/06/17 2:56, Chris Mason wrote: >>>>> On 06/16/2014 02:35 AM, Tsutomu Itoh wrote: >>>>>> I encountered soft lockup when executing 'xfstests btrfs/042' on 3.16-rc1. >>>>>> >>>>> >>>>> Did we recover, or was it stuck forever? >>>> >>>> The following messages are repeatedly output. >>>> And stuck forever. >>>> >>>> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >>>> [ 1147.967175] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u25:9:5194] >>>> [ 1147.979172] BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u25:15:5200] >>>> [ 1147.991169] BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u25:7:5192] >>>> [ 1148.064153] BUG: soft lockup - CPU#6 stuck for 23s! [kworker/u26:3:3182] >>> >>> Can you please capture a stack trace from all the cpus? >>> >> >> crash> log >> ... >> [ 1121.996312] BTRFS: creating UUID tree >> [ 1122.050762] BTRFS info (device sdc5): qgroup scan completed >> [ 1147.942181] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] >> [ 1147.949658] Modules linked in: loop ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc ip6table_filter ip6_tables ebtable_nat ebtables btrfs coretemp kvm_intel kvm xor raid6_pq crc32_pclmul crc32c_intel ghash_clmulni_intel nfsd igb microcode ptp pps_core dca iTCO_wdt shpchp auth_rpcgss iTCO_vendor_support i7core_edac pcspkr edac_core i2c_i801 lpc_ich mfd_core ipmi_si tpm_infineon ipmi_msghandler tpm_tis nfs_acl tpm acpi_power_meter lockd acpi_cpufreq sunrpc usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ata_generic pata_acpi drm megaraid_sas i2c_core ata_piix >> [ 1147.949695] CPU: 0 PID: 5189 Comm: kworker/u25:4 Not tainted 3.16.0-rc1 #1 >> [ 1147.949696] Hardware name: FUJITSU-SV PRIMERGY /D2619, BIOS 6.00 Rev. 1.03.2619.N1 04/06/2010 >> [ 1147.949717] Workqueue: btrfs-endio-write normal_work_helper [btrfs] >> [ 1147.949718] task: ffff88007408b840 ti: ffff8800740e4000 task.ti: ffff8800740e4000 >> [ 1147.949720] RIP: 0010:[<ffffffff810ba858>] [<ffffffff810ba858>] queue_read_lock_slowpath+0x68/0x90 >> [ 1147.949725] RSP: 0018:ffff8800740e7b08 EFLAGS: 00000202 >> [ 1147.949726] RAX: 00000000000000ee RBX: 0000000000000012 RCX: 0000000000000001 >> [ 1147.949727] RDX: 00000000000000ef RSI: 0000000000000004 RDI: ffff88003db221b0 >> [ 1147.949728] RBP: ffff8800740e7b08 R08: 0000000000000001 R09: ffff880071486b58 >> [ 1147.949729] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800715d1900 >> [ 1147.949730] R13: 0000000000000000 R14: 0000000000000003 R15: 0000000000000002 >> [ 1147.949732] FS: 0000000000000000(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000 >> [ 1147.949733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >> [ 1147.949734] CR2: 00000036c97b9490 CR3: 0000000001c0e000 CR4: 00000000000007f0 >> [ 1147.949735] Stack: >> [ 1147.949736] ffff8800740e7b18 ffffffff8166bcbc ffff8800740e7b38 ffffffffa070ffe4 >> [ 1147.949738] 0000000000000001 ffffffffffffffe8 ffff8800740e7b58 ffffffffa06b0c2a >> [ 1147.949740] ffff8800719de360 0000000000000000 ffff8800740e7bf8 ffffffffa06b5e32 >> [ 1147.949742] Call Trace: >> [ 1147.949748] [<ffffffff8166bcbc>] _raw_read_lock+0x1c/0x30 >> [ 1147.949760] [<ffffffffa070ffe4>] btrfs_clear_lock_blocking_rw+0x44/0x150 [btrfs] >> [ 1147.949767] [<ffffffffa06b0c2a>] btrfs_clear_path_blocking+0x3a/0x80 [btrfs] >> [ 1147.949774] [<ffffffffa06b5e32>] btrfs_search_slot+0x702/0xa10 [btrfs] > > Also, which line of ctree.c is this call? /Build/linux-3.16-rc1/fs/btrfs/ctree.c: 2898 0xffffffffa06b5e10 <btrfs_search_slot+0x6e0>: mov %rbx,%rdi 0xffffffffa06b5e13 <btrfs_search_slot+0x6e3>: callq 0xffffffffa06b0b70 <btrfs_set_path_blocking> /Build/linux-3.16-rc1/fs/btrfs/ctree.c: 2899 0xffffffffa06b5e18 <btrfs_search_slot+0x6e8>: mov -0x30(%rbp),%rdi 0xffffffffa06b5e1c <btrfs_search_slot+0x6ec>: callq 0xffffffffa0710450 <btrfs_tree_lock> /Build/linux-3.16-rc1/fs/btrfs/ctree.c: 2900 0xffffffffa06b5e21 <btrfs_search_slot+0x6f1>: mov -0x30(%rbp),%rsi 0xffffffffa06b5e25 <btrfs_search_slot+0x6f5>: mov $0x1,%edx 0xffffffffa06b5e2a <btrfs_search_slot+0x6fa>: mov %rbx,%rdi 0xffffffffa06b5e2d <btrfs_search_slot+0x6fd>: callq 0xffffffffa06b0bf0 <btrfs_clear_path_blocking> 0xffffffffa06b5e32 <btrfs_search_slot+0x702>: jmpq 0xffffffffa06b5b54 <btrfs_search_slot+0x424> 0xffffffffa06b5e37 <btrfs_search_slot+0x707>: nopw 0x0(%rax,%rax,1) 2686 int btrfs_search_slot(struct btrfs_trans_handle *trans, struct btrfs_root 2687 *root, struct btrfs_key *key, struct btrfs_path *p, int 2688 ins_len, int cow) 2689 { 2690 struct extent_buffer *b; 2691 int slot; 2692 int ret; 2693 int err; 2694 int level; 2695 int lowest_unlock = 1; 2696 int root_lock; ... ... 2892 2893 if (!p->skip_locking) { 2894 level = btrfs_header_level(b); 2895 if (level <= write_lock_level) { 2896 err = btrfs_try_tree_write_lock(b); 2897 if (!err) { 2898 btrfs_set_path_blocking(p); 2899 btrfs_tree_lock(b); 2900 btrfs_clear_path_blocking(p, b, <----- here 2901 BTRFS_WRITE_LOCK); 2902 } 2903 p->locks[level] = BTRFS_WRITE_LOCK; 2904 } else { 2905 err = btrfs_try_tree_read_lock(b); 2906 if (!err) { 2907 btrfs_set_path_blocking(p); 2908 btrfs_tree_read_lock(b); 2909 btrfs_clear_path_blocking(p, b, 2910 BTRFS_READ_LOCK); 2911 } ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2014-07-09 17:30 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-06-16 6:35 [3.16-rc1] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u25:4:5189] Tsutomu Itoh 2014-06-16 17:56 ` Chris Mason 2014-06-16 23:28 ` Tsutomu Itoh 2014-06-16 23:52 ` Chris Mason 2014-06-16 23:57 ` Tsutomu Itoh 2014-06-17 0:47 ` Chris Mason 2014-06-17 2:10 ` Tsutomu Itoh 2014-06-17 3:06 ` Tsutomu Itoh 2014-06-17 4:48 ` Chris Mason 2014-07-09 17:14 ` James Cloos 2014-06-17 4:52 ` Tsutomu Itoh 2014-06-17 1:11 ` Chris Mason 2014-06-17 1:36 ` Tsutomu Itoh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).