From: Torbjørn <lists@skagestad.org>
To: Chris Mason <clm@fb.com>
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: Blocked tasks on 3.15.1
Date: Tue, 22 Jul 2014 21:42:35 +0200 [thread overview]
Message-ID: <53CEBEAB.3010206@skagestad.org> (raw)
In-Reply-To: <53CE7ACF.7070109@fb.com>
On 07/22/2014 04:53 PM, Chris Mason wrote:
>
> On 07/19/2014 02:23 PM, Martin Steigerwald wrote:
>
>>> Running 3.15.6 with this patch applied on top:
>>> - still causes a hang with `rsync -hPaHAXx --del /mnt/home/nyx/ /home/nyx/`
>>> - no extra error messages printed (`dmesg | grep racing`) compared to
>>> without the patch
>> I got same results with 3.16-rc5 + this patch (see thread BTRFS hang with
>> 3.16-rc5). 3.16-rc4 still is fine with me. No hang whatsoever so far.
>>
>>> To recap some details (so I can have it all in one place):
>>> - /home/ is btrfs with compress=lzo
>> BTRFS RAID 1 with lzo.
>>
>>> - I have _not_ created any nodatacow files.
>> Me neither.
>>
>>> - Full stack is: sata <-> dmcrypt <-> lvm <-> btrfs (I noticed others
>>> mentioning the use of dmcrypt)
>> Same, except no dmcrypt.
>>
> Thanks for the help in tracking this down everyone. We'll get there!
> Are you all running multi-disk systems (from a btrfs POV, more than one
> device?) I don't care how many physical drives this maps to, just does
> btrfs think there's more than one drive.
>
> -chris
3.16-rc6 with your patch on top still causes hangs here.
No traces of "racing" in dmesg
Hang is on a btrfs raid 0 consisting of 3 drives.
Full stack is: sata <-> dmcrypt <-> btrfs raid0
Hang was caused by
1. Several rsync -av --inplace --delete <source> <backup subvol>
2. btrfs subvolume snapshot -r <backup subvol> <bacup snap>
The rsync jobs are done one at a time
btrfs is stuck when trying to create the read only snapshot
--
Torbjørn
All output via netconsole.
sysrq-w:
https://gist.github.com/anonymous/d1837187e261f9a4cbd2#file-gistfile1-txt
sysrq-t:
https://gist.github.com/anonymous/2bdb73f035ab9918c63d#file-gistfile1-txt
dmesg:
[ 9352.784136] INFO: task btrfs-transacti:3874 blocked for more than 120
seconds.
[ 9352.784222] Tainted: G E 3.16.0-rc6+ #64
[ 9352.784270] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9352.784354] btrfs-transacti D ffff88042fc943c0 0 3874 2
0x00000000
[ 9352.784413] ffff8803fb9dfca0 0000000000000002 ffff8800c4214b90
ffff8803fb9dffd8
[ 9352.784502] 00000000000143c0 00000000000143c0 ffff88041977b260
ffff8803d29f23a0
[ 9352.784592] ffff8803d29f23a8 7fffffffffffffff ffff8800c4214b90
ffff880232e2c0a8
[ 9352.784682] Call Trace:
[ 9352.784726] [<ffffffff8170eb59>] schedule+0x29/0x70
[ 9352.784774] [<ffffffff8170df99>] schedule_timeout+0x209/0x280
[ 9352.784827] [<ffffffff8170874b>] ? __slab_free+0xfe/0x2c3
[ 9352.784879] [<ffffffff810829f4>] ? wake_up_worker+0x24/0x30
[ 9352.784929] [<ffffffff8170f656>] wait_for_completion+0xa6/0x160
[ 9352.784981] [<ffffffff8109d4e0>] ? wake_up_state+0x20/0x20
[ 9352.785049] [<ffffffffc045b936>]
btrfs_wait_and_free_delalloc_work+0x16/0x30 [btrfs]
[ 9352.785141] [<ffffffffc04658be>]
btrfs_run_ordered_operations+0x1ee/0x2c0 [btrfs]
[ 9352.785260] [<ffffffffc044bbb7>] btrfs_commit_transaction+0x27/0xa40
[btrfs]
[ 9352.785324] [<ffffffffc0447d65>] transaction_kthread+0x1b5/0x240 [btrfs]
[ 9352.785385] [<ffffffffc0447bb0>] ?
btrfs_cleanup_transaction+0x560/0x560 [btrfs]
[ 9352.785469] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9352.785517] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.785571] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9352.785620] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.785678] INFO: task kworker/u16:3:6932 blocked for more than 120
seconds.
[ 9352.785732] Tainted: G E 3.16.0-rc6+ #64
[ 9352.785780] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9352.785863] kworker/u16:3 D ffff88042fd943c0 0 6932 2
0x00000000
[ 9352.785930] Workqueue: btrfs-flush_delalloc normal_work_helper [btrfs]
[ 9352.785983] ffff88035f1bbb58 0000000000000002 ffff880417e564c0
ffff88035f1bbfd8
[ 9352.786072] 00000000000143c0 00000000000143c0 ffff8800c1a03260
ffff88042fd94cd8
[ 9352.786160] ffff88042ffb4be8 ffff88035f1bbbe0 0000000000000002
ffffffff81159930
[ 9352.786250] Call Trace:
[ 9352.786292] [<ffffffff81159930>] ? wait_on_page_read+0x60/0x60
[ 9352.786343] [<ffffffff8170ee6d>] io_schedule+0x9d/0x130
[ 9352.786393] [<ffffffff8115993e>] sleep_on_page+0xe/0x20
[ 9352.786443] [<ffffffff8170f3e8>] __wait_on_bit_lock+0x48/0xb0
[ 9352.786495] [<ffffffff81159a4a>] __lock_page+0x6a/0x70
[ 9352.786544] [<ffffffff810b14a0>] ? autoremove_wake_function+0x40/0x40
[ 9352.786607] [<ffffffffc046711e>] ? flush_write_bio+0xe/0x10 [btrfs]
[ 9352.786669] [<ffffffffc046b0c0>]
extent_write_cache_pages.isra.28.constprop.46+0x3d0/0x3f0 [btrfs]
[ 9352.786766] [<ffffffffc046cd2d>] extent_writepages+0x4d/0x70 [btrfs]
[ 9352.786828] [<ffffffffc04506f0>] ? btrfs_submit_direct+0x6a0/0x6a0
[btrfs]
[ 9352.786883] [<ffffffff810b0d78>] ? __wake_up_common+0x58/0x90
[ 9352.786943] [<ffffffffc044e1d8>] btrfs_writepages+0x28/0x30 [btrfs]
[ 9352.786997] [<ffffffff811668ee>] do_writepages+0x1e/0x40
[ 9352.787045] [<ffffffff8115b409>] __filemap_fdatawrite_range+0x59/0x60
[ 9352.787097] [<ffffffff8115b4bc>] filemap_flush+0x1c/0x20
[ 9352.787151] [<ffffffffc045158a>] btrfs_run_delalloc_work+0x5a/0xa0
[btrfs]
[ 9352.787211] [<ffffffffc047accf>] normal_work_helper+0x11f/0x2b0 [btrfs]
[ 9352.787266] [<ffffffff810858d2>] process_one_work+0x182/0x450
[ 9352.787317] [<ffffffff81086043>] worker_thread+0x123/0x5a0
[ 9352.787464] [<ffffffff81085f20>] ? rescuer_thread+0x380/0x380
[ 9352.787515] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9352.787563] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.787616] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9352.787666] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.787744] INFO: task kworker/u16:9:8360 blocked for more than 120
seconds.
[ 9352.787799] Tainted: G E 3.16.0-rc6+ #64
[ 9352.787847] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9352.787949] kworker/u16:9 D ffff88042fd543c0 0 8360 2
0x00000000
[ 9352.788015] Workqueue: btrfs-delalloc normal_work_helper [btrfs]
[ 9352.788139] ffff88035f20f838 0000000000000002 ffff8803216c1930
ffff88035f20ffd8
[ 9352.788229] 00000000000143c0 00000000000143c0 ffff8804157e0000
ffff88042fd54cd8
[ 9352.788317] ffff88042ffad6e8 ffff88035f20f8c0 0000000000000002
ffffffff81159930
[ 9352.788406] Call Trace:
[ 9352.788445] [<ffffffff81159930>] ? wait_on_page_read+0x60/0x60
[ 9352.788495] [<ffffffff8170ee6d>] io_schedule+0x9d/0x130
[ 9352.788543] [<ffffffff8115993e>] sleep_on_page+0xe/0x20
[ 9352.788591] [<ffffffff8170f3e8>] __wait_on_bit_lock+0x48/0xb0
[ 9352.788642] [<ffffffff8133c8a4>] ? blk_finish_plug+0x14/0x40
[ 9352.788692] [<ffffffff81159a4a>] __lock_page+0x6a/0x70
[ 9352.788740] [<ffffffff810b14a0>] ? autoremove_wake_function+0x40/0x40
[ 9352.788792] [<ffffffff8115ac34>] pagecache_get_page+0x164/0x1f0
[ 9352.788849] [<ffffffffc048e447>] io_ctl_prepare_pages+0x67/0x180 [btrfs]
[ 9352.788933] [<ffffffffc048fd0f>] __load_free_space_cache+0x1ff/0x700
[btrfs]
[ 9352.788998] [<ffffffffc049030c>] load_free_space_cache+0xfc/0x1c0
[btrfs]
[ 9352.789059] [<ffffffffc0430292>] cache_block_group+0x192/0x390 [btrfs]
[ 9352.789113] [<ffffffff810b1460>] ? prepare_to_wait_event+0x100/0x100
[ 9352.789173] [<ffffffffc04390a2>] find_free_extent+0xc72/0xcb0 [btrfs]
[ 9352.789233] [<ffffffffc04392ef>] btrfs_reserve_extent+0xaf/0x1b0 [btrfs]
[ 9352.789295] [<ffffffffc04526cc>] cow_file_range+0x13c/0x430 [btrfs]
[ 9352.789355] [<ffffffffc0453c80>] ?
submit_compressed_extents+0x480/0x480 [btrfs]
[ 9352.789444] [<ffffffffc04539d2>]
submit_compressed_extents+0x1d2/0x480 [btrfs]
[ 9352.789533] [<ffffffffc0451524>] ? async_cow_free+0x24/0x30 [btrfs]
[ 9352.789592] [<ffffffffc0453c80>] ?
submit_compressed_extents+0x480/0x480 [btrfs]
[ 9352.789680] [<ffffffffc0453d06>] async_cow_submit+0x86/0x90 [btrfs]
[ 9352.789737] [<ffffffffc047ad43>] normal_work_helper+0x193/0x2b0 [btrfs]
[ 9352.789790] [<ffffffff810858d2>] process_one_work+0x182/0x450
[ 9352.789840] [<ffffffff81086043>] worker_thread+0x123/0x5a0
[ 9352.789889] [<ffffffff81085f20>] ? rescuer_thread+0x380/0x380
[ 9352.789938] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9352.789984] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.790038] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9352.790088] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.790144] INFO: task kworker/u16:0:11682 blocked for more than 120
seconds.
[ 9352.790199] Tainted: G E 3.16.0-rc6+ #64
[ 9352.795261] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9352.795347] kworker/u16:0 D ffff88042fd543c0 0 11682 2
0x00000000
[ 9352.795409] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-3)
[ 9352.795465] ffff8802161cf9d8 0000000000000002 ffff8803fc3c0000
ffff8802161cffd8
[ 9352.795554] 00000000000143c0 00000000000143c0 ffff8804197e0000
ffff88042fd54cd8
[ 9352.795644] ffff88042ffaa9e8 ffff8802161cfa60 0000000000000002
ffffffff81159930
[ 9352.795734] Call Trace:
[ 9352.795776] [<ffffffff81159930>] ? wait_on_page_read+0x60/0x60
[ 9352.795829] [<ffffffff8170ee6d>] io_schedule+0x9d/0x130
[ 9352.795879] [<ffffffff8115993e>] sleep_on_page+0xe/0x20
[ 9352.795928] [<ffffffff8170f3e8>] __wait_on_bit_lock+0x48/0xb0
[ 9352.795979] [<ffffffff81159a4a>] __lock_page+0x6a/0x70
[ 9352.796029] [<ffffffff810b14a0>] ? autoremove_wake_function+0x40/0x40
[ 9352.796150] [<ffffffffc046711e>] ? flush_write_bio+0xe/0x10 [btrfs]
[ 9352.796212] [<ffffffffc046b0c0>]
extent_write_cache_pages.isra.28.constprop.46+0x3d0/0x3f0 [btrfs]
[ 9352.796309] [<ffffffffc046cd2d>] extent_writepages+0x4d/0x70 [btrfs]
[ 9352.796369] [<ffffffffc04506f0>] ? btrfs_submit_direct+0x6a0/0x6a0
[btrfs]
[ 9352.796428] [<ffffffffc044e1d8>] btrfs_writepages+0x28/0x30 [btrfs]
[ 9352.796481] [<ffffffff811668ee>] do_writepages+0x1e/0x40
[ 9352.796530] [<ffffffff811f17d0>] __writeback_single_inode+0x40/0x210
[ 9352.796581] [<ffffffff811f22b7>] writeback_sb_inodes+0x247/0x3e0
[ 9352.796632] [<ffffffff811f24ef>] __writeback_inodes_wb+0x9f/0xd0
[ 9352.796682] [<ffffffff811f2763>] wb_writeback+0x243/0x2c0
[ 9352.796731] [<ffffffff811f4e09>] bdi_writeback_workfn+0x1b9/0x430
[ 9352.796784] [<ffffffff810858d2>] process_one_work+0x182/0x450
[ 9352.796833] [<ffffffff81086043>] worker_thread+0x123/0x5a0
[ 9352.796882] [<ffffffff81085f20>] ? rescuer_thread+0x380/0x380
[ 9352.796933] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9352.796979] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.797031] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9352.797079] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.797134] INFO: task kworker/u16:4:13743 blocked for more than 120
seconds.
[ 9352.797189] Tainted: G E 3.16.0-rc6+ #64
[ 9352.797299] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9352.797383] kworker/u16:4 D ffff88042fc943c0 0 13743 2
0x00000000
[ 9352.797451] Workqueue: btrfs-flush_delalloc normal_work_helper [btrfs]
[ 9352.797505] ffff880139843b58 0000000000000002 ffff880191e38000
ffff880139843fd8
[ 9352.797594] 00000000000143c0 00000000000143c0 ffff88041977b260
ffff88042fc94cd8
[ 9352.797684] ffff88042ffb4be8 ffff880139843be0 0000000000000002
ffffffff81159930
[ 9352.797773] Call Trace:
[ 9352.797813] [<ffffffff81159930>] ? wait_on_page_read+0x60/0x60
[ 9352.797865] [<ffffffff8170ee6d>] io_schedule+0x9d/0x130
[ 9352.797914] [<ffffffff8115993e>] sleep_on_page+0xe/0x20
[ 9352.797963] [<ffffffff8170f3e8>] __wait_on_bit_lock+0x48/0xb0
[ 9352.798013] [<ffffffff81159a4a>] __lock_page+0x6a/0x70
[ 9352.798060] [<ffffffff810b14a0>] ? autoremove_wake_function+0x40/0x40
[ 9352.798119] [<ffffffffc046711e>] ? flush_write_bio+0xe/0x10 [btrfs]
[ 9352.798176] [<ffffffffc046b0c0>]
extent_write_cache_pages.isra.28.constprop.46+0x3d0/0x3f0 [btrfs]
[ 9352.798269] [<ffffffffc046cd2d>] extent_writepages+0x4d/0x70 [btrfs]
[ 9352.798326] [<ffffffffc04506f0>] ? btrfs_submit_direct+0x6a0/0x6a0
[btrfs]
[ 9352.798385] [<ffffffffc044e1d8>] btrfs_writepages+0x28/0x30 [btrfs]
[ 9352.798437] [<ffffffff811668ee>] do_writepages+0x1e/0x40
[ 9352.798485] [<ffffffff8115b409>] __filemap_fdatawrite_range+0x59/0x60
[ 9352.798537] [<ffffffff8115b4bc>] filemap_flush+0x1c/0x20
[ 9352.798590] [<ffffffffc045158a>] btrfs_run_delalloc_work+0x5a/0xa0
[btrfs]
[ 9352.798650] [<ffffffffc047accf>] normal_work_helper+0x11f/0x2b0 [btrfs]
[ 9352.798703] [<ffffffff810858d2>] process_one_work+0x182/0x450
[ 9352.798752] [<ffffffff81086043>] worker_thread+0x123/0x5a0
[ 9352.798801] [<ffffffff81085f20>] ? rescuer_thread+0x380/0x380
[ 9352.798851] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9352.798897] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.798949] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9352.798997] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9352.799049] INFO: task btrfs:13823 blocked for more than 120 seconds.
[ 9352.799101] Tainted: G E 3.16.0-rc6+ #64
[ 9352.799148] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9352.799229] btrfs D ffff88042fd943c0 0 13823 13461
0x00000000
[ 9352.799286] ffff880177203a80 0000000000000002 ffff8800c1a01930
ffff880177203fd8
[ 9352.799375] 00000000000143c0 00000000000143c0 ffff8800c2f664c0
ffff8803d29f2438
[ 9352.799463] ffff8803d29f2440 7fffffffffffffff ffff8800c1a01930
ffff880177203bc0
[ 9352.799552] Call Trace:
[ 9352.799590] [<ffffffff8170eb59>] schedule+0x29/0x70
[ 9352.799637] [<ffffffff8170df99>] schedule_timeout+0x209/0x280
[ 9352.799687] [<ffffffff8109d4a3>] ? wake_up_process+0x23/0x40
[ 9352.799736] [<ffffffff810829f4>] ? wake_up_worker+0x24/0x30
[ 9352.799786] [<ffffffff810836ab>] ? insert_work+0x6b/0xb0
[ 9352.799877] [<ffffffff8170f656>] wait_for_completion+0xa6/0x160
[ 9352.799928] [<ffffffff8109d4e0>] ? wake_up_state+0x20/0x20
[ 9352.799989] [<ffffffffc045b936>]
btrfs_wait_and_free_delalloc_work+0x16/0x30 [btrfs]
[ 9352.800080] [<ffffffffc045bafe>] __start_delalloc_inodes+0x1ae/0x2a0
[btrfs]
[ 9352.800150] [<ffffffffc045bc2e>]
btrfs_start_delalloc_inodes+0x3e/0x120 [btrfs]
[ 9352.800235] [<ffffffff810b1228>] ? finish_wait+0x58/0x70
[ 9352.800293] [<ffffffffc047fa24>] btrfs_mksubvol.isra.29+0x1d4/0x570
[btrfs]
[ 9352.800348] [<ffffffff810b1460>] ? prepare_to_wait_event+0x100/0x100
[ 9352.800409] [<ffffffffc047ff46>]
btrfs_ioctl_snap_create_transid+0x186/0x190 [btrfs]
[ 9352.800501] [<ffffffffc04800ab>]
btrfs_ioctl_snap_create_v2+0xeb/0x130 [btrfs]
[ 9352.800592] [<ffffffffc048218f>] btrfs_ioctl+0xcaf/0x2ae0 [btrfs]
[ 9352.800645] [<ffffffff8118d933>] ? mmap_region+0x163/0x5d0
[ 9352.800696] [<ffffffff81057c8c>] ? __do_page_fault+0x20c/0x550
[ 9352.800748] [<ffffffff811dc840>] do_vfs_ioctl+0x2e0/0x4c0
[ 9352.800798] [<ffffffff810a0e64>] ? vtime_account_user+0x54/0x60
[ 9352.800850] [<ffffffff811dcaa1>] SyS_ioctl+0x81/0xa0
[ 9352.800899] [<ffffffff817130bf>] tracesys+0xe1/0xe6
[ 9354.863323] kvm [4191]: vcpu0 unhandled rdmsr: 0x345
[ 9354.869074] kvm_set_msr_common: 118 callbacks suppressed
[ 9354.869130] kvm [4191]: vcpu0 unhandled wrmsr: 0x680 data 0
[ 9354.869185] kvm [4191]: vcpu0 unhandled wrmsr: 0x6c0 data 0
[ 9354.869240] kvm [4191]: vcpu0 unhandled wrmsr: 0x681 data 0
[ 9354.869294] kvm [4191]: vcpu0 unhandled wrmsr: 0x6c1 data 0
[ 9354.869346] kvm [4191]: vcpu0 unhandled wrmsr: 0x682 data 0
[ 9354.869397] kvm [4191]: vcpu0 unhandled wrmsr: 0x6c2 data 0
[ 9354.869449] kvm [4191]: vcpu0 unhandled wrmsr: 0x683 data 0
[ 9354.869502] kvm [4191]: vcpu0 unhandled wrmsr: 0x6c3 data 0
[ 9354.869556] kvm [4191]: vcpu0 unhandled wrmsr: 0x684 data 0
[ 9354.869610] kvm [4191]: vcpu0 unhandled wrmsr: 0x6c4 data 0
[ 9376.448998] kvm [4191]: vcpu0 unhandled rdmsr: 0x611
[ 9376.449052] kvm [4191]: vcpu0 unhandled rdmsr: 0x639
[ 9376.449100] kvm [4191]: vcpu0 unhandled rdmsr: 0x641
[ 9376.449148] kvm [4191]: vcpu0 unhandled rdmsr: 0x619
[ 9472.705412] INFO: task btrfs-transacti:3874 blocked for more than 120
seconds.
[ 9472.705499] Tainted: G E 3.16.0-rc6+ #64
[ 9472.705547] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9472.705629] btrfs-transacti D ffff88042fc943c0 0 3874 2
0x00000000
[ 9472.705688] ffff8803fb9dfca0 0000000000000002 ffff8800c4214b90
ffff8803fb9dffd8
[ 9472.705778] 00000000000143c0 00000000000143c0 ffff88041977b260
ffff8803d29f23a0
[ 9472.705868] ffff8803d29f23a8 7fffffffffffffff ffff8800c4214b90
ffff880232e2c0a8
[ 9472.705976] Call Trace:
[ 9472.706019] [<ffffffff8170eb59>] schedule+0x29/0x70
[ 9472.706067] [<ffffffff8170df99>] schedule_timeout+0x209/0x280
[ 9472.706118] [<ffffffff8170874b>] ? __slab_free+0xfe/0x2c3
[ 9472.706169] [<ffffffff810829f4>] ? wake_up_worker+0x24/0x30
[ 9472.706219] [<ffffffff8170f656>] wait_for_completion+0xa6/0x160
[ 9472.706270] [<ffffffff8109d4e0>] ? wake_up_state+0x20/0x20
[ 9472.706338] [<ffffffffc045b936>]
btrfs_wait_and_free_delalloc_work+0x16/0x30 [btrfs]
[ 9472.706429] [<ffffffffc04658be>]
btrfs_run_ordered_operations+0x1ee/0x2c0 [btrfs]
[ 9472.706518] [<ffffffffc044bbb7>] btrfs_commit_transaction+0x27/0xa40
[btrfs]
[ 9472.706579] [<ffffffffc0447d65>] transaction_kthread+0x1b5/0x240 [btrfs]
[ 9472.706637] [<ffffffffc0447bb0>] ?
btrfs_cleanup_transaction+0x560/0x560 [btrfs]
[ 9472.706720] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9472.706767] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9472.706820] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9472.706869] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9472.706926] INFO: task kworker/u16:3:6932 blocked for more than 120
seconds.
[ 9472.706980] Tainted: G E 3.16.0-rc6+ #64
[ 9472.707029] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9472.707112] kworker/u16:3 D ffff88042fd943c0 0 6932 2
0x00000000
[ 9472.707179] Workqueue: btrfs-flush_delalloc normal_work_helper [btrfs]
[ 9472.707233] ffff88035f1bbb58 0000000000000002 ffff880417e564c0
ffff88035f1bbfd8
[ 9472.707322] 00000000000143c0 00000000000143c0 ffff8800c1a03260
ffff88042fd94cd8
[ 9472.707413] ffff88042ffb4be8 ffff88035f1bbbe0 0000000000000002
ffffffff81159930
[ 9472.707502] Call Trace:
[ 9472.707543] [<ffffffff81159930>] ? wait_on_page_read+0x60/0x60
[ 9472.707593] [<ffffffff8170ee6d>] io_schedule+0x9d/0x130
[ 9472.707643] [<ffffffff8115993e>] sleep_on_page+0xe/0x20
[ 9472.707697] [<ffffffff8170f3e8>] __wait_on_bit_lock+0x48/0xb0
[ 9472.707770] [<ffffffff81159a4a>] __lock_page+0x6a/0x70
[ 9472.707820] [<ffffffff810b14a0>] ? autoremove_wake_function+0x40/0x40
[ 9472.707884] [<ffffffffc046711e>] ? flush_write_bio+0xe/0x10 [btrfs]
[ 9472.707947] [<ffffffffc046b0c0>]
extent_write_cache_pages.isra.28.constprop.46+0x3d0/0x3f0 [btrfs]
[ 9472.713000] [<ffffffffc046cd2d>] extent_writepages+0x4d/0x70 [btrfs]
[ 9472.713092] [<ffffffffc04506f0>] ? btrfs_submit_direct+0x6a0/0x6a0
[btrfs]
[ 9472.713148] [<ffffffff810b0d78>] ? __wake_up_common+0x58/0x90
[ 9472.713203] [<ffffffffc044e1d8>] btrfs_writepages+0x28/0x30 [btrfs]
[ 9472.713256] [<ffffffff811668ee>] do_writepages+0x1e/0x40
[ 9472.713305] [<ffffffff8115b409>] __filemap_fdatawrite_range+0x59/0x60
[ 9472.713359] [<ffffffff8115b4bc>] filemap_flush+0x1c/0x20
[ 9472.713425] [<ffffffffc045158a>] btrfs_run_delalloc_work+0x5a/0xa0
[btrfs]
[ 9472.713491] [<ffffffffc047accf>] normal_work_helper+0x11f/0x2b0 [btrfs]
[ 9472.713547] [<ffffffff810858d2>] process_one_work+0x182/0x450
[ 9472.713598] [<ffffffff81086043>] worker_thread+0x123/0x5a0
[ 9472.713648] [<ffffffff81085f20>] ? rescuer_thread+0x380/0x380
[ 9472.713723] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9472.713794] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9472.713849] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9472.713898] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9472.713953] INFO: task kworker/u16:9:8360 blocked for more than 120
seconds.
[ 9472.714028] Tainted: G E 3.16.0-rc6+ #64
[ 9472.714076] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9472.714159] kworker/u16:9 D ffff88042fd543c0 0 8360 2
0x00000000
[ 9472.714225] Workqueue: btrfs-delalloc normal_work_helper [btrfs]
[ 9472.714277] ffff88035f20f838 0000000000000002 ffff8803216c1930
ffff88035f20ffd8
[ 9472.714366] 00000000000143c0 00000000000143c0 ffff8804157e0000
ffff88042fd54cd8
[ 9472.714455] ffff88042ffad6e8 ffff88035f20f8c0 0000000000000002
ffffffff81159930
[ 9472.714544] Call Trace:
[ 9472.714584] [<ffffffff81159930>] ? wait_on_page_read+0x60/0x60
[ 9472.714636] [<ffffffff8170ee6d>] io_schedule+0x9d/0x130
[ 9472.714684] [<ffffffff8115993e>] sleep_on_page+0xe/0x20
[ 9472.714734] [<ffffffff8170f3e8>] __wait_on_bit_lock+0x48/0xb0
[ 9472.714785] [<ffffffff8133c8a4>] ? blk_finish_plug+0x14/0x40
[ 9472.714835] [<ffffffff81159a4a>] __lock_page+0x6a/0x70
[ 9472.714884] [<ffffffff810b14a0>] ? autoremove_wake_function+0x40/0x40
[ 9472.714938] [<ffffffff8115ac34>] pagecache_get_page+0x164/0x1f0
[ 9472.714999] [<ffffffffc048e447>] io_ctl_prepare_pages+0x67/0x180 [btrfs]
[ 9472.715088] [<ffffffffc048fd0f>] __load_free_space_cache+0x1ff/0x700
[btrfs]
[ 9472.715152] [<ffffffffc049030c>] load_free_space_cache+0xfc/0x1c0
[btrfs]
[ 9472.715213] [<ffffffffc0430292>] cache_block_group+0x192/0x390 [btrfs]
[ 9472.715267] [<ffffffff810b1460>] ? prepare_to_wait_event+0x100/0x100
[ 9472.715328] [<ffffffffc04390a2>] find_free_extent+0xc72/0xcb0 [btrfs]
[ 9472.715388] [<ffffffffc04392ef>] btrfs_reserve_extent+0xaf/0x1b0 [btrfs]
[ 9472.715451] [<ffffffffc04526cc>] cow_file_range+0x13c/0x430 [btrfs]
[ 9472.715511] [<ffffffffc0453c80>] ?
submit_compressed_extents+0x480/0x480 [btrfs]
[ 9472.715602] [<ffffffffc04539d2>]
submit_compressed_extents+0x1d2/0x480 [btrfs]
[ 9472.715691] [<ffffffffc0451524>] ? async_cow_free+0x24/0x30 [btrfs]
[ 9472.715748] [<ffffffffc0453c80>] ?
submit_compressed_extents+0x480/0x480 [btrfs]
[ 9472.715835] [<ffffffffc0453d06>] async_cow_submit+0x86/0x90 [btrfs]
[ 9472.715894] [<ffffffffc047ad43>] normal_work_helper+0x193/0x2b0 [btrfs]
[ 9472.715948] [<ffffffff810858d2>] process_one_work+0x182/0x450
[ 9472.715998] [<ffffffff81086043>] worker_thread+0x123/0x5a0
[ 9472.716047] [<ffffffff81085f20>] ? rescuer_thread+0x380/0x380
[ 9472.716098] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9472.716145] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9472.716197] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9472.716246] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9472.716303] INFO: task kworker/u16:0:11682 blocked for more than 120
seconds.
[ 9472.716357] Tainted: G E 3.16.0-rc6+ #64
[ 9472.716405] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 9472.716488] kworker/u16:0 D ffff88042fd543c0 0 11682 2
0x00000000
[ 9472.716548] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-3)
[ 9472.716610] ffff8802161cf9d8 0000000000000002 ffff8803fc3c0000
ffff8802161cffd8
[ 9472.716699] 00000000000143c0 00000000000143c0 ffff8804197e0000
ffff88042fd54cd8
[ 9472.716788] ffff88042ffaa9e8 ffff8802161cfa60 0000000000000002
ffffffff81159930
[ 9472.716878] Call Trace:
[ 9472.716917] [<ffffffff81159930>] ? wait_on_page_read+0x60/0x60
[ 9472.716968] [<ffffffff8170ee6d>] io_schedule+0x9d/0x130
[ 9472.717017] [<ffffffff8115993e>] sleep_on_page+0xe/0x20
[ 9472.717067] [<ffffffff8170f3e8>] __wait_on_bit_lock+0x48/0xb0
[ 9472.717118] [<ffffffff81159a4a>] __lock_page+0x6a/0x70
[ 9472.717167] [<ffffffff810b14a0>] ? autoremove_wake_function+0x40/0x40
[ 9472.717231] [<ffffffffc046711e>] ? flush_write_bio+0xe/0x10 [btrfs]
[ 9472.717293] [<ffffffffc046b0c0>]
extent_write_cache_pages.isra.28.constprop.46+0x3d0/0x3f0 [btrfs]
[ 9472.717402] [<ffffffffc046cd2d>] extent_writepages+0x4d/0x70 [btrfs]
[ 9472.717465] [<ffffffffc04506f0>] ? btrfs_submit_direct+0x6a0/0x6a0
[btrfs]
[ 9472.717553] [<ffffffffc044e1d8>] btrfs_writepages+0x28/0x30 [btrfs]
[ 9472.717607] [<ffffffff811668ee>] do_writepages+0x1e/0x40
[ 9472.717657] [<ffffffff811f17d0>] __writeback_single_inode+0x40/0x210
[ 9472.717710] [<ffffffff811f22b7>] writeback_sb_inodes+0x247/0x3e0
[ 9472.717784] [<ffffffff811f24ef>] __writeback_inodes_wb+0x9f/0xd0
[ 9472.717836] [<ffffffff811f2763>] wb_writeback+0x243/0x2c0
[ 9472.717885] [<ffffffff811f4e09>] bdi_writeback_workfn+0x1b9/0x430
[ 9472.717938] [<ffffffff810858d2>] process_one_work+0x182/0x450
[ 9472.717990] [<ffffffff81086043>] worker_thread+0x123/0x5a0
[ 9472.718039] [<ffffffff81085f20>] ? rescuer_thread+0x380/0x380
[ 9472.718089] [<ffffffff8108cc52>] kthread+0xd2/0xf0
[ 9472.718136] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
[ 9472.718188] [<ffffffff81712dfc>] ret_from_fork+0x7c/0xb0
[ 9472.718237] [<ffffffff8108cb80>] ? kthread_create_on_node+0x180/0x180
next prev parent reply other threads:[~2014-07-22 19:42 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-06-27 10:02 Blocked tasks on 3.15.1 Tomasz Chmielewski
2014-06-27 13:06 ` Duncan
2014-06-27 15:14 ` Rich Freeman
2014-06-27 15:52 ` Chris Murphy
2014-06-27 17:20 ` Duncan
2014-06-28 0:22 ` Chris Samuel
2014-06-29 20:02 ` Cody P Schafer
2014-06-29 22:22 ` Cody P Schafer
2014-06-30 18:11 ` Chris Mason
2014-06-30 18:30 ` Chris Mason
2014-06-30 23:42 ` Cody P Schafer
2014-07-01 21:04 ` Chris Mason
2014-07-01 23:05 ` Cody P Schafer
2014-07-02 12:27 ` Cody P Schafer
2014-07-02 13:58 ` Chris Mason
2014-07-02 14:15 ` Chris Mason
2014-07-17 13:18 ` Chris Mason
2014-07-19 0:33 ` Blocked tasks on 3.15.1, raid1 btrfs is no ends of trouble for me Marc MERLIN
2014-07-19 0:44 ` Marc MERLIN
2014-07-19 1:58 ` Marc MERLIN
2014-07-19 1:59 ` Chris Samuel
2014-07-19 5:40 ` Marc MERLIN
2014-07-19 17:38 ` Blocked tasks on 3.15.1 Cody P Schafer
2014-07-19 18:23 ` Martin Steigerwald
2014-07-22 14:53 ` Chris Mason
2014-07-22 15:14 ` Torbjørn
2014-07-22 16:46 ` Marc MERLIN
2014-07-22 19:42 ` Torbjørn [this message]
2014-07-22 19:50 ` Chris Mason
2014-07-22 20:10 ` Torbjørn
2014-07-22 21:13 ` Martin Steigerwald
2014-07-22 21:15 ` Chris Mason
2014-07-23 11:13 ` Martin Steigerwald
2014-07-23 1:06 ` Rich Freeman
2014-07-23 6:38 ` Felix Seidel
2014-07-23 13:20 ` Charles Cazabon
2014-07-25 2:27 ` Cody P Schafer
2014-08-07 15:12 ` Tobias Holst
2014-08-07 16:05 ` Duncan
2014-08-12 2:55 ` Charles Cazabon
2014-08-12 2:56 ` Liu Bo
2014-08-12 4:18 ` Duncan
2014-08-12 4:49 ` Marc MERLIN
2014-08-18 20:34 ` James Cloos
2014-07-01 3:06 ` Charles Cazabon
2014-06-30 2:33 ` Rich Freeman
2014-06-27 18:33 ` Rich Freeman
-- strict thread matches above, loose matches on Subject: below --
2014-07-20 21:34 Matt
2014-06-27 1:37 Rich Freeman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53CEBEAB.3010206@skagestad.org \
--to=lists@skagestad.org \
--cc=clm@fb.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.