btrfs-transacti:516 blocked 120 seconds on 3.16-rc1

[Konstantinos Skarlatos <k.skarlatos@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2320 bytes --]

I am not sure this is related with the other reports for lockups etc on 
3.16-rc1, so i am sending it. full dmesg attached. this is after some 
heavy io on a multi disk btrfs filesystem.

[69932.966704] INFO: task btrfs-transacti:516 blocked for more than 120 
seconds.
[69932.966837]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[69932.966921] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[69932.967051] btrfs-transacti D 0000000000000001     0   516      2 
0x00000000
[69932.967060]  ffff8801f422fac0 0000000000000046 ffff880203f3bd20 
00000000000145c0
[69932.967069]  ffff8801f422ffd8 00000000000145c0 ffff880203f3bd20 
ffff8801f422fa30
[69932.967076]  ffffffffa062e392 ffff8800cda63300 ffff8802010b1e60 
00000c73d1920000
[69932.967083] Call Trace:
[69932.967133]  [<ffffffffa062e392>] ? add_delayed_tree_ref+0x102/0x1b0 
[btrfs]
[69932.967146]  [<ffffffff8119937a>] ? kmem_cache_alloc_trace+0x1fa/0x220
[69932.967155]  [<ffffffff814fd759>] schedule+0x29/0x70
[69932.967179]  [<ffffffffa05c8571>] cache_block_group+0x121/0x390 [btrfs]
[69932.967187]  [<ffffffff810b0990>] ? __wake_up_sync+0x20/0x20
[69932.967212]  [<ffffffffa05d16fa>] find_free_extent+0x5fa/0xc80 [btrfs]
[69932.967243]  [<ffffffffa0606f00>] ? free_extent_buffer+0x10/0xa0 [btrfs]
[69932.967269]  [<ffffffffa05d1f52>] btrfs_reserve_extent+0x62/0x140 [btrfs]
[69932.967298]  [<ffffffffa05ed388>] 
__btrfs_prealloc_file_range+0xe8/0x380 [btrfs]
[69932.967328]  [<ffffffffa05f52b0>] 
btrfs_prealloc_file_range_trans+0x30/0x40 [btrfs]
[69932.967353]  [<ffffffffa05d4a97>] 
btrfs_write_dirty_block_groups+0x5c7/0x700 [btrfs]
[69932.967380]  [<ffffffffa05e2b5d>] commit_cowonly_roots+0x18d/0x240 
[btrfs]
[69932.967408]  [<ffffffffa05e4c87>] 
btrfs_commit_transaction+0x4f7/0xa40 [btrfs]
[69932.967435]  [<ffffffffa05e0835>] transaction_kthread+0x1e5/0x250 [btrfs]
[69932.967462]  [<ffffffffa05e0650>] ? 
btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[69932.967471]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[69932.967478]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[69932.967486]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[69932.967493]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[69932.967505] INFO: task kworker/u16:15:30882 blocked for more than 120 
seconds.

-- 
Konstantinos Skarlatos


[-- Attachment #2: 3.16_rc1-blocked120seconds.txt --]
[-- Type: text/plain, Size: 25695 bytes --]

[  995.654816] BTRFS info (device sdh): force zlib compression
[  995.654827] BTRFS info (device sdh): disk space caching is enabled
[  995.654832] BTRFS: has skinny extents
[  995.785405] BTRFS: bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 0, gen 2
[69932.966704] INFO: task btrfs-transacti:516 blocked for more than 120 seconds.
[69932.966837]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[69932.966921] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69932.967051] btrfs-transacti D 0000000000000001     0   516      2 0x00000000
[69932.967060]  ffff8801f422fac0 0000000000000046 ffff880203f3bd20 00000000000145c0
[69932.967069]  ffff8801f422ffd8 00000000000145c0 ffff880203f3bd20 ffff8801f422fa30
[69932.967076]  ffffffffa062e392 ffff8800cda63300 ffff8802010b1e60 00000c73d1920000
[69932.967083] Call Trace:
[69932.967133]  [<ffffffffa062e392>] ? add_delayed_tree_ref+0x102/0x1b0 [btrfs]
[69932.967146]  [<ffffffff8119937a>] ? kmem_cache_alloc_trace+0x1fa/0x220
[69932.967155]  [<ffffffff814fd759>] schedule+0x29/0x70
[69932.967179]  [<ffffffffa05c8571>] cache_block_group+0x121/0x390 [btrfs]
[69932.967187]  [<ffffffff810b0990>] ? __wake_up_sync+0x20/0x20
[69932.967212]  [<ffffffffa05d16fa>] find_free_extent+0x5fa/0xc80 [btrfs]
[69932.967243]  [<ffffffffa0606f00>] ? free_extent_buffer+0x10/0xa0 [btrfs]
[69932.967269]  [<ffffffffa05d1f52>] btrfs_reserve_extent+0x62/0x140 [btrfs]
[69932.967298]  [<ffffffffa05ed388>] __btrfs_prealloc_file_range+0xe8/0x380 [btrfs]
[69932.967328]  [<ffffffffa05f52b0>] btrfs_prealloc_file_range_trans+0x30/0x40 [btrfs]
[69932.967353]  [<ffffffffa05d4a97>] btrfs_write_dirty_block_groups+0x5c7/0x700 [btrfs]
[69932.967380]  [<ffffffffa05e2b5d>] commit_cowonly_roots+0x18d/0x240 [btrfs]
[69932.967408]  [<ffffffffa05e4c87>] btrfs_commit_transaction+0x4f7/0xa40 [btrfs]
[69932.967435]  [<ffffffffa05e0835>] transaction_kthread+0x1e5/0x250 [btrfs]
[69932.967462]  [<ffffffffa05e0650>] ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[69932.967471]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[69932.967478]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[69932.967486]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[69932.967493]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[69932.967505] INFO: task kworker/u16:15:30882 blocked for more than 120 seconds.
[69932.967625]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[69932.967707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69932.967835] kworker/u16:15  D 0000000000000000     0 30882      2 0x00000000
[69932.967867] Workqueue: btrfs-delalloc normal_work_helper [btrfs]
[69932.967871]  ffff88003e537858 0000000000000046 ffff8801fc599e90 00000000000145c0
[69932.967878]  ffff88003e537fd8 00000000000145c0 ffff8801fc599e90 0000000000000000
[69932.967884]  0000000000000000 0000000000000000 ffff8802036bd968 ffff8802036bd848
[69932.967890] Call Trace:
[69932.967900]  [<ffffffff81263923>] ? __blk_run_queue+0x33/0x40
[69932.967908]  [<ffffffff81264bbb>] ? queue_unplugged+0x3b/0xd0
[69932.967916]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[69932.967924]  [<ffffffff81141410>] ? filemap_fdatawait+0x30/0x30
[69932.967930]  [<ffffffff814fd759>] schedule+0x29/0x70
[69932.967936]  [<ffffffff814fda34>] io_schedule+0x94/0xf0
[69932.967943]  [<ffffffff8114141e>] sleep_on_page+0xe/0x20
[69932.967949]  [<ffffffff814fdf68>] __wait_on_bit_lock+0x48/0xb0
[69932.967956]  [<ffffffff8114156a>] __lock_page+0x6a/0x70
[69932.967963]  [<ffffffff810b09d0>] ? autoremove_wake_function+0x40/0x40
[69932.967971]  [<ffffffff8114212c>] pagecache_get_page+0xac/0x1d0
[69932.968000]  [<ffffffffa0626887>] io_ctl_prepare_pages+0x67/0x180 [btrfs]
[69932.968030]  [<ffffffffa06298dd>] __load_free_space_cache+0x1bd/0x680 [btrfs]
[69932.968059]  [<ffffffffa0629e9c>] load_free_space_cache+0xfc/0x1c0 [btrfs]
[69932.968081]  [<ffffffffa05c85e2>] cache_block_group+0x192/0x390 [btrfs]
[69932.968088]  [<ffffffff810b0990>] ? __wake_up_sync+0x20/0x20
[69932.968112]  [<ffffffffa05d16fa>] find_free_extent+0x5fa/0xc80 [btrfs]
[69932.968121]  [<ffffffff8119a301>] ? kmem_cache_free+0x181/0x240
[69932.968145]  [<ffffffffa05d1f52>] btrfs_reserve_extent+0x62/0x140 [btrfs]
[69932.968173]  [<ffffffffa05eb563>] cow_file_range+0x123/0x400 [btrfs]
[69932.968202]  [<ffffffffa05ec855>] submit_compressed_extents+0x1f5/0x460 [btrfs]
[69932.968231]  [<ffffffffa05ecac0>] ? submit_compressed_extents+0x460/0x460 [btrfs]
[69932.968260]  [<ffffffffa05ecb46>] async_cow_submit+0x86/0x90 [btrfs]
[69932.968289]  [<ffffffffa0614735>] normal_work_helper+0x205/0x350 [btrfs]
[69932.968297]  [<ffffffff81085fc8>] process_one_work+0x168/0x450
[69932.968305]  [<ffffffff810865eb>] worker_thread+0x6b/0x550
[69932.968313]  [<ffffffff81086580>] ? init_pwq.part.22+0x10/0x10
[69932.968320]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[69932.968327]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[69932.968334]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[69932.968341]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[69932.968348] INFO: task rsync:30889 blocked for more than 120 seconds.
[69932.968455]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[69932.968537] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69932.968665] rsync           D 0000000000000001     0 30889  30887 0x00000000
[69932.968671]  ffff88009423bb58 0000000000000082 ffff8801cb03bd20 00000000000145c0
[69932.968677]  ffff88009423bfd8 00000000000145c0 ffff8801cb03bd20 ffff88009423bab0
[69932.968683]  ffffffffa0614ce8 ffff88020286e000 0000000000000001 000000000057afff
[69932.968689] Call Trace:
[69932.968719]  [<ffffffffa0614ce8>] ? btrfs_queue_work+0x88/0xf0 [btrfs]
[69932.968748]  [<ffffffffa05ec4a2>] ? run_delalloc_range+0x182/0x340 [btrfs]
[69932.968756]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[69932.968763]  [<ffffffff81141410>] ? filemap_fdatawait+0x30/0x30
[69932.968769]  [<ffffffff814fd759>] schedule+0x29/0x70
[69932.968775]  [<ffffffff814fda34>] io_schedule+0x94/0xf0
[69932.968782]  [<ffffffff8114141e>] sleep_on_page+0xe/0x20
[69932.968788]  [<ffffffff814fdf68>] __wait_on_bit_lock+0x48/0xb0
[69932.968795]  [<ffffffff8114156a>] __lock_page+0x6a/0x70
[69932.968802]  [<ffffffff810b09d0>] ? autoremove_wake_function+0x40/0x40
[69932.968831]  [<ffffffffa060036e>] ? flush_write_bio+0xe/0x10 [btrfs]
[69932.968860]  [<ffffffffa0604268>] extent_write_cache_pages.isra.29.constprop.46+0x2b8/0x3f0 [btrfs]
[69932.968870]  [<ffffffff81151d4d>] ? truncate_inode_pages_range+0x29d/0x740
[69932.968900]  [<ffffffffa060641d>] extent_writepages+0x4d/0x70 [btrfs]
[69932.968928]  [<ffffffffa05e8af0>] ? btrfs_direct_IO+0x360/0x360 [btrfs]
[69932.968956]  [<ffffffffa05e78d8>] btrfs_writepages+0x28/0x30 [btrfs]
[69932.968964]  [<ffffffff8114e92e>] do_writepages+0x1e/0x30
[69932.968972]  [<ffffffff81142d89>] __filemap_fdatawrite_range+0x59/0x60
[69932.968980]  [<ffffffff81142e53>] filemap_fdatawrite_range+0x13/0x20
[69932.969010]  [<ffffffffa05ff37f>] btrfs_wait_ordered_range+0xff/0x150 [btrfs]
[69932.969038]  [<ffffffffa05eee6a>] btrfs_truncate+0x4a/0x330 [btrfs]
[69932.969046]  [<ffffffff811522ca>] ? truncate_pagecache+0x5a/0x70
[69932.969074]  [<ffffffffa05efb98>] btrfs_setattr+0x228/0x2e0 [btrfs]
[69932.969083]  [<ffffffff811d1fb1>] notify_change+0x221/0x380
[69932.969091]  [<ffffffff811b4846>] do_truncate+0x66/0x90
[69932.969097]  [<ffffffff811b8d39>] ? __sb_start_write+0x49/0xf0
[69932.969105]  [<ffffffff811b4bbb>] do_sys_ftruncate.constprop.10+0x10b/0x160
[69932.969112]  [<ffffffff811b4c4e>] SyS_ftruncate+0xe/0x10
[69932.969119]  [<ffffffff81501429>] system_call_fastpath+0x16/0x1b
[69932.969126] INFO: task kworker/u16:20:31689 blocked for more than 120 seconds.
[69932.969245]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[69932.969326] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69932.969454] kworker/u16:20  D 0000000000000000     0 31689      2 0x00000000
[69932.969464] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-2)
[69932.969468]  ffff88009d0179c8 0000000000000046 ffff8802034bc750 00000000000145c0
[69932.969474]  ffff88009d017fd8 00000000000145c0 ffff8802034bc750 ffff8802035cb1c0
[69932.969480]  ffff88009d017928 ffffffffa0614ce8 ffff88020286e000 000000000000003c
[69932.969486] Call Trace:
[69932.969516]  [<ffffffffa0614ce8>] ? btrfs_queue_work+0x88/0xf0 [btrfs]
[69932.969545]  [<ffffffffa05ec4a2>] ? run_delalloc_range+0x182/0x340 [btrfs]
[69932.969552]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[69932.969560]  [<ffffffff810d37a8>] ? ktime_get_ts+0x48/0xf0
[69932.969567]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[69932.969574]  [<ffffffff81141410>] ? filemap_fdatawait+0x30/0x30
[69932.969580]  [<ffffffff814fd759>] schedule+0x29/0x70
[69932.969585]  [<ffffffff814fda34>] io_schedule+0x94/0xf0
[69932.969592]  [<ffffffff8114141e>] sleep_on_page+0xe/0x20
[69932.969598]  [<ffffffff814fdf68>] __wait_on_bit_lock+0x48/0xb0
[69932.969605]  [<ffffffff8114156a>] __lock_page+0x6a/0x70
[69932.969612]  [<ffffffff810b09d0>] ? autoremove_wake_function+0x40/0x40
[69932.969619]  [<ffffffff8114fad1>] ? pagevec_lookup_tag+0x21/0x30
[69932.969648]  [<ffffffffa060036e>] ? flush_write_bio+0xe/0x10 [btrfs]
[69932.969678]  [<ffffffffa0604268>] extent_write_cache_pages.isra.29.constprop.46+0x2b8/0x3f0 [btrfs]
[69932.969709]  [<ffffffffa060641d>] extent_writepages+0x4d/0x70 [btrfs]
[69932.969737]  [<ffffffffa05e8af0>] ? btrfs_direct_IO+0x360/0x360 [btrfs]
[69932.969764]  [<ffffffffa05e78d8>] btrfs_writepages+0x28/0x30 [btrfs]
[69932.969771]  [<ffffffff8114e92e>] do_writepages+0x1e/0x30
[69932.969778]  [<ffffffff811dfa70>] __writeback_single_inode+0x40/0x2b0
[69932.969785]  [<ffffffff811e0e97>] writeback_sb_inodes+0x247/0x400
[69932.969792]  [<ffffffff811e10ef>] __writeback_inodes_wb+0x9f/0xd0
[69932.969798]  [<ffffffff811e132b>] wb_writeback+0x20b/0x330
[69932.969805]  [<ffffffff811e18c4>] bdi_writeback_workfn+0x314/0x490
[69932.969814]  [<ffffffff81085fc8>] process_one_work+0x168/0x450
[69932.969821]  [<ffffffff810865eb>] worker_thread+0x6b/0x550
[69932.969829]  [<ffffffff81086580>] ? init_pwq.part.22+0x10/0x10
[69932.969836]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[69932.969843]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[69932.969850]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[69932.969857]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70053.125951] INFO: task btrfs-transacti:516 blocked for more than 120 seconds.
[70053.126012]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[70053.126069] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[70053.126123] btrfs-transacti D 0000000000000001     0   516      2 0x00000000
[70053.126127]  ffff8801f422fac0 0000000000000046 ffff880203f3bd20 00000000000145c0
[70053.126130]  ffff8801f422ffd8 00000000000145c0 ffff880203f3bd20 ffff8801f422fa30
[70053.126138]  ffffffffa062e392 ffff8800cda63300 ffff8802010b1e60 00000c73d1920000
[70053.126141] Call Trace:
[70053.126171]  [<ffffffffa062e392>] ? add_delayed_tree_ref+0x102/0x1b0 [btrfs]
[70053.126179]  [<ffffffff8119937a>] ? kmem_cache_alloc_trace+0x1fa/0x220
[70053.126183]  [<ffffffff814fd759>] schedule+0x29/0x70
[70053.126192]  [<ffffffffa05c8571>] cache_block_group+0x121/0x390 [btrfs]
[70053.126196]  [<ffffffff810b0990>] ? __wake_up_sync+0x20/0x20
[70053.126205]  [<ffffffffa05d16fa>] find_free_extent+0x5fa/0xc80 [btrfs]
[70053.126217]  [<ffffffffa0606f00>] ? free_extent_buffer+0x10/0xa0 [btrfs]
[70053.126227]  [<ffffffffa05d1f52>] btrfs_reserve_extent+0x62/0x140 [btrfs]
[70053.126238]  [<ffffffffa05ed388>] __btrfs_prealloc_file_range+0xe8/0x380 [btrfs]
[70053.126249]  [<ffffffffa05f52b0>] btrfs_prealloc_file_range_trans+0x30/0x40 [btrfs]
[70053.126259]  [<ffffffffa05d4a97>] btrfs_write_dirty_block_groups+0x5c7/0x700 [btrfs]
[70053.126269]  [<ffffffffa05e2b5d>] commit_cowonly_roots+0x18d/0x240 [btrfs]
[70053.126280]  [<ffffffffa05e4c87>] btrfs_commit_transaction+0x4f7/0xa40 [btrfs]
[70053.126290]  [<ffffffffa05e0835>] transaction_kthread+0x1e5/0x250 [btrfs]
[70053.126301]  [<ffffffffa05e0650>] ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[70053.126304]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[70053.126307]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70053.126310]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[70053.126313]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70053.126318] INFO: task kworker/u16:15:30882 blocked for more than 120 seconds.
[70053.126364]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[70053.126396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[70053.126444] kworker/u16:15  D 0000000000000000     0 30882      2 0x00000000
[70053.126457] Workqueue: btrfs-delalloc normal_work_helper [btrfs]
[70053.126459]  ffff88003e537858 0000000000000046 ffff8801fc599e90 00000000000145c0
[70053.126471]  ffff88003e537fd8 00000000000145c0 ffff8801fc599e90 0000000000000000
[70053.126474]  0000000000000000 0000000000000000 ffff8802036bd968 ffff8802036bd848
[70053.126476] Call Trace:
[70053.126480]  [<ffffffff81263923>] ? __blk_run_queue+0x33/0x40
[70053.126489]  [<ffffffff81264bbb>] ? queue_unplugged+0x3b/0xd0
[70053.126492]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[70053.126496]  [<ffffffff81141410>] ? filemap_fdatawait+0x30/0x30
[70053.126500]  [<ffffffff814fd759>] schedule+0x29/0x70
[70053.126503]  [<ffffffff814fda34>] io_schedule+0x94/0xf0
[70053.126505]  [<ffffffff8114141e>] sleep_on_page+0xe/0x20
[70053.126508]  [<ffffffff814fdf68>] __wait_on_bit_lock+0x48/0xb0
[70053.126511]  [<ffffffff8114156a>] __lock_page+0x6a/0x70
[70053.126513]  [<ffffffff810b09d0>] ? autoremove_wake_function+0x40/0x40
[70053.126516]  [<ffffffff8114212c>] pagecache_get_page+0xac/0x1d0
[70053.126527]  [<ffffffffa0626887>] io_ctl_prepare_pages+0x67/0x180 [btrfs]
[70053.126539]  [<ffffffffa06298dd>] __load_free_space_cache+0x1bd/0x680 [btrfs]
[70053.126550]  [<ffffffffa0629e9c>] load_free_space_cache+0xfc/0x1c0 [btrfs]
[70053.126558]  [<ffffffffa05c85e2>] cache_block_group+0x192/0x390 [btrfs]
[70053.126561]  [<ffffffff810b0990>] ? __wake_up_sync+0x20/0x20
[70053.126570]  [<ffffffffa05d16fa>] find_free_extent+0x5fa/0xc80 [btrfs]
[70053.126573]  [<ffffffff8119a301>] ? kmem_cache_free+0x181/0x240
[70053.126583]  [<ffffffffa05d1f52>] btrfs_reserve_extent+0x62/0x140 [btrfs]
[70053.126593]  [<ffffffffa05eb563>] cow_file_range+0x123/0x400 [btrfs]
[70053.126604]  [<ffffffffa05ec855>] submit_compressed_extents+0x1f5/0x460 [btrfs]
[70053.126616]  [<ffffffffa05ecac0>] ? submit_compressed_extents+0x460/0x460 [btrfs]
[70053.126626]  [<ffffffffa05ecb46>] async_cow_submit+0x86/0x90 [btrfs]
[70053.126637]  [<ffffffffa0614735>] normal_work_helper+0x205/0x350 [btrfs]
[70053.126641]  [<ffffffff81085fc8>] process_one_work+0x168/0x450
[70053.126644]  [<ffffffff810865eb>] worker_thread+0x6b/0x550
[70053.126647]  [<ffffffff81086580>] ? init_pwq.part.22+0x10/0x10
[70053.126649]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[70053.126652]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70053.126655]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[70053.126657]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70053.126660] INFO: task rsync:30889 blocked for more than 120 seconds.
[70053.126701]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[70053.126732] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[70053.126781] rsync           D 0000000000000001     0 30889  30887 0x00000000
[70053.126783]  ffff88009423bb58 0000000000000082 ffff8801cb03bd20 00000000000145c0
[70053.126786]  ffff88009423bfd8 00000000000145c0 ffff8801cb03bd20 ffff88009423bab0
[70053.126788]  ffffffffa0614ce8 ffff88020286e000 0000000000000001 000000000057afff
[70053.126790] Call Trace:
[70053.126802]  [<ffffffffa0614ce8>] ? btrfs_queue_work+0x88/0xf0 [btrfs]
[70053.126813]  [<ffffffffa05ec4a2>] ? run_delalloc_range+0x182/0x340 [btrfs]
[70053.126816]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[70053.126819]  [<ffffffff81141410>] ? filemap_fdatawait+0x30/0x30
[70053.126821]  [<ffffffff814fd759>] schedule+0x29/0x70
[70053.126823]  [<ffffffff814fda34>] io_schedule+0x94/0xf0
[70053.126826]  [<ffffffff8114141e>] sleep_on_page+0xe/0x20
[70053.126828]  [<ffffffff814fdf68>] __wait_on_bit_lock+0x48/0xb0
[70053.126831]  [<ffffffff8114156a>] __lock_page+0x6a/0x70
[70053.126833]  [<ffffffff810b09d0>] ? autoremove_wake_function+0x40/0x40
[70053.126844]  [<ffffffffa060036e>] ? flush_write_bio+0xe/0x10 [btrfs]
[70053.126856]  [<ffffffffa0604268>] extent_write_cache_pages.isra.29.constprop.46+0x2b8/0x3f0 [btrfs]
[70053.126859]  [<ffffffff81151d4d>] ? truncate_inode_pages_range+0x29d/0x740
[70053.126871]  [<ffffffffa060641d>] extent_writepages+0x4d/0x70 [btrfs]
[70053.126882]  [<ffffffffa05e8af0>] ? btrfs_direct_IO+0x360/0x360 [btrfs]
[70053.126893]  [<ffffffffa05e78d8>] btrfs_writepages+0x28/0x30 [btrfs]
[70053.126895]  [<ffffffff8114e92e>] do_writepages+0x1e/0x30
[70053.126899]  [<ffffffff81142d89>] __filemap_fdatawrite_range+0x59/0x60
[70053.126902]  [<ffffffff81142e53>] filemap_fdatawrite_range+0x13/0x20
[70053.126913]  [<ffffffffa05ff37f>] btrfs_wait_ordered_range+0xff/0x150 [btrfs]
[70053.126924]  [<ffffffffa05eee6a>] btrfs_truncate+0x4a/0x330 [btrfs]
[70053.126927]  [<ffffffff811522ca>] ? truncate_pagecache+0x5a/0x70
[70053.126938]  [<ffffffffa05efb98>] btrfs_setattr+0x228/0x2e0 [btrfs]
[70053.126941]  [<ffffffff811d1fb1>] notify_change+0x221/0x380
[70053.126945]  [<ffffffff811b4846>] do_truncate+0x66/0x90
[70053.126947]  [<ffffffff811b8d39>] ? __sb_start_write+0x49/0xf0
[70053.126950]  [<ffffffff811b4bbb>] do_sys_ftruncate.constprop.10+0x10b/0x160
[70053.126953]  [<ffffffff811b4c4e>] SyS_ftruncate+0xe/0x10
[70053.126955]  [<ffffffff81501429>] system_call_fastpath+0x16/0x1b
[70053.126958] INFO: task kworker/u16:20:31689 blocked for more than 120 seconds.
[70053.127004]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[70053.127035] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[70053.127083] kworker/u16:20  D 0000000000000000     0 31689      2 0x00000000
[70053.127088] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-2)
[70053.127090]  ffff88009d0179c8 0000000000000046 ffff8802034bc750 00000000000145c0
[70053.127092]  ffff88009d017fd8 00000000000145c0 ffff8802034bc750 ffff8802035cb1c0
[70053.127094]  ffff88009d017928 ffffffffa0614ce8 ffff88020286e000 000000000000003c
[70053.127097] Call Trace:
[70053.127108]  [<ffffffffa0614ce8>] ? btrfs_queue_work+0x88/0xf0 [btrfs]
[70053.127119]  [<ffffffffa05ec4a2>] ? run_delalloc_range+0x182/0x340 [btrfs]
[70053.127121]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[70053.127125]  [<ffffffff810d37a8>] ? ktime_get_ts+0x48/0xf0
[70053.127127]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[70053.127130]  [<ffffffff81141410>] ? filemap_fdatawait+0x30/0x30
[70053.127132]  [<ffffffff814fd759>] schedule+0x29/0x70
[70053.127134]  [<ffffffff814fda34>] io_schedule+0x94/0xf0
[70053.127137]  [<ffffffff8114141e>] sleep_on_page+0xe/0x20
[70053.127139]  [<ffffffff814fdf68>] __wait_on_bit_lock+0x48/0xb0
[70053.127142]  [<ffffffff8114156a>] __lock_page+0x6a/0x70
[70053.127144]  [<ffffffff810b09d0>] ? autoremove_wake_function+0x40/0x40
[70053.127147]  [<ffffffff8114fad1>] ? pagevec_lookup_tag+0x21/0x30
[70053.127158]  [<ffffffffa060036e>] ? flush_write_bio+0xe/0x10 [btrfs]
[70053.127170]  [<ffffffffa0604268>] extent_write_cache_pages.isra.29.constprop.46+0x2b8/0x3f0 [btrfs]
[70053.127182]  [<ffffffffa060641d>] extent_writepages+0x4d/0x70 [btrfs]
[70053.127192]  [<ffffffffa05e8af0>] ? btrfs_direct_IO+0x360/0x360 [btrfs]
[70053.127203]  [<ffffffffa05e78d8>] btrfs_writepages+0x28/0x30 [btrfs]
[70053.127205]  [<ffffffff8114e92e>] do_writepages+0x1e/0x30
[70053.127208]  [<ffffffff811dfa70>] __writeback_single_inode+0x40/0x2b0
[70053.127210]  [<ffffffff811e0e97>] writeback_sb_inodes+0x247/0x400
[70053.127213]  [<ffffffff811e10ef>] __writeback_inodes_wb+0x9f/0xd0
[70053.127216]  [<ffffffff811e132b>] wb_writeback+0x20b/0x330
[70053.127218]  [<ffffffff811e18c4>] bdi_writeback_workfn+0x314/0x490
[70053.127222]  [<ffffffff81085fc8>] process_one_work+0x168/0x450
[70053.127224]  [<ffffffff810865eb>] worker_thread+0x6b/0x550
[70053.127227]  [<ffffffff81086580>] ? init_pwq.part.22+0x10/0x10
[70053.127230]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[70053.127233]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70053.127235]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[70053.127238]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70173.288526] INFO: task btrfs-transacti:516 blocked for more than 120 seconds.
[70173.288580]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[70173.288612] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[70173.288662] btrfs-transacti D 0000000000000001     0   516      2 0x00000000
[70173.288667]  ffff8801f422fac0 0000000000000046 ffff880203f3bd20 00000000000145c0
[70173.288670]  ffff8801f422ffd8 00000000000145c0 ffff880203f3bd20 ffff8801f422fa30
[70173.288673]  ffffffffa062e392 ffff8800cda63300 ffff8802010b1e60 00000c73d1920000
[70173.288676] Call Trace:
[70173.288701]  [<ffffffffa062e392>] ? add_delayed_tree_ref+0x102/0x1b0 [btrfs]
[70173.288707]  [<ffffffff8119937a>] ? kmem_cache_alloc_trace+0x1fa/0x220
[70173.288711]  [<ffffffff814fd759>] schedule+0x29/0x70
[70173.288720]  [<ffffffffa05c8571>] cache_block_group+0x121/0x390 [btrfs]
[70173.288724]  [<ffffffff810b0990>] ? __wake_up_sync+0x20/0x20
[70173.288733]  [<ffffffffa05d16fa>] find_free_extent+0x5fa/0xc80 [btrfs]
[70173.288745]  [<ffffffffa0606f00>] ? free_extent_buffer+0x10/0xa0 [btrfs]
[70173.288755]  [<ffffffffa05d1f52>] btrfs_reserve_extent+0x62/0x140 [btrfs]
[70173.288766]  [<ffffffffa05ed388>] __btrfs_prealloc_file_range+0xe8/0x380 [btrfs]
[70173.288777]  [<ffffffffa05f52b0>] btrfs_prealloc_file_range_trans+0x30/0x40 [btrfs]
[70173.288787]  [<ffffffffa05d4a97>] btrfs_write_dirty_block_groups+0x5c7/0x700 [btrfs]
[70173.288798]  [<ffffffffa05e2b5d>] commit_cowonly_roots+0x18d/0x240 [btrfs]
[70173.288808]  [<ffffffffa05e4c87>] btrfs_commit_transaction+0x4f7/0xa40 [btrfs]
[70173.288819]  [<ffffffffa05e0835>] transaction_kthread+0x1e5/0x250 [btrfs]
[70173.288829]  [<ffffffffa05e0650>] ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[70173.288833]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[70173.288836]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70173.288839]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[70173.288842]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70173.288847] INFO: task kworker/u16:15:30882 blocked for more than 120 seconds.
[70173.288893]       Not tainted 3.16.0-rc1-ge99cfa2 #1
[70173.288924] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[70173.289004] kworker/u16:15  D 0000000000000000     0 30882      2 0x00000000
[70173.289017] Workqueue: btrfs-delalloc normal_work_helper [btrfs]
[70173.289019]  ffff88003e537858 0000000000000046 ffff8801fc599e90 00000000000145c0
[70173.289022]  ffff88003e537fd8 00000000000145c0 ffff8801fc599e90 0000000000000000
[70173.289024]  0000000000000000 0000000000000000 ffff8802036bd968 ffff8802036bd848
[70173.289027] Call Trace:
[70173.289031]  [<ffffffff81263923>] ? __blk_run_queue+0x33/0x40
[70173.289034]  [<ffffffff81264bbb>] ? queue_unplugged+0x3b/0xd0
[70173.289038]  [<ffffffff8101dda9>] ? read_tsc+0x9/0x20
[70173.289042]  [<ffffffff81141410>] ? filemap_fdatawait+0x30/0x30
[70173.289044]  [<ffffffff814fd759>] schedule+0x29/0x70
[70173.289046]  [<ffffffff814fda34>] io_schedule+0x94/0xf0
[70173.289049]  [<ffffffff8114141e>] sleep_on_page+0xe/0x20
[70173.289051]  [<ffffffff814fdf68>] __wait_on_bit_lock+0x48/0xb0
[70173.289054]  [<ffffffff8114156a>] __lock_page+0x6a/0x70
[70173.289057]  [<ffffffff810b09d0>] ? autoremove_wake_function+0x40/0x40
[70173.289060]  [<ffffffff8114212c>] pagecache_get_page+0xac/0x1d0
[70173.289071]  [<ffffffffa0626887>] io_ctl_prepare_pages+0x67/0x180 [btrfs]
[70173.289082]  [<ffffffffa06298dd>] __load_free_space_cache+0x1bd/0x680 [btrfs]
[70173.289094]  [<ffffffffa0629e9c>] load_free_space_cache+0xfc/0x1c0 [btrfs]
[70173.289102]  [<ffffffffa05c85e2>] cache_block_group+0x192/0x390 [btrfs]
[70173.289105]  [<ffffffff810b0990>] ? __wake_up_sync+0x20/0x20
[70173.289114]  [<ffffffffa05d16fa>] find_free_extent+0x5fa/0xc80 [btrfs]
[70173.289117]  [<ffffffff8119a301>] ? kmem_cache_free+0x181/0x240
[70173.289127]  [<ffffffffa05d1f52>] btrfs_reserve_extent+0x62/0x140 [btrfs]
[70173.289137]  [<ffffffffa05eb563>] cow_file_range+0x123/0x400 [btrfs]
[70173.289149]  [<ffffffffa05ec855>] submit_compressed_extents+0x1f5/0x460 [btrfs]
[70173.289160]  [<ffffffffa05ecac0>] ? submit_compressed_extents+0x460/0x460 [btrfs]
[70173.289170]  [<ffffffffa05ecb46>] async_cow_submit+0x86/0x90 [btrfs]
[70173.289181]  [<ffffffffa0614735>] normal_work_helper+0x205/0x350 [btrfs]
[70173.289185]  [<ffffffff81085fc8>] process_one_work+0x168/0x450
[70173.289188]  [<ffffffff810865eb>] worker_thread+0x6b/0x550
[70173.289192]  [<ffffffff81086580>] ? init_pwq.part.22+0x10/0x10
[70173.289194]  [<ffffffff8108c97b>] kthread+0xdb/0x100
[70173.289197]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180
[70173.289200]  [<ffffffff8150137c>] ret_from_fork+0x7c/0xb0
[70173.289202]  [<ffffffff8108c8a0>] ? kthread_create_on_node+0x180/0x180