* Blcoked for more than 120 seconds @ 2011-11-28 8:16 Tobias 2011-11-28 9:29 ` Chris Samuel 0 siblings, 1 reply; 15+ messages in thread From: Tobias @ 2011-11-28 8:16 UTC (permalink / raw) To: linux-btrfs Hi! I have two Hardware-Raid Arrays. Each encrypted with dmcrypt and on each one btrfs on it (without snapshots). They are half-filled I do rsyncs on these fs over night and i get these errors: INFO: task flush-btrfs-2:1472 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-btrfs-2 D ffff8802313fb1a0 0 1472 2 0x00000000 ffff880227c318d0 0000000000000046 ffff88006320b798 ffff88006320b768 ffff8802313fade0 ffff880227c31fd8 ffff880227c31fd8 ffff880227c31fd8 ffff880211da5bc0 ffff8802313fade0 ffff880227c318d0 ffffffff8108f1fd Call Trace: [<ffffffff8108f1fd>] ? ktime_get_ts+0xad/0xe0 [<ffffffff81109a30>] ? __lock_page+0x70/0x70 [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffff815ffc4f>] io_schedule+0x8f/0xd0 [<ffffffff81109a3e>] sleep_on_page+0xe/0x20 [<ffffffff8160031a>] __wait_on_bit_lock+0x5a/0xc0 [<ffffffff81109a27>] __lock_page+0x67/0x70 [<ffffffff81085440>] ? autoremove_wake_function+0x40/0x40 [<ffffffffa02430e1>] extent_write_cache_pages.isra.23.constprop.32+0x1e1/0x3b0 [btrfs] [<ffffffffa02434f5>] extent_writepages+0x45/0x60 [btrfs] [<ffffffffa0228710>] ? acls_after_inode_item+0xc0/0xc0 [btrfs] [<ffffffff81084fd4>] ? bit_waitqueue+0x14/0xc0 [<ffffffffa0227958>] btrfs_writepages+0x28/0x30 [btrfs] [<ffffffff81114e81>] do_writepages+0x21/0x40 [<ffffffff8118f0b0>] writeback_single_inode+0x180/0x430 [<ffffffff8118f776>] writeback_sb_inodes+0x1b6/0x270 [<ffffffff8118f8ce>] __writeback_inodes_wb+0x9e/0xd0 [<ffffffff8118fb6b>] wb_writeback+0x26b/0x350 [<ffffffff8118fce8>] wb_check_old_data_flush+0x98/0xa0 [<ffffffff8119006c>] wb_do_writeback+0x15c/0x200 [<ffffffff816000b5>] ? schedule_timeout+0x175/0x320 [<ffffffff81071840>] ? usleep_range+0x50/0x50 [<ffffffff81190193>] bdi_writeback_thread+0x83/0x2a0 [<ffffffff81190110>] ? wb_do_writeback+0x200/0x200 [<ffffffff8108496c>] kthread+0x8c/0xa0 [<ffffffff8160bb74>] kernel_thread_helper+0x4/0x10 [<ffffffff810848e0>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8160bb70>] ? gs_change+0x13/0x13 INFO: task rsync:1887 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsync D ffff880227d49ab0 0 1887 1884 0x00000000 ffff8801a2bd7b68 0000000000000086 ffff880200000000 ffff8801a2bd7b08 ffff880227d496f0 ffff8801a2bd7fd8 ffff8801a2bd7fd8 ffff8801a2bd7fd8 ffffffff81c0d020 ffff880227d496f0 0000000000000286 000000012fd942f0 Call Trace: [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffffa022422d>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [<ffffffff81085400>] ? add_wait_queue+0x60/0x60 [<ffffffffa0225a50>] start_transaction+0x1f0/0x2b0 [btrfs] [<ffffffffa0225b65>] btrfs_join_transaction+0x15/0x20 [btrfs] [<ffffffffa0230788>] btrfs_dirty_inode+0x48/0x160 [btrfs] [<ffffffff8118dfb0>] __mark_inode_dirty+0x40/0x230 [<ffffffff8117fed7>] file_update_time+0xe7/0x150 [<ffffffffa02348a0>] btrfs_file_aio_write+0x1c0/0x510 [btrfs] [<ffffffff8112f11b>] ? handle_pte_fault+0x61b/0xac0 [<ffffffff81166c62>] do_sync_write+0xd2/0x110 [<ffffffff8128dacc>] ? security_file_permission+0x2c/0xb0 [<ffffffff81167101>] ? rw_verify_area+0x61/0xf0 [<ffffffff81167463>] vfs_write+0xb3/0x180 [<ffffffff8116778a>] sys_write+0x4a/0x90 [<ffffffff81609a02>] system_call_fastpath+0x16/0x1b INFO: task btrfs-delalloc-:2148 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. btrfs-delalloc- D ffff880089b8c890 0 2148 2 0x00000000 ffff880039dbbb40 0000000000000046 0000000000000000 0000000000001000 ffff880089b8c4d0 ffff880039dbbfd8 ffff880039dbbfd8 ffff880039dbbfd8 ffff8802321444d0 ffff880089b8c4d0 0000000000000286 000000012fd942f0 Call Trace: [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffffa022422d>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [<ffffffff81085400>] ? add_wait_queue+0x60/0x60 [<ffffffffa0225a50>] start_transaction+0x1f0/0x2b0 [btrfs] [<ffffffffa0225b65>] btrfs_join_transaction+0x15/0x20 [btrfs] [<ffffffffa022c07c>] cow_file_range+0x7c/0x3a0 [btrfs] [<ffffffffa022d0f2>] submit_compressed_extents+0x242/0x470 [btrfs] [<ffffffffa022d7cc>] ? compress_file_range+0x41c/0x610 [btrfs] [<ffffffffa022d3a4>] async_cow_submit+0x84/0x90 [btrfs] [<ffffffffa024c490>] run_ordered_completions+0x80/0xe0 [btrfs] [<ffffffffa024ce80>] worker_loop+0x180/0x560 [btrfs] [<ffffffffa024cd00>] ? btrfs_queue_worker+0x300/0x300 [btrfs] [<ffffffff8108496c>] kthread+0x8c/0xa0 [<ffffffff8160bb74>] kernel_thread_helper+0x4/0x10 [<ffffffff810848e0>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8160bb70>] ? gs_change+0x13/0x13 INFO: task rsync:2217 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsync D ffff880212624890 0 2217 2212 0x00000000 ffff8801acaabd98 0000000000000082 0000000000000000 0000000000000000 ffff8802126244d0 ffff8801acaabfd8 ffff8801acaabfd8 ffff8801acaabfd8 ffff88023212dbc0 ffff8802126244d0 0000000000000282 000000012fd942f0 Call Trace: [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffffa022422d>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [<ffffffff81085400>] ? add_wait_queue+0x60/0x60 [<ffffffffa0225995>] start_transaction+0x135/0x2b0 [btrfs] [<ffffffff8117651a>] ? kern_path_create+0x8a/0x120 [<ffffffffa0225de3>] btrfs_start_transaction+0x13/0x20 [btrfs] [<ffffffffa0231cb5>] btrfs_link+0xa5/0x1a0 [btrfs] [<ffffffff81173211>] vfs_link+0x101/0x190 [<ffffffff81177648>] sys_linkat+0x168/0x180 [<ffffffff8117767e>] sys_link+0x1e/0x20 [<ffffffff81609a02>] system_call_fastpath+0x16/0x1b INFO: task flush-btrfs-2:1472 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-btrfs-2 D ffff8802313fb1a0 0 1472 2 0x00000000 ffff880227c318d0 0000000000000046 ffff88006320b798 ffff88006320b768 ffff8802313fade0 ffff880227c31fd8 ffff880227c31fd8 ffff880227c31fd8 ffff880211da5bc0 ffff8802313fade0 ffff880227c318d0 ffffffff8108f1fd Call Trace: [<ffffffff8108f1fd>] ? ktime_get_ts+0xad/0xe0 [<ffffffff81109a30>] ? __lock_page+0x70/0x70 [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffff815ffc4f>] io_schedule+0x8f/0xd0 [<ffffffff81109a3e>] sleep_on_page+0xe/0x20 [<ffffffff8160031a>] __wait_on_bit_lock+0x5a/0xc0 [<ffffffff81109a27>] __lock_page+0x67/0x70 [<ffffffff81085440>] ? autoremove_wake_function+0x40/0x40 [<ffffffffa02430e1>] extent_write_cache_pages.isra.23.constprop.32+0x1e1/0x3b0 [btrfs] [<ffffffffa02434f5>] extent_writepages+0x45/0x60 [btrfs] [<ffffffffa0228710>] ? acls_after_inode_item+0xc0/0xc0 [btrfs] [<ffffffff81084fd4>] ? bit_waitqueue+0x14/0xc0 [<ffffffffa0227958>] btrfs_writepages+0x28/0x30 [btrfs] [<ffffffff81114e81>] do_writepages+0x21/0x40 [<ffffffff8118f0b0>] writeback_single_inode+0x180/0x430 [<ffffffff8118f776>] writeback_sb_inodes+0x1b6/0x270 [<ffffffff8118f8ce>] __writeback_inodes_wb+0x9e/0xd0 [<ffffffff8118fb6b>] wb_writeback+0x26b/0x350 [<ffffffff8118fce8>] wb_check_old_data_flush+0x98/0xa0 [<ffffffff8119006c>] wb_do_writeback+0x15c/0x200 [<ffffffff816000b5>] ? schedule_timeout+0x175/0x320 [<ffffffff81071840>] ? usleep_range+0x50/0x50 [<ffffffff81190193>] bdi_writeback_thread+0x83/0x2a0 [<ffffffff81190110>] ? wb_do_writeback+0x200/0x200 [<ffffffff8108496c>] kthread+0x8c/0xa0 [<ffffffff8160bb74>] kernel_thread_helper+0x4/0x10 [<ffffffff810848e0>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8160bb70>] ? gs_change+0x13/0x13 INFO: task rsync:1887 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsync D ffff880227d49ab0 0 1887 1884 0x00000000 ffff8801a2bd7b68 0000000000000086 ffff880200000000 ffff8801a2bd7b08 ffff880227d496f0 ffff8801a2bd7fd8 ffff8801a2bd7fd8 ffff8801a2bd7fd8 ffffffff81c0d020 ffff880227d496f0 0000000000000286 000000012fd942f0 Call Trace: [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffffa022422d>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [<ffffffff81085400>] ? add_wait_queue+0x60/0x60 [<ffffffffa0225a50>] start_transaction+0x1f0/0x2b0 [btrfs] [<ffffffffa0225b65>] btrfs_join_transaction+0x15/0x20 [btrfs] [<ffffffffa0230788>] btrfs_dirty_inode+0x48/0x160 [btrfs] [<ffffffff8118dfb0>] __mark_inode_dirty+0x40/0x230 [<ffffffff8117fed7>] file_update_time+0xe7/0x150 [<ffffffffa02348a0>] btrfs_file_aio_write+0x1c0/0x510 [btrfs] [<ffffffff8112f11b>] ? handle_pte_fault+0x61b/0xac0 [<ffffffff81166c62>] do_sync_write+0xd2/0x110 [<ffffffff8128dacc>] ? security_file_permission+0x2c/0xb0 [<ffffffff81167101>] ? rw_verify_area+0x61/0xf0 [<ffffffff81167463>] vfs_write+0xb3/0x180 [<ffffffff8116778a>] sys_write+0x4a/0x90 [<ffffffff81609a02>] system_call_fastpath+0x16/0x1b INFO: task btrfs-delalloc-:2148 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. btrfs-delalloc- D ffff880089b8c890 0 2148 2 0x00000000 ffff880039dbbb40 0000000000000046 0000000000000000 0000000000001000 ffff880089b8c4d0 ffff880039dbbfd8 ffff880039dbbfd8 ffff880039dbbfd8 ffff8802321444d0 ffff880089b8c4d0 0000000000000286 000000012fd942f0 Call Trace: [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffffa022422d>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [<ffffffff81085400>] ? add_wait_queue+0x60/0x60 [<ffffffffa0225a50>] start_transaction+0x1f0/0x2b0 [btrfs] [<ffffffffa0225b65>] btrfs_join_transaction+0x15/0x20 [btrfs] [<ffffffffa022c07c>] cow_file_range+0x7c/0x3a0 [btrfs] [<ffffffffa022d0f2>] submit_compressed_extents+0x242/0x470 [btrfs] [<ffffffffa022d7cc>] ? compress_file_range+0x41c/0x610 [btrfs] [<ffffffffa022d3a4>] async_cow_submit+0x84/0x90 [btrfs] [<ffffffffa024c490>] run_ordered_completions+0x80/0xe0 [btrfs] [<ffffffffa024ce80>] worker_loop+0x180/0x560 [btrfs] [<ffffffffa024cd00>] ? btrfs_queue_worker+0x300/0x300 [btrfs] [<ffffffff8108496c>] kthread+0x8c/0xa0 [<ffffffff8160bb74>] kernel_thread_helper+0x4/0x10 [<ffffffff810848e0>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8160bb70>] ? gs_change+0x13/0x13 INFO: task rsync:2217 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsync D ffff880212624890 0 2217 2212 0x00000000 ffff8801acaabd98 0000000000000082 0000000000000000 0000000000000000 ffff8802126244d0 ffff8801acaabfd8 ffff8801acaabfd8 ffff8801acaabfd8 ffff88023212dbc0 ffff8802126244d0 0000000000000282 000000012fd942f0 Call Trace: [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffffa022422d>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [<ffffffff81085400>] ? add_wait_queue+0x60/0x60 [<ffffffffa0225995>] start_transaction+0x135/0x2b0 [btrfs] [<ffffffff8117651a>] ? kern_path_create+0x8a/0x120 [<ffffffffa0225de3>] btrfs_start_transaction+0x13/0x20 [btrfs] [<ffffffffa0231cb5>] btrfs_link+0xa5/0x1a0 [btrfs] [<ffffffff81173211>] vfs_link+0x101/0x190 [<ffffffff81177648>] sys_linkat+0x168/0x180 [<ffffffff8117767e>] sys_link+0x1e/0x20 [<ffffffff81609a02>] system_call_fastpath+0x16/0x1b INFO: task flush-btrfs-2:1472 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-btrfs-2 D ffff8802313fb1a0 0 1472 2 0x00000000 ffff880227c318d0 0000000000000046 ffff88006320b798 ffff88006320b768 ffff8802313fade0 ffff880227c31fd8 ffff880227c31fd8 ffff880227c31fd8 ffff880211da5bc0 ffff8802313fade0 ffff880227c318d0 ffffffff8108f1fd Call Trace: [<ffffffff8108f1fd>] ? ktime_get_ts+0xad/0xe0 [<ffffffff81109a30>] ? __lock_page+0x70/0x70 [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffff815ffc4f>] io_schedule+0x8f/0xd0 [<ffffffff81109a3e>] sleep_on_page+0xe/0x20 [<ffffffff8160031a>] __wait_on_bit_lock+0x5a/0xc0 [<ffffffff81109a27>] __lock_page+0x67/0x70 [<ffffffff81085440>] ? autoremove_wake_function+0x40/0x40 [<ffffffffa02430e1>] extent_write_cache_pages.isra.23.constprop.32+0x1e1/0x3b0 [btrfs] [<ffffffffa02434f5>] extent_writepages+0x45/0x60 [btrfs] [<ffffffffa0228710>] ? acls_after_inode_item+0xc0/0xc0 [btrfs] [<ffffffff81084fd4>] ? bit_waitqueue+0x14/0xc0 [<ffffffffa0227958>] btrfs_writepages+0x28/0x30 [btrfs] [<ffffffff81114e81>] do_writepages+0x21/0x40 [<ffffffff8118f0b0>] writeback_single_inode+0x180/0x430 [<ffffffff8118f776>] writeback_sb_inodes+0x1b6/0x270 [<ffffffff8118f8ce>] __writeback_inodes_wb+0x9e/0xd0 [<ffffffff8118fb6b>] wb_writeback+0x26b/0x350 [<ffffffff8118fce8>] wb_check_old_data_flush+0x98/0xa0 [<ffffffff8119006c>] wb_do_writeback+0x15c/0x200 [<ffffffff816000b5>] ? schedule_timeout+0x175/0x320 [<ffffffff81071840>] ? usleep_range+0x50/0x50 [<ffffffff81190193>] bdi_writeback_thread+0x83/0x2a0 [<ffffffff81190110>] ? wb_do_writeback+0x200/0x200 [<ffffffff8108496c>] kthread+0x8c/0xa0 [<ffffffff8160bb74>] kernel_thread_helper+0x4/0x10 [<ffffffff810848e0>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8160bb70>] ? gs_change+0x13/0x13 INFO: task rsync:1887 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsync D ffff880227d49ab0 0 1887 1884 0x00000004 ffff8801a2bd7b68 0000000000000086 ffff880200000000 ffff8801a2bd7b08 ffff880227d496f0 ffff8801a2bd7fd8 ffff8801a2bd7fd8 ffff8801a2bd7fd8 ffffffff81c0d020 ffff880227d496f0 0000000000000286 000000012fd942f0 Call Trace: [<ffffffff815ffb9f>] schedule+0x3f/0x60 [<ffffffffa022422d>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [<ffffffff81085400>] ? add_wait_queue+0x60/0x60 [<ffffffffa0225a50>] start_transaction+0x1f0/0x2b0 [btrfs] [<ffffffffa0225b65>] btrfs_join_transaction+0x15/0x20 [btrfs] [<ffffffffa0230788>] btrfs_dirty_inode+0x48/0x160 [btrfs] [<ffffffff8118dfb0>] __mark_inode_dirty+0x40/0x230 [<ffffffff8117fed7>] file_update_time+0xe7/0x150 [<ffffffffa02348a0>] btrfs_file_aio_write+0x1c0/0x510 [btrfs] [<ffffffff8112f11b>] ? handle_pte_fault+0x61b/0xac0 [<ffffffff81166c62>] do_sync_write+0xd2/0x110 [<ffffffff8128dacc>] ? security_file_permission+0x2c/0xb0 [<ffffffff81167101>] ? rw_verify_area+0x61/0xf0 [<ffffffff81167463>] vfs_write+0xb3/0x180 [<ffffffff8116778a>] sys_write+0x4a/0x90 [<ffffffff81609a02>] system_call_fastpath+0x16/0x1b I seems that the problem got worse the more data i put on the fs... Sometime the system is still usable; sometimes i cant access the fs anymore (no even sync or umount) and have to hard-reset. The problem occurs on the stock ubuntu kernel 2.6.38-8, 3.0.0-12, 3.0.0-13 and on my self-compiled 3.1.2. Can anyone help? Tobias ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blcoked for more than 120 seconds 2011-11-28 8:16 Blcoked for more than 120 seconds Tobias @ 2011-11-28 9:29 ` Chris Samuel 2011-11-30 9:44 ` Blocked " Tobias 0 siblings, 1 reply; 15+ messages in thread From: Chris Samuel @ 2011-11-28 9:29 UTC (permalink / raw) To: linux-btrfs Hi Tobias, On Mon, 28 Nov 2011, 19:16:25 EST, Tobias <tracer@robotech.de> wrote: > The problem occurs on the stock ubuntu kernel 2.6.38-8, 3.0.0-12, > 3.0.0-13 and on my self-compiled 3.1.2. There's a lot of work gone into btrfs in 3.2, it would be interesting to know (speaking as just another user) whether it still occurs with 3.2-rc3. cheers, Chris -- Chris Samuel - http://www.csamuel.org/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-11-28 9:29 ` Chris Samuel @ 2011-11-30 9:44 ` Tobias 2011-11-30 14:10 ` Chris Mason 0 siblings, 1 reply; 15+ messages in thread From: Tobias @ 2011-11-30 9:44 UTC (permalink / raw) To: linux-btrfs Am 28.11.2011 10:29, schrieb Chris Samuel: > Hi Tobias, > > On Mon, 28 Nov 2011, 19:16:25 EST, Tobias<tracer@robotech.de> wrote: > >> The problem occurs on the stock ubuntu kernel 2.6.38-8, 3.0.0-12, >> 3.0.0-13 and on my self-compiled 3.1.2. > There's a lot of work gone into btrfs in 3.2, > it would be interesting to know (speaking as > just another user) whether it still occurs > with 3.2-rc3. > I tried 3.2-rc3 tonight but the messages are still there: [46203.412044] INFO: task rsync:1653 blocked for more than 120 seconds. [46203.412056] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [46203.412066] rsync D ffff8801d7d51aa0 0 1653 1647 0x00000000 [46203.412073] ffff8800042b1d98 0000000000000086 0000000000000000 0000000000000000 [46203.412079] ffff8801d7d516e0 ffff8800042b1fd8 ffff8800042b1fd8 ffff8800042b1fd8 [46203.412084] ffff88023212db80 ffff8801d7d516e0 0000000000000282 0000000122103228 [46203.412090] Call Trace: [46203.412101] [<ffffffff8161259f>] schedule+0x3f/0x60 [46203.412126] [<ffffffffa01fd1bd>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [46203.412132] [<ffffffff81085a40>] ? add_wait_queue+0x60/0x60 [46203.412148] [<ffffffffa01fe7f5>] start_transaction+0x135/0x2b0 [btrfs] [46203.412154] [<ffffffff8117bd5a>] ? kern_path_create+0x8a/0x120 [46203.412171] [<ffffffffa01fec43>] btrfs_start_transaction+0x13/0x20 [btrfs] [46203.412188] [<ffffffffa020a885>] btrfs_link+0xa5/0x1a0 [btrfs] [46203.412193] [<ffffffff81178a91>] vfs_link+0x101/0x190 [46203.412197] [<ffffffff8117ce88>] sys_linkat+0x168/0x180 [46203.412200] [<ffffffff8117cebe>] sys_link+0x1e/0x20 [46203.412205] [<ffffffff8161c442>] system_call_fastpath+0x16/0x1b [46563.412042] INFO: task btrfs-delalloc-:31614 blocked for more than 120 seconds. [46563.412054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [46563.412064] btrfs-delalloc- D ffff8801f8529aa0 0 31614 2 0x00000000 [46563.412071] ffff8801330bdc50 0000000000000046 0000000000000000 0000000000000004 [46563.412077] ffff8801f85296e0 ffff8801330bdfd8 ffff8801330bdfd8 ffff8801330bdfd8 [46563.412082] ffff88023212db80 ffff8801f85296e0 0000000000000282 0000000122103228 [46563.412088] Call Trace: [46563.412098] [<ffffffff8161259f>] schedule+0x3f/0x60 [46563.412124] [<ffffffffa01fd1bd>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [46563.412130] [<ffffffff81085a40>] ? add_wait_queue+0x60/0x60 [46563.412146] [<ffffffffa01fe8b0>] start_transaction+0x1f0/0x2b0 [btrfs] [46563.412163] [<ffffffffa01fe9c5>] btrfs_join_transaction+0x15/0x20 [btrfs] [46563.412179] [<ffffffffa0206423>] compress_file_range+0x2d3/0x610 [btrfs] [46563.412197] [<ffffffffa0206795>] async_cow_start+0x35/0x50 [btrfs] [46563.412213] [<ffffffffa02269ba>] worker_loop+0x16a/0x560 [btrfs] [46563.412231] [<ffffffffa0226850>] ? btrfs_queue_worker+0x300/0x300 [btrfs] [46563.412236] [<ffffffff81084fac>] kthread+0x8c/0xa0 [46563.412241] [<ffffffff8161e5b4>] kernel_thread_helper+0x4/0x10 [46563.412246] [<ffffffff81084f20>] ? flush_kthread_worker+0xa0/0xa0 [46563.412250] [<ffffffff8161e5b0>] ? gs_change+0x13/0x13 [46563.412255] INFO: task flush-btrfs-1:323 blocked for more than 120 seconds. [46563.412263] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [46563.412273] flush-btrfs-1 D ffff8802129bb180 0 323 2 0x00000000 [46563.412278] ffff8800209c58e0 0000000000000046 ffff880000000000 ffff8800218a2710 [46563.412284] ffff8802129badc0 ffff8800209c5fd8 ffff8800209c5fd8 ffff8800209c5fd8 [46563.412290] ffff8802321444a0 ffff8802129badc0 ffff8800209c58e0 000000018108f20d [46563.412295] Call Trace: [46563.412300] [<ffffffff8110dc90>] ? __lock_page+0x70/0x70 [46563.412305] [<ffffffff8161259f>] schedule+0x3f/0x60 [46563.412309] [<ffffffff8161264f>] io_schedule+0x8f/0xd0 [46563.412313] [<ffffffff8110dc9e>] sleep_on_page+0xe/0x20 [46563.412317] [<ffffffff81612d1a>] __wait_on_bit_lock+0x5a/0xc0 [46563.412321] [<ffffffff8110dc87>] __lock_page+0x67/0x70 [46563.412325] [<ffffffff81085a80>] ? autoremove_wake_function+0x40/0x40 [46563.412342] [<ffffffffa021c945>] extent_write_cache_pages.isra.21.constprop.31+0x215/0x3f0 [btrfs] [46563.412361] [<ffffffffa021cd65>] extent_writepages+0x45/0x60 [btrfs] [46563.412378] [<ffffffffa0201350>] ? acls_after_inode_item+0xc0/0xc0 [btrfs] [46563.412382] [<ffffffff81085614>] ? bit_waitqueue+0x14/0xc0 [46563.412398] [<ffffffffa0200448>] btrfs_writepages+0x28/0x30 [btrfs] [46563.412403] [<ffffffff811195f1>] do_writepages+0x21/0x40 [46563.412409] [<ffffffff81194c70>] writeback_single_inode+0x180/0x430 [46563.412413] [<ffffffff81195336>] writeback_sb_inodes+0x1b6/0x270 [46563.412418] [<ffffffff8119548e>] __writeback_inodes_wb+0x9e/0xd0 [46563.412422] [<ffffffff8119573b>] wb_writeback+0x27b/0x330 [46563.412427] [<ffffffff81187352>] ? get_nr_dirty_inodes+0x52/0x80 [46563.412432] [<ffffffff8119588f>] wb_check_old_data_flush+0x9f/0xb0 [46563.412436] [<ffffffff81196731>] wb_do_writeback+0x151/0x1d0 [46563.412441] [<ffffffff81611fd4>] ? __schedule+0x3d4/0x8d0 [46563.412446] [<ffffffff81071ae0>] ? usleep_range+0x50/0x50 [46563.412450] [<ffffffff81196833>] bdi_writeback_thread+0x83/0x2a0 [46563.412455] [<ffffffff811967b0>] ? wb_do_writeback+0x1d0/0x1d0 [46563.412459] [<ffffffff81084fac>] kthread+0x8c/0xa0 [46563.412463] [<ffffffff8161e5b4>] kernel_thread_helper+0x4/0x10 [46563.412468] [<ffffffff81084f20>] ? flush_kthread_worker+0xa0/0xa0 [46563.412472] [<ffffffff8161e5b0>] ? gs_change+0x13/0x13 [46563.412475] INFO: task rsync:1653 blocked for more than 120 seconds. [46563.412483] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [46563.412493] rsync D ffff8801d7d51aa0 0 1653 1647 0x00000000 [46563.412498] ffff8800042b1d98 0000000000000086 0000000000000000 0000000000000000 [46563.412504] ffff8801d7d516e0 ffff8800042b1fd8 ffff8800042b1fd8 ffff8800042b1fd8 [46563.412510] ffff88023212db80 ffff8801d7d516e0 0000000000000282 0000000122103228 [46563.412515] Call Trace: [46563.412520] [<ffffffff8161259f>] schedule+0x3f/0x60 [46563.412535] [<ffffffffa01fd1bd>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [46563.412540] [<ffffffff81085a40>] ? add_wait_queue+0x60/0x60 [46563.412555] [<ffffffffa01fe7f5>] start_transaction+0x135/0x2b0 [btrfs] [46563.412561] [<ffffffff8117bd5a>] ? kern_path_create+0x8a/0x120 [46563.412577] [<ffffffffa01fec43>] btrfs_start_transaction+0x13/0x20 [btrfs] [46563.412594] [<ffffffffa020a885>] btrfs_link+0xa5/0x1a0 [btrfs] [46563.412599] [<ffffffff81178a91>] vfs_link+0x101/0x190 [46563.412603] [<ffffffff8117ce88>] sys_linkat+0x168/0x180 [46563.412606] [<ffffffff8117cebe>] sys_link+0x1e/0x20 [46563.412610] [<ffffffff8161c442>] system_call_fastpath+0x16/0x1b [46563.412616] INFO: task rsync:2240 blocked for more than 120 seconds. [46563.412624] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [46563.412633] rsync D ffff8802286d1aa0 0 2240 1760 0x00000000 [46563.412638] ffff88001b9b7d98 0000000000000086 0000000000000000 0000000000000000 [46563.412644] ffff8802286d16e0 ffff88001b9b7fd8 ffff88001b9b7fd8 ffff88001b9b7fd8 [46563.412650] ffff88023212db80 ffff8802286d16e0 0000000000000282 0000000122103228 [46563.412655] Call Trace: [46563.412660] [<ffffffff8161259f>] schedule+0x3f/0x60 [46563.412676] [<ffffffffa01fd1bd>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [46563.412680] [<ffffffff81085a40>] ? add_wait_queue+0x60/0x60 [46563.412696] [<ffffffffa01fe7f5>] start_transaction+0x135/0x2b0 [btrfs] [46563.412700] [<ffffffff8117bd5a>] ? kern_path_create+0x8a/0x120 [46563.412716] [<ffffffffa01fec43>] btrfs_start_transaction+0x13/0x20 [btrfs] [46563.412733] [<ffffffffa020a885>] btrfs_link+0xa5/0x1a0 [btrfs] [46563.412738] [<ffffffff81178a91>] vfs_link+0x101/0x190 [46563.412742] [<ffffffff8117ce88>] sys_linkat+0x168/0x180 [46563.412746] [<ffffffff8117cebe>] sys_link+0x1e/0x20 [46563.412749] [<ffffffff8161c442>] system_call_fastpath+0x16/0x1b [46563.412754] INFO: task rsync:2257 blocked for more than 120 seconds. [46563.412762] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [46563.412771] rsync D ffff880232145f40 0 2257 2254 0x00000000 [46563.412777] ffff88009c5c3b58 0000000000000086 ffff880000000000 ffffffff8108572e [46563.412782] ffff880232145b80 ffff88009c5c3fd8 ffff88009c5c3fd8 ffff88009c5c3fd8 [46563.412788] ffff880232128000 ffff880232145b80 0000000000000286 0000000122103228 [46563.412793] Call Trace: [46563.412797] [<ffffffff8108572e>] ? wake_up_bit+0x2e/0x40 [46563.412802] [<ffffffff8161259f>] schedule+0x3f/0x60 [46563.412818] [<ffffffffa01fd1bd>] wait_current_trans.isra.22+0x9d/0x100 [btrfs] [46563.412822] [<ffffffff81085a40>] ? add_wait_queue+0x60/0x60 [46563.412838] [<ffffffffa01fe8b0>] start_transaction+0x1f0/0x2b0 [btrfs] [46563.412854] [<ffffffffa01fe9c5>] btrfs_join_transaction+0x15/0x20 [btrfs] [46563.412871] [<ffffffffa0209368>] btrfs_dirty_inode+0x48/0x160 [btrfs] [46563.412876] [<ffffffff81194440>] __mark_inode_dirty+0x40/0x230 [46563.412880] [<ffffffff81185814>] file_update_time+0xe4/0x150 [46563.412897] [<ffffffffa020d500>] btrfs_file_aio_write+0x1c0/0x510 [btrfs] [46563.412903] [<ffffffff8125e857>] ? jbd2_journal_stop+0x1b7/0x2a0 [46563.412908] [<ffffffff81177d03>] ? putname+0x33/0x50 [46563.412913] [<ffffffff8116c422>] do_sync_write+0xd2/0x110 [46563.412918] [<ffffffff812d5e98>] ? apparmor_file_permission+0x18/0x20 [46563.412924] [<ffffffff8129c57c>] ? security_file_permission+0x2c/0xb0 [46563.412928] [<ffffffff8116c9b1>] ? rw_verify_area+0x61/0xf0 [46563.412932] [<ffffffff8116cd13>] vfs_write+0xb3/0x180 [46563.412936] [<ffffffff8116d03a>] sys_write+0x4a/0x90 [46563.412940] [<ffffffff8161c442>] system_call_fastpath+0x16/0x1b Can anyone help? Tobias ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-11-30 9:44 ` Blocked " Tobias @ 2011-11-30 14:10 ` Chris Mason [not found] ` <4ED738E5.3080200@robotech.de> 0 siblings, 1 reply; 15+ messages in thread From: Chris Mason @ 2011-11-30 14:10 UTC (permalink / raw) To: Tobias; +Cc: linux-btrfs On Wed, Nov 30, 2011 at 10:44:15AM +0100, Tobias wrote: > Am 28.11.2011 10:29, schrieb Chris Samuel: > >Hi Tobias, > > > >On Mon, 28 Nov 2011, 19:16:25 EST, Tobias<tracer@robotech.de> wrote: > > > >>The problem occurs on the stock ubuntu kernel 2.6.38-8, 3.0.0-12, > >>3.0.0-13 and on my self-compiled 3.1.2. > >There's a lot of work gone into btrfs in 3.2, > >it would be interesting to know (speaking as > >just another user) whether it still occurs > >with 3.2-rc3. > > > I tried 3.2-rc3 tonight but the messages are still there: We see a bunch of procs stuck waiting to start a transaction, but we don't see why they are waiting. Could you please capture a sysrq-t during this? That will show us all the waiters everywhere. We're really looking for the one proc stuck in btrfs_commit_transaction, he's the key to the stalls. -chris ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <4ED738E5.3080200@robotech.de>]
* Re: Blocked for more than 120 seconds [not found] ` <4ED738E5.3080200@robotech.de> @ 2011-12-01 18:41 ` Chris Mason 2011-12-02 13:46 ` Tobias 0 siblings, 1 reply; 15+ messages in thread From: Chris Mason @ 2011-12-01 18:41 UTC (permalink / raw) To: Tobias; +Cc: linux-btrfs On Thu, Dec 01, 2011 at 09:20:53AM +0100, Tobias wrote: > Hi Chris > > Am 30.11.2011 15:10, schrieb Chris Mason: > >We see a bunch of procs stuck waiting to start a transaction, but we > >don't see why they are waiting. Could you please capture a sysrq-t > >during this? That will show us all the waiters everywhere. We're > >really looking for the one proc stuck in btrfs_commit_transaction, he's > >the key to the stalls. > > > > This is my first time working with sysrq... i hope i did it right... > Here is the logoutput (quite much) So, the transaction close is in btrfs_evict_inode, which sounds like a deadlock recently fixed by this commit: http://git.kernel.org/?p=linux/kernel/git/mason/linux-btrfs.git;a=commit;h=aa38a711a893accf5b5192f3d705a120deaa81e0 If you pull the for-linus branch from today, hopefully the problem will be gone. -chris ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-01 18:41 ` Chris Mason @ 2011-12-02 13:46 ` Tobias 2011-12-02 14:01 ` Chris Mason 2011-12-02 15:22 ` Konstantinos Skarlatos 0 siblings, 2 replies; 15+ messages in thread From: Tobias @ 2011-12-02 13:46 UTC (permalink / raw) To: Chris Mason, linux-btrfs Hi Chris! Am 01.12.2011 19:41, schrieb Chris Mason: > > So, the transaction close is in btrfs_evict_inode, which sounds like a > deadlock recently fixed by this commit: > > http://git.kernel.org/?p=linux/kernel/git/mason/linux-btrfs.git;a=commit;h=aa38a711a893accf5b5192f3d705a120deaa81e0 > > If you pull the for-linus branch from today, hopefully the problem will > be gone. > This looks very good. With this Kernel i still have some hangs, but only in rsync, only under high load and they don't lock up the system - so i guess it's ok now. Thank You very much for Your help! When will this patches go into the main Kernel? Tobias ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-02 13:46 ` Tobias @ 2011-12-02 14:01 ` Chris Mason 2011-12-04 12:11 ` Konstantinos Skarlatos 2011-12-02 15:22 ` Konstantinos Skarlatos 1 sibling, 1 reply; 15+ messages in thread From: Chris Mason @ 2011-12-02 14:01 UTC (permalink / raw) To: Tobias; +Cc: linux-btrfs, miaox On Fri, Dec 02, 2011 at 02:46:48PM +0100, Tobias wrote: > Hi Chris! > > Am 01.12.2011 19:41, schrieb Chris Mason: > > > >So, the transaction close is in btrfs_evict_inode, which sounds like a > >deadlock recently fixed by this commit: > > > >http://git.kernel.org/?p=linux/kernel/git/mason/linux-btrfs.git;a=commit;h=aa38a711a893accf5b5192f3d705a120deaa81e0 > > > >If you pull the for-linus branch from today, hopefully the problem will > >be gone. > > > > This looks very good. With this Kernel i still have some hangs, but > only in rsync, only under high load and they don't lock up the > system - so i guess it's ok now. > > Thank You very much for Your help! Glad to hear this is working. All the credit to Miao, who found the deadlock. > > When will this patches go into the main Kernel? Linus pulled them in yesterday. -chris ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-02 14:01 ` Chris Mason @ 2011-12-04 12:11 ` Konstantinos Skarlatos 0 siblings, 0 replies; 15+ messages in thread From: Konstantinos Skarlatos @ 2011-12-04 12:11 UTC (permalink / raw) To: Chris Mason, Tobias, linux-btrfs, miaox even more kernel messages from btrfs crashing when rsyncing large=20 amounts of data on 3.2rc4 Dec 3 15:12:14 mail kernel: [15481.100564] loop0 D=20 000000010044b6c5 0 1729 2 0x00000000 Dec 3 15:12:14 mail kernel: [15481.101550] ffff8801f9b31b30=20 0000000000000046 0000000000000000 0000000000000000 Dec 3 15:12:14 mail kernel: [15481.102548] ffff880200950e40=20 ffff8801f9b31fd8 ffff8801f9b31fd8 ffff8801f9b31fd8 Dec 3 15:12:14 mail kernel: [15481.103539] ffff880202cb7200=20 ffff880200950e40 0000000000000002 ffff8801f9b31b78 Dec 3 15:12:14 mail kernel: [15481.104533] Call Trace: Dec 3 15:12:14 mail kernel: [15481.105531] [<ffffffff81101a55>] ?=20 find_get_pages_tag+0x125/0x150 Dec 3 15:12:14 mail kernel: [15481.106541] [<ffffffff8110e205>] ?=20 pagevec_lookup_tag+0x25/0x40 Dec 3 15:12:14 mail kernel: [15481.107552] [<ffffffff8101d639>] ?=20 read_tsc+0x9/0x20 Dec 3 15:12:14 mail kernel: [15481.108576] [<ffffffff8108f14d>] ?=20 ktime_get_ts+0xad/0xe0 Dec 3 15:12:14 mail kernel: [15481.109592] [<ffffffff81101d60>] ?=20 __lock_page+0x70/0x70 Dec 3 15:12:14 mail kernel: [15481.110607] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:12:14 mail kernel: [15481.111619] [<ffffffff8141416f>]=20 io_schedule+0x8f/0xd0 Dec 3 15:12:14 mail kernel: [15481.112641] [<ffffffff81101d6e>]=20 sleep_on_page+0xe/0x20 Dec 3 15:12:14 mail kernel: [15481.113639] [<ffffffff8141491f>]=20 __wait_on_bit+0x5f/0x90 Dec 3 15:12:14 mail kernel: [15481.114629] [<ffffffff81101f58>]=20 wait_on_page_bit+0x78/0x80 Dec 3 15:12:14 mail kernel: [15481.115628] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 Dec 3 15:12:14 mail kernel: [15481.116614] [<ffffffff811020cc>]=20 filemap_fdatawait_range+0x10c/0x1a0 Dec 3 15:12:14 mail kernel: [15481.117613] [<ffffffff811030c8>]=20 filemap_write_and_wait_range+0x68/0x80 Dec 3 15:12:14 mail kernel: [15481.118630] [<ffffffffa03a7234>]=20 xfs_file_fsync+0x54/0x340 [xfs] Dec 3 15:12:14 mail kernel: [15481.119629] [<ffffffff8119148b>]=20 vfs_fsync+0x2b/0x40 Dec 3 15:12:14 mail kernel: [15481.120627] [<ffffffffa04dacf2>]=20 do_bio_filebacked+0x1b2/0x320 [loop] Dec 3 15:12:14 mail kernel: [15481.121645] [<ffffffffa050efac>] ?=20 end_workqueue_bio+0x9c/0xa0 [btrfs] Dec 3 15:12:14 mail kernel: [15481.122668] [<ffffffffa04daf1b>]=20 loop_thread+0xbb/0x260 [loop] Dec 3 15:12:14 mail kernel: [15481.123674] [<ffffffff81085750>] ?=20 abort_exclusive_wait+0xb0/0xb0 Dec 3 15:12:14 mail kernel: [15481.124676] [<ffffffffa04dae60>] ?=20 do_bio_filebacked+0x320/0x320 [loop] Dec 3 15:12:14 mail kernel: [15481.125698] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:12:14 mail kernel: [15481.126710] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:12:14 mail kernel: [15481.127721] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:12:14 mail kernel: [15481.128742] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:12:14 mail kernel: [15481.131702] btrfs-transacti D=20 ffff8801f9ab7200 0 1756 2 0x00000000 Dec 3 15:12:14 mail kernel: [15481.132723] ffff8801e7533bc0=20 0000000000000046 ffff88020fc93400 0000000000000002 Dec 3 15:12:14 mail kernel: [15481.133744] ffff8801f9ab7200=20 ffff8801e7533fd8 ffff8801e7533fd8 ffff8801e7533fd8 Dec 3 15:12:14 mail kernel: [15481.134771] ffff880200950e40=20 ffff8801f9ab7200 ffff8801e7533b10 0000000081051ae2 Dec 3 15:12:14 mail kernel: [15481.135813] Call Trace: Dec 3 15:12:14 mail kernel: [15481.136828] [<ffffffff8105ad36>] ?=20 ttwu_do_activate.constprop.172+0x66/0x70 Dec 3 15:12:14 mail kernel: [15481.137863] [<ffffffff8105bd6e>] ?=20 try_to_wake_up+0x1de/0x290 Dec 3 15:12:14 mail kernel: [15481.138914] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:12:14 mail kernel: [15481.139956] [<ffffffff814147d5>]=20 schedule_timeout+0x305/0x390 Dec 3 15:12:14 mail kernel: [15481.141007] [<ffffffff8104d003>] ?=20 __wake_up+0x53/0x70 Dec 3 15:12:14 mail kernel: [15481.142074] [<ffffffff81413348>]=20 wait_for_common+0xc8/0x160 Dec 3 15:12:14 mail kernel: [15481.143124] [<ffffffff8105be20>] ?=20 try_to_wake_up+0x290/0x290 Dec 3 15:12:14 mail kernel: [15481.144170] [<ffffffff814133fd>]=20 wait_for_completion+0x1d/0x20 Dec 3 15:12:14 mail kernel: [15481.145229] [<ffffffffa050f0bb>]=20 write_dev_flush+0x4b/0x140 [btrfs] Dec 3 15:12:14 mail kernel: [15481.146275] [<ffffffffa0511086>]=20 write_all_supers+0x6f6/0x800 [btrfs] Dec 3 15:12:14 mail kernel: [15481.147317] [<ffffffffa05111a3>]=20 write_ctree_super+0x13/0x20 [btrfs] Dec 3 15:12:14 mail kernel: [15481.148354] [<ffffffffa05164dd>]=20 btrfs_commit_transaction+0x63d/0x880 [btrfs] Dec 3 15:12:14 mail kernel: [15481.149397] [<ffffffff81085750>] ?=20 abort_exclusive_wait+0xb0/0xb0 Dec 3 15:12:14 mail kernel: [15481.150416] [<ffffffffa0516b74>] ?=20 start_transaction+0x94/0x2b0 [btrfs] Dec 3 15:12:14 mail kernel: [15481.151444] [<ffffffffa050ed4d>]=20 transaction_kthread+0x26d/0x290 [btrfs] Dec 3 15:12:14 mail kernel: [15481.152492] [<ffffffffa050eae0>] ?=20 btrfs_congested_fn+0xd0/0xd0 [btrfs] Dec 3 15:12:14 mail kernel: [15481.153519] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:12:14 mail kernel: [15481.154542] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:12:14 mail kernel: [15481.155553] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:12:14 mail kernel: [15481.156522] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:12:14 mail kernel: [15481.159519] smbd D=20 000000010044b8d7 0 2058 823 0x00000004 Dec 3 15:12:14 mail kernel: [15481.160544] ffff88005cf37a08=20 0000000000000082 ffff88005cf37938 ffffffff81371188 Dec 3 15:12:14 mail kernel: [15481.161588] ffff8802012e63c0=20 ffff88005cf37fd8 ffff88005cf37fd8 ffff88005cf37fd8 Dec 3 15:12:14 mail kernel: [15481.162658] ffff880202d21c80=20 ffff8802012e63c0 ffff8802012769c0 0000000000000246 Dec 3 15:12:14 mail kernel: [15481.163708] Call Trace: Dec 3 15:12:14 mail kernel: [15481.164736] [<ffffffff81371188>] ?=20 sch_direct_xmit+0x68/0x1d0 Dec 3 15:12:14 mail kernel: [15481.165781] [<ffffffff81355a00>] ?=20 dev_queue_xmit+0x200/0x680 Dec 3 15:12:14 mail kernel: [15481.166805] [<ffffffff81389200>] ?=20 ip_forward_options+0x1c0/0x1c0 Dec 3 15:12:14 mail kernel: [15481.167822] [<ffffffff8138adbe>] ?=20 ip_finish_output+0x18e/0x310 Dec 3 15:12:14 mail kernel: [15481.168850] [<ffffffff8101d639>] ?=20 read_tsc+0x9/0x20 Dec 3 15:12:14 mail kernel: [15481.169853] [<ffffffff8108f14d>] ?=20 ktime_get_ts+0xad/0xe0 Dec 3 15:12:14 mail kernel: [15481.170849] [<ffffffff81101d60>] ?=20 __lock_page+0x70/0x70 Dec 3 15:12:14 mail kernel: [15481.171858] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:12:14 mail kernel: [15481.172851] [<ffffffff8141416f>]=20 io_schedule+0x8f/0xd0 Dec 3 15:12:14 mail kernel: [15481.173844] [<ffffffff81101d6e>]=20 sleep_on_page+0xe/0x20 Dec 3 15:12:14 mail kernel: [15481.174836] [<ffffffff8141491f>]=20 __wait_on_bit+0x5f/0x90 Dec 3 15:12:14 mail kernel: [15481.175845] [<ffffffff81101f58>]=20 wait_on_page_bit+0x78/0x80 Dec 3 15:12:14 mail kernel: [15481.176842] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 Dec 3 15:12:14 mail kernel: [15481.177845] [<ffffffff81102845>]=20 grab_cache_page_write_begin+0x95/0xe0 Dec 3 15:12:14 mail kernel: [15481.178872] [<ffffffffa03a1150>] ?=20 xfs_get_blocks_direct+0x20/0x20 [xfs] Dec 3 15:12:14 mail kernel: [15481.179888] [<ffffffff811967b8>]=20 block_write_begin+0x38/0xa0 Dec 3 15:12:14 mail kernel: [15481.180904] [<ffffffffa03a1213>]=20 xfs_vm_write_begin+0x43/0x70 [xfs] Dec 3 15:12:14 mail kernel: [15481.181928] [<ffffffff8110233c>]=20 generic_file_buffered_write+0x10c/0x270 Dec 3 15:12:14 mail kernel: [15481.182955] [<ffffffffa03aad66>] ?=20 xfs_iunlock+0x116/0x180 [xfs] Dec 3 15:12:14 mail kernel: [15481.183979] [<ffffffffa03a7fef>]=20 xfs_file_buffered_aio_write+0x10f/0x200 [xfs] Dec 3 15:12:14 mail kernel: [15481.185011] [<ffffffffa03a8252>]=20 xfs_file_aio_write+0x172/0x2a0 [xfs] Dec 3 15:12:14 mail kernel: [15481.186052] [<ffffffff81162d62>]=20 do_sync_write+0xd2/0x110 Dec 3 15:12:14 mail kernel: [15481.187073] [<ffffffff811f0fcc>] ?=20 security_file_permission+0x2c/0xb0 Dec 3 15:12:14 mail kernel: [15481.188096] [<ffffffff81163311>] ?=20 rw_verify_area+0x61/0xf0 Dec 3 15:12:14 mail kernel: [15481.189127] [<ffffffff8116366f>]=20 vfs_write+0xaf/0x180 Dec 3 15:12:14 mail kernel: [15481.190141] [<ffffffff81163b12>]=20 sys_pwrite64+0x82/0xb0 Dec 3 15:12:14 mail kernel: [15481.191149] [<ffffffff814178c2>]=20 system_call_fastpath+0x16/0x1b Dec 3 15:12:14 mail kernel: [15481.194224] kworker/2:3 D=20 0000000000000000 0 3713 2 0x00000000 Dec 3 15:12:14 mail kernel: [15481.195273] ffff880005d49cc0=20 0000000000000046 0000000246417c40 0000000800000000 Dec 3 15:12:14 mail kernel: [15481.196304] ffff8802013eaac0=20 ffff880005d49fd8 ffff880005d49fd8 ffff880005d49fd8 Dec 3 15:12:14 mail kernel: [15481.197329] ffff8802013eeae0=20 ffff8802013eaac0 ffff880005d49c20 ffffffffa03a3fcb Dec 3 15:12:14 mail kernel: [15481.198345] Call Trace: Dec 3 15:12:14 mail kernel: [15481.199365] [<ffffffffa03a3fcb>] ?=20 xfs_buf_iorequest+0x6b/0x100 [xfs] Dec 3 15:12:14 mail kernel: [15481.200393] [<ffffffffa03fa603>] ?=20 xlog_bdstrat+0x23/0x60 [xfs] Dec 3 15:12:14 mail kernel: [15481.201408] [<ffffffffa03fb287>] ?=20 xlog_sync+0x1d7/0x3c0 [xfs] Dec 3 15:12:14 mail kernel: [15481.202419] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:12:14 mail kernel: [15481.203418] [<ffffffffa03fe62b>]=20 _xfs_log_force_lsn+0x29b/0x2d0 [xfs] Dec 3 15:12:14 mail kernel: [15481.204420] [<ffffffff8105be20>] ?=20 try_to_wake_up+0x290/0x290 Dec 3 15:12:14 mail kernel: [15481.205438] [<ffffffffa03fa23b>]=20 xfs_trans_commit+0x29b/0x2b0 [xfs] Dec 3 15:12:14 mail kernel: [15481.206450] [<ffffffffa03b4c60>] ?=20 xfs_sync_inode_attr+0xf0/0xf0 [xfs] Dec 3 15:12:14 mail kernel: [15481.207466] [<ffffffffa03aa162>]=20 xfs_fs_log_dummy+0x62/0x90 [xfs] Dec 3 15:12:14 mail kernel: [15481.208486] [<ffffffff8111ff90>] ?=20 refresh_cpu_vm_stats+0x1a0/0x1a0 Dec 3 15:12:14 mail kernel: [15481.209504] [<ffffffffa03b4ce4>]=20 xfs_sync_worker+0x84/0x90 [xfs] Dec 3 15:12:14 mail kernel: [15481.210518] [<ffffffff8107f446>]=20 process_one_work+0x116/0x4d0 Dec 3 15:12:14 mail kernel: [15481.211531] [<ffffffff8107fdde>]=20 worker_thread+0x15e/0x350 Dec 3 15:12:14 mail kernel: [15481.212559] [<ffffffff8107fc80>] ?=20 manage_workers.isra.29+0x230/0x230 Dec 3 15:12:14 mail kernel: [15481.213573] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:12:14 mail kernel: [15481.214577] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:12:14 mail kernel: [15481.215596] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:12:14 mail kernel: [15481.216600] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:14:14 mail kernel: [15601.217226] xfsbufd/md127 D=20 00000001004562e8 0 527 2 0x00000000 Dec 3 15:14:14 mail kernel: [15601.218264] ffff8801f912fab0=20 0000000000000046 0000000100000000 0000000300000001 Dec 3 15:14:14 mail kernel: [15601.219326] ffff8802009dc740=20 ffff8801f912ffd8 ffff8801f912ffd8 ffff8801f912ffd8 Dec 3 15:14:14 mail kernel: [15601.220375] ffffffff8180d020=20 ffff8802009dc740 ffff8801f912fa10 ffffffff8104d003 Dec 3 15:14:14 mail kernel: [15601.221411] Call Trace: Dec 3 15:14:14 mail kernel: [15601.222441] [<ffffffff8104d003>] ?=20 __wake_up+0x53/0x70 Dec 3 15:14:14 mail kernel: [15601.223461] [<ffffffff8114d48b>] ?=20 kfree+0x3b/0x170 Dec 3 15:14:14 mail kernel: [15601.224479] [<ffffffffa01c732b>] ?=20 plugger_unplug+0x3b/0x50 [md_mod] Dec 3 15:14:14 mail kernel: [15601.225507] [<ffffffff8121b457>] ?=20 blk_flush_plug_list+0xa7/0x250 Dec 3 15:14:14 mail kernel: [15601.226516] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:14:14 mail kernel: [15601.227516] [<ffffffffa0378209>]=20 get_active_stripe+0x2e9/0x600 [raid456] Dec 3 15:14:14 mail kernel: [15601.228540] [<ffffffff8105be20>] ?=20 try_to_wake_up+0x290/0x290 Dec 3 15:14:14 mail kernel: [15601.229552] [<ffffffffa037be22>]=20 make_request+0x192/0x430 [raid456] Dec 3 15:14:14 mail kernel: [15601.230569] [<ffffffff81085750>] ?=20 abort_exclusive_wait+0xb0/0xb0 Dec 3 15:14:14 mail kernel: [15601.231588] [<ffffffffa01c964a>]=20 md_make_request+0xfa/0x240 [md_mod] Dec 3 15:14:14 mail kernel: [15601.232628] [<ffffffff811047a5>] ?=20 mempool_alloc_slab+0x15/0x20 Dec 3 15:14:14 mail kernel: [15601.233654] [<ffffffff81219917>]=20 generic_make_request+0xc7/0x100 Dec 3 15:14:14 mail kernel: [15601.234682] [<ffffffff812199d7>]=20 submit_bio+0x87/0x110 Dec 3 15:14:14 mail kernel: [15601.235731] [<ffffffffa03a3e6c>]=20 _xfs_buf_ioapply+0x15c/0x1c0 [xfs] Dec 3 15:14:14 mail kernel: [15601.236772] [<ffffffffa03a4d75>] ?=20 xfs_bdstrat_cb+0x65/0x120 [xfs] Dec 3 15:14:14 mail kernel: [15601.237815] [<ffffffffa03a3fb9>]=20 xfs_buf_iorequest+0x59/0x100 [xfs] Dec 3 15:14:14 mail kernel: [15601.238874] [<ffffffffa03a4d75>]=20 xfs_bdstrat_cb+0x65/0x120 [xfs] Dec 3 15:14:14 mail kernel: [15601.239916] [<ffffffffa03a4f3e>]=20 xfsbufd+0x10e/0x170 [xfs] Dec 3 15:14:14 mail kernel: [15601.240949] [<ffffffff8105be32>] ?=20 default_wake_function+0x12/0x20 Dec 3 15:14:14 mail kernel: [15601.241995] [<ffffffffa03a4e30>] ?=20 xfs_bdstrat_cb+0x120/0x120 [xfs] Dec 3 15:14:14 mail kernel: [15601.243026] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:14:14 mail kernel: [15601.244051] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:14:14 mail kernel: [15601.245076] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:14:14 mail kernel: [15601.246114] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:14:14 mail kernel: [15601.249209] md127_resync D=20 00000001004562fa 0 529 2 0x00000000 Dec 3 15:14:14 mail kernel: [15601.250259] ffff8801f9073b80=20 0000000000000046 0000000100000000 ffffffff81233ce7 Dec 3 15:14:14 mail kernel: [15601.251316] ffff8802013e8720=20 ffff8801f9073fd8 ffff8801f9073fd8 ffff8801f9073fd8 Dec 3 15:14:14 mail kernel: [15601.252392] ffff880202cb4e60=20 ffff8802013e8720 ffff880200b70138 ffff880004bff300 Dec 3 15:14:14 mail kernel: [15601.253459] Call Trace: Dec 3 15:14:14 mail kernel: [15601.254516] [<ffffffff81233ce7>] ?=20 kobject_put+0x27/0x60 Dec 3 15:14:14 mail kernel: [15601.255590] [<ffffffff8121b913>] ?=20 blk_queue_bio+0x2c3/0x410 Dec 3 15:14:14 mail kernel: [15601.256642] [<ffffffff814171a6>] ?=20 retint_kernel+0x26/0x30 Dec 3 15:14:14 mail kernel: [15601.257679] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:14:14 mail kernel: [15601.258721] [<ffffffffa0378209>]=20 get_active_stripe+0x2e9/0x600 [raid456] Dec 3 15:14:14 mail kernel: [15601.259756] [<ffffffff8105be20>] ?=20 try_to_wake_up+0x290/0x290 Dec 3 15:14:14 mail kernel: [15601.260792] [<ffffffffa037559a>]=20 sync_request+0x35a/0x370 [raid456] Dec 3 15:14:14 mail kernel: [15601.261838] [<ffffffffa01c9a08>] ?=20 is_mddev_idle+0x128/0x140 [md_mod] Dec 3 15:14:14 mail kernel: [15601.262867] [<ffffffffa01cd2bf>]=20 md_do_sync+0x7ff/0xc70 [md_mod] Dec 3 15:14:14 mail kernel: [15601.263895] [<ffffffff81085750>] ?=20 abort_exclusive_wait+0xb0/0xb0 Dec 3 15:14:14 mail kernel: [15601.264926] [<ffffffff810748cb>] ?=20 recalc_sigpending+0x1b/0x50 Dec 3 15:14:14 mail kernel: [15601.265972] [<ffffffffa01c9dce>]=20 md_thread+0x10e/0x140 [md_mod] Dec 3 15:14:14 mail kernel: [15601.267002] [<ffffffffa01c9cc0>] ?=20 md_register_thread+0x110/0x110 [md_mod] Dec 3 15:14:14 mail kernel: [15601.268033] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:14:14 mail kernel: [15601.269074] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:14:14 mail kernel: [15601.270103] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:14:14 mail kernel: [15601.271132] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:14:14 mail kernel: [15601.274249] loop0 D=20 000000010044b6c5 0 1729 2 0x00000000 Dec 3 15:14:14 mail kernel: [15601.275320] ffff8801f9b31b30=20 0000000000000046 0000000000000000 0000000000000000 Dec 3 15:14:14 mail kernel: [15601.276385] ffff880200950e40=20 ffff8801f9b31fd8 ffff8801f9b31fd8 ffff8801f9b31fd8 Dec 3 15:14:14 mail kernel: [15601.277454] ffff880202cb7200=20 ffff880200950e40 0000000000000002 ffff8801f9b31b78 Dec 3 15:14:14 mail kernel: [15601.278536] Call Trace: Dec 3 15:14:14 mail kernel: [15601.279594] [<ffffffff81101a55>] ?=20 find_get_pages_tag+0x125/0x150 Dec 3 15:14:14 mail kernel: [15601.280671] [<ffffffff8110e205>] ?=20 pagevec_lookup_tag+0x25/0x40 Dec 3 15:14:14 mail kernel: [15601.281736] [<ffffffff8101d639>] ?=20 read_tsc+0x9/0x20 Dec 3 15:14:14 mail kernel: [15601.282801] [<ffffffff8108f14d>] ?=20 ktime_get_ts+0xad/0xe0 Dec 3 15:14:14 mail kernel: [15601.283840] [<ffffffff81101d60>] ?=20 __lock_page+0x70/0x70 Dec 3 15:14:14 mail kernel: [15601.284870] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:14:14 mail kernel: [15601.285902] [<ffffffff8141416f>]=20 io_schedule+0x8f/0xd0 Dec 3 15:14:14 mail kernel: [15601.286919] [<ffffffff81101d6e>]=20 sleep_on_page+0xe/0x20 Dec 3 15:14:14 mail kernel: [15601.287924] [<ffffffff8141491f>]=20 __wait_on_bit+0x5f/0x90 Dec 3 15:14:14 mail kernel: [15601.288938] [<ffffffff81101f58>]=20 wait_on_page_bit+0x78/0x80 Dec 3 15:14:14 mail kernel: [15601.289944] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 Dec 3 15:14:14 mail kernel: [15601.290953] [<ffffffff811020cc>]=20 filemap_fdatawait_range+0x10c/0x1a0 Dec 3 15:14:14 mail kernel: [15601.291972] [<ffffffff811030c8>]=20 filemap_write_and_wait_range+0x68/0x80 Dec 3 15:14:14 mail kernel: [15601.292985] [<ffffffffa03a7234>]=20 xfs_file_fsync+0x54/0x340 [xfs] Dec 3 15:14:14 mail kernel: [15601.293993] [<ffffffff8119148b>]=20 vfs_fsync+0x2b/0x40 Dec 3 15:14:14 mail kernel: [15601.294993] [<ffffffffa04dacf2>]=20 do_bio_filebacked+0x1b2/0x320 [loop] Dec 3 15:14:14 mail kernel: [15601.296022] [<ffffffffa050efac>] ?=20 end_workqueue_bio+0x9c/0xa0 [btrfs] Dec 3 15:14:14 mail kernel: [15601.297029] [<ffffffffa04daf1b>]=20 loop_thread+0xbb/0x260 [loop] Dec 3 15:14:14 mail kernel: [15601.298032] [<ffffffff81085750>] ?=20 abort_exclusive_wait+0xb0/0xb0 Dec 3 15:14:14 mail kernel: [15601.299049] [<ffffffffa04dae60>] ?=20 do_bio_filebacked+0x320/0x320 [loop] Dec 3 15:14:14 mail kernel: [15601.300057] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:14:14 mail kernel: [15601.301063] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:14:14 mail kernel: [15601.302084] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:14:14 mail kernel: [15601.303093] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:14:14 mail kernel: [15601.306162] btrfs-transacti D=20 ffff8801f9ab7200 0 1756 2 0x00000000 Dec 3 15:14:14 mail kernel: [15601.307205] ffff8801e7533bc0=20 0000000000000046 ffff88020fc93400 0000000000000002 Dec 3 15:14:14 mail kernel: [15601.308257] ffff8801f9ab7200=20 ffff8801e7533fd8 ffff8801e7533fd8 ffff8801e7533fd8 Dec 3 15:14:14 mail kernel: [15601.309327] ffff880200950e40=20 ffff8801f9ab7200 ffff8801e7533b10 0000000081051ae2 Dec 3 15:14:14 mail kernel: [15601.310384] Call Trace: Dec 3 15:14:14 mail kernel: [15601.311427] [<ffffffff8105ad36>] ?=20 ttwu_do_activate.constprop.172+0x66/0x70 Dec 3 15:14:14 mail kernel: [15601.312509] [<ffffffff8105bd6e>] ?=20 try_to_wake_up+0x1de/0x290 Dec 3 15:14:14 mail kernel: [15601.313568] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:14:14 mail kernel: [15601.314612] [<ffffffff814147d5>]=20 schedule_timeout+0x305/0x390 Dec 3 15:14:14 mail kernel: [15601.315658] [<ffffffff8104d003>] ?=20 __wake_up+0x53/0x70 Dec 3 15:14:14 mail kernel: [15601.316680] [<ffffffff81413348>]=20 wait_for_common+0xc8/0x160 Dec 3 15:14:14 mail kernel: [15601.317699] [<ffffffff8105be20>] ?=20 try_to_wake_up+0x290/0x290 Dec 3 15:14:14 mail kernel: [15601.318730] [<ffffffff814133fd>]=20 wait_for_completion+0x1d/0x20 Dec 3 15:14:14 mail kernel: [15601.319745] [<ffffffffa050f0bb>]=20 write_dev_flush+0x4b/0x140 [btrfs] Dec 3 15:14:14 mail kernel: [15601.320760] [<ffffffffa0511086>]=20 write_all_supers+0x6f6/0x800 [btrfs] Dec 3 15:14:14 mail kernel: [15601.321791] [<ffffffffa05111a3>]=20 write_ctree_super+0x13/0x20 [btrfs] Dec 3 15:14:14 mail kernel: [15601.322809] [<ffffffffa05164dd>]=20 btrfs_commit_transaction+0x63d/0x880 [btrfs] Dec 3 15:14:14 mail kernel: [15601.323831] [<ffffffff81085750>] ?=20 abort_exclusive_wait+0xb0/0xb0 Dec 3 15:14:14 mail kernel: [15601.324855] [<ffffffffa0516b74>] ?=20 start_transaction+0x94/0x2b0 [btrfs] Dec 3 15:14:14 mail kernel: [15601.325894] [<ffffffffa050ed4d>]=20 transaction_kthread+0x26d/0x290 [btrfs] Dec 3 15:14:14 mail kernel: [15601.326918] [<ffffffffa050eae0>] ?=20 btrfs_congested_fn+0xd0/0xd0 [btrfs] Dec 3 15:14:14 mail kernel: [15601.327937] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:14:14 mail kernel: [15601.328963] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:14:14 mail kernel: [15601.329979] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:14:14 mail kernel: [15601.330995] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:14:14 mail kernel: [15601.334083] flush-9:127 D=20 00000000a01fc200 0 1891 2 0x00000000 Dec 3 15:14:14 mail kernel: [15601.335141] ffff880171771540=20 0000000000000046 0000000100000000 0000000300000001 Dec 3 15:14:14 mail kernel: [15601.336194] ffff8802009d9c80=20 ffff880171771fd8 ffff880171771fd8 ffff880171771fd8 Dec 3 15:14:14 mail kernel: [15601.337252] ffff8802009dc740=20 ffff8802009d9c80 ffff8801717714a0 ffffffff8104d003 Dec 3 15:14:14 mail kernel: [15601.338304] Call Trace: Dec 3 15:14:14 mail kernel: [15601.339347] [<ffffffff8104d003>] ?=20 __wake_up+0x53/0x70 Dec 3 15:14:14 mail kernel: [15601.340376] [<ffffffff8114d48b>] ?=20 kfree+0x3b/0x170 Dec 3 15:14:14 mail kernel: [15601.341398] [<ffffffffa01c732b>] ?=20 plugger_unplug+0x3b/0x50 [md_mod] Dec 3 15:14:14 mail kernel: [15601.342432] [<ffffffff8121b457>] ?=20 blk_flush_plug_list+0xa7/0x250 Dec 3 15:14:14 mail kernel: [15601.343453] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:14:14 mail kernel: [15601.344460] [<ffffffffa0378209>]=20 get_active_stripe+0x2e9/0x600 [raid456] Dec 3 15:14:14 mail kernel: [15601.345487] [<ffffffffa01c6400>] ?=20 md_wakeup_thread+0x10/0x30 [md_mod] Dec 3 15:14:14 mail kernel: [15601.346503] [<ffffffff8105be20>] ?=20 try_to_wake_up+0x290/0x290 Dec 3 15:14:14 mail kernel: [15601.347519] [<ffffffffa037be22>]=20 make_request+0x192/0x430 [raid456] Dec 3 15:14:14 mail kernel: [15601.348555] [<ffffffff81085750>] ?=20 abort_exclusive_wait+0xb0/0xb0 Dec 3 15:14:14 mail kernel: [15601.348559] [<ffffffffa01c964a>]=20 md_make_request+0xfa/0x240 [md_mod] Dec 3 15:14:14 mail kernel: [15601.348561] [<ffffffff81219917>]=20 generic_make_request+0xc7/0x100 Dec 3 15:14:14 mail kernel: [15601.348563] [<ffffffff812199d7>]=20 submit_bio+0x87/0x110 Dec 3 15:14:14 mail kernel: [15601.348565] [<ffffffff8118b300>] ?=20 __mark_inode_dirty+0x40/0x230 Dec 3 15:14:14 mail kernel: [15601.348571] [<ffffffffa03a16f7>]=20 xfs_submit_ioend_bio.isra.11+0x57/0x80 [xfs] Dec 3 15:14:14 mail kernel: [15601.348578] [<ffffffffa03a180e>]=20 xfs_submit_ioend+0xee/0x110 [xfs] Dec 3 15:14:14 mail kernel: [15601.348585] [<ffffffffa03a1a65>]=20 xfs_vm_writepage+0x235/0x530 [xfs] Dec 3 15:14:14 mail kernel: [15601.348588] [<ffffffff8110b75a>]=20 __writepage+0x1a/0x50 Dec 3 15:14:14 mail kernel: [15601.348591] [<ffffffff8110bc21>]=20 write_cache_pages+0x1e1/0x4e0 Dec 3 15:14:14 mail kernel: [15601.348599] [<ffffffffa03bab27>] ?=20 kmem_alloc+0x67/0xe0 [xfs] Dec 3 15:14:14 mail kernel: [15601.348602] [<ffffffff8110b740>] ?=20 bdi_set_max_ratio+0x90/0x90 Dec 3 15:14:14 mail kernel: [15601.348606] [<ffffffff8110bf6d>]=20 generic_writepages+0x4d/0x70 Dec 3 15:14:14 mail kernel: [15601.348612] [<ffffffffa03a06ad>]=20 xfs_vm_writepages+0x4d/0x60 [xfs] Dec 3 15:14:14 mail kernel: [15601.348615] [<ffffffff8110d6a4>]=20 do_writepages+0x24/0x40 Dec 3 15:14:14 mail kernel: [15601.348618] [<ffffffff8118b76f>]=20 writeback_single_inode+0x10f/0x3f0 Dec 3 15:14:14 mail kernel: [15601.348621] [<ffffffff8118be4a>]=20 writeback_sb_inodes+0x1aa/0x270 Dec 3 15:14:14 mail kernel: [15601.348624] [<ffffffff8118bfae>]=20 __writeback_inodes_wb+0x9e/0xd0 Dec 3 15:14:14 mail kernel: [15601.348627] [<ffffffff8118c5cb>]=20 wb_writeback+0x37b/0x3b0 Dec 3 15:14:14 mail kernel: [15601.348631] [<ffffffff8117d562>] ?=20 get_nr_inodes+0x52/0x70 Dec 3 15:14:14 mail kernel: [15601.348634] [<ffffffff8117e262>] ?=20 get_nr_dirty_inodes+0x52/0x80 Dec 3 15:14:14 mail kernel: [15601.348636] [<ffffffff8118c69f>]=20 wb_check_old_data_flush+0x9f/0xb0 Dec 3 15:14:14 mail kernel: [15601.348639] [<ffffffff8118db7c>]=20 wb_do_writeback+0x19c/0x230 Dec 3 15:14:14 mail kernel: [15601.348642] [<ffffffff81414643>] ?=20 schedule_timeout+0x173/0x390 Dec 3 15:14:14 mail kernel: [15601.348645] [<ffffffff81071b10>] ?=20 init_timer_deferrable_key+0x30/0x30 Dec 3 15:14:14 mail kernel: [15601.348648] [<ffffffff8118dc9c>]=20 bdi_writeback_thread+0x8c/0x340 Dec 3 15:14:14 mail kernel: [15601.348651] [<ffffffff8118dc10>] ?=20 wb_do_writeback+0x230/0x230 Dec 3 15:14:14 mail kernel: [15601.348654] [<ffffffff81084e0c>]=20 kthread+0x8c/0xa0 Dec 3 15:14:14 mail kernel: [15601.348657] [<ffffffff81419a34>]=20 kernel_thread_helper+0x4/0x10 Dec 3 15:14:14 mail kernel: [15601.348660] [<ffffffff81084d80>] ?=20 kthread_worker_fn+0x190/0x190 Dec 3 15:14:14 mail kernel: [15601.348663] [<ffffffff81419a30>] ?=20 gs_change+0x13/0x13 Dec 3 15:14:14 mail kernel: [15601.348669] smbd D=20 000000010044b8d7 0 2058 823 0x00000004 Dec 3 15:14:14 mail kernel: [15601.348672] ffff88005cf37a08=20 0000000000000082 ffff88005cf37938 ffffffff81371188 Dec 3 15:14:14 mail kernel: [15601.348676] ffff8802012e63c0=20 ffff88005cf37fd8 ffff88005cf37fd8 ffff88005cf37fd8 Dec 3 15:14:14 mail kernel: [15601.348680] ffff880202d21c80=20 ffff8802012e63c0 ffff8802012769c0 0000000000000246 Dec 3 15:14:14 mail kernel: [15601.348684] Call Trace: Dec 3 15:14:14 mail kernel: [15601.348687] [<ffffffff81371188>] ?=20 sch_direct_xmit+0x68/0x1d0 Dec 3 15:14:14 mail kernel: [15601.348691] [<ffffffff81355a00>] ?=20 dev_queue_xmit+0x200/0x680 Dec 3 15:14:14 mail kernel: [15601.348694] [<ffffffff81389200>] ?=20 ip_forward_options+0x1c0/0x1c0 Dec 3 15:14:14 mail kernel: [15601.348697] [<ffffffff8138adbe>] ?=20 ip_finish_output+0x18e/0x310 Dec 3 15:14:14 mail kernel: [15601.348700] [<ffffffff8101d639>] ?=20 read_tsc+0x9/0x20 Dec 3 15:14:14 mail kernel: [15601.348703] [<ffffffff8108f14d>] ?=20 ktime_get_ts+0xad/0xe0 Dec 3 15:14:14 mail kernel: [15601.348705] [<ffffffff81101d60>] ?=20 __lock_page+0x70/0x70 Dec 3 15:14:14 mail kernel: [15601.348708] [<ffffffff814140bf>]=20 schedule+0x3f/0x60 Dec 3 15:14:14 mail kernel: [15601.348711] [<ffffffff8141416f>]=20 io_schedule+0x8f/0xd0 Dec 3 15:14:14 mail kernel: [15601.348714] [<ffffffff81101d6e>]=20 sleep_on_page+0xe/0x20 Dec 3 15:14:14 mail kernel: [15601.348716] [<ffffffff8141491f>]=20 __wait_on_bit+0x5f/0x90 Dec 3 15:14:14 mail kernel: [15601.348719] [<ffffffff81101f58>]=20 wait_on_page_bit+0x78/0x80 Dec 3 15:14:14 mail kernel: [15601.348722] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 Dec 3 15:14:14 mail kernel: [15601.348725] [<ffffffff81102845>]=20 grab_cache_page_write_begin+0x95/0xe0 Dec 3 15:14:14 mail kernel: [15601.348732] [<ffffffffa03a1150>] ?=20 xfs_get_blocks_direct+0x20/0x20 [xfs] Dec 3 15:14:14 mail kernel: [15601.348736] [<ffffffff811967b8>]=20 block_write_begin+0x38/0xa0 Dec 3 15:14:14 mail kernel: [15601.348743] [<ffffffffa03a1213>]=20 xfs_vm_write_begin+0x43/0x70 [xfs] Dec 3 15:14:14 mail kernel: [15601.348746] [<ffffffff8110233c>]=20 generic_file_buffered_write+0x10c/0x270 Dec 3 15:14:14 mail kernel: [15601.348754] [<ffffffffa03aad66>] ?=20 xfs_iunlock+0x116/0x180 [xfs] Dec 3 15:14:14 mail kernel: [15601.348761] [<ffffffffa03a7fef>]=20 xfs_file_buffered_aio_write+0x10f/0x200 [xfs] Dec 3 15:14:14 mail kernel: [15601.348768] [<ffffffffa03a8252>]=20 xfs_file_aio_write+0x172/0x2a0 [xfs] Dec 3 15:14:14 mail kernel: [15601.348772] [<ffffffff81162d62>]=20 do_sync_write+0xd2/0x110 Dec 3 15:14:14 mail kernel: [15601.348775] [<ffffffff811f0fcc>] ?=20 security_file_permission+0x2c/0xb0 Dec 3 15:14:14 mail kernel: [15601.348778] [<ffffffff81163311>] ?=20 rw_verify_area+0x61/0xf0 Dec 3 15:14:14 mail kernel: [15601.348781] [<ffffffff8116366f>]=20 vfs_write+0xaf/0x180 Dec 3 15:14:14 mail kernel: [15601.348784] [<ffffffff81163b12>]=20 sys_pwrite64+0x82/0xb0 Dec 3 15:14:14 mail kernel: [15601.348787] [<ffffffff814178c2>]=20 system_call_fastpath+0x16/0x1b On 2/12/2011 4:01 =CE=BC=CE=BC, Chris Mason wrote: > On Fri, Dec 02, 2011 at 02:46:48PM +0100, Tobias wrote: >> Hi Chris! >> >> Am 01.12.2011 19:41, schrieb Chris Mason: >>> So, the transaction close is in btrfs_evict_inode, which sounds lik= e a >>> deadlock recently fixed by this commit: >>> >>> http://git.kernel.org/?p=3Dlinux/kernel/git/mason/linux-btrfs.git;a= =3Dcommit;h=3Daa38a711a893accf5b5192f3d705a120deaa81e0 >>> >>> If you pull the for-linus branch from today, hopefully the problem = will >>> be gone. >>> >> This looks very good. With this Kernel i still have some hangs, but >> only in rsync, only under high load and they don't lock up the >> system - so i guess it's ok now. >> >> Thank You very much for Your help! > Glad to hear this is working. All the credit to Miao, who found the > deadlock. > >> When will this patches go into the main Kernel? > Linus pulled them in yesterday. > > -chris > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs= " in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-02 13:46 ` Tobias 2011-12-02 14:01 ` Chris Mason @ 2011-12-02 15:22 ` Konstantinos Skarlatos 2011-12-02 15:48 ` Tobias 1 sibling, 1 reply; 15+ messages in thread From: Konstantinos Skarlatos @ 2011-12-02 15:22 UTC (permalink / raw) To: Tobias; +Cc: Chris Mason, linux-btrfs Hi all On 2/12/2011 3:46 =CE=BC=CE=BC, Tobias wrote: > Hi Chris! > > Am 01.12.2011 19:41, schrieb Chris Mason: >> >> So, the transaction close is in btrfs_evict_inode, which sounds like= a >> deadlock recently fixed by this commit: >> >> http://git.kernel.org/?p=3Dlinux/kernel/git/mason/linux-btrfs.git;a=3D= commit;h=3Daa38a711a893accf5b5192f3d705a120deaa81e0=20 >> >> >> If you pull the for-linus branch from today, hopefully the problem w= ill >> be gone. >> > > This looks very good. With this Kernel i still have some hangs, but=20 > only in rsync, only under high load and they don't lock up the system= =20 > - so i guess it's ok now. I still have hangs and lock ups under the same situation (rsync of many= =20 files) under 3.2rc3. rc3 made the hang appear after 200gb of files,=20 while in rc2 i had hangs after only 11gb . > > Thank You very much for Your help! > > When will this patches go into the main Kernel? > > Tobias > > --=20 > To unsubscribe from this list: send the line "unsubscribe linux-btrfs= " in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-02 15:22 ` Konstantinos Skarlatos @ 2011-12-02 15:48 ` Tobias 2011-12-02 19:53 ` Konstantinos Skarlatos 0 siblings, 1 reply; 15+ messages in thread From: Tobias @ 2011-12-02 15:48 UTC (permalink / raw) To: Konstantinos Skarlatos; +Cc: linux-btrfs Am 02.12.2011 16:22, schrieb Konstantinos Skarlatos: >>> So, the transaction close is in btrfs_evict_inode, which sounds like a >>> deadlock recently fixed by this commit: >>> >>> http://git.kernel.org/?p=linux/kernel/git/mason/linux-btrfs.git;a=commit;h=aa38a711a893accf5b5192f3d705a120deaa81e0 >>> >>> >>> If you pull the for-linus branch from today, hopefully the problem will >>> be gone. >>> >> >> This looks very good. With this Kernel i still have some hangs, but >> only in rsync, only under high load and they don't lock up the system >> - so i guess it's ok now. > > I still have hangs and lock ups under the same situation (rsync of > many files) under 3.2rc3. rc3 made the hang appear after 200gb of > files, while in rc2 i had hangs after only 11gb . Yes, i had them too in 3.2rc3! The problem where solved with patches from the "btrfs-for-linus" -branch. (see link above). Tobias ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-02 15:48 ` Tobias @ 2011-12-02 19:53 ` Konstantinos Skarlatos 2011-12-03 0:35 ` Konstantinos Skarlatos 0 siblings, 1 reply; 15+ messages in thread From: Konstantinos Skarlatos @ 2011-12-02 19:53 UTC (permalink / raw) To: Tobias; +Cc: linux-btrfs I see they got into 3.2rc4, so I am now compiling it. I will report=20 back in a few hours On =CE=A0=CE=B1=CF=81=CE=B1=CF=83=CE=BA=CE=B5=CF=85=CE=AE, 2 =CE=94=CE=B5= =CE=BA=CE=AD=CE=BC=CE=B2=CF=81=CE=B9=CE=BF=CF=82 2011 5:48:31 =CE=BC=CE= =BC, Tobias wrote: > Am 02.12.2011 16:22, schrieb Konstantinos Skarlatos: >>>> So, the transaction close is in btrfs_evict_inode, which sounds li= ke a >>>> deadlock recently fixed by this commit: >>>> >>>> http://git.kernel.org/?p=3Dlinux/kernel/git/mason/linux-btrfs.git;= a=3Dcommit;h=3Daa38a711a893accf5b5192f3d705a120deaa81e0=20 >>>> >>>> >>>> If you pull the for-linus branch from today, hopefully the problem= =20 >>>> will >>>> be gone. >>>> >>> >>> This looks very good. With this Kernel i still have some hangs, but= =20 >>> only in rsync, only under high load and they don't lock up the=20 >>> system - so i guess it's ok now. >> >> I still have hangs and lock ups under the same situation (rsync of=20 >> many files) under 3.2rc3. rc3 made the hang appear after 200gb of=20 >> files, while in rc2 i had hangs after only 11gb . > > Yes, i had them too in 3.2rc3! The problem where solved with patches=20 > from the "btrfs-for-linus" -branch. (see link above). > > Tobias > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-02 19:53 ` Konstantinos Skarlatos @ 2011-12-03 0:35 ` Konstantinos Skarlatos 2011-12-03 14:36 ` Konstantinos Skarlatos 0 siblings, 1 reply; 15+ messages in thread From: Konstantinos Skarlatos @ 2011-12-03 0:35 UTC (permalink / raw) To: Tobias; +Cc: linux-btrfs After about 1TB of rsyncs from multiple servers at the same time, plus=20 some heavy filesystem loading, i believe that 3.2rc4 solves the problem= =20 for me. Now if only we had deduplication and an fsck tool :) On =CE=A0=CE=B1=CF=81=CE=B1=CF=83=CE=BA=CE=B5=CF=85=CE=AE, 2 =CE=94=CE=B5= =CE=BA=CE=AD=CE=BC=CE=B2=CF=81=CE=B9=CE=BF=CF=82 2011 9:53:10 =CE=BC=CE= =BC, Konstantinos Skarlatos=20 wrote: > I see they got into 3.2rc4, so I am now compiling it. I will report=20 > back in a few hours > > On =CE=A0=CE=B1=CF=81=CE=B1=CF=83=CE=BA=CE=B5=CF=85=CE=AE, 2 =CE=94=CE= =B5=CE=BA=CE=AD=CE=BC=CE=B2=CF=81=CE=B9=CE=BF=CF=82 2011 5:48:31 =CE=BC= =CE=BC, Tobias wrote: >> Am 02.12.2011 16:22, schrieb Konstantinos Skarlatos: >>>>> So, the transaction close is in btrfs_evict_inode, which sounds=20 >>>>> like a >>>>> deadlock recently fixed by this commit: >>>>> >>>>> http://git.kernel.org/?p=3Dlinux/kernel/git/mason/linux-btrfs.git= ;a=3Dcommit;h=3Daa38a711a893accf5b5192f3d705a120deaa81e0=20 >>>>> >>>>> >>>>> If you pull the for-linus branch from today, hopefully the proble= m=20 >>>>> will >>>>> be gone. >>>>> >>>> >>>> This looks very good. With this Kernel i still have some hangs, bu= t=20 >>>> only in rsync, only under high load and they don't lock up the=20 >>>> system - so i guess it's ok now. >>> >>> I still have hangs and lock ups under the same situation (rsync of=20 >>> many files) under 3.2rc3. rc3 made the hang appear after 200gb of=20 >>> files, while in rc2 i had hangs after only 11gb . >> >> Yes, i had them too in 3.2rc3! The problem where solved with patches= =20 >> from the "btrfs-for-linus" -branch. (see link above). >> >> Tobias >> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-03 0:35 ` Konstantinos Skarlatos @ 2011-12-03 14:36 ` Konstantinos Skarlatos 2011-12-05 13:12 ` Chris Mason 0 siblings, 1 reply; 15+ messages in thread From: Konstantinos Skarlatos @ 2011-12-03 14:36 UTC (permalink / raw) To: Tobias; +Cc: linux-btrfs unfortunately i was wrong. rc4 does not fix this issue for me when=20 rsyncing large amounts of data... my mount options: mount -o loop,compress=3Dzlib,compress-force btrfs_test /storage/btrfs the filesystem is a file on a raid5 xfs volume. [15481.098588] INFO: task loop0:1729 blocked for more than 120 seconds. [15481.099571] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15481.100564] loop0 D 000000010044b6c5 0 1729 2=20 0x00000000 [15481.101550] ffff8801f9b31b30 0000000000000046 0000000000000000=20 0000000000000000 [15481.102548] ffff880200950e40 ffff8801f9b31fd8 ffff8801f9b31fd8=20 ffff8801f9b31fd8 [15481.103539] ffff880202cb7200 ffff880200950e40 0000000000000002=20 ffff8801f9b31b78 [15481.104533] Call Trace: [15481.105531] [<ffffffff81101a55>] ? find_get_pages_tag+0x125/0x150 [15481.106541] [<ffffffff8110e205>] ? pagevec_lookup_tag+0x25/0x40 [15481.107552] [<ffffffff8101d639>] ? read_tsc+0x9/0x20 [15481.108576] [<ffffffff8108f14d>] ? ktime_get_ts+0xad/0xe0 [15481.109592] [<ffffffff81101d60>] ? __lock_page+0x70/0x70 [15481.110607] [<ffffffff814140bf>] schedule+0x3f/0x60 [15481.111619] [<ffffffff8141416f>] io_schedule+0x8f/0xd0 [15481.112641] [<ffffffff81101d6e>] sleep_on_page+0xe/0x20 [15481.113639] [<ffffffff8141491f>] __wait_on_bit+0x5f/0x90 [15481.114629] [<ffffffff81101f58>] wait_on_page_bit+0x78/0x80 [15481.115628] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 [15481.116614] [<ffffffff811020cc>] filemap_fdatawait_range+0x10c/0x1a= 0 [15481.117613] [<ffffffff811030c8>]=20 filemap_write_and_wait_range+0x68/0x80 [15481.118630] [<ffffffffa03a7234>] xfs_file_fsync+0x54/0x340 [xfs] [15481.119629] [<ffffffff8119148b>] vfs_fsync+0x2b/0x40 [15481.120627] [<ffffffffa04dacf2>] do_bio_filebacked+0x1b2/0x320=20 [loop] [15481.121645] [<ffffffffa050efac>] ? end_workqueue_bio+0x9c/0xa0=20 [btrfs] [15481.122668] [<ffffffffa04daf1b>] loop_thread+0xbb/0x260 [loop] [15481.123674] [<ffffffff81085750>] ? abort_exclusive_wait+0xb0/0xb0 [15481.124676] [<ffffffffa04dae60>] ? do_bio_filebacked+0x320/0x320=20 [loop] [15481.125698] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15481.126710] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15481.127721] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15481.128742] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15481.129728] INFO: task btrfs-transacti:1756 blocked for more than=20 120 seconds. [15481.130706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15481.131702] btrfs-transacti D ffff8801f9ab7200 0 1756 2=20 0x00000000 [15481.132723] ffff8801e7533bc0 0000000000000046 ffff88020fc93400=20 0000000000000002 [15481.133744] ffff8801f9ab7200 ffff8801e7533fd8 ffff8801e7533fd8=20 ffff8801e7533fd8 [15481.134771] ffff880200950e40 ffff8801f9ab7200 ffff8801e7533b10=20 0000000081051ae2 [15481.135813] Call Trace: [15481.136828] [<ffffffff8105ad36>] ?=20 ttwu_do_activate.constprop.172+0x66/0x70 [15481.137863] [<ffffffff8105bd6e>] ? try_to_wake_up+0x1de/0x290 [15481.138914] [<ffffffff814140bf>] schedule+0x3f/0x60 [15481.139956] [<ffffffff814147d5>] schedule_timeout+0x305/0x390 [15481.141007] [<ffffffff8104d003>] ? __wake_up+0x53/0x70 [15481.142074] [<ffffffff81413348>] wait_for_common+0xc8/0x160 [15481.143124] [<ffffffff8105be20>] ? try_to_wake_up+0x290/0x290 [15481.144170] [<ffffffff814133fd>] wait_for_completion+0x1d/0x20 [15481.145229] [<ffffffffa050f0bb>] write_dev_flush+0x4b/0x140 [btrfs] [15481.146275] [<ffffffffa0511086>] write_all_supers+0x6f6/0x800=20 [btrfs] [15481.147317] [<ffffffffa05111a3>] write_ctree_super+0x13/0x20 [btrfs= ] [15481.148354] [<ffffffffa05164dd>]=20 btrfs_commit_transaction+0x63d/0x880 [btrfs] [15481.149397] [<ffffffff81085750>] ? abort_exclusive_wait+0xb0/0xb0 [15481.150416] [<ffffffffa0516b74>] ? start_transaction+0x94/0x2b0=20 [btrfs] [15481.151444] [<ffffffffa050ed4d>] transaction_kthread+0x26d/0x290=20 [btrfs] [15481.152492] [<ffffffffa050eae0>] ? btrfs_congested_fn+0xd0/0xd0=20 [btrfs] [15481.153519] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15481.154542] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15481.155553] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15481.156522] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15481.157501] INFO: task smbd:2058 blocked for more than 120 seconds. [15481.158513] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15481.159519] smbd D 000000010044b8d7 0 2058 823=20 0x00000004 [15481.160544] ffff88005cf37a08 0000000000000082 ffff88005cf37938=20 ffffffff81371188 [15481.161588] ffff8802012e63c0 ffff88005cf37fd8 ffff88005cf37fd8=20 ffff88005cf37fd8 [15481.162658] ffff880202d21c80 ffff8802012e63c0 ffff8802012769c0=20 0000000000000246 [15481.163708] Call Trace: [15481.164736] [<ffffffff81371188>] ? sch_direct_xmit+0x68/0x1d0 [15481.165781] [<ffffffff81355a00>] ? dev_queue_xmit+0x200/0x680 [15481.166805] [<ffffffff81389200>] ? ip_forward_options+0x1c0/0x1c0 [15481.167822] [<ffffffff8138adbe>] ? ip_finish_output+0x18e/0x310 [15481.168850] [<ffffffff8101d639>] ? read_tsc+0x9/0x20 [15481.169853] [<ffffffff8108f14d>] ? ktime_get_ts+0xad/0xe0 [15481.170849] [<ffffffff81101d60>] ? __lock_page+0x70/0x70 [15481.171858] [<ffffffff814140bf>] schedule+0x3f/0x60 [15481.172851] [<ffffffff8141416f>] io_schedule+0x8f/0xd0 [15481.173844] [<ffffffff81101d6e>] sleep_on_page+0xe/0x20 [15481.174836] [<ffffffff8141491f>] __wait_on_bit+0x5f/0x90 [15481.175845] [<ffffffff81101f58>] wait_on_page_bit+0x78/0x80 [15481.176842] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 [15481.177845] [<ffffffff81102845>]=20 grab_cache_page_write_begin+0x95/0xe0 [15481.178872] [<ffffffffa03a1150>] ? xfs_get_blocks_direct+0x20/0x20=20 [xfs] [15481.179888] [<ffffffff811967b8>] block_write_begin+0x38/0xa0 [15481.180904] [<ffffffffa03a1213>] xfs_vm_write_begin+0x43/0x70 [xfs] [15481.181928] [<ffffffff8110233c>]=20 generic_file_buffered_write+0x10c/0x270 [15481.182955] [<ffffffffa03aad66>] ? xfs_iunlock+0x116/0x180 [xfs] [15481.183979] [<ffffffffa03a7fef>]=20 xfs_file_buffered_aio_write+0x10f/0x200 [xfs] [15481.185011] [<ffffffffa03a8252>] xfs_file_aio_write+0x172/0x2a0=20 [xfs] [15481.186052] [<ffffffff81162d62>] do_sync_write+0xd2/0x110 [15481.187073] [<ffffffff811f0fcc>] ?=20 security_file_permission+0x2c/0xb0 [15481.188096] [<ffffffff81163311>] ? rw_verify_area+0x61/0xf0 [15481.189127] [<ffffffff8116366f>] vfs_write+0xaf/0x180 [15481.190141] [<ffffffff81163b12>] sys_pwrite64+0x82/0xb0 [15481.191149] [<ffffffff814178c2>] system_call_fastpath+0x16/0x1b [15481.192171] INFO: task kworker/2:3:3713 blocked for more than 120=20 seconds. [15481.193191] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15481.194224] kworker/2:3 D 0000000000000000 0 3713 2=20 0x00000000 [15481.195273] ffff880005d49cc0 0000000000000046 0000000246417c40=20 0000000800000000 [15481.196304] ffff8802013eaac0 ffff880005d49fd8 ffff880005d49fd8=20 ffff880005d49fd8 [15481.197329] ffff8802013eeae0 ffff8802013eaac0 ffff880005d49c20=20 ffffffffa03a3fcb [15481.198345] Call Trace: [15481.199365] [<ffffffffa03a3fcb>] ? xfs_buf_iorequest+0x6b/0x100=20 [xfs] [15481.200393] [<ffffffffa03fa603>] ? xlog_bdstrat+0x23/0x60 [xfs] [15481.201408] [<ffffffffa03fb287>] ? xlog_sync+0x1d7/0x3c0 [xfs] [15481.202419] [<ffffffff814140bf>] schedule+0x3f/0x60 [15481.203418] [<ffffffffa03fe62b>] _xfs_log_force_lsn+0x29b/0x2d0=20 [xfs] [15481.204420] [<ffffffff8105be20>] ? try_to_wake_up+0x290/0x290 [15481.205438] [<ffffffffa03fa23b>] xfs_trans_commit+0x29b/0x2b0 [xfs] [15481.206450] [<ffffffffa03b4c60>] ? xfs_sync_inode_attr+0xf0/0xf0=20 [xfs] [15481.207466] [<ffffffffa03aa162>] xfs_fs_log_dummy+0x62/0x90 [xfs] [15481.208486] [<ffffffff8111ff90>] ? refresh_cpu_vm_stats+0x1a0/0x1a0 [15481.209504] [<ffffffffa03b4ce4>] xfs_sync_worker+0x84/0x90 [xfs] [15481.210518] [<ffffffff8107f446>] process_one_work+0x116/0x4d0 [15481.211531] [<ffffffff8107fdde>] worker_thread+0x15e/0x350 [15481.212559] [<ffffffff8107fc80>] ?=20 manage_workers.isra.29+0x230/0x230 [15481.213573] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15481.214577] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15481.215596] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15481.216600] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15601.215172] INFO: task xfsbufd/md127:527 blocked for more than 120=20 seconds. [15601.216198] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15601.217226] xfsbufd/md127 D 00000001004562e8 0 527 2=20 0x00000000 [15601.218264] ffff8801f912fab0 0000000000000046 0000000100000000=20 0000000300000001 [15601.219326] ffff8802009dc740 ffff8801f912ffd8 ffff8801f912ffd8=20 ffff8801f912ffd8 [15601.220375] ffffffff8180d020 ffff8802009dc740 ffff8801f912fa10=20 ffffffff8104d003 [15601.221411] Call Trace: [15601.222441] [<ffffffff8104d003>] ? __wake_up+0x53/0x70 [15601.223461] [<ffffffff8114d48b>] ? kfree+0x3b/0x170 [15601.224479] [<ffffffffa01c732b>] ? plugger_unplug+0x3b/0x50 [md_mod= ] [15601.225507] [<ffffffff8121b457>] ? blk_flush_plug_list+0xa7/0x250 [15601.226516] [<ffffffff814140bf>] schedule+0x3f/0x60 [15601.227516] [<ffffffffa0378209>] get_active_stripe+0x2e9/0x600=20 [raid456] [15601.228540] [<ffffffff8105be20>] ? try_to_wake_up+0x290/0x290 [15601.229552] [<ffffffffa037be22>] make_request+0x192/0x430 [raid456] [15601.230569] [<ffffffff81085750>] ? abort_exclusive_wait+0xb0/0xb0 [15601.231588] [<ffffffffa01c964a>] md_make_request+0xfa/0x240 [md_mod= ] [15601.232628] [<ffffffff811047a5>] ? mempool_alloc_slab+0x15/0x20 [15601.233654] [<ffffffff81219917>] generic_make_request+0xc7/0x100 [15601.234682] [<ffffffff812199d7>] submit_bio+0x87/0x110 [15601.235731] [<ffffffffa03a3e6c>] _xfs_buf_ioapply+0x15c/0x1c0 [xfs] [15601.236772] [<ffffffffa03a4d75>] ? xfs_bdstrat_cb+0x65/0x120 [xfs] [15601.237815] [<ffffffffa03a3fb9>] xfs_buf_iorequest+0x59/0x100 [xfs] [15601.238874] [<ffffffffa03a4d75>] xfs_bdstrat_cb+0x65/0x120 [xfs] [15601.239916] [<ffffffffa03a4f3e>] xfsbufd+0x10e/0x170 [xfs] [15601.240949] [<ffffffff8105be32>] ? default_wake_function+0x12/0x20 [15601.241995] [<ffffffffa03a4e30>] ? xfs_bdstrat_cb+0x120/0x120 [xfs] [15601.243026] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15601.244051] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15601.245076] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15601.246114] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15601.247129] INFO: task md127_resync:529 blocked for more than 120=20 seconds. [15601.248156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15601.249209] md127_resync D 00000001004562fa 0 529 2=20 0x00000000 [15601.250259] ffff8801f9073b80 0000000000000046 0000000100000000=20 ffffffff81233ce7 [15601.251316] ffff8802013e8720 ffff8801f9073fd8 ffff8801f9073fd8=20 ffff8801f9073fd8 [15601.252392] ffff880202cb4e60 ffff8802013e8720 ffff880200b70138=20 ffff880004bff300 [15601.253459] Call Trace: [15601.254516] [<ffffffff81233ce7>] ? kobject_put+0x27/0x60 [15601.255590] [<ffffffff8121b913>] ? blk_queue_bio+0x2c3/0x410 [15601.256642] [<ffffffff814171a6>] ? retint_kernel+0x26/0x30 [15601.257679] [<ffffffff814140bf>] schedule+0x3f/0x60 [15601.258721] [<ffffffffa0378209>] get_active_stripe+0x2e9/0x600=20 [raid456] [15601.259756] [<ffffffff8105be20>] ? try_to_wake_up+0x290/0x290 [15601.260792] [<ffffffffa037559a>] sync_request+0x35a/0x370 [raid456] [15601.261838] [<ffffffffa01c9a08>] ? is_mddev_idle+0x128/0x140=20 [md_mod] [15601.262867] [<ffffffffa01cd2bf>] md_do_sync+0x7ff/0xc70 [md_mod] [15601.263895] [<ffffffff81085750>] ? abort_exclusive_wait+0xb0/0xb0 [15601.264926] [<ffffffff810748cb>] ? recalc_sigpending+0x1b/0x50 [15601.265972] [<ffffffffa01c9dce>] md_thread+0x10e/0x140 [md_mod] [15601.267002] [<ffffffffa01c9cc0>] ? md_register_thread+0x110/0x110=20 [md_mod] [15601.268033] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15601.269074] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15601.270103] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15601.271132] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15601.272170] INFO: task loop0:1729 blocked for more than 120 seconds. [15601.273202] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15601.274249] loop0 D 000000010044b6c5 0 1729 2=20 0x00000000 [15601.275320] ffff8801f9b31b30 0000000000000046 0000000000000000=20 0000000000000000 [15601.276385] ffff880200950e40 ffff8801f9b31fd8 ffff8801f9b31fd8=20 ffff8801f9b31fd8 [15601.277454] ffff880202cb7200 ffff880200950e40 0000000000000002=20 ffff8801f9b31b78 [15601.278536] Call Trace: [15601.279594] [<ffffffff81101a55>] ? find_get_pages_tag+0x125/0x150 [15601.280671] [<ffffffff8110e205>] ? pagevec_lookup_tag+0x25/0x40 [15601.281736] [<ffffffff8101d639>] ? read_tsc+0x9/0x20 [15601.282801] [<ffffffff8108f14d>] ? ktime_get_ts+0xad/0xe0 [15601.283840] [<ffffffff81101d60>] ? __lock_page+0x70/0x70 [15601.284870] [<ffffffff814140bf>] schedule+0x3f/0x60 [15601.285902] [<ffffffff8141416f>] io_schedule+0x8f/0xd0 [15601.286919] [<ffffffff81101d6e>] sleep_on_page+0xe/0x20 [15601.287924] [<ffffffff8141491f>] __wait_on_bit+0x5f/0x90 [15601.288938] [<ffffffff81101f58>] wait_on_page_bit+0x78/0x80 [15601.289944] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 [15601.290953] [<ffffffff811020cc>] filemap_fdatawait_range+0x10c/0x1a= 0 [15601.291972] [<ffffffff811030c8>]=20 filemap_write_and_wait_range+0x68/0x80 [15601.292985] [<ffffffffa03a7234>] xfs_file_fsync+0x54/0x340 [xfs] [15601.293993] [<ffffffff8119148b>] vfs_fsync+0x2b/0x40 [15601.294993] [<ffffffffa04dacf2>] do_bio_filebacked+0x1b2/0x320=20 [loop] [15601.296022] [<ffffffffa050efac>] ? end_workqueue_bio+0x9c/0xa0=20 [btrfs] [15601.297029] [<ffffffffa04daf1b>] loop_thread+0xbb/0x260 [loop] [15601.298032] [<ffffffff81085750>] ? abort_exclusive_wait+0xb0/0xb0 [15601.299049] [<ffffffffa04dae60>] ? do_bio_filebacked+0x320/0x320=20 [loop] [15601.300057] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15601.301063] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15601.302084] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15601.303093] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15601.304098] INFO: task btrfs-transacti:1756 blocked for more than=20 120 seconds. [15601.305126] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15601.306162] btrfs-transacti D ffff8801f9ab7200 0 1756 2=20 0x00000000 [15601.307205] ffff8801e7533bc0 0000000000000046 ffff88020fc93400=20 0000000000000002 [15601.308257] ffff8801f9ab7200 ffff8801e7533fd8 ffff8801e7533fd8=20 ffff8801e7533fd8 [15601.309327] ffff880200950e40 ffff8801f9ab7200 ffff8801e7533b10=20 0000000081051ae2 [15601.310384] Call Trace: [15601.311427] [<ffffffff8105ad36>] ?=20 ttwu_do_activate.constprop.172+0x66/0x70 [15601.312509] [<ffffffff8105bd6e>] ? try_to_wake_up+0x1de/0x290 [15601.313568] [<ffffffff814140bf>] schedule+0x3f/0x60 [15601.314612] [<ffffffff814147d5>] schedule_timeout+0x305/0x390 [15601.315658] [<ffffffff8104d003>] ? __wake_up+0x53/0x70 [15601.316680] [<ffffffff81413348>] wait_for_common+0xc8/0x160 [15601.317699] [<ffffffff8105be20>] ? try_to_wake_up+0x290/0x290 [15601.318730] [<ffffffff814133fd>] wait_for_completion+0x1d/0x20 [15601.319745] [<ffffffffa050f0bb>] write_dev_flush+0x4b/0x140 [btrfs] [15601.320760] [<ffffffffa0511086>] write_all_supers+0x6f6/0x800=20 [btrfs] [15601.321791] [<ffffffffa05111a3>] write_ctree_super+0x13/0x20 [btrfs= ] [15601.322809] [<ffffffffa05164dd>]=20 btrfs_commit_transaction+0x63d/0x880 [btrfs] [15601.323831] [<ffffffff81085750>] ? abort_exclusive_wait+0xb0/0xb0 [15601.324855] [<ffffffffa0516b74>] ? start_transaction+0x94/0x2b0=20 [btrfs] [15601.325894] [<ffffffffa050ed4d>] transaction_kthread+0x26d/0x290=20 [btrfs] [15601.326918] [<ffffffffa050eae0>] ? btrfs_congested_fn+0xd0/0xd0=20 [btrfs] [15601.327937] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15601.328963] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15601.329979] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15601.330995] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15601.332024] INFO: task flush-9:127:1891 blocked for more than 120=20 seconds. [15601.333049] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15601.334083] flush-9:127 D 00000000a01fc200 0 1891 2=20 0x00000000 [15601.335141] ffff880171771540 0000000000000046 0000000100000000=20 0000000300000001 [15601.336194] ffff8802009d9c80 ffff880171771fd8 ffff880171771fd8=20 ffff880171771fd8 [15601.337252] ffff8802009dc740 ffff8802009d9c80 ffff8801717714a0=20 ffffffff8104d003 [15601.338304] Call Trace: [15601.339347] [<ffffffff8104d003>] ? __wake_up+0x53/0x70 [15601.340376] [<ffffffff8114d48b>] ? kfree+0x3b/0x170 [15601.341398] [<ffffffffa01c732b>] ? plugger_unplug+0x3b/0x50 [md_mod= ] [15601.342432] [<ffffffff8121b457>] ? blk_flush_plug_list+0xa7/0x250 [15601.343453] [<ffffffff814140bf>] schedule+0x3f/0x60 [15601.344460] [<ffffffffa0378209>] get_active_stripe+0x2e9/0x600=20 [raid456] [15601.345487] [<ffffffffa01c6400>] ? md_wakeup_thread+0x10/0x30=20 [md_mod] [15601.346503] [<ffffffff8105be20>] ? try_to_wake_up+0x290/0x290 [15601.347519] [<ffffffffa037be22>] make_request+0x192/0x430 [raid456] [15601.348555] [<ffffffff81085750>] ? abort_exclusive_wait+0xb0/0xb0 [15601.348559] [<ffffffffa01c964a>] md_make_request+0xfa/0x240 [md_mod= ] [15601.348561] [<ffffffff81219917>] generic_make_request+0xc7/0x100 [15601.348563] [<ffffffff812199d7>] submit_bio+0x87/0x110 [15601.348565] [<ffffffff8118b300>] ? __mark_inode_dirty+0x40/0x230 [15601.348571] [<ffffffffa03a16f7>]=20 xfs_submit_ioend_bio.isra.11+0x57/0x80 [xfs] [15601.348578] [<ffffffffa03a180e>] xfs_submit_ioend+0xee/0x110 [xfs] [15601.348585] [<ffffffffa03a1a65>] xfs_vm_writepage+0x235/0x530 [xfs] [15601.348588] [<ffffffff8110b75a>] __writepage+0x1a/0x50 [15601.348591] [<ffffffff8110bc21>] write_cache_pages+0x1e1/0x4e0 [15601.348599] [<ffffffffa03bab27>] ? kmem_alloc+0x67/0xe0 [xfs] [15601.348602] [<ffffffff8110b740>] ? bdi_set_max_ratio+0x90/0x90 [15601.348606] [<ffffffff8110bf6d>] generic_writepages+0x4d/0x70 [15601.348612] [<ffffffffa03a06ad>] xfs_vm_writepages+0x4d/0x60 [xfs] [15601.348615] [<ffffffff8110d6a4>] do_writepages+0x24/0x40 [15601.348618] [<ffffffff8118b76f>] writeback_single_inode+0x10f/0x3f0 [15601.348621] [<ffffffff8118be4a>] writeback_sb_inodes+0x1aa/0x270 [15601.348624] [<ffffffff8118bfae>] __writeback_inodes_wb+0x9e/0xd0 [15601.348627] [<ffffffff8118c5cb>] wb_writeback+0x37b/0x3b0 [15601.348631] [<ffffffff8117d562>] ? get_nr_inodes+0x52/0x70 [15601.348634] [<ffffffff8117e262>] ? get_nr_dirty_inodes+0x52/0x80 [15601.348636] [<ffffffff8118c69f>] wb_check_old_data_flush+0x9f/0xb0 [15601.348639] [<ffffffff8118db7c>] wb_do_writeback+0x19c/0x230 [15601.348642] [<ffffffff81414643>] ? schedule_timeout+0x173/0x390 [15601.348645] [<ffffffff81071b10>] ?=20 init_timer_deferrable_key+0x30/0x30 [15601.348648] [<ffffffff8118dc9c>] bdi_writeback_thread+0x8c/0x340 [15601.348651] [<ffffffff8118dc10>] ? wb_do_writeback+0x230/0x230 [15601.348654] [<ffffffff81084e0c>] kthread+0x8c/0xa0 [15601.348657] [<ffffffff81419a34>] kernel_thread_helper+0x4/0x10 [15601.348660] [<ffffffff81084d80>] ? kthread_worker_fn+0x190/0x190 [15601.348663] [<ffffffff81419a30>] ? gs_change+0x13/0x13 [15601.348665] INFO: task smbd:2058 blocked for more than 120 seconds. [15601.348667] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [15601.348669] smbd D 000000010044b8d7 0 2058 823=20 0x00000004 [15601.348672] ffff88005cf37a08 0000000000000082 ffff88005cf37938=20 ffffffff81371188 [15601.348676] ffff8802012e63c0 ffff88005cf37fd8 ffff88005cf37fd8=20 ffff88005cf37fd8 [15601.348680] ffff880202d21c80 ffff8802012e63c0 ffff8802012769c0=20 0000000000000246 [15601.348684] Call Trace: [15601.348687] [<ffffffff81371188>] ? sch_direct_xmit+0x68/0x1d0 [15601.348691] [<ffffffff81355a00>] ? dev_queue_xmit+0x200/0x680 [15601.348694] [<ffffffff81389200>] ? ip_forward_options+0x1c0/0x1c0 [15601.348697] [<ffffffff8138adbe>] ? ip_finish_output+0x18e/0x310 [15601.348700] [<ffffffff8101d639>] ? read_tsc+0x9/0x20 [15601.348703] [<ffffffff8108f14d>] ? ktime_get_ts+0xad/0xe0 [15601.348705] [<ffffffff81101d60>] ? __lock_page+0x70/0x70 [15601.348708] [<ffffffff814140bf>] schedule+0x3f/0x60 [15601.348711] [<ffffffff8141416f>] io_schedule+0x8f/0xd0 [15601.348714] [<ffffffff81101d6e>] sleep_on_page+0xe/0x20 [15601.348716] [<ffffffff8141491f>] __wait_on_bit+0x5f/0x90 [15601.348719] [<ffffffff81101f58>] wait_on_page_bit+0x78/0x80 [15601.348722] [<ffffffff81085790>] ?=20 autoremove_wake_function+0x40/0x40 [15601.348725] [<ffffffff81102845>]=20 grab_cache_page_write_begin+0x95/0xe0 [15601.348732] [<ffffffffa03a1150>] ? xfs_get_blocks_direct+0x20/0x20=20 [xfs] [15601.348736] [<ffffffff811967b8>] block_write_begin+0x38/0xa0 [15601.348743] [<ffffffffa03a1213>] xfs_vm_write_begin+0x43/0x70 [xfs] [15601.348746] [<ffffffff8110233c>]=20 generic_file_buffered_write+0x10c/0x270 [15601.348754] [<ffffffffa03aad66>] ? xfs_iunlock+0x116/0x180 [xfs] [15601.348761] [<ffffffffa03a7fef>]=20 xfs_file_buffered_aio_write+0x10f/0x200 [xfs] [15601.348768] [<ffffffffa03a8252>] xfs_file_aio_write+0x172/0x2a0=20 [xfs] [15601.348772] [<ffffffff81162d62>] do_sync_write+0xd2/0x110 [15601.348775] [<ffffffff811f0fcc>] ?=20 security_file_permission+0x2c/0xb0 [15601.348778] [<ffffffff81163311>] ? rw_verify_area+0x61/0xf0 [15601.348781] [<ffffffff8116366f>] vfs_write+0xaf/0x180 [15601.348784] [<ffffffff81163b12>] sys_pwrite64+0x82/0xb0 [15601.348787] [<ffffffff814178c2>] system_call_fastpath+0x16/0x1b On =CE=A3=CE=AC=CE=B2=CE=B2=CE=B1=CF=84=CE=BF, 3 =CE=94=CE=B5=CE=BA=CE=AD= =CE=BC=CE=B2=CF=81=CE=B9=CE=BF=CF=82 2011 2:35:50 =CF=80=CE=BC, Konstan= tinos Skarlatos wrote: > After about 1TB of rsyncs from multiple servers at the same time, plu= s=20 > some heavy filesystem loading, i believe that 3.2rc4 solves the=20 > problem for me. Now if only we had deduplication and an fsck tool :) > On =CE=A0=CE=B1=CF=81=CE=B1=CF=83=CE=BA=CE=B5=CF=85=CE=AE, 2 =CE=94=CE= =B5=CE=BA=CE=AD=CE=BC=CE=B2=CF=81=CE=B9=CE=BF=CF=82 2011 9:53:10 =CE=BC= =CE=BC, Konstantinos Skarlatos wrote: >> I see they got into 3.2rc4, so I am now compiling it. I will report=20 >> back in a few hours >> >> On =CE=A0=CE=B1=CF=81=CE=B1=CF=83=CE=BA=CE=B5=CF=85=CE=AE, 2 =CE=94=CE= =B5=CE=BA=CE=AD=CE=BC=CE=B2=CF=81=CE=B9=CE=BF=CF=82 2011 5:48:31 =CE=BC= =CE=BC, Tobias wrote: >>> Am 02.12.2011 16:22, schrieb Konstantinos Skarlatos: >>>>>> So, the transaction close is in btrfs_evict_inode, which sounds=20 >>>>>> like a >>>>>> deadlock recently fixed by this commit: >>>>>> >>>>>> http://git.kernel.org/?p=3Dlinux/kernel/git/mason/linux-btrfs.gi= t;a=3Dcommit;h=3Daa38a711a893accf5b5192f3d705a120deaa81e0=20 >>>>>> >>>>>> >>>>>> If you pull the for-linus branch from today, hopefully the=20 >>>>>> problem will >>>>>> be gone. >>>>>> >>>>> >>>>> This looks very good. With this Kernel i still have some hangs,=20 >>>>> but only in rsync, only under high load and they don't lock up th= e=20 >>>>> system - so i guess it's ok now. >>>> >>>> I still have hangs and lock ups under the same situation (rsync of= =20 >>>> many files) under 3.2rc3. rc3 made the hang appear after 200gb of=20 >>>> files, while in rc2 i had hangs after only 11gb . >>> >>> Yes, i had them too in 3.2rc3! The problem where solved with patche= s=20 >>> from the "btrfs-for-linus" -branch. (see link above). >>> >>> Tobias >>> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-03 14:36 ` Konstantinos Skarlatos @ 2011-12-05 13:12 ` Chris Mason 2011-12-25 1:06 ` Konstantinos Skarlatos 0 siblings, 1 reply; 15+ messages in thread From: Chris Mason @ 2011-12-05 13:12 UTC (permalink / raw) To: Konstantinos Skarlatos; +Cc: Tobias, linux-btrfs On Sat, Dec 03, 2011 at 04:36:44PM +0200, Konstantinos Skarlatos wrote: > unfortunately i was wrong. rc4 does not fix this issue for me when > rsyncing large amounts of data... > > my mount options: > mount -o loop,compress=zlib,compress-force btrfs_test /storage/btrfs > the filesystem is a file on a raid5 xfs volume. Oh, the loop + raid5 + xfs is going to cause problems. The loop driver is fine for testing but I wouldn't be using it in a production environment. -chris ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Blocked for more than 120 seconds 2011-12-05 13:12 ` Chris Mason @ 2011-12-25 1:06 ` Konstantinos Skarlatos 0 siblings, 0 replies; 15+ messages in thread From: Konstantinos Skarlatos @ 2011-12-25 1:06 UTC (permalink / raw) To: Chris Mason; +Cc: Tobias, linux-btrfs I got the same message on a new btrfs filesystem on a raid5 volume=20 (without loop this time). kernel is 3.2rc6 this happened after a day of rsyncing at about 10mbyte/sec, only way to= =20 reboot is with echo b > /proc/sysrq-trigger [181201.285259] INFO: task btrfs-submit-1:768 blocked for more than 120= =20 seconds. [181201.285297] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181201.285332] btrfs-submit-1 D 0000000021102920 0 768 2=20 0x00000000 [181201.285368] ffff88006d1b7aa0 0000000000000046 0000000100000000=20 0000000300000001 [181201.285406] ffff880072c64020 ffff88006d1b7fd8 ffff88006d1b7fd8=20 ffff88006d1b7fd8 [181201.285443] ffff88006d900000 ffff880072c64020 ffff88006d1b7a00=20 ffffffff8104e053 [181201.285481] Call Trace: [181201.285498] [<ffffffff8104e053>] ? __wake_up+0x53/0x70 [181201.285524] [<ffffffff8114e95b>] ? kfree+0x3b/0x170 [181201.285552] [<ffffffffa053535b>] ? plugger_unplug+0x3b/0x50=20 [md_mod] [181201.285583] [<ffffffff8121ca57>] ? blk_flush_plug_list+0xa7/0x250 [181201.285612] [<ffffffff814158bf>] schedule+0x3f/0x60 [181201.285637] [<ffffffffa058b209>] get_active_stripe+0x2e9/0x600=20 [raid456] [181201.285669] [<ffffffffa0534400>] ? max_sync_store+0xc0/0xe0=20 [md_mod] [181201.285700] [<ffffffff8105cf10>] ? try_to_wake_up+0x290/0x290 [181201.285727] [<ffffffffa058ee22>] make_request+0x192/0x430 [raid456= ] [181201.285758] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181201.285787] [<ffffffffa053752a>] md_make_request+0xfa/0x240=20 [md_mod] [181201.285816] [<ffffffff8114e95b>] ? kfree+0x3b/0x170 [181201.285840] [<ffffffff8121af47>] generic_make_request+0xc7/0x100 [181201.285868] [<ffffffff8121b007>] submit_bio+0x87/0x110 [181201.285900] [<ffffffffa074dcfb>] run_scheduled_bios+0x26b/0x590=20 [btrfs] [181201.285936] [<ffffffffa074e035>] pending_bios_fn+0x15/0x20 [btrfs] [181201.285969] [<ffffffffa0754870>] worker_loop+0x150/0x520 [btrfs] [181201.285997] [<ffffffff8104c568>] ? __wake_up_common+0x58/0x90 [181201.286029] [<ffffffffa0754720>] ? btrfs_queue_worker+0x300/0x300=20 [btrfs] [181201.286060] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181201.286084] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181201.286111] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181201.286139] [<ffffffff8141b230>] ? gs_change+0x13/0x13 [181201.286164] INFO: task btrfs-transacti:780 blocked for more than=20 120 seconds. [181201.286195] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181201.286230] btrfs-transacti D 00000001033b48f7 0 780 2=20 0x00000000 [181201.286264] ffff88006d1e9430 0000000000000046 ffff880000000000=20 ffffffff810146ef [181201.286301] ffff88006d905ca0 ffff88006d1e9fd8 ffff88006d1e9fd8=20 ffff88006d1e9fd8 [181201.286338] ffff880073e97200 ffff88006d905ca0 ffff88006d1e9380=20 ffffffff8108cb65 [181201.287224] Call Trace: [181201.288093] [<ffffffff810146ef>] ? __switch_to+0xbf/0x2f0 [181201.288986] [<ffffffff8108cb65>] ? sched_clock_local+0x25/0x90 [181201.289867] [<ffffffffa0726640>] ? btrfs_buffer_uptodate+0x50/0x70= =20 [btrfs] [181201.290744] [<ffffffffa070c48c>] ? comp_keys+0x2c/0x30 [btrfs] [181201.291613] [<ffffffff814158bf>] schedule+0x3f/0x60 [181201.292470] [<ffffffffa058b209>] get_active_stripe+0x2e9/0x600=20 [raid456] [181201.293317] [<ffffffff8105cf10>] ? try_to_wake_up+0x290/0x290 [181201.294170] [<ffffffffa058ee22>] make_request+0x192/0x430 [raid456= ] [181201.295030] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181201.295897] [<ffffffffa053752a>] md_make_request+0xfa/0x240=20 [md_mod] [181201.296764] [<ffffffffa074e14e>] ? __btrfs_map_block+0x10e/0x840=20 [btrfs] [181201.297620] [<ffffffff8121af47>] generic_make_request+0xc7/0x100 [181201.298463] [<ffffffff8121b007>] submit_bio+0x87/0x110 [181201.299303] [<ffffffffa0751ebe>] btrfs_map_bio+0x13e/0x1d0 [btrfs] [181201.300137] [<ffffffffa072d799>] btrfs_submit_bio_hook+0xa9/0x170=20 [btrfs] [181201.300966] [<ffffffff81199eb1>] ?=20 __bio_add_page.part.14+0x101/0x240 [181201.301803] [<ffffffffa0745c9a>] submit_one_bio+0x6a/0xa0 [btrfs] [181201.302644] [<ffffffffa07492fd>]=20 submit_extent_page.isra.28+0xdd/0x200 [btrfs] [181201.302651] [<ffffffffa074a0ac>] __extent_writepage+0x52c/0x740=20 [btrfs] [181201.302658] [<ffffffffa0746e80>] ? clear_extent_bit+0x3d0/0x3d0=20 [btrfs] [181201.302666] [<ffffffffa074a58a>]=20 extent_write_cache_pages.isra.22.constprop.30+0x2ca/0x3f0 [btrfs] [181201.302674] [<ffffffffa074a8f5>] extent_writepages+0x45/0x60=20 [btrfs] [181201.302681] [<ffffffffa072f780>] ?=20 uncompress_inline.isra.47+0x1d0/0x1d0 [btrfs] [181201.302687] [<ffffffffa0746c7a>] ? clear_extent_bit+0x1ca/0x3d0=20 [btrfs] [181201.302694] [<ffffffffa072ea68>] btrfs_writepages+0x28/0x30 [btrfs= ] [181201.302697] [<ffffffff8110eb64>] do_writepages+0x24/0x40 [181201.302700] [<ffffffff811043cb>]=20 __filemap_fdatawrite_range+0x5b/0x60 [181201.302702] [<ffffffff811043ef>] filemap_fdatawrite+0x1f/0x30 [181201.302704] [<ffffffff81104435>] filemap_write_and_wait+0x35/0x60 [181201.302711] [<ffffffffa07616c7>]=20 __btrfs_write_out_cache+0x807/0xa80 [btrfs] [181201.302717] [<ffffffffa0767008>] ?=20 btrfs_find_ref_cluster+0x68/0x190 [btrfs] [181201.302723] [<ffffffffa07619d5>] btrfs_write_out_cache+0x95/0xf0=20 [btrfs] [181201.302729] [<ffffffffa071abe1>]=20 btrfs_write_dirty_block_groups+0x481/0x580 [btrfs] [181201.302736] [<ffffffffa0729c05>] commit_cowonly_roots+0x115/0x1e0=20 [btrfs] [181201.302742] [<ffffffffa072b3ec>]=20 btrfs_commit_transaction+0x3fc/0x880 [btrfs] [181201.302745] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181201.302751] [<ffffffffa072bcc4>] ? start_transaction+0x94/0x2b0=20 [btrfs] [181201.302758] [<ffffffffa0723e6d>] transaction_kthread+0x26d/0x290=20 [btrfs] [181201.302764] [<ffffffffa0723c00>] ? btrfs_congested_fn+0xd0/0xd0=20 [btrfs] [181201.302766] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181201.302768] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181201.302770] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181201.302772] [<ffffffff8141b230>] ? gs_change+0x13/0x13 [181201.302775] INFO: task rsync:30933 blocked for more than 120=20 seconds. [181201.302776] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181201.302777] rsync D 00000001033b48f2 0 30933 30932=20 0x00000000 [181201.302779] ffff880072ce5b98 0000000000000082 0000000000000000=20 0000000000000001 [181201.302781] ffff88006d9023a0 ffff880072ce5fd8 ffff880072ce5fd8=20 ffff880072ce5fd8 [181201.302783] ffffffff8180d020 ffff88006d9023a0 ffffffff81177b50=20 00000000000000db [181201.302785] Call Trace: [181201.302788] [<ffffffff81177b50>] ? poll_freewait+0xe0/0xe0 [181201.302790] [<ffffffff81177c40>] ? __pollwait+0xf0/0xf0 [181201.302792] [<ffffffff81051970>] ? select_task_rq_fair+0x540/0xa60 [181201.302794] [<ffffffff814158bf>] schedule+0x3f/0x60 [181201.302800] [<ffffffffa072a705>]=20 wait_current_trans.isra.22+0xa5/0xf0 [btrfs] [181201.302803] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181201.302809] [<ffffffffa072be20>] start_transaction+0x1f0/0x2b0=20 [btrfs] [181201.302816] [<ffffffffa072bf35>] btrfs_join_transaction+0x15/0x20=20 [btrfs] [181201.302823] [<ffffffffa0736980>] btrfs_dirty_inode+0x50/0xd0=20 [btrfs] [181201.302830] [<ffffffffa0736c86>] btrfs_update_time+0xd6/0x170=20 [btrfs] [181201.302837] [<ffffffffa073aee8>] btrfs_file_aio_write+0x1c8/0x520=20 [btrfs] [181201.302840] [<ffffffff8116e8f2>] ? pipe_read+0x2d2/0x4e0 [181201.302842] [<ffffffff81164302>] do_sync_write+0xd2/0x110 [181201.302844] [<ffffffff81051019>] ? finish_task_switch+0x49/0xd0 [181201.302848] [<ffffffff811f266c>] ?=20 security_file_permission+0x2c/0xb0 [181201.302849] [<ffffffff811648b1>] ? rw_verify_area+0x61/0xf0 [181201.302851] [<ffffffff81164c0f>] vfs_write+0xaf/0x180 [181201.302853] [<ffffffff81164f3a>] sys_write+0x4a/0x90 [181201.302856] [<ffffffff814190c2>] system_call_fastpath+0x16/0x1b [181201.302858] INFO: task flush-btrfs-1:30940 blocked for more than=20 120 seconds. [181201.302859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181201.302860] flush-btrfs-1 D 00000001033b5a85 0 30940 2=20 0x00000000 [181201.302862] ffff8800236218d0 0000000000000046 ffff880000000000=20 ffffffffa075460e [181201.302864] ffff88007222b1e0 ffff880023621fd8 ffff880023621fd8=20 ffff880023621fd8 [181201.302866] ffff880073e97200 ffff88007222b1e0 ffff88000dad1550=20 ffff880023621870 [181201.302868] Call Trace: [181201.302874] [<ffffffffa075460e>] ? btrfs_queue_worker+0x1ee/0x300=20 [btrfs] [181201.302881] [<ffffffffa07330b5>] ? run_delalloc_range+0x185/0x360=20 [btrfs] [181201.302883] [<ffffffff8101d639>] ? read_tsc+0x9/0x20 [181201.302885] [<ffffffff81103140>] ? __lock_page+0x70/0x70 [181201.302887] [<ffffffff814158bf>] schedule+0x3f/0x60 [181201.302889] [<ffffffff8141596f>] io_schedule+0x8f/0xd0 [181201.302891] [<ffffffff8110314e>] sleep_on_page+0xe/0x20 [181201.302892] [<ffffffff8141623a>] __wait_on_bit_lock+0x5a/0xc0 [181201.302894] [<ffffffff81103137>] __lock_page+0x67/0x70 [181201.302897] [<ffffffff81086890>] ?=20 autoremove_wake_function+0x40/0x40 [181201.302903] [<ffffffffa074a4d5>]=20 extent_write_cache_pages.isra.22.constprop.30+0x215/0x3f0 [btrfs] [181201.302911] [<ffffffffa074a8f5>] extent_writepages+0x45/0x60=20 [btrfs] [181201.302917] [<ffffffffa072f780>] ?=20 uncompress_inline.isra.47+0x1d0/0x1d0 [btrfs] [181201.302924] [<ffffffffa072ea68>] btrfs_writepages+0x28/0x30 [btrfs= ] [181201.302926] [<ffffffff8110eb64>] do_writepages+0x24/0x40 [181201.302928] [<ffffffff8118ce0f>] writeback_single_inode+0x10f/0x3f= 0 [181201.302930] [<ffffffff8118d4ea>] writeback_sb_inodes+0x1aa/0x270 [181201.302932] [<ffffffff8118d64e>] __writeback_inodes_wb+0x9e/0xd0 [181201.302934] [<ffffffff8118dc6b>] wb_writeback+0x37b/0x3b0 [181201.302937] [<ffffffff8117ebd2>] ? get_nr_inodes+0x52/0x70 [181201.302939] [<ffffffff8117f8d2>] ? get_nr_dirty_inodes+0x52/0x80 [181201.302941] [<ffffffff8118dd3f>] wb_check_old_data_flush+0x9f/0xb0 [181201.302943] [<ffffffff8118f21c>] wb_do_writeback+0x19c/0x230 [181201.302944] [<ffffffff81415e43>] ? schedule_timeout+0x173/0x390 [181201.302947] [<ffffffff81072c10>] ?=20 init_timer_deferrable_key+0x30/0x30 [181201.302949] [<ffffffff8118f33c>] bdi_writeback_thread+0x8c/0x340 [181201.302951] [<ffffffff8118f2b0>] ? wb_do_writeback+0x230/0x230 [181201.302953] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181201.302955] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181201.302957] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181201.302959] [<ffffffff8141b230>] ? gs_change+0x13/0x13 [181201.302960] INFO: task btrfs-delalloc-:30957 blocked for more than=20 120 seconds. [181201.302962] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181201.302963] btrfs-delalloc- D 00000001033b5a86 0 30957 2=20 0x00000000 [181201.302965] ffff88005147fc50 0000000000000046 0080004000000000=20 0800040002000100 [181201.302967] ffff880072ab8e40 ffff88005147ffd8 ffff88005147ffd8=20 ffff88005147ffd8 [181201.302969] ffff880073f09c80 ffff880072ab8e40 0000000000000001=20 0000000000000000 [181201.302970] Call Trace: [181201.302973] [<ffffffffa06fea84>] ? zlib_tr_flush_block+0x5f4/0x6f0= =20 [zlib_deflate] [181201.302975] [<ffffffffa06fb6cc>] ? flush_pending+0x4c/0x90=20 [zlib_deflate] [181201.302978] [<ffffffffa06fb6cc>] ? flush_pending+0x4c/0x90=20 [zlib_deflate] [181201.302980] [<ffffffffa06fc506>] ? zlib_deflate+0x1b6/0x410=20 [zlib_deflate] [181201.302982] [<ffffffff814158bf>] schedule+0x3f/0x60 [181201.302988] [<ffffffffa072a705>]=20 wait_current_trans.isra.22+0xa5/0xf0 [btrfs] [181201.302990] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181201.302997] [<ffffffffa072be20>] start_transaction+0x1f0/0x2b0=20 [btrfs] [181201.303003] [<ffffffffa072bf35>] btrfs_join_transaction+0x15/0x20=20 [btrfs] [181201.303010] [<ffffffffa0733a55>] compress_file_range+0x2c5/0x660=20 [btrfs] [181201.303017] [<ffffffffa0733e25>] async_cow_start+0x35/0x50 [btrfs] [181201.303023] [<ffffffffa0754870>] worker_loop+0x150/0x520 [btrfs] [181201.303029] [<ffffffffa0754720>] ? btrfs_queue_worker+0x300/0x300=20 [btrfs] [181201.303032] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181201.303033] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181201.303036] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181201.303037] [<ffffffff8141b230>] ? gs_change+0x13/0x13 [181321.301930] INFO: task btrfs-submit-1:768 blocked for more than 120= =20 seconds. [181321.302761] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181321.303594] btrfs-submit-1 D 0000000021102920 0 768 2=20 0x00000000 [181321.304428] ffff88006d1b7aa0 0000000000000046 0000000100000000=20 0000000300000001 [181321.305272] ffff880072c64020 ffff88006d1b7fd8 ffff88006d1b7fd8=20 ffff88006d1b7fd8 [181321.306110] ffff88006d900000 ffff880072c64020 ffff88006d1b7a00=20 ffffffff8104e053 [181321.306940] Call Trace: [181321.307759] [<ffffffff8104e053>] ? __wake_up+0x53/0x70 [181321.308595] [<ffffffff8114e95b>] ? kfree+0x3b/0x170 [181321.309428] [<ffffffffa053535b>] ? plugger_unplug+0x3b/0x50=20 [md_mod] [181321.310266] [<ffffffff8121ca57>] ? blk_flush_plug_list+0xa7/0x250 [181321.311107] [<ffffffff814158bf>] schedule+0x3f/0x60 [181321.311951] [<ffffffffa058b209>] get_active_stripe+0x2e9/0x600=20 [raid456] [181321.312805] [<ffffffffa0534400>] ? max_sync_store+0xc0/0xe0=20 [md_mod] [181321.313657] [<ffffffff8105cf10>] ? try_to_wake_up+0x290/0x290 [181321.314510] [<ffffffffa058ee22>] make_request+0x192/0x430 [raid456= ] [181321.315376] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181321.316234] [<ffffffffa053752a>] md_make_request+0xfa/0x240=20 [md_mod] [181321.317094] [<ffffffff8114e95b>] ? kfree+0x3b/0x170 [181321.317952] [<ffffffff8121af47>] generic_make_request+0xc7/0x100 [181321.318821] [<ffffffff8121b007>] submit_bio+0x87/0x110 [181321.319690] [<ffffffffa074dcfb>] run_scheduled_bios+0x26b/0x590=20 [btrfs] [181321.320564] [<ffffffffa074e035>] pending_bios_fn+0x15/0x20 [btrfs] [181321.321441] [<ffffffffa0754870>] worker_loop+0x150/0x520 [btrfs] [181321.322311] [<ffffffff8104c568>] ? __wake_up_common+0x58/0x90 [181321.323167] [<ffffffffa0754720>] ? btrfs_queue_worker+0x300/0x300=20 [btrfs] [181321.324015] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181321.324856] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181321.325700] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181321.326537] [<ffffffff8141b230>] ? gs_change+0x13/0x13 [181321.327360] INFO: task btrfs-transacti:780 blocked for more than=20 120 seconds. [181321.328189] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181321.329035] btrfs-transacti D 00000001033b48f7 0 780 2=20 0x00000000 [181321.329900] ffff88006d1e9430 0000000000000046 ffff880000000000=20 ffffffff810146ef [181321.330774] ffff88006d905ca0 ffff88006d1e9fd8 ffff88006d1e9fd8=20 ffff88006d1e9fd8 [181321.331646] ffff880073e97200 ffff88006d905ca0 ffff88006d1e9380=20 ffffffff8108cb65 [181321.332527] Call Trace: [181321.333375] [<ffffffff810146ef>] ? __switch_to+0xbf/0x2f0 [181321.334238] [<ffffffff8108cb65>] ? sched_clock_local+0x25/0x90 [181321.335104] [<ffffffffa0726640>] ? btrfs_buffer_uptodate+0x50/0x70= =20 [btrfs] [181321.335980] [<ffffffffa070c48c>] ? comp_keys+0x2c/0x30 [btrfs] [181321.336843] [<ffffffff814158bf>] schedule+0x3f/0x60 [181321.337704] [<ffffffffa058b209>] get_active_stripe+0x2e9/0x600=20 [raid456] [181321.338581] [<ffffffff8105cf10>] ? try_to_wake_up+0x290/0x290 [181321.339447] [<ffffffffa058ee22>] make_request+0x192/0x430 [raid456= ] [181321.340320] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181321.341194] [<ffffffffa053752a>] md_make_request+0xfa/0x240=20 [md_mod] [181321.342084] [<ffffffffa074e14e>] ? __btrfs_map_block+0x10e/0x840=20 [btrfs] [181321.342958] [<ffffffff8121af47>] generic_make_request+0xc7/0x100 [181321.343818] [<ffffffff8121b007>] submit_bio+0x87/0x110 [181321.344668] [<ffffffffa0751ebe>] btrfs_map_bio+0x13e/0x1d0 [btrfs] [181321.345524] [<ffffffffa072d799>] btrfs_submit_bio_hook+0xa9/0x170=20 [btrfs] [181321.346365] [<ffffffff81199eb1>] ?=20 __bio_add_page.part.14+0x101/0x240 [181321.347216] [<ffffffffa0745c9a>] submit_one_bio+0x6a/0xa0 [btrfs] [181321.348059] [<ffffffffa07492fd>]=20 submit_extent_page.isra.28+0xdd/0x200 [btrfs] [181321.348912] [<ffffffffa074a0ac>] __extent_writepage+0x52c/0x740=20 [btrfs] [181321.349763] [<ffffffffa0746e80>] ? clear_extent_bit+0x3d0/0x3d0=20 [btrfs] [181321.350613] [<ffffffffa074a58a>]=20 extent_write_cache_pages.isra.22.constprop.30+0x2ca/0x3f0 [btrfs] [181321.351480] [<ffffffffa074a8f5>] extent_writepages+0x45/0x60=20 [btrfs] [181321.352351] [<ffffffffa072f780>] ?=20 uncompress_inline.isra.47+0x1d0/0x1d0 [btrfs] [181321.353220] [<ffffffffa0746c7a>] ? clear_extent_bit+0x1ca/0x3d0=20 [btrfs] [181321.354089] [<ffffffffa072ea68>] btrfs_writepages+0x28/0x30 [btrfs= ] [181321.354955] [<ffffffff8110eb64>] do_writepages+0x24/0x40 [181321.355823] [<ffffffff811043cb>]=20 __filemap_fdatawrite_range+0x5b/0x60 [181321.356688] [<ffffffff811043ef>] filemap_fdatawrite+0x1f/0x30 [181321.357546] [<ffffffff81104435>] filemap_write_and_wait+0x35/0x60 [181321.358408] [<ffffffffa07616c7>]=20 __btrfs_write_out_cache+0x807/0xa80 [btrfs] [181321.359281] [<ffffffffa0767008>] ?=20 btrfs_find_ref_cluster+0x68/0x190 [btrfs] [181321.360147] [<ffffffffa07619d5>] btrfs_write_out_cache+0x95/0xf0=20 [btrfs] [181321.361012] [<ffffffffa071abe1>]=20 btrfs_write_dirty_block_groups+0x481/0x580 [btrfs] [181321.361888] [<ffffffffa0729c05>] commit_cowonly_roots+0x115/0x1e0=20 [btrfs] [181321.362759] [<ffffffffa072b3ec>]=20 btrfs_commit_transaction+0x3fc/0x880 [btrfs] [181321.363632] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181321.364508] [<ffffffffa072bcc4>] ? start_transaction+0x94/0x2b0=20 [btrfs] [181321.365388] [<ffffffffa0723e6d>] transaction_kthread+0x26d/0x290=20 [btrfs] [181321.366265] [<ffffffffa0723c00>] ? btrfs_congested_fn+0xd0/0xd0=20 [btrfs] [181321.367138] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181321.368005] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181321.368875] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181321.369726] [<ffffffff8141b230>] ? gs_change+0x13/0x13 [181321.370560] INFO: task rsync:30933 blocked for more than 120=20 seconds. [181321.371399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181321.372258] rsync D 00000001033b48f2 0 30933 30932=20 0x00000000 [181321.373125] ffff880072ce5b98 0000000000000082 0000000000000000=20 0000000000000001 [181321.373990] ffff88006d9023a0 ffff880072ce5fd8 ffff880072ce5fd8=20 ffff880072ce5fd8 [181321.374857] ffffffff8180d020 ffff88006d9023a0 ffffffff81177b50=20 00000000000000db [181321.375735] Call Trace: [181321.376591] [<ffffffff81177b50>] ? poll_freewait+0xe0/0xe0 [181321.377461] [<ffffffff81177c40>] ? __pollwait+0xf0/0xf0 [181321.378325] [<ffffffff81051970>] ? select_task_rq_fair+0x540/0xa60 [181321.379197] [<ffffffff814158bf>] schedule+0x3f/0x60 [181321.380058] [<ffffffffa072a705>]=20 wait_current_trans.isra.22+0xa5/0xf0 [btrfs] [181321.380929] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181321.381802] [<ffffffffa072be20>] start_transaction+0x1f0/0x2b0=20 [btrfs] [181321.382684] [<ffffffffa072bf35>] btrfs_join_transaction+0x15/0x20=20 [btrfs] [181321.383561] [<ffffffffa0736980>] btrfs_dirty_inode+0x50/0xd0=20 [btrfs] [181321.384434] [<ffffffffa0736c86>] btrfs_update_time+0xd6/0x170=20 [btrfs] [181321.385311] [<ffffffffa073aee8>] btrfs_file_aio_write+0x1c8/0x520=20 [btrfs] [181321.386179] [<ffffffff8116e8f2>] ? pipe_read+0x2d2/0x4e0 [181321.387043] [<ffffffff81164302>] do_sync_write+0xd2/0x110 [181321.387899] [<ffffffff81051019>] ? finish_task_switch+0x49/0xd0 [181321.388763] [<ffffffff811f266c>] ?=20 security_file_permission+0x2c/0xb0 [181321.389623] [<ffffffff811648b1>] ? rw_verify_area+0x61/0xf0 [181321.390475] [<ffffffff81164c0f>] vfs_write+0xaf/0x180 [181321.391324] [<ffffffff81164f3a>] sys_write+0x4a/0x90 [181321.392163] [<ffffffff814190c2>] system_call_fastpath+0x16/0x1b [181321.392993] INFO: task flush-btrfs-1:30940 blocked for more than=20 120 seconds. [181321.393826] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181321.394670] flush-btrfs-1 D 00000001033b5a85 0 30940 2=20 0x00000000 [181321.395531] ffff8800236218d0 0000000000000046 ffff880000000000=20 ffffffffa075460e [181321.396394] ffff88007222b1e0 ffff880023621fd8 ffff880023621fd8=20 ffff880023621fd8 [181321.397258] ffff880073e97200 ffff88007222b1e0 ffff88000dad1550=20 ffff880023621870 [181321.398118] Call Trace: [181321.398977] [<ffffffffa075460e>] ? btrfs_queue_worker+0x1ee/0x300=20 [btrfs] [181321.399856] [<ffffffffa07330b5>] ? run_delalloc_range+0x185/0x360=20 [btrfs] [181321.400728] [<ffffffff8101d639>] ? read_tsc+0x9/0x20 [181321.401593] [<ffffffff81103140>] ? __lock_page+0x70/0x70 [181321.402460] [<ffffffff814158bf>] schedule+0x3f/0x60 [181321.403310] [<ffffffff8141596f>] io_schedule+0x8f/0xd0 [181321.404153] [<ffffffff8110314e>] sleep_on_page+0xe/0x20 [181321.404988] [<ffffffff8141623a>] __wait_on_bit_lock+0x5a/0xc0 [181321.405829] [<ffffffff81103137>] __lock_page+0x67/0x70 [181321.406660] [<ffffffff81086890>] ?=20 autoremove_wake_function+0x40/0x40 [181321.407500] [<ffffffffa074a4d5>]=20 extent_write_cache_pages.isra.22.constprop.30+0x215/0x3f0 [btrfs] [181321.408359] [<ffffffffa074a8f5>] extent_writepages+0x45/0x60=20 [btrfs] [181321.409220] [<ffffffffa072f780>] ?=20 uncompress_inline.isra.47+0x1d0/0x1d0 [btrfs] [181321.410085] [<ffffffffa072ea68>] btrfs_writepages+0x28/0x30 [btrfs= ] [181321.410946] [<ffffffff8110eb64>] do_writepages+0x24/0x40 [181321.411799] [<ffffffff8118ce0f>] writeback_single_inode+0x10f/0x3f= 0 [181321.412651] [<ffffffff8118d4ea>] writeback_sb_inodes+0x1aa/0x270 [181321.413480] [<ffffffff8118d64e>] __writeback_inodes_wb+0x9e/0xd0 [181321.414304] [<ffffffff8118dc6b>] wb_writeback+0x37b/0x3b0 [181321.415118] [<ffffffff8117ebd2>] ? get_nr_inodes+0x52/0x70 [181321.415934] [<ffffffff8117f8d2>] ? get_nr_dirty_inodes+0x52/0x80 [181321.416731] [<ffffffff8118dd3f>] wb_check_old_data_flush+0x9f/0xb0 [181321.417525] [<ffffffff8118f21c>] wb_do_writeback+0x19c/0x230 [181321.418322] [<ffffffff81415e43>] ? schedule_timeout+0x173/0x390 [181321.419127] [<ffffffff81072c10>] ?=20 init_timer_deferrable_key+0x30/0x30 [181321.419928] [<ffffffff8118f33c>] bdi_writeback_thread+0x8c/0x340 [181321.420724] [<ffffffff8118f2b0>] ? wb_do_writeback+0x230/0x230 [181321.421513] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181321.422300] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181321.423086] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181321.423868] [<ffffffff8141b230>] ? gs_change+0x13/0x13 [181321.424650] INFO: task btrfs-delalloc-:30957 blocked for more than=20 120 seconds. [181321.425455] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"=20 disables this message. [181321.426264] btrfs-delalloc- D 00000001033b5a86 0 30957 2=20 0x00000000 [181321.427088] ffff88005147fc50 0000000000000046 0080004000000000=20 0800040002000100 [181321.427922] ffff880072ab8e40 ffff88005147ffd8 ffff88005147ffd8=20 ffff88005147ffd8 [181321.428764] ffff880073f09c80 ffff880072ab8e40 0000000000000001=20 0000000000000000 [181321.429604] Call Trace: [181321.430433] [<ffffffffa06fea84>] ? zlib_tr_flush_block+0x5f4/0x6f0= =20 [zlib_deflate] [181321.431299] [<ffffffffa06fb6cc>] ? flush_pending+0x4c/0x90=20 [zlib_deflate] [181321.432176] [<ffffffffa06fb6cc>] ? flush_pending+0x4c/0x90=20 [zlib_deflate] [181321.433026] [<ffffffffa06fc506>] ? zlib_deflate+0x1b6/0x410=20 [zlib_deflate] [181321.433865] [<ffffffff814158bf>] schedule+0x3f/0x60 [181321.434692] [<ffffffffa072a705>]=20 wait_current_trans.isra.22+0xa5/0xf0 [btrfs] [181321.435529] [<ffffffff81086850>] ? abort_exclusive_wait+0xb0/0xb0 [181321.436362] [<ffffffffa072be20>] start_transaction+0x1f0/0x2b0=20 [btrfs] [181321.437204] [<ffffffffa072bf35>] btrfs_join_transaction+0x15/0x20=20 [btrfs] [181321.438039] [<ffffffffa0733a55>] compress_file_range+0x2c5/0x660=20 [btrfs] [181321.438876] [<ffffffffa0733e25>] async_cow_start+0x35/0x50 [btrfs] [181321.439707] [<ffffffffa0754870>] worker_loop+0x150/0x520 [btrfs] [181321.440535] [<ffffffffa0754720>] ? btrfs_queue_worker+0x300/0x300=20 [btrfs] [181321.441366] [<ffffffff81085f0c>] kthread+0x8c/0xa0 [181321.442202] [<ffffffff8141b234>] kernel_thread_helper+0x4/0x10 [181321.443039] [<ffffffff81085e80>] ? kthread_worker_fn+0x190/0x190 [181321.443880] [<ffffffff8141b230>] ? gs_change+0x13/0x13 On =CE=94=CE=B5=CF=85=CF=84=CE=AD=CF=81=CE=B1, 5 =CE=94=CE=B5=CE=BA=CE=AD= =CE=BC=CE=B2=CF=81=CE=B9=CE=BF=CF=82 2011 3:12:52 =CE=BC=CE=BC, Chris M= ason wrote: > On Sat, Dec 03, 2011 at 04:36:44PM +0200, Konstantinos Skarlatos wrot= e: >> unfortunately i was wrong. rc4 does not fix this issue for me when >> rsyncing large amounts of data... >> >> my mount options: >> mount -o loop,compress=3Dzlib,compress-force btrfs_test /storage/btr= fs >> the filesystem is a file on a raid5 xfs volume. > > Oh, the loop + raid5 + xfs is going to cause problems. The loop driv= er > is fine for testing but I wouldn't be using it in a production > environment. > > -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2011-12-25 1:06 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-11-28 8:16 Blcoked for more than 120 seconds Tobias
2011-11-28 9:29 ` Chris Samuel
2011-11-30 9:44 ` Blocked " Tobias
2011-11-30 14:10 ` Chris Mason
[not found] ` <4ED738E5.3080200@robotech.de>
2011-12-01 18:41 ` Chris Mason
2011-12-02 13:46 ` Tobias
2011-12-02 14:01 ` Chris Mason
2011-12-04 12:11 ` Konstantinos Skarlatos
2011-12-02 15:22 ` Konstantinos Skarlatos
2011-12-02 15:48 ` Tobias
2011-12-02 19:53 ` Konstantinos Skarlatos
2011-12-03 0:35 ` Konstantinos Skarlatos
2011-12-03 14:36 ` Konstantinos Skarlatos
2011-12-05 13:12 ` Chris Mason
2011-12-25 1:06 ` Konstantinos Skarlatos
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).