* kswapd hung tasks in 2.6.38-rc1 [not found] <2056664954.83361.1295513496286.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> @ 2011-01-20 8:55 ` CAI Qian 0 siblings, 0 replies; 14+ messages in thread From: CAI Qian @ 2011-01-20 8:55 UTC (permalink / raw) To: linux-mm; +Cc: linux-kernel When running LTP oom01 [1] testing, the allocation process stopped processing right after starting to swap. # free -m total used free shared buffers cached Mem: 64307 64059 248 0 4 61 -/+ buffers/cache: 63993 314 Swap: 66139 754 65385 # ./oom01 oom01 0 TINFO : start testing overcommit_memory=2. oom01 0 TINFO : expected victim is 7276. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. <hung...> INFO: task kswapd0:274 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd0 D ffff88045dd07000 0 274 2 0x00000000 ffff88045dd0d810 0000000000000046 0000000000000000 ffff88002e366e40 0000000000014d40 ffff88045dd06a70 ffff88045dd07000 ffff88045dd0dfd8 ffff88045dd07008 0000000000014d40 ffff88045dd0c010 0000000000014d40 Call Trace: [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff8120f285>] get_request_wait+0xc5/0x190 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff81207db7>] ? elv_merge+0x1d7/0x210 [<ffffffff8120f3bb>] __make_request+0x6b/0x4c0 [<ffffffff8120d28a>] generic_make_request+0x2ea/0x5d0 [<ffffffff810423a6>] ? native_flush_tlb_others+0x76/0x90 [<ffffffff8120d5f6>] submit_bio+0x86/0x110 [<ffffffff811040b6>] ? test_set_page_writeback+0x106/0x190 [<ffffffff8112d0d3>] swap_writepage+0x83/0xd0 [<ffffffff81108f7e>] pageout+0x12e/0x310 [<ffffffff8110957a>] shrink_page_list+0x41a/0x5a0 [<ffffffff81109dc2>] shrink_inactive_list+0x172/0x460 [<ffffffff81102cfa>] ? determine_dirtyable_memory+0x1a/0x30 [<ffffffff8110a76b>] shrink_zone+0x36b/0x520 [<ffffffff810fe805>] ? zone_watermark_ok_safe+0xb5/0xd0 [<ffffffff8110bee9>] kswapd+0x969/0xc20 [<ffffffff8110b580>] ? kswapd+0x0/0xc20 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task kswapd2:276 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd2 D ffff88045dd13a70 0 276 2 0x00000000 ffff88045dd15810 0000000000000046 0000000000000000 ffff880c5fa63240 0000000000014d40 ffff88045dd134e0 ffff88045dd13a70 ffff88045dd15fd8 ffff88045dd13a78 0000000000014d40 ffff88045dd14010 0000000000014d40 Call Trace: [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff8120f285>] get_request_wait+0xc5/0x190 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff81207db7>] ? elv_merge+0x1d7/0x210 [<ffffffff8120f3bb>] __make_request+0x6b/0x4c0 [<ffffffff8120d28a>] generic_make_request+0x2ea/0x5d0 [<ffffffff810423a6>] ? native_flush_tlb_others+0x76/0x90 [<ffffffff8120d5f6>] submit_bio+0x86/0x110 [<ffffffff811040b6>] ? test_set_page_writeback+0x106/0x190 [<ffffffff8112d0d3>] swap_writepage+0x83/0xd0 [<ffffffff81108f7e>] pageout+0x12e/0x310 [<ffffffff8110957a>] shrink_page_list+0x41a/0x5a0 [<ffffffff81109dc2>] shrink_inactive_list+0x172/0x460 [<ffffffff81071cda>] ? del_timer_sync+0x3a/0x60 [<ffffffff8110a76b>] shrink_zone+0x36b/0x520 [<ffffffff810fe805>] ? zone_watermark_ok_safe+0xb5/0xd0 [<ffffffff8110bee9>] kswapd+0x969/0xc20 [<ffffffff8110b580>] ? kswapd+0x0/0xc20 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task kswapd3:277 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd3 D ffff88045dd13040 0 277 2 0x00000000 ffff88045dd19810 0000000000000046 0000000000000000 ffff880857ed9d40 0000000000014d40 ffff88045dd12ab0 ffff88045dd13040 ffff88045dd19fd8 ffff88045dd13048 0000000000014d40 ffff88045dd18010 0000000000014d40 Call Trace: [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff8120f285>] get_request_wait+0xc5/0x190 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff81207c27>] ? elv_merge+0x47/0x210 [<ffffffff8120f3bb>] __make_request+0x6b/0x4c0 [<ffffffff8120d28a>] generic_make_request+0x2ea/0x5d0 [<ffffffff810423a6>] ? native_flush_tlb_others+0x76/0x90 [<ffffffff8120d5f6>] submit_bio+0x86/0x110 [<ffffffff811040b6>] ? test_set_page_writeback+0x106/0x190 [<ffffffff8112d0d3>] swap_writepage+0x83/0xd0 [<ffffffff81108f7e>] pageout+0x12e/0x310 [<ffffffff8110957a>] shrink_page_list+0x41a/0x5a0 [<ffffffff81109dc2>] shrink_inactive_list+0x172/0x460 [<ffffffff81071cda>] ? del_timer_sync+0x3a/0x60 [<ffffffff8110a76b>] shrink_zone+0x36b/0x520 [<ffffffff810fe805>] ? zone_watermark_ok_safe+0xb5/0xd0 [<ffffffff8110bee9>] kswapd+0x969/0xc20 [<ffffffff8110b580>] ? kswapd+0x0/0xc20 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task jbd2/dm-0-8:1002 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/dm-0-8 D ffff88045d33d080 0 1002 2 0x00000000 ffff88045d53bc20 0000000000000046 ffff88085f044900 ffff88085f0da4f8 0000000000014d40 ffff88045d33caf0 ffff88045d33d080 ffff88045d53bfd8 ffff88045d33d088 0000000000014d40 ffff88045d53a010 0000000000014d40 Call Trace: [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff811809d0>] sync_buffer+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814aed88>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff8118098e>] __wait_on_buffer+0x2e/0x30 [<ffffffffa0082fd8>] jbd2_journal_commit_transaction+0x8e8/0x13d0 [jbd2] [<ffffffff81071c41>] ? try_to_del_timer_sync+0x81/0xe0 [<ffffffffa00882b8>] kjournald2+0xb8/0x220 [jbd2] [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa0088200>] ? kjournald2+0x0/0x220 [jbd2] [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task flush-253:0:2030 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:0 D ffff88045d2a0610 0 2030 2 0x00000000 ffff88045e5ab4f0 0000000000000046 ffff88085f044900 ffff88085f0da4f8 0000000000014d40 ffff88045d2a0080 ffff88045d2a0610 ffff88045e5abfd8 ffff88045d2a0618 0000000000014d40 ffff88045e5aa010 0000000000014d40 Call Trace: [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff811809d0>] sync_buffer+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814aed88>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff8118098e>] __wait_on_buffer+0x2e/0x30 [<ffffffffa00d9275>] ext4_mb_init_cache+0x345/0x9f0 [ext4] [<ffffffffa00d9b28>] ext4_mb_init_group+0x208/0x390 [ext4] [<ffffffffa00d9d7e>] ext4_mb_good_group+0xce/0x110 [ext4] [<ffffffffa00dd5ab>] ext4_mb_regular_allocator+0x19b/0x410 [ext4] [<ffffffffa00ddbc5>] ext4_mb_new_blocks+0x3a5/0x510 [ext4] [<ffffffffa00d020b>] ? ext4_ext_find_extent+0x2bb/0x320 [ext4] [<ffffffffa00d2a1e>] ext4_ext_map_blocks+0x58e/0x1f40 [ext4] [<ffffffff8113edf6>] ? transfer_objects+0x56/0x80 [<ffffffff81104a25>] ? pagevec_lookup_tag+0x25/0x40 [<ffffffffa00b0ec4>] ext4_map_blocks+0xf4/0x210 [ext4] [<ffffffffa00b2379>] mpage_da_map_and_submit+0xb9/0x450 [ext4] [<ffffffffa0081393>] ? jbd2_journal_start+0x13/0x20 [jbd2] [<ffffffffa00c6c70>] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [<ffffffffa00b2fe9>] ext4_da_writepages+0x349/0x680 [ext4] [<ffffffff811026b0>] ? __writepage+0x0/0x40 [<ffffffff81103c01>] do_writepages+0x21/0x40 [<ffffffff81178238>] writeback_single_inode+0x98/0x240 [<ffffffff8117887e>] writeback_sb_inodes+0xce/0x170 [<ffffffff81179289>] writeback_inodes_wb+0x99/0x160 [<ffffffff81179660>] wb_writeback+0x310/0x440 [<ffffffff810710fc>] ? lock_timer_base+0x3c/0x70 [<ffffffff811799ef>] wb_do_writeback+0x25f/0x270 [<ffffffff81179aa2>] bdi_writeback_thread+0xa2/0x280 [<ffffffff81179a00>] ? bdi_writeback_thread+0x0/0x280 [<ffffffff81179a00>] ? bdi_writeback_thread+0x0/0x280 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task rsyslogd:3246 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsyslogd D ffff880c5e992610 0 3246 1 0x00000080 ffff880c5cdbda28 0000000000000082 00000037ffffffc8 ffff88107ffd9c00 0000000000014d40 ffff880c5e992080 ffff880c5e992610 ffff880c5cdbdfd8 ffff880c5e992618 0000000000014d40 ffff880c5cdbc010 0000000000014d40 Call Trace: [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff81058c5a>] ? load_balance+0x10a/0x880 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8100a876>] ? __switch_to+0x256/0x320 [<ffffffff814b0c55>] page_fault+0x25/0x30 [<ffffffff81063926>] ? do_syslog+0x406/0x540 [<ffffffff810638d5>] ? do_syslog+0x3b5/0x540 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811b90f7>] kmsg_read+0x37/0x70 [<ffffffff811ae2e6>] proc_reg_read+0x76/0xb0 [<ffffffff811542a5>] vfs_read+0xc5/0x190 [<ffffffff81154471>] sys_read+0x51/0x90 [<ffffffff8100bf82>] system_call_fastpath+0x16/0x1b INFO: task rs:main Q:Reg:7286 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rs:main Q:Reg D ffff88045d5bc650 0 7286 1 0x00000080 ffff88045d35bb08 0000000000000082 ffff880400000000 ffffffffa00040ac 0000000000014d40 ffff88045d5bc0c0 ffff88045d5bc650 ffff88045d35bfd8 ffff88045d5bc658 0000000000014d40 ffff88045d35a010 0000000000014d40 Call Trace: [<ffffffffa00040ac>] ? dm_table_unplug_all+0x5c/0x110 [dm_mod] [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff810fa847>] filemap_fault+0x2d7/0x4c0 [<ffffffff8111e454>] __do_fault+0x54/0x570 [<ffffffff8111ea67>] handle_pte_fault+0xf7/0xb20 [<ffffffff8110236d>] ? __free_pages+0x2d/0x40 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8109768b>] ? sys_futex+0x7b/0x180 [<ffffffff814b0c55>] page_fault+0x25/0x30 INFO: task automount:6638 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. automount D ffff88105cd6ba70 0 6638 1 0x00000080 ffff88105b8ebc18 0000000000000082 00000037ffffffff ffff88087ffdac00 0000000000014d40 ffff88105cd6b4e0 ffff88105cd6ba70 ffff88105b8ebfd8 ffff88105cd6ba78 0000000000014d40 ffff88105b8ea010 0000000000014d40 Call Trace: [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8109768b>] ? sys_futex+0x7b/0x180 [<ffffffff814b0c55>] page_fault+0x25/0x30 INFO: task master:6753 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. master D ffff88085d4a65d0 0 6753 1 0x00000080 ffff88085da87c18 0000000000000082 0000003700000000 ffffffff8113edf6 0000000000014d40 ffff88085d4a6040 ffff88085d4a65d0 ffff88085da87fd8 ffff88085d4a65d8 0000000000014d40 ffff88085da86010 0000000000014d40 Call Trace: [<ffffffff8113edf6>] ? transfer_objects+0x56/0x80 [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8118fae0>] ? sys_epoll_wait+0xa0/0x450 [<ffffffff81059c90>] ? default_wake_function+0x0/0x20 [<ffffffff814b0c55>] page_fault+0x25/0x30 INFO: task qmgr:6760 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. qmgr D ffff88105bb5baf0 0 6760 6753 0x00000084 ffff88105ce93c18 0000000000000082 0000003700000000 ffff88047ffdac00 0000000000014d40 ffff88105bb5b560 ffff88105bb5baf0 ffff88105ce93fd8 ffff88105bb5baf8 0000000000014d40 ffff88105ce92010 0000000000014d40 Call Trace: [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8118fae0>] ? sys_epoll_wait+0xa0/0x450 [<ffffffff81059c90>] ? default_wake_function+0x0/0x20 [<ffffffff814b0c55>] page_fault+0x25/0x30 [1] http://ltp.git.sourceforge.net/git/gitweb.cgi?p=ltp/ltp.git;a=blob;f=testcases/kernel/mem/oom/oom01.c ^ permalink raw reply [flat|nested] 14+ messages in thread
* kswapd hung tasks in 2.6.38-rc1 @ 2011-01-20 8:55 ` CAI Qian 0 siblings, 0 replies; 14+ messages in thread From: CAI Qian @ 2011-01-20 8:55 UTC (permalink / raw) To: linux-mm; +Cc: linux-kernel When running LTP oom01 [1] testing, the allocation process stopped processing right after starting to swap. # free -m total used free shared buffers cached Mem: 64307 64059 248 0 4 61 -/+ buffers/cache: 63993 314 Swap: 66139 754 65385 # ./oom01 oom01 0 TINFO : start testing overcommit_memory=2. oom01 0 TINFO : expected victim is 7276. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. <hung...> INFO: task kswapd0:274 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd0 D ffff88045dd07000 0 274 2 0x00000000 ffff88045dd0d810 0000000000000046 0000000000000000 ffff88002e366e40 0000000000014d40 ffff88045dd06a70 ffff88045dd07000 ffff88045dd0dfd8 ffff88045dd07008 0000000000014d40 ffff88045dd0c010 0000000000014d40 Call Trace: [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff8120f285>] get_request_wait+0xc5/0x190 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff81207db7>] ? elv_merge+0x1d7/0x210 [<ffffffff8120f3bb>] __make_request+0x6b/0x4c0 [<ffffffff8120d28a>] generic_make_request+0x2ea/0x5d0 [<ffffffff810423a6>] ? native_flush_tlb_others+0x76/0x90 [<ffffffff8120d5f6>] submit_bio+0x86/0x110 [<ffffffff811040b6>] ? test_set_page_writeback+0x106/0x190 [<ffffffff8112d0d3>] swap_writepage+0x83/0xd0 [<ffffffff81108f7e>] pageout+0x12e/0x310 [<ffffffff8110957a>] shrink_page_list+0x41a/0x5a0 [<ffffffff81109dc2>] shrink_inactive_list+0x172/0x460 [<ffffffff81102cfa>] ? determine_dirtyable_memory+0x1a/0x30 [<ffffffff8110a76b>] shrink_zone+0x36b/0x520 [<ffffffff810fe805>] ? zone_watermark_ok_safe+0xb5/0xd0 [<ffffffff8110bee9>] kswapd+0x969/0xc20 [<ffffffff8110b580>] ? kswapd+0x0/0xc20 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task kswapd2:276 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd2 D ffff88045dd13a70 0 276 2 0x00000000 ffff88045dd15810 0000000000000046 0000000000000000 ffff880c5fa63240 0000000000014d40 ffff88045dd134e0 ffff88045dd13a70 ffff88045dd15fd8 ffff88045dd13a78 0000000000014d40 ffff88045dd14010 0000000000014d40 Call Trace: [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff8120f285>] get_request_wait+0xc5/0x190 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff81207db7>] ? elv_merge+0x1d7/0x210 [<ffffffff8120f3bb>] __make_request+0x6b/0x4c0 [<ffffffff8120d28a>] generic_make_request+0x2ea/0x5d0 [<ffffffff810423a6>] ? native_flush_tlb_others+0x76/0x90 [<ffffffff8120d5f6>] submit_bio+0x86/0x110 [<ffffffff811040b6>] ? test_set_page_writeback+0x106/0x190 [<ffffffff8112d0d3>] swap_writepage+0x83/0xd0 [<ffffffff81108f7e>] pageout+0x12e/0x310 [<ffffffff8110957a>] shrink_page_list+0x41a/0x5a0 [<ffffffff81109dc2>] shrink_inactive_list+0x172/0x460 [<ffffffff81071cda>] ? del_timer_sync+0x3a/0x60 [<ffffffff8110a76b>] shrink_zone+0x36b/0x520 [<ffffffff810fe805>] ? zone_watermark_ok_safe+0xb5/0xd0 [<ffffffff8110bee9>] kswapd+0x969/0xc20 [<ffffffff8110b580>] ? kswapd+0x0/0xc20 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task kswapd3:277 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd3 D ffff88045dd13040 0 277 2 0x00000000 ffff88045dd19810 0000000000000046 0000000000000000 ffff880857ed9d40 0000000000014d40 ffff88045dd12ab0 ffff88045dd13040 ffff88045dd19fd8 ffff88045dd13048 0000000000014d40 ffff88045dd18010 0000000000014d40 Call Trace: [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff8120f285>] get_request_wait+0xc5/0x190 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff81207c27>] ? elv_merge+0x47/0x210 [<ffffffff8120f3bb>] __make_request+0x6b/0x4c0 [<ffffffff8120d28a>] generic_make_request+0x2ea/0x5d0 [<ffffffff810423a6>] ? native_flush_tlb_others+0x76/0x90 [<ffffffff8120d5f6>] submit_bio+0x86/0x110 [<ffffffff811040b6>] ? test_set_page_writeback+0x106/0x190 [<ffffffff8112d0d3>] swap_writepage+0x83/0xd0 [<ffffffff81108f7e>] pageout+0x12e/0x310 [<ffffffff8110957a>] shrink_page_list+0x41a/0x5a0 [<ffffffff81109dc2>] shrink_inactive_list+0x172/0x460 [<ffffffff81071cda>] ? del_timer_sync+0x3a/0x60 [<ffffffff8110a76b>] shrink_zone+0x36b/0x520 [<ffffffff810fe805>] ? zone_watermark_ok_safe+0xb5/0xd0 [<ffffffff8110bee9>] kswapd+0x969/0xc20 [<ffffffff8110b580>] ? kswapd+0x0/0xc20 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task jbd2/dm-0-8:1002 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/dm-0-8 D ffff88045d33d080 0 1002 2 0x00000000 ffff88045d53bc20 0000000000000046 ffff88085f044900 ffff88085f0da4f8 0000000000014d40 ffff88045d33caf0 ffff88045d33d080 ffff88045d53bfd8 ffff88045d33d088 0000000000014d40 ffff88045d53a010 0000000000014d40 Call Trace: [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff811809d0>] sync_buffer+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814aed88>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff8118098e>] __wait_on_buffer+0x2e/0x30 [<ffffffffa0082fd8>] jbd2_journal_commit_transaction+0x8e8/0x13d0 [jbd2] [<ffffffff81071c41>] ? try_to_del_timer_sync+0x81/0xe0 [<ffffffffa00882b8>] kjournald2+0xb8/0x220 [jbd2] [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa0088200>] ? kjournald2+0x0/0x220 [jbd2] [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task flush-253:0:2030 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:0 D ffff88045d2a0610 0 2030 2 0x00000000 ffff88045e5ab4f0 0000000000000046 ffff88085f044900 ffff88085f0da4f8 0000000000014d40 ffff88045d2a0080 ffff88045d2a0610 ffff88045e5abfd8 ffff88045d2a0618 0000000000014d40 ffff88045e5aa010 0000000000014d40 Call Trace: [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff811809d0>] sync_buffer+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff81180990>] ? sync_buffer+0x0/0x50 [<ffffffff814aed88>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff8118098e>] __wait_on_buffer+0x2e/0x30 [<ffffffffa00d9275>] ext4_mb_init_cache+0x345/0x9f0 [ext4] [<ffffffffa00d9b28>] ext4_mb_init_group+0x208/0x390 [ext4] [<ffffffffa00d9d7e>] ext4_mb_good_group+0xce/0x110 [ext4] [<ffffffffa00dd5ab>] ext4_mb_regular_allocator+0x19b/0x410 [ext4] [<ffffffffa00ddbc5>] ext4_mb_new_blocks+0x3a5/0x510 [ext4] [<ffffffffa00d020b>] ? ext4_ext_find_extent+0x2bb/0x320 [ext4] [<ffffffffa00d2a1e>] ext4_ext_map_blocks+0x58e/0x1f40 [ext4] [<ffffffff8113edf6>] ? transfer_objects+0x56/0x80 [<ffffffff81104a25>] ? pagevec_lookup_tag+0x25/0x40 [<ffffffffa00b0ec4>] ext4_map_blocks+0xf4/0x210 [ext4] [<ffffffffa00b2379>] mpage_da_map_and_submit+0xb9/0x450 [ext4] [<ffffffffa0081393>] ? jbd2_journal_start+0x13/0x20 [jbd2] [<ffffffffa00c6c70>] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [<ffffffffa00b2fe9>] ext4_da_writepages+0x349/0x680 [ext4] [<ffffffff811026b0>] ? __writepage+0x0/0x40 [<ffffffff81103c01>] do_writepages+0x21/0x40 [<ffffffff81178238>] writeback_single_inode+0x98/0x240 [<ffffffff8117887e>] writeback_sb_inodes+0xce/0x170 [<ffffffff81179289>] writeback_inodes_wb+0x99/0x160 [<ffffffff81179660>] wb_writeback+0x310/0x440 [<ffffffff810710fc>] ? lock_timer_base+0x3c/0x70 [<ffffffff811799ef>] wb_do_writeback+0x25f/0x270 [<ffffffff81179aa2>] bdi_writeback_thread+0xa2/0x280 [<ffffffff81179a00>] ? bdi_writeback_thread+0x0/0x280 [<ffffffff81179a00>] ? bdi_writeback_thread+0x0/0x280 [<ffffffff81083046>] kthread+0x96/0xa0 [<ffffffff8100cdc4>] kernel_thread_helper+0x4/0x10 [<ffffffff81082fb0>] ? kthread+0x0/0xa0 [<ffffffff8100cdc0>] ? kernel_thread_helper+0x0/0x10 INFO: task rsyslogd:3246 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsyslogd D ffff880c5e992610 0 3246 1 0x00000080 ffff880c5cdbda28 0000000000000082 00000037ffffffc8 ffff88107ffd9c00 0000000000014d40 ffff880c5e992080 ffff880c5e992610 ffff880c5cdbdfd8 ffff880c5e992618 0000000000014d40 ffff880c5cdbc010 0000000000014d40 Call Trace: [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff81058c5a>] ? load_balance+0x10a/0x880 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8100a876>] ? __switch_to+0x256/0x320 [<ffffffff814b0c55>] page_fault+0x25/0x30 [<ffffffff81063926>] ? do_syslog+0x406/0x540 [<ffffffff810638d5>] ? do_syslog+0x3b5/0x540 [<ffffffff810836e0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811b90f7>] kmsg_read+0x37/0x70 [<ffffffff811ae2e6>] proc_reg_read+0x76/0xb0 [<ffffffff811542a5>] vfs_read+0xc5/0x190 [<ffffffff81154471>] sys_read+0x51/0x90 [<ffffffff8100bf82>] system_call_fastpath+0x16/0x1b INFO: task rs:main Q:Reg:7286 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rs:main Q:Reg D ffff88045d5bc650 0 7286 1 0x00000080 ffff88045d35bb08 0000000000000082 ffff880400000000 ffffffffa00040ac 0000000000014d40 ffff88045d5bc0c0 ffff88045d5bc650 ffff88045d35bfd8 ffff88045d5bc658 0000000000014d40 ffff88045d35a010 0000000000014d40 Call Trace: [<ffffffffa00040ac>] ? dm_table_unplug_all+0x5c/0x110 [dm_mod] [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff810fa847>] filemap_fault+0x2d7/0x4c0 [<ffffffff8111e454>] __do_fault+0x54/0x570 [<ffffffff8111ea67>] handle_pte_fault+0xf7/0xb20 [<ffffffff8110236d>] ? __free_pages+0x2d/0x40 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8109768b>] ? sys_futex+0x7b/0x180 [<ffffffff814b0c55>] page_fault+0x25/0x30 INFO: task automount:6638 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. automount D ffff88105cd6ba70 0 6638 1 0x00000080 ffff88105b8ebc18 0000000000000082 00000037ffffffff ffff88087ffdac00 0000000000014d40 ffff88105cd6b4e0 ffff88105cd6ba70 ffff88105b8ebfd8 ffff88105cd6ba78 0000000000014d40 ffff88105b8ea010 0000000000014d40 Call Trace: [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8109768b>] ? sys_futex+0x7b/0x180 [<ffffffff814b0c55>] page_fault+0x25/0x30 INFO: task master:6753 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. master D ffff88085d4a65d0 0 6753 1 0x00000080 ffff88085da87c18 0000000000000082 0000003700000000 ffffffff8113edf6 0000000000014d40 ffff88085d4a6040 ffff88085d4a65d0 ffff88085da87fd8 ffff88085d4a65d8 0000000000014d40 ffff88085da86010 0000000000014d40 Call Trace: [<ffffffff8113edf6>] ? transfer_objects+0x56/0x80 [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8118fae0>] ? sys_epoll_wait+0xa0/0x450 [<ffffffff81059c90>] ? default_wake_function+0x0/0x20 [<ffffffff814b0c55>] page_fault+0x25/0x30 INFO: task qmgr:6760 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. qmgr D ffff88105bb5baf0 0 6760 6753 0x00000084 ffff88105ce93c18 0000000000000082 0000003700000000 ffff88047ffdac00 0000000000014d40 ffff88105bb5b560 ffff88105bb5baf0 ffff88105ce93fd8 ffff88105bb5baf8 0000000000014d40 ffff88105ce92010 0000000000014d40 Call Trace: [<ffffffff810f9570>] ? sync_page+0x0/0x50 [<ffffffff814ae420>] io_schedule+0x70/0xc0 [<ffffffff810f95b0>] sync_page+0x40/0x50 [<ffffffff814aecdf>] __wait_on_bit+0x5f/0x90 [<ffffffff8112d3bf>] ? read_swap_cache_async+0x4f/0x140 [<ffffffff810f9773>] wait_on_page_bit+0x73/0x80 [<ffffffff81083720>] ? wake_bit_function+0x0/0x50 [<ffffffff810f981a>] __lock_page_or_retry+0x3a/0x60 [<ffffffff8111f433>] handle_pte_fault+0xac3/0xb20 [<ffffffff8111f641>] handle_mm_fault+0x1b1/0x320 [<ffffffff814b3ee5>] do_page_fault+0x135/0x460 [<ffffffff8118fae0>] ? sys_epoll_wait+0xa0/0x450 [<ffffffff81059c90>] ? default_wake_function+0x0/0x20 [<ffffffff814b0c55>] page_fault+0x25/0x30 [1] http://ltp.git.sourceforge.net/git/gitweb.cgi?p=ltp/ltp.git;a=blob;f=testcases/kernel/mem/oom/oom01.c -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 2011-01-20 8:55 ` CAI Qian @ 2011-01-31 20:58 ` Dave Hansen -1 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-01-31 20:58 UTC (permalink / raw) To: CAI Qian; +Cc: linux-mm, linux-kernel On Thu, 2011-01-20 at 03:55 -0500, CAI Qian wrote: > When running LTP oom01 [1] testing, the allocation process stopped > processing right after starting to swap. I'm seeing the same stuff, but on -rc2. I thought it was transparent-hugepage-related, but I don't see much of a trace of it in the stack dumps. http://sr71.net/~dave/ibm/config-v2.6.38-rc2 It happened to me as well around the time that things started to hit swap. [ 723.702046] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 723.708478] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 723.716289] kswapd0 D 000000010000f58a 0 706 2 0x00000000 [ 723.723358] ffff88083e183920 0000000000000046 0000000000000001 0000000000011c40 [ 723.730815] ffff88083f9af460 ffff88083f9af100 ffffffff81925020 ffffffff00000000 [ 723.738270] ffff88083fac94f0 000000103e183880 ffff88083dfa1d80 ffff88083d891308 [ 723.745727] Call Trace: [ 723.748177] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 723.753825] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 723.759388] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 723.764603] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 723.770337] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 723.776764] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 723.782154] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 723.787716] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 723.793885] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 723.799013] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 723.804141] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 723.810656] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 723.816131] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 723.821954] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 723.828120] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 723.833509] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 723.839848] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 723.844802] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 723.849756] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 723.854628] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 723.860533] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 723.865489] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 723.871567] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 723.877992] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 723.885800] kswapd1 D 000000010000f4ec 0 707 2 0x00000000 [ 723.892869] ffff88083e145920 0000000000000046 0000000000000001 0000000000011c40 [ 723.900324] ffff88083f9afb60 ffff88083f9af800 ffff88083f80b8c0 ffffffff0000000c [ 723.907781] ffff88083fac94f0 000000103e145880 ffff88103e94c3c0 ffff88083d891308 [ 723.915239] Call Trace: [ 723.917683] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 723.923329] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 723.928891] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 723.934104] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 723.939836] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 723.946264] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 723.951651] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 723.957212] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 723.963378] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 723.968509] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 723.973636] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 723.980151] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 723.985625] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 723.991448] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 723.997613] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 724.004298] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.009686] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 724.016024] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 724.020981] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 724.025937] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 724.030805] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 724.036713] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 724.041667] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 724.047748] INFO: task kjournald:1908 blocked for more than 120 seconds. [ 724.054431] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.062240] kjournald D 000000010000f8dd 0 1908 2 0x00000000 [ 724.069309] ffff88083e3795d0 0000000000000046 0000000000000000 0000000000011c40 [ 724.076763] ffff88083f93dbe0 ffff88083f93d880 ffff880840350840 ffffea0000000004 [ 724.084214] 0000000000000001 ffff88083e3795f8 0000000000000202 0000000000000000 [ 724.091670] Call Trace: [ 724.094116] [<ffffffff812ecb9c>] ? cfq_may_queue+0x2c/0xe0 [ 724.099676] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.104889] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.110624] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.117049] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.122437] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.127999] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.134164] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.139292] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.144418] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.150931] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.156404] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.162223] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.168388] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.173775] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.179853] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.185674] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.192015] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 724.197921] [<ffffffff810e9800>] __page_cache_alloc+0x10/0x20 [ 724.203740] [<ffffffff810e9ca1>] find_or_create_page+0x51/0xb0 [ 724.209650] [<ffffffff8116067f>] __getblk+0xdf/0x270 [ 724.214693] [<ffffffff811f7f56>] journal_get_descriptor_buffer+0x46/0xa0 [ 724.221467] [<ffffffff811f401a>] journal_commit_transaction+0xd7a/0xf30 [ 724.228157] [<ffffffff81087ec9>] ? try_to_del_timer_sync+0x49/0xe0 [ 724.234411] [<ffffffff811f73e0>] kjournald+0xe0/0x250 [ 724.239540] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.245968] [<ffffffff811f7300>] ? kjournald+0x0/0x250 [ 724.251182] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 724.256051] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 724.261960] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 724.266913] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 724.272993] INFO: task jbd2/sdc3-8:2298 blocked for more than 120 seconds. [ 724.279851] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.287660] jbd2/sdc3-8 D 000000010000f9d4 0 2298 2 0x00000000 [ 724.294731] ffff88083ef3d560 0000000000000046 ffff881000000000 0000000000011c40 [ 724.302184] ffff88083e3a7b60 ffff88083e3a7800 ffff880840350840 0000000000000004 [ 724.309637] ffff88103e8c8520 ffff88083d891308 ffff88083ef3d4c0 ffffffff812f52ba [ 724.317092] Call Trace: [ 724.319537] [<ffffffff812f52ba>] ? rb_insert_color+0x8a/0xf0 [ 724.325272] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 724.330832] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.336046] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.341781] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.348207] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.353597] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.359156] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.365324] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.370450] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.375579] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.382091] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.387567] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.393387] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.399556] [<ffffffff81070a99>] ? load_balance+0x1e9/0x1560 [ 724.405293] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.410681] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.416762] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.422582] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.428926] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 724.434832] [<ffffffff810e9800>] __page_cache_alloc+0x10/0x20 [ 724.440653] [<ffffffff810e9ca1>] find_or_create_page+0x51/0xb0 [ 724.446560] [<ffffffff8116067f>] __getblk+0xdf/0x270 [ 724.451603] [<ffffffff811ff807>] jbd2_journal_get_descriptor_buffer+0x47/0xa0 [ 724.458809] [<ffffffff811fb5d7>] jbd2_journal_commit_transaction+0x1247/0x1400 [ 724.466099] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.472528] [<ffffffff811ffcf7>] kjournald2+0xb7/0x230 [ 724.477743] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.484172] [<ffffffff811ffc40>] ? kjournald2+0x0/0x230 [ 724.489474] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 724.494343] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 724.500248] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 724.505206] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 724.511286] INFO: task cron:2380 blocked for more than 120 seconds. [ 724.517537] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.525346] cron D 00000001000126d4 0 2380 1 0x00000000 [ 724.532418] ffff88083ea235b8 0000000000000082 00e8c03100000000 0000000000011c40 [ 724.539871] ffff88083f86d520 ffff88083f86d1c0 ffff880840350840 90b88d4800000004 [ 724.547327] ffff88083ea23518 0000000000000046 00c7c748c68948da 0000000000000001 [ 724.554782] Call Trace: [ 724.557226] [<ffffffff812ecb9c>] ? cfq_may_queue+0x2c/0xe0 [ 724.562789] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.568006] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.573741] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.580166] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.585554] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.591114] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.597281] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.602409] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.607538] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.614053] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.619528] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.625350] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.631515] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.636903] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.642985] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.648805] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.655148] [<ffffffff8112703c>] kmem_getpages+0x6c/0x160 [ 724.660621] [<ffffffff81127689>] fallback_alloc+0x119/0x1c0 [ 724.666269] [<ffffffff811277c5>] ____cache_alloc_node+0x95/0x150 [ 724.672349] [<ffffffff811281a3>] kmem_cache_alloc+0x103/0x170 [ 724.678170] [<ffffffff811439cb>] getname+0x3b/0x230 [ 724.683124] [<ffffffff81145b4a>] user_path_at+0x2a/0xa0 [ 724.688425] [<ffffffff812f68b0>] ? timerqueue_add+0x60/0xb0 [ 724.694075] [<ffffffff8109c891>] ? lock_hrtimer_base+0x31/0x60 [ 724.699983] [<ffffffff8109c961>] ? hrtimer_try_to_cancel+0x51/0xd0 [ 724.706236] [<ffffffff8109c9fd>] ? hrtimer_cancel+0x1d/0x30 [ 724.711887] [<ffffffff8113ba0a>] vfs_fstatat+0x4a/0x80 [ 724.717100] [<ffffffff8113ba7b>] vfs_stat+0x1b/0x20 [ 724.722055] [<ffffffff8113bc64>] sys_newstat+0x24/0x50 [ 724.727270] [<ffffffff8109cdb7>] ? sys_nanosleep+0x77/0x80 [ 724.732832] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 724.738829] INFO: task sshd:2536 blocked for more than 120 seconds. [ 724.745080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.752890] sshd D 0000000100010811 0 2536 2508 0x00000000 [ 724.759959] ffff88083ddd9438 0000000000000082 ffff88083ddd9378 0000000000011c40 [ 724.767413] ffff88083fb0ebe0 ffff88083fb0e880 ffff88084039d880 ffff881000000008 [ 724.774870] 0000000000000000 0000000000000000 0000000000000020 0000000000000046 [ 724.782324] Call Trace: [ 724.784769] [<ffffffff812e25ab>] ? alloc_io_context+0x1b/0x80 [ 724.790591] [<ffffffff812e263f>] ? current_io_context+0x2f/0x40 [ 724.796586] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.801801] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.807536] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.813961] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.819351] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.824914] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.831080] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.836210] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.841337] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.847850] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.853325] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.859147] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.865315] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 724.872001] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.877388] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.883466] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.889290] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.895632] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 724.901539] [<ffffffff81063a4b>] pte_alloc_one+0x1b/0x50 [ 724.906926] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 724.911969] [<ffffffff81109fa7>] __pte_alloc+0x27/0x110 [ 724.917270] [<ffffffff8110a19f>] handle_mm_fault+0x10f/0x270 [ 724.923004] [<ffffffff815f52f1>] do_page_fault+0x161/0x480 [ 724.928567] [<ffffffff815f1ea4>] ? _raw_spin_unlock_bh+0x14/0x20 [ 724.934646] [<ffffffff81518519>] ? release_sock+0xd9/0x110 [ 724.940210] [<ffffffff81559339>] ? tcp_sendmsg+0x819/0xb80 [ 724.945770] [<ffffffff815f251f>] page_fault+0x1f/0x30 [ 724.950897] [<ffffffff812f992d>] ? copy_user_generic_string+0x2d/0x40 [ 724.957411] [<ffffffff81149243>] ? core_sys_select+0x1e3/0x2c0 [ 724.963317] [<ffffffff81137585>] ? do_sync_write+0xd5/0x120 [ 724.968968] [<ffffffff8116aa05>] ? fsnotify+0x1d5/0x2e0 [ 724.974272] [<ffffffff8114953b>] sys_select+0x4b/0x100 [ 724.979485] [<ffffffff811383f5>] ? sys_write+0x55/0x90 [ 724.984701] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 724.990696] INFO: task qemu-system-x86:2731 blocked for more than 120 seconds. [ 724.997900] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.005710] qemu-system-x86 D 000000010000f685 0 2731 2692 0x00000000 [ 725.012780] ffff88083fa472c8 0000000000000082 0000000000000001 0000000000011c40 [ 725.020237] ffff88083fa693e0 ffff88083fa69080 ffff88084039d880 ffffffff00000008 [ 725.027692] ffff88083fac94f0 000000103fa47228 ffff88083fb947b0 ffff88083d891308 [ 725.035146] Call Trace: [ 725.037589] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 725.043236] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.048796] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.054010] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.059746] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.066170] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.071560] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.077123] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.083289] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.088415] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.093545] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.100059] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.105535] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.111358] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.117523] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.124211] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 725.129599] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 725.135682] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 725.141503] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 725.147841] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 725.153746] [<ffffffff810ef25e>] __get_free_pages+0xe/0x50 [ 725.159306] [<ffffffff81148285>] __pollwait+0xc5/0x100 [ 725.164521] [<ffffffff815579d8>] tcp_poll+0x28/0x180 [ 725.169564] [<ffffffff815136fd>] sock_poll+0x1d/0x20 [ 725.174604] [<ffffffff81148d93>] do_select+0x3b3/0x680 [ 725.179821] [<ffffffff811481c0>] ? __pollwait+0x0/0x100 [ 725.185125] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.190165] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.195207] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.200250] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.205290] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.210331] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.215371] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.220412] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.225453] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.230495] [<ffffffff8114921e>] core_sys_select+0x1be/0x2c0 [ 725.236229] [<ffffffff8116f605>] ? signalfd_read+0xd5/0x500 [ 725.241875] [<ffffffff812f68b0>] ? timerqueue_add+0x60/0xb0 [ 725.247522] [<ffffffff81170305>] ? eventfd_ctx_read+0x95/0x190 [ 725.253431] [<ffffffff8107fb39>] ? timespec_add_safe+0x39/0x70 [ 725.259340] [<ffffffff8114953b>] sys_select+0x4b/0x100 [ 725.264556] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 725.270550] INFO: task qemu-system-x86:2733 blocked for more than 120 seconds. [ 725.277755] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.285562] qemu-system-x86 D 000000010000fbd6 0 2733 2692 0x00000000 [ 725.292631] ffff88103b95d238 0000000000000082 0000000000000001 0000000000011c40 [ 725.300086] ffff88103ea603a0 ffff88103ea60040 ffff88083f80b8c0 ffffffff0000000c [ 725.307541] ffff88083fac94f0 000000103b95d198 ffff880102069b60 ffff88083d891308 [ 725.314997] Call Trace: [ 725.317441] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 725.323088] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.328650] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.333865] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.339601] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.346026] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.351415] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.356977] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.363145] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.368274] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.373400] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.379914] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.385386] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.391208] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.397373] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.404060] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 725.409448] [<ffffffff810f70f9>] ? shrink_slab+0x149/0x180 [ 725.415008] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 725.421089] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 725.426909] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 725.433251] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 725.439159] [<ffffffff81063a4b>] pte_alloc_one+0x1b/0x50 [ 725.444548] [<ffffffff81109fa7>] __pte_alloc+0x27/0x110 [ 725.449848] [<ffffffff8110a19f>] handle_mm_fault+0x10f/0x270 [ 725.455582] [<ffffffff8110a44f>] __get_user_pages+0x14f/0x490 [ 725.461406] [<ffffffff81001e69>] ? kvm_read_guest_page+0x39/0x80 [ 725.467487] [<ffffffff8110a7e2>] get_user_pages+0x52/0x60 [ 725.472963] [<ffffffff810642c4>] get_user_pages_fast+0x134/0x160 [ 725.479042] [<ffffffff810029a1>] hva_to_pfn+0xc1/0x280 [ 725.484257] [<ffffffff81002c4f>] __gfn_to_pfn+0xbf/0xe0 [ 725.489560] [<ffffffff81002cca>] gfn_to_pfn_async+0x1a/0x20 [ 725.495208] [<ffffffff8101f6fb>] try_async_pf+0x4b/0x1e0 [ 725.500598] [<ffffffff81001232>] ? gfn_to_memslot+0x12/0x20 [ 725.506244] [<ffffffff810012af>] ? gfn_to_hva+0x1f/0x30 [ 725.511543] [<ffffffff81004296>] ? kvm_host_page_size+0x86/0x90 [ 725.517538] [<ffffffff810241ff>] tdp_page_fault+0xef/0x1d0 [ 725.523100] [<ffffffff8103316a>] ? vmx_set_interrupt_shadow+0x1a/0x50 [ 725.529614] [<ffffffff8102080c>] kvm_mmu_page_fault+0x2c/0x90 [ 725.535435] [<ffffffff81033cc0>] handle_ept_violation+0x80/0x150 [ 725.541515] [<ffffffff8103716a>] vmx_handle_exit+0xba/0x4e0 [ 725.547162] [<ffffffff8102f65f>] ? apic_update_ppr+0x3f/0x90 [ 725.552897] [<ffffffff8109ea74>] ? sched_clock_cpu+0x24/0xf0 [ 725.558632] [<ffffffff810180d1>] kvm_arch_vcpu_ioctl_run+0x4e1/0xe00 [ 725.565057] [<ffffffff81017797>] ? kvm_arch_vcpu_load+0x47/0x120 [ 725.571140] [<ffffffff81006bf2>] kvm_vcpu_ioctl+0x4d2/0x5c0 [ 725.576787] [<ffffffff81003ca0>] ? kvm_vm_ioctl+0xa0/0x3c0 [ 725.582346] [<ffffffff81515cc7>] ? sys_sendto+0x137/0x140 [ 725.587819] [<ffffffff81146e1d>] vfs_ioctl+0x1d/0x50 [ 725.592861] [<ffffffff81146fb5>] do_vfs_ioctl+0xa5/0x4e0 [ 725.598251] [<ffffffff810a8d43>] ? sys_futex+0x83/0x140 [ 725.603553] [<ffffffff8114743f>] sys_ioctl+0x4f/0x80 [ 725.608596] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 725.614587] INFO: task qemu-system-x86:2736 blocked for more than 120 seconds. [ 725.621790] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.629600] qemu-system-x86 D 000000010000f67d 0 2736 2692 0x00000000 [ 725.636669] ffff88102de7f508 0000000000000082 ffff88083d891308 0000000000011c40 [ 725.644128] ffff88103ea5fc20 ffff88103ea5f8c0 ffff88084039d880 ffff880800000008 [ 725.651584] ffffffff8192d4a0 ffffffff811269b8 ffff88102de7f478 ffff88083faaecc0 [ 725.659040] Call Trace: [ 725.661483] [<ffffffff811269b8>] ? transfer_objects+0x58/0x80 [ 725.667304] [<ffffffff811279af>] ? cache_alloc_refill+0x9f/0x260 [ 725.673383] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.678944] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.684159] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.689893] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.696319] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.701708] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.707271] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.713437] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.718568] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.723695] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.730209] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.735685] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.741506] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.747673] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.754360] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 725.759750] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 725.765831] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 725.771653] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 725.778000] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 725.783906] [<ffffffff810e9800>] __page_cache_alloc+0x10/0x20 [ 725.789725] [<ffffffff810f2f82>] __do_page_cache_readahead+0xe2/0x220 [ 725.796236] [<ffffffff810f30e1>] ra_submit+0x21/0x30 [ 725.801278] [<ffffffff810f3288>] ondemand_readahead+0xa8/0x1d0 [ 725.807186] [<ffffffff810f34a6>] page_cache_sync_readahead+0x36/0x50 [ 725.813614] [<ffffffff810ead0a>] generic_file_aio_read+0x57a/0x6f0 [ 725.819865] [<ffffffff8106a72b>] ? __wake_up_common+0x5b/0x90 [ 725.825688] [<ffffffff811376a5>] do_sync_read+0xd5/0x120 [ 725.831075] [<ffffffff8108c0e2>] ? group_send_sig_info+0x52/0x60 [ 725.837157] [<ffffffff8108c13b>] ? kill_pid_info+0x4b/0x70 [ 725.842717] [<ffffffff8108ca44>] ? sys_kill+0x94/0x190 [ 725.847930] [<ffffffff81137ebb>] vfs_read+0xcb/0x170 [ 725.852972] [<ffffffff811384ba>] sys_pread64+0x8a/0x90 [ 725.858186] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 725.864180] INFO: task memknobs:2751 blocked for more than 120 seconds. [ 725.870778] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.878588] memknobs D 000000010000f603 0 2751 2617 0x00000000 [ 725.885656] ffff88083e57d6b8 0000000000000082 0000000000000001 0000000000011c40 [ 725.893111] ffff88083fb0ab20 ffff88083fb0a7c0 ffffffff81925020 ffffffff00000000 [ 725.900567] ffff88083fac94f0 000000103e57d618 ffff88083d8a15e0 ffff88083d891308 [ 725.908021] Call Trace: [ 725.910467] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 725.916115] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.921676] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.926890] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.932623] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.939050] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.944437] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.949998] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.956165] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.961292] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.966418] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.972933] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.978406] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.984228] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.990394] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.997081] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 726.002471] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 726.008551] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 726.014373] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 726.020714] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 726.025845] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 726.031751] [<ffffffff81063a4b>] pte_alloc_one+0x1b/0x50 [ 726.037141] [<ffffffff81109fa7>] __pte_alloc+0x27/0x110 [ 726.042445] [<ffffffff8110a19f>] handle_mm_fault+0x10f/0x270 [ 726.048179] [<ffffffff815f52f1>] do_page_fault+0x161/0x480 [ 726.053740] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 726.059388] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 726.065207] [<ffffffff815f251f>] page_fault+0x1f/0x30 -- Dave ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 @ 2011-01-31 20:58 ` Dave Hansen 0 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-01-31 20:58 UTC (permalink / raw) To: CAI Qian; +Cc: linux-mm, linux-kernel On Thu, 2011-01-20 at 03:55 -0500, CAI Qian wrote: > When running LTP oom01 [1] testing, the allocation process stopped > processing right after starting to swap. I'm seeing the same stuff, but on -rc2. I thought it was transparent-hugepage-related, but I don't see much of a trace of it in the stack dumps. http://sr71.net/~dave/ibm/config-v2.6.38-rc2 It happened to me as well around the time that things started to hit swap. [ 723.702046] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 723.708478] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 723.716289] kswapd0 D 000000010000f58a 0 706 2 0x00000000 [ 723.723358] ffff88083e183920 0000000000000046 0000000000000001 0000000000011c40 [ 723.730815] ffff88083f9af460 ffff88083f9af100 ffffffff81925020 ffffffff00000000 [ 723.738270] ffff88083fac94f0 000000103e183880 ffff88083dfa1d80 ffff88083d891308 [ 723.745727] Call Trace: [ 723.748177] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 723.753825] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 723.759388] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 723.764603] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 723.770337] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 723.776764] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 723.782154] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 723.787716] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 723.793885] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 723.799013] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 723.804141] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 723.810656] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 723.816131] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 723.821954] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 723.828120] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 723.833509] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 723.839848] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 723.844802] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 723.849756] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 723.854628] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 723.860533] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 723.865489] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 723.871567] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 723.877992] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 723.885800] kswapd1 D 000000010000f4ec 0 707 2 0x00000000 [ 723.892869] ffff88083e145920 0000000000000046 0000000000000001 0000000000011c40 [ 723.900324] ffff88083f9afb60 ffff88083f9af800 ffff88083f80b8c0 ffffffff0000000c [ 723.907781] ffff88083fac94f0 000000103e145880 ffff88103e94c3c0 ffff88083d891308 [ 723.915239] Call Trace: [ 723.917683] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 723.923329] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 723.928891] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 723.934104] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 723.939836] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 723.946264] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 723.951651] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 723.957212] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 723.963378] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 723.968509] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 723.973636] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 723.980151] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 723.985625] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 723.991448] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 723.997613] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 724.004298] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.009686] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 724.016024] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 724.020981] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 724.025937] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 724.030805] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 724.036713] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 724.041667] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 724.047748] INFO: task kjournald:1908 blocked for more than 120 seconds. [ 724.054431] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.062240] kjournald D 000000010000f8dd 0 1908 2 0x00000000 [ 724.069309] ffff88083e3795d0 0000000000000046 0000000000000000 0000000000011c40 [ 724.076763] ffff88083f93dbe0 ffff88083f93d880 ffff880840350840 ffffea0000000004 [ 724.084214] 0000000000000001 ffff88083e3795f8 0000000000000202 0000000000000000 [ 724.091670] Call Trace: [ 724.094116] [<ffffffff812ecb9c>] ? cfq_may_queue+0x2c/0xe0 [ 724.099676] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.104889] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.110624] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.117049] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.122437] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.127999] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.134164] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.139292] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.144418] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.150931] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.156404] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.162223] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.168388] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.173775] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.179853] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.185674] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.192015] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 724.197921] [<ffffffff810e9800>] __page_cache_alloc+0x10/0x20 [ 724.203740] [<ffffffff810e9ca1>] find_or_create_page+0x51/0xb0 [ 724.209650] [<ffffffff8116067f>] __getblk+0xdf/0x270 [ 724.214693] [<ffffffff811f7f56>] journal_get_descriptor_buffer+0x46/0xa0 [ 724.221467] [<ffffffff811f401a>] journal_commit_transaction+0xd7a/0xf30 [ 724.228157] [<ffffffff81087ec9>] ? try_to_del_timer_sync+0x49/0xe0 [ 724.234411] [<ffffffff811f73e0>] kjournald+0xe0/0x250 [ 724.239540] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.245968] [<ffffffff811f7300>] ? kjournald+0x0/0x250 [ 724.251182] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 724.256051] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 724.261960] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 724.266913] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 724.272993] INFO: task jbd2/sdc3-8:2298 blocked for more than 120 seconds. [ 724.279851] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.287660] jbd2/sdc3-8 D 000000010000f9d4 0 2298 2 0x00000000 [ 724.294731] ffff88083ef3d560 0000000000000046 ffff881000000000 0000000000011c40 [ 724.302184] ffff88083e3a7b60 ffff88083e3a7800 ffff880840350840 0000000000000004 [ 724.309637] ffff88103e8c8520 ffff88083d891308 ffff88083ef3d4c0 ffffffff812f52ba [ 724.317092] Call Trace: [ 724.319537] [<ffffffff812f52ba>] ? rb_insert_color+0x8a/0xf0 [ 724.325272] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 724.330832] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.336046] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.341781] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.348207] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.353597] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.359156] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.365324] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.370450] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.375579] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.382091] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.387567] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.393387] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.399556] [<ffffffff81070a99>] ? load_balance+0x1e9/0x1560 [ 724.405293] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.410681] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.416762] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.422582] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.428926] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 724.434832] [<ffffffff810e9800>] __page_cache_alloc+0x10/0x20 [ 724.440653] [<ffffffff810e9ca1>] find_or_create_page+0x51/0xb0 [ 724.446560] [<ffffffff8116067f>] __getblk+0xdf/0x270 [ 724.451603] [<ffffffff811ff807>] jbd2_journal_get_descriptor_buffer+0x47/0xa0 [ 724.458809] [<ffffffff811fb5d7>] jbd2_journal_commit_transaction+0x1247/0x1400 [ 724.466099] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.472528] [<ffffffff811ffcf7>] kjournald2+0xb7/0x230 [ 724.477743] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.484172] [<ffffffff811ffc40>] ? kjournald2+0x0/0x230 [ 724.489474] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 724.494343] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 724.500248] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 724.505206] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 724.511286] INFO: task cron:2380 blocked for more than 120 seconds. [ 724.517537] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.525346] cron D 00000001000126d4 0 2380 1 0x00000000 [ 724.532418] ffff88083ea235b8 0000000000000082 00e8c03100000000 0000000000011c40 [ 724.539871] ffff88083f86d520 ffff88083f86d1c0 ffff880840350840 90b88d4800000004 [ 724.547327] ffff88083ea23518 0000000000000046 00c7c748c68948da 0000000000000001 [ 724.554782] Call Trace: [ 724.557226] [<ffffffff812ecb9c>] ? cfq_may_queue+0x2c/0xe0 [ 724.562789] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.568006] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.573741] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.580166] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.585554] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.591114] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.597281] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.602409] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.607538] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.614053] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.619528] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.625350] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.631515] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.636903] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.642985] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.648805] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.655148] [<ffffffff8112703c>] kmem_getpages+0x6c/0x160 [ 724.660621] [<ffffffff81127689>] fallback_alloc+0x119/0x1c0 [ 724.666269] [<ffffffff811277c5>] ____cache_alloc_node+0x95/0x150 [ 724.672349] [<ffffffff811281a3>] kmem_cache_alloc+0x103/0x170 [ 724.678170] [<ffffffff811439cb>] getname+0x3b/0x230 [ 724.683124] [<ffffffff81145b4a>] user_path_at+0x2a/0xa0 [ 724.688425] [<ffffffff812f68b0>] ? timerqueue_add+0x60/0xb0 [ 724.694075] [<ffffffff8109c891>] ? lock_hrtimer_base+0x31/0x60 [ 724.699983] [<ffffffff8109c961>] ? hrtimer_try_to_cancel+0x51/0xd0 [ 724.706236] [<ffffffff8109c9fd>] ? hrtimer_cancel+0x1d/0x30 [ 724.711887] [<ffffffff8113ba0a>] vfs_fstatat+0x4a/0x80 [ 724.717100] [<ffffffff8113ba7b>] vfs_stat+0x1b/0x20 [ 724.722055] [<ffffffff8113bc64>] sys_newstat+0x24/0x50 [ 724.727270] [<ffffffff8109cdb7>] ? sys_nanosleep+0x77/0x80 [ 724.732832] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 724.738829] INFO: task sshd:2536 blocked for more than 120 seconds. [ 724.745080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.752890] sshd D 0000000100010811 0 2536 2508 0x00000000 [ 724.759959] ffff88083ddd9438 0000000000000082 ffff88083ddd9378 0000000000011c40 [ 724.767413] ffff88083fb0ebe0 ffff88083fb0e880 ffff88084039d880 ffff881000000008 [ 724.774870] 0000000000000000 0000000000000000 0000000000000020 0000000000000046 [ 724.782324] Call Trace: [ 724.784769] [<ffffffff812e25ab>] ? alloc_io_context+0x1b/0x80 [ 724.790591] [<ffffffff812e263f>] ? current_io_context+0x2f/0x40 [ 724.796586] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 724.801801] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 724.807536] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 724.813961] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 724.819351] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 724.824914] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 724.831080] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 724.836210] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 724.841337] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 724.847850] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 724.853325] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 724.859147] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 724.865315] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 724.872001] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 724.877388] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 724.883466] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 724.889290] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 724.895632] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 724.901539] [<ffffffff81063a4b>] pte_alloc_one+0x1b/0x50 [ 724.906926] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 724.911969] [<ffffffff81109fa7>] __pte_alloc+0x27/0x110 [ 724.917270] [<ffffffff8110a19f>] handle_mm_fault+0x10f/0x270 [ 724.923004] [<ffffffff815f52f1>] do_page_fault+0x161/0x480 [ 724.928567] [<ffffffff815f1ea4>] ? _raw_spin_unlock_bh+0x14/0x20 [ 724.934646] [<ffffffff81518519>] ? release_sock+0xd9/0x110 [ 724.940210] [<ffffffff81559339>] ? tcp_sendmsg+0x819/0xb80 [ 724.945770] [<ffffffff815f251f>] page_fault+0x1f/0x30 [ 724.950897] [<ffffffff812f992d>] ? copy_user_generic_string+0x2d/0x40 [ 724.957411] [<ffffffff81149243>] ? core_sys_select+0x1e3/0x2c0 [ 724.963317] [<ffffffff81137585>] ? do_sync_write+0xd5/0x120 [ 724.968968] [<ffffffff8116aa05>] ? fsnotify+0x1d5/0x2e0 [ 724.974272] [<ffffffff8114953b>] sys_select+0x4b/0x100 [ 724.979485] [<ffffffff811383f5>] ? sys_write+0x55/0x90 [ 724.984701] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 724.990696] INFO: task qemu-system-x86:2731 blocked for more than 120 seconds. [ 724.997900] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.005710] qemu-system-x86 D 000000010000f685 0 2731 2692 0x00000000 [ 725.012780] ffff88083fa472c8 0000000000000082 0000000000000001 0000000000011c40 [ 725.020237] ffff88083fa693e0 ffff88083fa69080 ffff88084039d880 ffffffff00000008 [ 725.027692] ffff88083fac94f0 000000103fa47228 ffff88083fb947b0 ffff88083d891308 [ 725.035146] Call Trace: [ 725.037589] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 725.043236] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.048796] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.054010] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.059746] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.066170] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.071560] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.077123] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.083289] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.088415] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.093545] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.100059] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.105535] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.111358] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.117523] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.124211] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 725.129599] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 725.135682] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 725.141503] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 725.147841] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 725.153746] [<ffffffff810ef25e>] __get_free_pages+0xe/0x50 [ 725.159306] [<ffffffff81148285>] __pollwait+0xc5/0x100 [ 725.164521] [<ffffffff815579d8>] tcp_poll+0x28/0x180 [ 725.169564] [<ffffffff815136fd>] sock_poll+0x1d/0x20 [ 725.174604] [<ffffffff81148d93>] do_select+0x3b3/0x680 [ 725.179821] [<ffffffff811481c0>] ? __pollwait+0x0/0x100 [ 725.185125] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.190165] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.195207] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.200250] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.205290] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.210331] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.215371] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.220412] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.225453] [<ffffffff811482c0>] ? pollwake+0x0/0x70 [ 725.230495] [<ffffffff8114921e>] core_sys_select+0x1be/0x2c0 [ 725.236229] [<ffffffff8116f605>] ? signalfd_read+0xd5/0x500 [ 725.241875] [<ffffffff812f68b0>] ? timerqueue_add+0x60/0xb0 [ 725.247522] [<ffffffff81170305>] ? eventfd_ctx_read+0x95/0x190 [ 725.253431] [<ffffffff8107fb39>] ? timespec_add_safe+0x39/0x70 [ 725.259340] [<ffffffff8114953b>] sys_select+0x4b/0x100 [ 725.264556] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 725.270550] INFO: task qemu-system-x86:2733 blocked for more than 120 seconds. [ 725.277755] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.285562] qemu-system-x86 D 000000010000fbd6 0 2733 2692 0x00000000 [ 725.292631] ffff88103b95d238 0000000000000082 0000000000000001 0000000000011c40 [ 725.300086] ffff88103ea603a0 ffff88103ea60040 ffff88083f80b8c0 ffffffff0000000c [ 725.307541] ffff88083fac94f0 000000103b95d198 ffff880102069b60 ffff88083d891308 [ 725.314997] Call Trace: [ 725.317441] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 725.323088] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.328650] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.333865] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.339601] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.346026] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.351415] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.356977] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.363145] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.368274] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.373400] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.379914] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.385386] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.391208] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.397373] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.404060] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 725.409448] [<ffffffff810f70f9>] ? shrink_slab+0x149/0x180 [ 725.415008] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 725.421089] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 725.426909] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 725.433251] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 725.439159] [<ffffffff81063a4b>] pte_alloc_one+0x1b/0x50 [ 725.444548] [<ffffffff81109fa7>] __pte_alloc+0x27/0x110 [ 725.449848] [<ffffffff8110a19f>] handle_mm_fault+0x10f/0x270 [ 725.455582] [<ffffffff8110a44f>] __get_user_pages+0x14f/0x490 [ 725.461406] [<ffffffff81001e69>] ? kvm_read_guest_page+0x39/0x80 [ 725.467487] [<ffffffff8110a7e2>] get_user_pages+0x52/0x60 [ 725.472963] [<ffffffff810642c4>] get_user_pages_fast+0x134/0x160 [ 725.479042] [<ffffffff810029a1>] hva_to_pfn+0xc1/0x280 [ 725.484257] [<ffffffff81002c4f>] __gfn_to_pfn+0xbf/0xe0 [ 725.489560] [<ffffffff81002cca>] gfn_to_pfn_async+0x1a/0x20 [ 725.495208] [<ffffffff8101f6fb>] try_async_pf+0x4b/0x1e0 [ 725.500598] [<ffffffff81001232>] ? gfn_to_memslot+0x12/0x20 [ 725.506244] [<ffffffff810012af>] ? gfn_to_hva+0x1f/0x30 [ 725.511543] [<ffffffff81004296>] ? kvm_host_page_size+0x86/0x90 [ 725.517538] [<ffffffff810241ff>] tdp_page_fault+0xef/0x1d0 [ 725.523100] [<ffffffff8103316a>] ? vmx_set_interrupt_shadow+0x1a/0x50 [ 725.529614] [<ffffffff8102080c>] kvm_mmu_page_fault+0x2c/0x90 [ 725.535435] [<ffffffff81033cc0>] handle_ept_violation+0x80/0x150 [ 725.541515] [<ffffffff8103716a>] vmx_handle_exit+0xba/0x4e0 [ 725.547162] [<ffffffff8102f65f>] ? apic_update_ppr+0x3f/0x90 [ 725.552897] [<ffffffff8109ea74>] ? sched_clock_cpu+0x24/0xf0 [ 725.558632] [<ffffffff810180d1>] kvm_arch_vcpu_ioctl_run+0x4e1/0xe00 [ 725.565057] [<ffffffff81017797>] ? kvm_arch_vcpu_load+0x47/0x120 [ 725.571140] [<ffffffff81006bf2>] kvm_vcpu_ioctl+0x4d2/0x5c0 [ 725.576787] [<ffffffff81003ca0>] ? kvm_vm_ioctl+0xa0/0x3c0 [ 725.582346] [<ffffffff81515cc7>] ? sys_sendto+0x137/0x140 [ 725.587819] [<ffffffff81146e1d>] vfs_ioctl+0x1d/0x50 [ 725.592861] [<ffffffff81146fb5>] do_vfs_ioctl+0xa5/0x4e0 [ 725.598251] [<ffffffff810a8d43>] ? sys_futex+0x83/0x140 [ 725.603553] [<ffffffff8114743f>] sys_ioctl+0x4f/0x80 [ 725.608596] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 725.614587] INFO: task qemu-system-x86:2736 blocked for more than 120 seconds. [ 725.621790] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.629600] qemu-system-x86 D 000000010000f67d 0 2736 2692 0x00000000 [ 725.636669] ffff88102de7f508 0000000000000082 ffff88083d891308 0000000000011c40 [ 725.644128] ffff88103ea5fc20 ffff88103ea5f8c0 ffff88084039d880 ffff880800000008 [ 725.651584] ffffffff8192d4a0 ffffffff811269b8 ffff88102de7f478 ffff88083faaecc0 [ 725.659040] Call Trace: [ 725.661483] [<ffffffff811269b8>] ? transfer_objects+0x58/0x80 [ 725.667304] [<ffffffff811279af>] ? cache_alloc_refill+0x9f/0x260 [ 725.673383] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.678944] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.684159] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.689893] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.696319] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.701708] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.707271] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.713437] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.718568] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.723695] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.730209] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.735685] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.741506] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.747673] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.754360] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 725.759750] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 725.765831] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 725.771653] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 725.778000] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 725.783906] [<ffffffff810e9800>] __page_cache_alloc+0x10/0x20 [ 725.789725] [<ffffffff810f2f82>] __do_page_cache_readahead+0xe2/0x220 [ 725.796236] [<ffffffff810f30e1>] ra_submit+0x21/0x30 [ 725.801278] [<ffffffff810f3288>] ondemand_readahead+0xa8/0x1d0 [ 725.807186] [<ffffffff810f34a6>] page_cache_sync_readahead+0x36/0x50 [ 725.813614] [<ffffffff810ead0a>] generic_file_aio_read+0x57a/0x6f0 [ 725.819865] [<ffffffff8106a72b>] ? __wake_up_common+0x5b/0x90 [ 725.825688] [<ffffffff811376a5>] do_sync_read+0xd5/0x120 [ 725.831075] [<ffffffff8108c0e2>] ? group_send_sig_info+0x52/0x60 [ 725.837157] [<ffffffff8108c13b>] ? kill_pid_info+0x4b/0x70 [ 725.842717] [<ffffffff8108ca44>] ? sys_kill+0x94/0x190 [ 725.847930] [<ffffffff81137ebb>] vfs_read+0xcb/0x170 [ 725.852972] [<ffffffff811384ba>] sys_pread64+0x8a/0x90 [ 725.858186] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 725.864180] INFO: task memknobs:2751 blocked for more than 120 seconds. [ 725.870778] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.878588] memknobs D 000000010000f603 0 2751 2617 0x00000000 [ 725.885656] ffff88083e57d6b8 0000000000000082 0000000000000001 0000000000011c40 [ 725.893111] ffff88083fb0ab20 ffff88083fb0a7c0 ffffffff81925020 ffffffff00000000 [ 725.900567] ffff88083fac94f0 000000103e57d618 ffff88083d8a15e0 ffff88083d891308 [ 725.908021] Call Trace: [ 725.910467] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 725.916115] [<ffffffff812ecbc6>] ? cfq_may_queue+0x56/0xe0 [ 725.921676] [<ffffffff815f0424>] io_schedule+0x44/0x60 [ 725.926890] [<ffffffff812dd13b>] get_request_wait+0xdb/0x170 [ 725.932623] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 725.939050] [<ffffffff812d8e5b>] ? elv_merge+0x17b/0x1d0 [ 725.944437] [<ffffffff812df93e>] __make_request+0x6e/0x480 [ 725.949998] [<ffffffff812db9d2>] generic_make_request+0x282/0x3f0 [ 725.956165] [<ffffffff81164a2d>] ? bio_init+0x1d/0x40 [ 725.961292] [<ffffffff812df807>] submit_bio+0x67/0xe0 [ 725.966418] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 725.972933] [<ffffffff81118cb0>] swap_writepage+0xa0/0xc0 [ 725.978406] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 725.984228] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 725.990394] [<ffffffff810f1cfa>] ? determine_dirtyable_memory+0x1a/0x40 [ 725.997081] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 726.002471] [<ffffffff810fa9db>] do_try_to_free_pages+0x9b/0x300 [ 726.008551] [<ffffffff810fae00>] try_to_free_pages+0x90/0x110 [ 726.014373] [<ffffffff810f02f5>] __alloc_pages_nodemask+0x355/0x730 [ 726.020714] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 726.025845] [<ffffffff81123a9c>] alloc_pages_current+0x8c/0xd0 [ 726.031751] [<ffffffff81063a4b>] pte_alloc_one+0x1b/0x50 [ 726.037141] [<ffffffff81109fa7>] __pte_alloc+0x27/0x110 [ 726.042445] [<ffffffff8110a19f>] handle_mm_fault+0x10f/0x270 [ 726.048179] [<ffffffff815f52f1>] do_page_fault+0x161/0x480 [ 726.053740] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 726.059388] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 726.065207] [<ffffffff815f251f>] page_fault+0x1f/0x30 -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 2011-01-31 20:58 ` Dave Hansen @ 2011-01-31 22:40 ` Dave Hansen -1 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-01-31 22:40 UTC (permalink / raw) To: CAI Qian; +Cc: linux-mm, linux-kernel, aarcange On Mon, 2011-01-31 at 12:58 -0800, Dave Hansen wrote: > On Thu, 2011-01-20 at 03:55 -0500, CAI Qian wrote: > > When running LTP oom01 [1] testing, the allocation process stopped > > processing right after starting to swap. > > I'm seeing the same stuff, but on -rc2. I thought it was > transparent-hugepage-related, but I don't see much of a trace of it in > the stack dumps. > > http://sr71.net/~dave/ibm/config-v2.6.38-rc2 > > It happened to me as well around the time that things started to hit > swap. Still not a very good data point, but I ran a heavy swap load for an hour or so without reproducing this. But, it happened again after I enabled transparent huge pages. I managed to get a sysrq-t dump out of it: http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt khugepaged is one of the three running tasks. Note, I set both its sleep timeouts to zero to stress it out a bit. I'll keep trying to reproduce without THP. -- Dave ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 @ 2011-01-31 22:40 ` Dave Hansen 0 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-01-31 22:40 UTC (permalink / raw) To: CAI Qian; +Cc: linux-mm, linux-kernel, aarcange On Mon, 2011-01-31 at 12:58 -0800, Dave Hansen wrote: > On Thu, 2011-01-20 at 03:55 -0500, CAI Qian wrote: > > When running LTP oom01 [1] testing, the allocation process stopped > > processing right after starting to swap. > > I'm seeing the same stuff, but on -rc2. I thought it was > transparent-hugepage-related, but I don't see much of a trace of it in > the stack dumps. > > http://sr71.net/~dave/ibm/config-v2.6.38-rc2 > > It happened to me as well around the time that things started to hit > swap. Still not a very good data point, but I ran a heavy swap load for an hour or so without reproducing this. But, it happened again after I enabled transparent huge pages. I managed to get a sysrq-t dump out of it: http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt khugepaged is one of the three running tasks. Note, I set both its sleep timeouts to zero to stress it out a bit. I'll keep trying to reproduce without THP. -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 2011-01-31 22:40 ` Dave Hansen @ 2011-01-31 23:13 ` Andrea Arcangeli -1 siblings, 0 replies; 14+ messages in thread From: Andrea Arcangeli @ 2011-01-31 23:13 UTC (permalink / raw) To: Dave Hansen; +Cc: CAI Qian, linux-mm, linux-kernel On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote: > Still not a very good data point, but I ran a heavy swap load for an > hour or so without reproducing this. But, it happened again after I > enabled transparent huge pages. I managed to get a sysrq-t dump out of > it: > > http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt > > khugepaged is one of the three running tasks. Note, I set both its > sleep timeouts to zero to stress it out a bit. sysrq+l? sysrq+t doesn't provide interesting info for running tasks. > I'll keep trying to reproduce without THP. BTW, do you have prove locking enabled? With recent git I get a deadlock inside _raw_spin_unlock_irq with prove locking enabled and it goes away when I disable it. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 @ 2011-01-31 23:13 ` Andrea Arcangeli 0 siblings, 0 replies; 14+ messages in thread From: Andrea Arcangeli @ 2011-01-31 23:13 UTC (permalink / raw) To: Dave Hansen; +Cc: CAI Qian, linux-mm, linux-kernel On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote: > Still not a very good data point, but I ran a heavy swap load for an > hour or so without reproducing this. But, it happened again after I > enabled transparent huge pages. I managed to get a sysrq-t dump out of > it: > > http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt > > khugepaged is one of the three running tasks. Note, I set both its > sleep timeouts to zero to stress it out a bit. sysrq+l? sysrq+t doesn't provide interesting info for running tasks. > I'll keep trying to reproduce without THP. BTW, do you have prove locking enabled? With recent git I get a deadlock inside _raw_spin_unlock_irq with prove locking enabled and it goes away when I disable it. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 2011-01-31 23:13 ` Andrea Arcangeli @ 2011-01-31 23:20 ` Dave Hansen -1 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-01-31 23:20 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: CAI Qian, linux-mm, linux-kernel On Tue, 2011-02-01 at 00:13 +0100, Andrea Arcangeli wrote: > On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote: > > Still not a very good data point, but I ran a heavy swap load for an > > hour or so without reproducing this. But, it happened again after I > > enabled transparent huge pages. I managed to get a sysrq-t dump out of > > it: > > > > http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt > > > > khugepaged is one of the three running tasks. Note, I set both its > > sleep timeouts to zero to stress it out a bit. > > sysrq+l? sysrq+t doesn't provide interesting info for running tasks. I'll try to get a sysrq-l dump. > > I'll keep trying to reproduce without THP. > > BTW, do you have prove locking enabled? With recent git I get a > deadlock inside _raw_spin_unlock_irq with prove locking enabled and it > goes away when I disable it. Nope: $ grep PROVE ../mhp-build/x86_64-elm3b82/.config # CONFIG_PROVE_LOCKING is not set Full .config is here http://sr71.net/~dave/ibm/config-v2.6.38-rc2 -- Dave ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 @ 2011-01-31 23:20 ` Dave Hansen 0 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-01-31 23:20 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: CAI Qian, linux-mm, linux-kernel On Tue, 2011-02-01 at 00:13 +0100, Andrea Arcangeli wrote: > On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote: > > Still not a very good data point, but I ran a heavy swap load for an > > hour or so without reproducing this. But, it happened again after I > > enabled transparent huge pages. I managed to get a sysrq-t dump out of > > it: > > > > http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt > > > > khugepaged is one of the three running tasks. Note, I set both its > > sleep timeouts to zero to stress it out a bit. > > sysrq+l? sysrq+t doesn't provide interesting info for running tasks. I'll try to get a sysrq-l dump. > > I'll keep trying to reproduce without THP. > > BTW, do you have prove locking enabled? With recent git I get a > deadlock inside _raw_spin_unlock_irq with prove locking enabled and it > goes away when I disable it. Nope: $ grep PROVE ../mhp-build/x86_64-elm3b82/.config # CONFIG_PROVE_LOCKING is not set Full .config is here http://sr71.net/~dave/ibm/config-v2.6.38-rc2 -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 2011-01-31 23:13 ` Andrea Arcangeli @ 2011-02-01 1:43 ` Dave Hansen -1 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-02-01 1:43 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: CAI Qian, linux-mm, linux-kernel On Tue, 2011-02-01 at 00:13 +0100, Andrea Arcangeli wrote: > On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote: > > Still not a very good data point, but I ran a heavy swap load for an > > hour or so without reproducing this. But, it happened again after I > > enabled transparent huge pages. I managed to get a sysrq-t dump out of > > it: > > > > http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt > > > > khugepaged is one of the three running tasks. Note, I set both its > > sleep timeouts to zero to stress it out a bit. > > sysrq+l? sysrq+t doesn't provide interesting info for running tasks. Got one again. Took ~3 hours to reproduce this time, but I don't think I was hitting swap very hard most of the time. Here's sysrq+l pasted. sysrq-t is here: http://sr71.net/~dave/ibm/2.6.38-rc2-hang-1.txt [ 9834.262374] SysRq : Show backtrace of all active CPUs [ 9834.267437] CPU0: [ 9834.269360] CPU 0 [ 9834.271197] Modules linked in: [ 9834.274455] [ 9834.275948] Pid: 0, comm: swapper Not tainted 2.6.38-rc2-dirty #119 49Y6512 /System x3650 M2 -[7947AC1]- [ 9834.285748] RIP: 0010:[<ffffffff81348d3d>] [<ffffffff81348d3d>] acpi_idle_enter_bm+0x22e/0x265 [ 9834.294443] RSP: 0000:ffffffff818c9ec8 EFLAGS: 00000246 [ 9834.299742] RAX: 00000000000009db RBX: ffffffff8109dbce RCX: 00000000000003e8 [ 9834.306857] RDX: 0000000000000119 RSI: 0000000000000001 RDI: 0000000000000000 [ 9834.313971] RBP: ffffffff818c9ef8 R08: 0000000000000001 R09: 0000000000000020 [ 9834.321086] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff815f1ece [ 9834.328201] R13: ffff88083e786800 R14: ffff88083e786820 R15: ffff88083e78680c [ 9834.335317] FS: 0000000000000000(0000) GS:ffff88007a800000(0000) knlGS:0000000000000000 [ 9834.343386] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 9834.349116] CR2: 00007f9b2eaf5720 CR3: 000000000191b000 CR4: 00000000000006f0 [ 9834.356231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9834.363346] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9834.370461] Process swapper (pid: 0, threadinfo ffffffff818c8000, task ffffffff81923020) [ 9834.378530] Stack: [ 9834.380540] ffffffff818c9ef8 00000000814f76b6 ffff88083e786820 ffff88083e7868f0 [ 9834.387992] 0000000000000000 0000000000000002 ffffffff818c9f38 ffffffff814f68a9 [ 9834.395442] ffffffff818c9f18 ffffffff819aaf80 6db6db6db6db6db7 000000000000001f [ 9834.402893] Call Trace: [ 9834.405338] [<ffffffff814f68a9>] cpuidle_idle_call+0xb9/0x210 [ 9834.411159] [<ffffffff81039c87>] cpu_idle+0x47/0x80 [ 9834.416113] [<ffffffff815c61f2>] rest_init+0x72/0x80 [ 9834.421152] [<ffffffff819c3dff>] start_kernel+0x2ff/0x3e0 [ 9834.426626] [<ffffffff819c3271>] x86_64_start_reservations+0x81/0xd0 [ 9834.433049] [<ffffffff819c339d>] x86_64_start_kernel+0xdd/0x100 [ 9834.439041] Code: 89 d5 ff 48 89 c6 ba e8 03 00 00 48 29 de 48 89 d1 31 d2 48 89 f0 48 89 f7 48 f7 f1 48 89 c3 e8 ba 5e d5 ff fb 41 80 7c 24 08 01 <74> 10 65 48 8b 04 25 08 b7 00 00 83 88 3c e0 ff ff 04 41 ff 44 [ 9834.459137] Call Trace: [ 9834.461580] [<ffffffff814f68a9>] cpuidle_idle_call+0xb9/0x210 [ 9834.467399] [<ffffffff81039c87>] cpu_idle+0x47/0x80 [ 9834.472353] [<ffffffff815c61f2>] rest_init+0x72/0x80 [ 9834.477391] [<ffffffff819c3dff>] start_kernel+0x2ff/0x3e0 [ 9834.482864] [<ffffffff819c3271>] x86_64_start_reservations+0x81/0xd0 [ 9834.489287] [<ffffffff819c339d>] x86_64_start_kernel+0xdd/0x100 [ 9834.495337] CPU4: [ 9834.497266] ffff88107fc03ef8 0000000000000046 0000000000000004 ffffffff8135edc0 [ 9834.504721] ffff88087fffc2d8 ffff88083e067bd0 ffff88107fc03f38 ffffffff8103e74a [ 9834.512175] ffff88107fc03f58 ffffffff8135ee0d ffff88107fc03f68 ffff88007a812580 [ 9834.519633] Call Trace: [ 9834.522077] <IRQ> [<ffffffff8135edc0>] ? showacpu+0x0/0x70 [ 9834.527747] [<ffffffff8103e74a>] ? show_stack+0x1a/0x20 [ 9834.533049] [<ffffffff8135ee0d>] ? showacpu+0x4d/0x70 [ 9834.538179] [<ffffffff810aabcb>] ? generic_smp_call_function_interrupt+0x7b/0x170 [ 9834.545733] [<ffffffff81053c27>] ? smp_call_function_interrupt+0x27/0x40 [ 9834.552505] [<ffffffff8103b7d3>] ? call_function_interrupt+0x13/0x20 [ 9834.558927] <EOI> [<ffffffff810f99cd>] ? shrink_zone+0x16d/0x4c0 [ 9834.565113] [<ffffffff810fa9db>] ? do_try_to_free_pages+0x9b/0x300 [ 9834.571365] [<ffffffff810fae00>] ? try_to_free_pages+0x90/0x110 [ 9834.577360] [<ffffffff810f02f5>] ? __alloc_pages_nodemask+0x355/0x730 [ 9834.583873] [<ffffffff815f018e>] ? schedule_timeout+0x11e/0x250 [ 9834.589866] [<ffffffff811223e2>] ? alloc_pages_vma+0x82/0x130 [ 9834.595687] [<ffffffff8112de30>] ? khugepaged+0x8c0/0x10c0 [ 9834.601249] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9834.607673] [<ffffffff8112d570>] ? khugepaged+0x0/0x10c0 [ 9834.613059] [<ffffffff81098c77>] ? kthread+0x97/0xa0 [ 9834.618099] [<ffffffff8103ba54>] ? kernel_thread_helper+0x4/0x10 [ 9834.624176] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9834.629132] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 And the hung task messages: [ 9687.185858] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 9687.192291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.200102] kswapd0 D 000000010022f817 0 706 2 0x00000000 [ 9687.207173] ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40 [ 9687.214627] ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001 [ 9687.222080] ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308 [ 9687.229532] Call Trace: [ 9687.231986] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9687.237634] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9687.243201] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.248421] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9687.254154] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9687.260581] [<ffffffff812d8927>] ? elv_merge+0x47/0x1d0 [ 9687.265882] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9687.271446] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9687.277612] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9687.282741] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9687.287869] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9687.294386] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9687.299860] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9687.305682] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9687.311849] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9687.317239] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9687.323579] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9687.328534] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9687.333488] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9687.338362] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9687.344269] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9687.349225] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9687.355303] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 9687.361725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.369537] kswapd1 D 000000010022f812 0 707 2 0x00000000 [ 9687.376609] ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40 [ 9687.384062] ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c [ 9687.391518] ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308 [ 9687.398971] Call Trace: [ 9687.401414] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9687.407062] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9687.412625] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.417842] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9687.423576] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9687.430002] [<ffffffff812d8a5b>] ? elv_merge+0x17b/0x1d0 [ 9687.435392] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9687.440952] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9687.447121] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9687.452251] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9687.457377] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9687.463889] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9687.469364] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9687.475183] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9687.481349] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9687.486741] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9687.493082] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9687.498038] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9687.502992] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9687.507862] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9687.513768] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9687.518723] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9687.524805] INFO: task memknobs:2741 blocked for more than 120 seconds. [ 9687.531403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.539211] memknobs D 000000010022f7f5 0 2741 2674 0x00000000 [ 9687.546282] ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9687.553735] ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e [ 9687.561189] 0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2 [ 9687.568644] Call Trace: [ 9687.571089] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9687.577429] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9687.582556] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.587769] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9687.592811] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9687.598197] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9687.603324] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9687.608973] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9687.614795] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9687.621048] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9687.627043] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9687.632864] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9687.638340] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9687.643813] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9687.649547] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9687.655108] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 9687.660758] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 9687.666578] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9687.671708] INFO: task memknobs:2794 blocked for more than 120 seconds. [ 9687.678308] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.686117] memknobs D 000000010022f90a 0 2794 2724 0x00000000 [ 9687.693187] ffff880c66627c68 0000000000000082 ffff88083faa8f40 0000000000011c40 [ 9687.700639] ffff88103e26cae0 ffff88103e26c780 ffffffff81923020 ffff880800000000 [ 9687.708095] 0000000000000008 0000000000821ba7 ffff880c66627c38 ffffffff812db5d2 [ 9687.715551] Call Trace: [ 9687.717998] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9687.724338] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.729555] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9687.734597] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9687.739984] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9687.745112] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9687.750759] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9687.756578] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9687.762830] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9687.768824] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9687.774645] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9687.780118] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9687.785591] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9687.791324] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9687.796884] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 9687.802530] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 9687.808351] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9807.612419] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 9807.618845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.626656] kswapd0 D 000000010022f817 0 706 2 0x00000000 [ 9807.633728] ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40 [ 9807.641187] ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001 [ 9807.648640] ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308 [ 9807.656097] Call Trace: [ 9807.658544] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9807.664190] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9807.669754] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9807.674969] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9807.680705] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9807.687131] [<ffffffff812d8927>] ? elv_merge+0x47/0x1d0 [ 9807.692432] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9807.697995] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9807.704160] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9807.709288] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9807.714417] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9807.720932] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9807.726406] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9807.732231] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9807.738396] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9807.743788] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9807.750127] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9807.755081] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9807.760037] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9807.764908] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9807.770814] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9807.775770] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9807.781851] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 9807.788277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.796084] kswapd1 D 000000010022f812 0 707 2 0x00000000 [ 9807.803158] ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40 [ 9807.810615] ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c [ 9807.818070] ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308 [ 9807.825524] Call Trace: [ 9807.827967] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9807.833614] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9807.839176] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9807.844393] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9807.850126] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9807.856551] [<ffffffff812d8a5b>] ? elv_merge+0x17b/0x1d0 [ 9807.861940] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9807.867503] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9807.873669] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9807.878796] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9807.883926] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9807.890439] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9807.895915] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9807.901737] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9807.907905] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9807.913335] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9807.919675] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9807.924632] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9807.929588] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9807.934457] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9807.940365] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9807.945321] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9807.951404] INFO: task avahi-daemon:2353 blocked for more than 120 seconds. [ 9807.958346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.966155] avahi-daemon D 00000001002332a4 0 2353 1 0x00000000 [ 9807.973227] ffff88103f5437a8 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9807.980682] ffff88103eb764e0 ffff88103eb76180 ffffffff81923020 ffff880800000000 [ 9807.988134] 0000000000000008 00000000007e89ff ffff88103f543778 ffffffff812db5d2 [ 9807.995587] Call Trace: [ 9807.998034] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.004374] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.009501] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.014716] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.019759] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.025146] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.030273] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.035922] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.041745] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.047999] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.053994] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.059814] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.065290] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.070766] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.076500] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.082063] [<ffffffff812f4eba>] ? rb_insert_color+0x8a/0xf0 [ 9808.087800] [<ffffffff812f64b0>] ? timerqueue_add+0x60/0xb0 [ 9808.093449] [<ffffffff8109c891>] ? lock_hrtimer_base+0x31/0x60 [ 9808.099356] [<ffffffff8109c961>] ? hrtimer_try_to_cancel+0x51/0xd0 [ 9808.105610] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9808.110741] [<ffffffff8114848c>] ? do_sys_poll+0x3ac/0x460 [ 9808.116301] [<ffffffff81148450>] ? do_sys_poll+0x370/0x460 [ 9808.121863] [<ffffffff81147e80>] ? __pollwait+0x0/0x100 [ 9808.127164] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.132206] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.137248] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.142291] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.147331] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.152373] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.157416] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.162457] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.167530] [<ffffffff81137365>] ? do_sync_read+0xd5/0x120 [ 9808.173093] [<ffffffff8107fb39>] ? timespec_add_safe+0x39/0x70 [ 9808.179000] [<ffffffff81147d34>] ? poll_select_set_timeout+0x84/0xa0 [ 9808.185426] [<ffffffff811485bc>] sys_poll+0x7c/0x100 [ 9808.190469] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 9808.196463] INFO: task rsyslogd:2523 blocked for more than 120 seconds. [ 9808.203061] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.210872] rsyslogd D 0000000100237de3 0 2523 1 0x00000000 [ 9808.217944] ffff88083ec11a88 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.225403] ffff88083f93a460 ffff88083f93a100 ffffffff81923020 ffff880800000000 [ 9808.232858] 0000000000000008 00000000006f52cf ffff88083ec11a58 ffffffff812db5d2 [ 9808.240311] Call Trace: [ 9808.242754] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.249092] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.254220] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.259571] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.264616] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.270002] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.275131] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.280780] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.286600] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.292855] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.298849] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.304668] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.310142] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.315618] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.321352] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.326913] [<ffffffff81039996>] ? __switch_to+0x1e6/0x2a0 [ 9808.332472] [<ffffffff815ef808>] ? schedule+0x3f8/0xaf0 [ 9808.337775] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9808.342904] [<ffffffff8107bd47>] ? do_syslog+0x347/0x4e0 [ 9808.348292] [<ffffffff8107bd02>] ? do_syslog+0x302/0x4e0 [ 9808.353681] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9808.360107] [<ffffffff81193f36>] kmsg_read+0x36/0x70 [ 9808.365149] [<ffffffff8118a324>] proc_reg_read+0x74/0xb0 [ 9808.370538] [<ffffffff81137b7b>] vfs_read+0xcb/0x170 [ 9808.375581] [<ffffffff81138025>] sys_read+0x55/0x90 [ 9808.380539] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 9808.386532] INFO: task memknobs:2741 blocked for more than 120 seconds. [ 9808.393133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.400942] memknobs D 000000010022f7f5 0 2741 2674 0x00000000 [ 9808.408014] ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.415471] ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e [ 9808.422926] 0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2 [ 9808.430377] Call Trace: [ 9808.432824] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.439165] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.444295] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.449511] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.454553] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.459943] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.465070] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.470715] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.476535] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.482788] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.488781] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.494604] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.500077] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.505549] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.511284] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.516848] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 9808.522498] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 9808.528318] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9808.533449] INFO: task sshd:2755 blocked for more than 120 seconds. [ 9808.539700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.547511] sshd D 00000001002310ff 0 2755 2510 0x00000000 [ 9808.554582] ffff880079c27c68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.562039] ffff88083e2f1aa0 ffff88083e2f1740 ffffffff81923020 ffff880800000000 [ 9808.569493] 0000000000000008 00000000007de1b7 ffff880079c27c38 ffffffff812db5d2 [ 9808.576948] Call Trace: [ 9808.579393] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.585731] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.590859] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.596075] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.601115] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.606502] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.611630] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.617277] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.623096] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.629350] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.635345] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.641166] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.646643] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.652118] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.657851] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.663413] [<ffffffff81137bde>] ? vfs_read+0x12e/0x170 [ 9808.668713] [<ffffffff815f20df>] page_fault+0x1f/0x30 -- Dave ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 @ 2011-02-01 1:43 ` Dave Hansen 0 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-02-01 1:43 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: CAI Qian, linux-mm, linux-kernel On Tue, 2011-02-01 at 00:13 +0100, Andrea Arcangeli wrote: > On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote: > > Still not a very good data point, but I ran a heavy swap load for an > > hour or so without reproducing this. But, it happened again after I > > enabled transparent huge pages. I managed to get a sysrq-t dump out of > > it: > > > > http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt > > > > khugepaged is one of the three running tasks. Note, I set both its > > sleep timeouts to zero to stress it out a bit. > > sysrq+l? sysrq+t doesn't provide interesting info for running tasks. Got one again. Took ~3 hours to reproduce this time, but I don't think I was hitting swap very hard most of the time. Here's sysrq+l pasted. sysrq-t is here: http://sr71.net/~dave/ibm/2.6.38-rc2-hang-1.txt [ 9834.262374] SysRq : Show backtrace of all active CPUs [ 9834.267437] CPU0: [ 9834.269360] CPU 0 [ 9834.271197] Modules linked in: [ 9834.274455] [ 9834.275948] Pid: 0, comm: swapper Not tainted 2.6.38-rc2-dirty #119 49Y6512 /System x3650 M2 -[7947AC1]- [ 9834.285748] RIP: 0010:[<ffffffff81348d3d>] [<ffffffff81348d3d>] acpi_idle_enter_bm+0x22e/0x265 [ 9834.294443] RSP: 0000:ffffffff818c9ec8 EFLAGS: 00000246 [ 9834.299742] RAX: 00000000000009db RBX: ffffffff8109dbce RCX: 00000000000003e8 [ 9834.306857] RDX: 0000000000000119 RSI: 0000000000000001 RDI: 0000000000000000 [ 9834.313971] RBP: ffffffff818c9ef8 R08: 0000000000000001 R09: 0000000000000020 [ 9834.321086] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff815f1ece [ 9834.328201] R13: ffff88083e786800 R14: ffff88083e786820 R15: ffff88083e78680c [ 9834.335317] FS: 0000000000000000(0000) GS:ffff88007a800000(0000) knlGS:0000000000000000 [ 9834.343386] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 9834.349116] CR2: 00007f9b2eaf5720 CR3: 000000000191b000 CR4: 00000000000006f0 [ 9834.356231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9834.363346] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9834.370461] Process swapper (pid: 0, threadinfo ffffffff818c8000, task ffffffff81923020) [ 9834.378530] Stack: [ 9834.380540] ffffffff818c9ef8 00000000814f76b6 ffff88083e786820 ffff88083e7868f0 [ 9834.387992] 0000000000000000 0000000000000002 ffffffff818c9f38 ffffffff814f68a9 [ 9834.395442] ffffffff818c9f18 ffffffff819aaf80 6db6db6db6db6db7 000000000000001f [ 9834.402893] Call Trace: [ 9834.405338] [<ffffffff814f68a9>] cpuidle_idle_call+0xb9/0x210 [ 9834.411159] [<ffffffff81039c87>] cpu_idle+0x47/0x80 [ 9834.416113] [<ffffffff815c61f2>] rest_init+0x72/0x80 [ 9834.421152] [<ffffffff819c3dff>] start_kernel+0x2ff/0x3e0 [ 9834.426626] [<ffffffff819c3271>] x86_64_start_reservations+0x81/0xd0 [ 9834.433049] [<ffffffff819c339d>] x86_64_start_kernel+0xdd/0x100 [ 9834.439041] Code: 89 d5 ff 48 89 c6 ba e8 03 00 00 48 29 de 48 89 d1 31 d2 48 89 f0 48 89 f7 48 f7 f1 48 89 c3 e8 ba 5e d5 ff fb 41 80 7c 24 08 01 <74> 10 65 48 8b 04 25 08 b7 00 00 83 88 3c e0 ff ff 04 41 ff 44 [ 9834.459137] Call Trace: [ 9834.461580] [<ffffffff814f68a9>] cpuidle_idle_call+0xb9/0x210 [ 9834.467399] [<ffffffff81039c87>] cpu_idle+0x47/0x80 [ 9834.472353] [<ffffffff815c61f2>] rest_init+0x72/0x80 [ 9834.477391] [<ffffffff819c3dff>] start_kernel+0x2ff/0x3e0 [ 9834.482864] [<ffffffff819c3271>] x86_64_start_reservations+0x81/0xd0 [ 9834.489287] [<ffffffff819c339d>] x86_64_start_kernel+0xdd/0x100 [ 9834.495337] CPU4: [ 9834.497266] ffff88107fc03ef8 0000000000000046 0000000000000004 ffffffff8135edc0 [ 9834.504721] ffff88087fffc2d8 ffff88083e067bd0 ffff88107fc03f38 ffffffff8103e74a [ 9834.512175] ffff88107fc03f58 ffffffff8135ee0d ffff88107fc03f68 ffff88007a812580 [ 9834.519633] Call Trace: [ 9834.522077] <IRQ> [<ffffffff8135edc0>] ? showacpu+0x0/0x70 [ 9834.527747] [<ffffffff8103e74a>] ? show_stack+0x1a/0x20 [ 9834.533049] [<ffffffff8135ee0d>] ? showacpu+0x4d/0x70 [ 9834.538179] [<ffffffff810aabcb>] ? generic_smp_call_function_interrupt+0x7b/0x170 [ 9834.545733] [<ffffffff81053c27>] ? smp_call_function_interrupt+0x27/0x40 [ 9834.552505] [<ffffffff8103b7d3>] ? call_function_interrupt+0x13/0x20 [ 9834.558927] <EOI> [<ffffffff810f99cd>] ? shrink_zone+0x16d/0x4c0 [ 9834.565113] [<ffffffff810fa9db>] ? do_try_to_free_pages+0x9b/0x300 [ 9834.571365] [<ffffffff810fae00>] ? try_to_free_pages+0x90/0x110 [ 9834.577360] [<ffffffff810f02f5>] ? __alloc_pages_nodemask+0x355/0x730 [ 9834.583873] [<ffffffff815f018e>] ? schedule_timeout+0x11e/0x250 [ 9834.589866] [<ffffffff811223e2>] ? alloc_pages_vma+0x82/0x130 [ 9834.595687] [<ffffffff8112de30>] ? khugepaged+0x8c0/0x10c0 [ 9834.601249] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9834.607673] [<ffffffff8112d570>] ? khugepaged+0x0/0x10c0 [ 9834.613059] [<ffffffff81098c77>] ? kthread+0x97/0xa0 [ 9834.618099] [<ffffffff8103ba54>] ? kernel_thread_helper+0x4/0x10 [ 9834.624176] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9834.629132] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 And the hung task messages: [ 9687.185858] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 9687.192291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.200102] kswapd0 D 000000010022f817 0 706 2 0x00000000 [ 9687.207173] ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40 [ 9687.214627] ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001 [ 9687.222080] ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308 [ 9687.229532] Call Trace: [ 9687.231986] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9687.237634] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9687.243201] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.248421] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9687.254154] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9687.260581] [<ffffffff812d8927>] ? elv_merge+0x47/0x1d0 [ 9687.265882] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9687.271446] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9687.277612] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9687.282741] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9687.287869] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9687.294386] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9687.299860] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9687.305682] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9687.311849] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9687.317239] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9687.323579] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9687.328534] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9687.333488] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9687.338362] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9687.344269] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9687.349225] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9687.355303] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 9687.361725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.369537] kswapd1 D 000000010022f812 0 707 2 0x00000000 [ 9687.376609] ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40 [ 9687.384062] ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c [ 9687.391518] ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308 [ 9687.398971] Call Trace: [ 9687.401414] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9687.407062] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9687.412625] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.417842] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9687.423576] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9687.430002] [<ffffffff812d8a5b>] ? elv_merge+0x17b/0x1d0 [ 9687.435392] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9687.440952] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9687.447121] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9687.452251] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9687.457377] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9687.463889] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9687.469364] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9687.475183] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9687.481349] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9687.486741] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9687.493082] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9687.498038] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9687.502992] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9687.507862] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9687.513768] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9687.518723] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9687.524805] INFO: task memknobs:2741 blocked for more than 120 seconds. [ 9687.531403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.539211] memknobs D 000000010022f7f5 0 2741 2674 0x00000000 [ 9687.546282] ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9687.553735] ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e [ 9687.561189] 0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2 [ 9687.568644] Call Trace: [ 9687.571089] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9687.577429] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9687.582556] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.587769] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9687.592811] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9687.598197] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9687.603324] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9687.608973] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9687.614795] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9687.621048] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9687.627043] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9687.632864] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9687.638340] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9687.643813] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9687.649547] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9687.655108] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 9687.660758] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 9687.666578] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9687.671708] INFO: task memknobs:2794 blocked for more than 120 seconds. [ 9687.678308] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.686117] memknobs D 000000010022f90a 0 2794 2724 0x00000000 [ 9687.693187] ffff880c66627c68 0000000000000082 ffff88083faa8f40 0000000000011c40 [ 9687.700639] ffff88103e26cae0 ffff88103e26c780 ffffffff81923020 ffff880800000000 [ 9687.708095] 0000000000000008 0000000000821ba7 ffff880c66627c38 ffffffff812db5d2 [ 9687.715551] Call Trace: [ 9687.717998] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9687.724338] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9687.729555] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9687.734597] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9687.739984] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9687.745112] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9687.750759] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9687.756578] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9687.762830] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9687.768824] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9687.774645] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9687.780118] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9687.785591] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9687.791324] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9687.796884] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 9687.802530] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 9687.808351] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9807.612419] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 9807.618845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.626656] kswapd0 D 000000010022f817 0 706 2 0x00000000 [ 9807.633728] ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40 [ 9807.641187] ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001 [ 9807.648640] ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308 [ 9807.656097] Call Trace: [ 9807.658544] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9807.664190] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9807.669754] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9807.674969] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9807.680705] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9807.687131] [<ffffffff812d8927>] ? elv_merge+0x47/0x1d0 [ 9807.692432] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9807.697995] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9807.704160] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9807.709288] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9807.714417] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9807.720932] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9807.726406] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9807.732231] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9807.738396] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9807.743788] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9807.750127] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9807.755081] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9807.760037] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9807.764908] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9807.770814] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9807.775770] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9807.781851] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 9807.788277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.796084] kswapd1 D 000000010022f812 0 707 2 0x00000000 [ 9807.803158] ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40 [ 9807.810615] ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c [ 9807.818070] ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308 [ 9807.825524] Call Trace: [ 9807.827967] [<ffffffff810ebb27>] ? mempool_alloc+0x47/0x120 [ 9807.833614] [<ffffffff812ec7c6>] ? cfq_may_queue+0x56/0xe0 [ 9807.839176] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9807.844393] [<ffffffff812dcd3b>] get_request_wait+0xdb/0x170 [ 9807.850126] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9807.856551] [<ffffffff812d8a5b>] ? elv_merge+0x17b/0x1d0 [ 9807.861940] [<ffffffff812df53e>] __make_request+0x6e/0x480 [ 9807.867503] [<ffffffff812db5d2>] generic_make_request+0x282/0x3f0 [ 9807.873669] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9807.878796] [<ffffffff812df407>] submit_bio+0x67/0xe0 [ 9807.883926] [<ffffffff810f2955>] ? test_set_page_writeback+0xc5/0x180 [ 9807.890439] [<ffffffff81118ca0>] swap_writepage+0xa0/0xc0 [ 9807.895915] [<ffffffff810f7a7c>] shrink_page_list+0x45c/0x800 [ 9807.901737] [<ffffffff810f95be>] shrink_inactive_list+0x11e/0x3c0 [ 9807.907905] [<ffffffff810f9af1>] shrink_zone+0x291/0x4c0 [ 9807.913335] [<ffffffff810eff86>] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9807.919675] [<ffffffff810fa575>] kswapd+0x605/0x8a0 [ 9807.924632] [<ffffffff810f9f70>] ? kswapd+0x0/0x8a0 [ 9807.929588] [<ffffffff81098c77>] kthread+0x97/0xa0 [ 9807.934457] [<ffffffff8103ba54>] kernel_thread_helper+0x4/0x10 [ 9807.940365] [<ffffffff81098be0>] ? kthread+0x0/0xa0 [ 9807.945321] [<ffffffff8103ba50>] ? kernel_thread_helper+0x0/0x10 [ 9807.951404] INFO: task avahi-daemon:2353 blocked for more than 120 seconds. [ 9807.958346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.966155] avahi-daemon D 00000001002332a4 0 2353 1 0x00000000 [ 9807.973227] ffff88103f5437a8 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9807.980682] ffff88103eb764e0 ffff88103eb76180 ffffffff81923020 ffff880800000000 [ 9807.988134] 0000000000000008 00000000007e89ff ffff88103f543778 ffffffff812db5d2 [ 9807.995587] Call Trace: [ 9807.998034] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.004374] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.009501] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.014716] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.019759] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.025146] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.030273] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.035922] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.041745] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.047999] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.053994] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.059814] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.065290] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.070766] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.076500] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.082063] [<ffffffff812f4eba>] ? rb_insert_color+0x8a/0xf0 [ 9808.087800] [<ffffffff812f64b0>] ? timerqueue_add+0x60/0xb0 [ 9808.093449] [<ffffffff8109c891>] ? lock_hrtimer_base+0x31/0x60 [ 9808.099356] [<ffffffff8109c961>] ? hrtimer_try_to_cancel+0x51/0xd0 [ 9808.105610] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9808.110741] [<ffffffff8114848c>] ? do_sys_poll+0x3ac/0x460 [ 9808.116301] [<ffffffff81148450>] ? do_sys_poll+0x370/0x460 [ 9808.121863] [<ffffffff81147e80>] ? __pollwait+0x0/0x100 [ 9808.127164] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.132206] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.137248] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.142291] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.147331] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.152373] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.157416] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.162457] [<ffffffff81147f80>] ? pollwake+0x0/0x70 [ 9808.167530] [<ffffffff81137365>] ? do_sync_read+0xd5/0x120 [ 9808.173093] [<ffffffff8107fb39>] ? timespec_add_safe+0x39/0x70 [ 9808.179000] [<ffffffff81147d34>] ? poll_select_set_timeout+0x84/0xa0 [ 9808.185426] [<ffffffff811485bc>] sys_poll+0x7c/0x100 [ 9808.190469] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 9808.196463] INFO: task rsyslogd:2523 blocked for more than 120 seconds. [ 9808.203061] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.210872] rsyslogd D 0000000100237de3 0 2523 1 0x00000000 [ 9808.217944] ffff88083ec11a88 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.225403] ffff88083f93a460 ffff88083f93a100 ffffffff81923020 ffff880800000000 [ 9808.232858] 0000000000000008 00000000006f52cf ffff88083ec11a58 ffffffff812db5d2 [ 9808.240311] Call Trace: [ 9808.242754] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.249092] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.254220] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.259571] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.264616] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.270002] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.275131] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.280780] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.286600] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.292855] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.298849] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.304668] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.310142] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.315618] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.321352] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.326913] [<ffffffff81039996>] ? __switch_to+0x1e6/0x2a0 [ 9808.332472] [<ffffffff815ef808>] ? schedule+0x3f8/0xaf0 [ 9808.337775] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9808.342904] [<ffffffff8107bd47>] ? do_syslog+0x347/0x4e0 [ 9808.348292] [<ffffffff8107bd02>] ? do_syslog+0x302/0x4e0 [ 9808.353681] [<ffffffff81099150>] ? autoremove_wake_function+0x0/0x40 [ 9808.360107] [<ffffffff81193f36>] kmsg_read+0x36/0x70 [ 9808.365149] [<ffffffff8118a324>] proc_reg_read+0x74/0xb0 [ 9808.370538] [<ffffffff81137b7b>] vfs_read+0xcb/0x170 [ 9808.375581] [<ffffffff81138025>] sys_read+0x55/0x90 [ 9808.380539] [<ffffffff8103ad2b>] system_call_fastpath+0x16/0x1b [ 9808.386532] INFO: task memknobs:2741 blocked for more than 120 seconds. [ 9808.393133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.400942] memknobs D 000000010022f7f5 0 2741 2674 0x00000000 [ 9808.408014] ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.415471] ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e [ 9808.422926] 0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2 [ 9808.430377] Call Trace: [ 9808.432824] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.439165] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.444295] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.449511] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.454553] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.459943] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.465070] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.470715] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.476535] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.482788] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.488781] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.494604] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.500077] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.505549] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.511284] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.516848] [<ffffffff8108ae63>] ? do_sigaction+0x103/0x1a0 [ 9808.522498] [<ffffffff8108d9e9>] ? sys_rt_sigaction+0x89/0xc0 [ 9808.528318] [<ffffffff815f20df>] page_fault+0x1f/0x30 [ 9808.533449] INFO: task sshd:2755 blocked for more than 120 seconds. [ 9808.539700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.547511] sshd D 00000001002310ff 0 2755 2510 0x00000000 [ 9808.554582] ffff880079c27c68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.562039] ffff88083e2f1aa0 ffff88083e2f1740 ffffffff81923020 ffff880800000000 [ 9808.569493] 0000000000000008 00000000007de1b7 ffff880079c27c38 ffffffff812db5d2 [ 9808.576948] Call Trace: [ 9808.579393] [<ffffffff812db5d2>] ? generic_make_request+0x282/0x3f0 [ 9808.585731] [<ffffffff811646ed>] ? bio_init+0x1d/0x40 [ 9808.590859] [<ffffffff815effe4>] io_schedule+0x44/0x60 [ 9808.596075] [<ffffffff810e9525>] sync_page+0x45/0x60 [ 9808.601115] [<ffffffff815f04be>] __wait_on_bit+0x5e/0x90 [ 9808.606502] [<ffffffff810e94e0>] ? sync_page+0x0/0x60 [ 9808.611630] [<ffffffff810e9724>] wait_on_page_bit+0x74/0x80 [ 9808.617277] [<ffffffff81099190>] ? wake_bit_function+0x0/0x40 [ 9808.623096] [<ffffffff812f3abe>] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.629350] [<ffffffff810e97dc>] __lock_page_or_retry+0x4c/0x60 [ 9808.635345] [<ffffffff81108adf>] handle_pte_fault+0x79f/0x7c0 [ 9808.641166] [<ffffffff810ef167>] ? __free_pages+0x27/0x30 [ 9808.646643] [<ffffffff8110a031>] ? __pte_alloc+0xb1/0x110 [ 9808.652118] [<ffffffff8110a1f2>] handle_mm_fault+0x162/0x270 [ 9808.657851] [<ffffffff815f4eb1>] do_page_fault+0x161/0x480 [ 9808.663413] [<ffffffff81137bde>] ? vfs_read+0x12e/0x170 [ 9808.668713] [<ffffffff815f20df>] page_fault+0x1f/0x30 -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 2011-01-31 23:13 ` Andrea Arcangeli @ 2011-02-01 17:17 ` Dave Hansen -1 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-02-01 17:17 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: CAI Qian, linux-mm, linux-kernel Andrea, One more data point: Same kernel binary, but with THP disabled in sysfs, it ran all night, no problems. I'll go dig in to it a bit more. -- Dave ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kswapd hung tasks in 2.6.38-rc1 @ 2011-02-01 17:17 ` Dave Hansen 0 siblings, 0 replies; 14+ messages in thread From: Dave Hansen @ 2011-02-01 17:17 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: CAI Qian, linux-mm, linux-kernel Andrea, One more data point: Same kernel binary, but with THP disabled in sysfs, it ran all night, no problems. I'll go dig in to it a bit more. -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2011-02-01 17:17 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <2056664954.83361.1295513496286.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com>
2011-01-20 8:55 ` kswapd hung tasks in 2.6.38-rc1 CAI Qian
2011-01-20 8:55 ` CAI Qian
2011-01-31 20:58 ` Dave Hansen
2011-01-31 20:58 ` Dave Hansen
2011-01-31 22:40 ` Dave Hansen
2011-01-31 22:40 ` Dave Hansen
2011-01-31 23:13 ` Andrea Arcangeli
2011-01-31 23:13 ` Andrea Arcangeli
2011-01-31 23:20 ` Dave Hansen
2011-01-31 23:20 ` Dave Hansen
2011-02-01 1:43 ` Dave Hansen
2011-02-01 1:43 ` Dave Hansen
2011-02-01 17:17 ` Dave Hansen
2011-02-01 17:17 ` Dave Hansen
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.