From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with SMTP id 3FAF08D003A for ; Thu, 20 Jan 2011 03:55:50 -0500 (EST) Date: Thu, 20 Jan 2011 03:55:48 -0500 (EST) From: CAI Qian Message-ID: <1150342867.83404.1295513748640.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> In-Reply-To: <2056664954.83361.1295513496286.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> Subject: kswapd hung tasks in 2.6.38-rc1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: linux-mm Cc: linux-kernel@vger.kernel.org List-ID: When running LTP oom01 [1] testing, the allocation process stopped processing right after starting to swap. # free -m total used free shared buffers cached Mem: 64307 64059 248 0 4 61 -/+ buffers/cache: 63993 314 Swap: 66139 754 65385 # ./oom01 oom01 0 TINFO : start testing overcommit_memory=2. oom01 0 TINFO : expected victim is 7276. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. oom01 0 TINFO : allocating 3221225472 bytes. INFO: task kswapd0:274 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd0 D ffff88045dd07000 0 274 2 0x00000000 ffff88045dd0d810 0000000000000046 0000000000000000 ffff88002e366e40 0000000000014d40 ffff88045dd06a70 ffff88045dd07000 ffff88045dd0dfd8 ffff88045dd07008 0000000000014d40 ffff88045dd0c010 0000000000014d40 Call Trace: [] io_schedule+0x70/0xc0 [] get_request_wait+0xc5/0x190 [] ? autoremove_wake_function+0x0/0x40 [] ? elv_merge+0x1d7/0x210 [] __make_request+0x6b/0x4c0 [] generic_make_request+0x2ea/0x5d0 [] ? native_flush_tlb_others+0x76/0x90 [] submit_bio+0x86/0x110 [] ? test_set_page_writeback+0x106/0x190 [] swap_writepage+0x83/0xd0 [] pageout+0x12e/0x310 [] shrink_page_list+0x41a/0x5a0 [] shrink_inactive_list+0x172/0x460 [] ? determine_dirtyable_memory+0x1a/0x30 [] shrink_zone+0x36b/0x520 [] ? zone_watermark_ok_safe+0xb5/0xd0 [] kswapd+0x969/0xc20 [] ? kswapd+0x0/0xc20 [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task kswapd2:276 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd2 D ffff88045dd13a70 0 276 2 0x00000000 ffff88045dd15810 0000000000000046 0000000000000000 ffff880c5fa63240 0000000000014d40 ffff88045dd134e0 ffff88045dd13a70 ffff88045dd15fd8 ffff88045dd13a78 0000000000014d40 ffff88045dd14010 0000000000014d40 Call Trace: [] io_schedule+0x70/0xc0 [] get_request_wait+0xc5/0x190 [] ? autoremove_wake_function+0x0/0x40 [] ? elv_merge+0x1d7/0x210 [] __make_request+0x6b/0x4c0 [] generic_make_request+0x2ea/0x5d0 [] ? native_flush_tlb_others+0x76/0x90 [] submit_bio+0x86/0x110 [] ? test_set_page_writeback+0x106/0x190 [] swap_writepage+0x83/0xd0 [] pageout+0x12e/0x310 [] shrink_page_list+0x41a/0x5a0 [] shrink_inactive_list+0x172/0x460 [] ? del_timer_sync+0x3a/0x60 [] shrink_zone+0x36b/0x520 [] ? zone_watermark_ok_safe+0xb5/0xd0 [] kswapd+0x969/0xc20 [] ? kswapd+0x0/0xc20 [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task kswapd3:277 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd3 D ffff88045dd13040 0 277 2 0x00000000 ffff88045dd19810 0000000000000046 0000000000000000 ffff880857ed9d40 0000000000014d40 ffff88045dd12ab0 ffff88045dd13040 ffff88045dd19fd8 ffff88045dd13048 0000000000014d40 ffff88045dd18010 0000000000014d40 Call Trace: [] io_schedule+0x70/0xc0 [] get_request_wait+0xc5/0x190 [] ? autoremove_wake_function+0x0/0x40 [] ? elv_merge+0x47/0x210 [] __make_request+0x6b/0x4c0 [] generic_make_request+0x2ea/0x5d0 [] ? native_flush_tlb_others+0x76/0x90 [] submit_bio+0x86/0x110 [] ? test_set_page_writeback+0x106/0x190 [] swap_writepage+0x83/0xd0 [] pageout+0x12e/0x310 [] shrink_page_list+0x41a/0x5a0 [] shrink_inactive_list+0x172/0x460 [] ? del_timer_sync+0x3a/0x60 [] shrink_zone+0x36b/0x520 [] ? zone_watermark_ok_safe+0xb5/0xd0 [] kswapd+0x969/0xc20 [] ? kswapd+0x0/0xc20 [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task jbd2/dm-0-8:1002 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/dm-0-8 D ffff88045d33d080 0 1002 2 0x00000000 ffff88045d53bc20 0000000000000046 ffff88085f044900 ffff88085f0da4f8 0000000000014d40 ffff88045d33caf0 ffff88045d33d080 ffff88045d53bfd8 ffff88045d33d088 0000000000014d40 ffff88045d53a010 0000000000014d40 Call Trace: [] ? sync_buffer+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_buffer+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? sync_buffer+0x0/0x50 [] out_of_line_wait_on_bit+0x78/0x90 [] ? wake_bit_function+0x0/0x50 [] __wait_on_buffer+0x2e/0x30 [] jbd2_journal_commit_transaction+0x8e8/0x13d0 [jbd2] [] ? try_to_del_timer_sync+0x81/0xe0 [] kjournald2+0xb8/0x220 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] ? kjournald2+0x0/0x220 [jbd2] [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task flush-253:0:2030 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:0 D ffff88045d2a0610 0 2030 2 0x00000000 ffff88045e5ab4f0 0000000000000046 ffff88085f044900 ffff88085f0da4f8 0000000000014d40 ffff88045d2a0080 ffff88045d2a0610 ffff88045e5abfd8 ffff88045d2a0618 0000000000014d40 ffff88045e5aa010 0000000000014d40 Call Trace: [] ? sync_buffer+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_buffer+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? sync_buffer+0x0/0x50 [] out_of_line_wait_on_bit+0x78/0x90 [] ? wake_bit_function+0x0/0x50 [] __wait_on_buffer+0x2e/0x30 [] ext4_mb_init_cache+0x345/0x9f0 [ext4] [] ext4_mb_init_group+0x208/0x390 [ext4] [] ext4_mb_good_group+0xce/0x110 [ext4] [] ext4_mb_regular_allocator+0x19b/0x410 [ext4] [] ext4_mb_new_blocks+0x3a5/0x510 [ext4] [] ? ext4_ext_find_extent+0x2bb/0x320 [ext4] [] ext4_ext_map_blocks+0x58e/0x1f40 [ext4] [] ? transfer_objects+0x56/0x80 [] ? pagevec_lookup_tag+0x25/0x40 [] ext4_map_blocks+0xf4/0x210 [ext4] [] mpage_da_map_and_submit+0xb9/0x450 [ext4] [] ? jbd2_journal_start+0x13/0x20 [jbd2] [] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [] ext4_da_writepages+0x349/0x680 [ext4] [] ? __writepage+0x0/0x40 [] do_writepages+0x21/0x40 [] writeback_single_inode+0x98/0x240 [] writeback_sb_inodes+0xce/0x170 [] writeback_inodes_wb+0x99/0x160 [] wb_writeback+0x310/0x440 [] ? lock_timer_base+0x3c/0x70 [] wb_do_writeback+0x25f/0x270 [] bdi_writeback_thread+0xa2/0x280 [] ? bdi_writeback_thread+0x0/0x280 [] ? bdi_writeback_thread+0x0/0x280 [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task rsyslogd:3246 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rsyslogd D ffff880c5e992610 0 3246 1 0x00000080 ffff880c5cdbda28 0000000000000082 00000037ffffffc8 ffff88107ffd9c00 0000000000014d40 ffff880c5e992080 ffff880c5e992610 ffff880c5cdbdfd8 ffff880c5e992618 0000000000014d40 ffff880c5cdbc010 0000000000014d40 Call Trace: [] ? sync_page+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_page+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? read_swap_cache_async+0x4f/0x140 [] wait_on_page_bit+0x73/0x80 [] ? wake_bit_function+0x0/0x50 [] __lock_page_or_retry+0x3a/0x60 [] handle_pte_fault+0xac3/0xb20 [] handle_mm_fault+0x1b1/0x320 [] ? load_balance+0x10a/0x880 [] do_page_fault+0x135/0x460 [] ? __switch_to+0x256/0x320 [] page_fault+0x25/0x30 [] ? do_syslog+0x406/0x540 [] ? do_syslog+0x3b5/0x540 [] ? autoremove_wake_function+0x0/0x40 [] kmsg_read+0x37/0x70 [] proc_reg_read+0x76/0xb0 [] vfs_read+0xc5/0x190 [] sys_read+0x51/0x90 [] system_call_fastpath+0x16/0x1b INFO: task rs:main Q:Reg:7286 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. rs:main Q:Reg D ffff88045d5bc650 0 7286 1 0x00000080 ffff88045d35bb08 0000000000000082 ffff880400000000 ffffffffa00040ac 0000000000014d40 ffff88045d5bc0c0 ffff88045d5bc650 ffff88045d35bfd8 ffff88045d5bc658 0000000000014d40 ffff88045d35a010 0000000000014d40 Call Trace: [] ? dm_table_unplug_all+0x5c/0x110 [dm_mod] [] ? sync_page+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_page+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] wait_on_page_bit+0x73/0x80 [] ? wake_bit_function+0x0/0x50 [] __lock_page_or_retry+0x3a/0x60 [] filemap_fault+0x2d7/0x4c0 [] __do_fault+0x54/0x570 [] handle_pte_fault+0xf7/0xb20 [] ? __free_pages+0x2d/0x40 [] handle_mm_fault+0x1b1/0x320 [] do_page_fault+0x135/0x460 [] ? sys_futex+0x7b/0x180 [] page_fault+0x25/0x30 INFO: task automount:6638 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. automount D ffff88105cd6ba70 0 6638 1 0x00000080 ffff88105b8ebc18 0000000000000082 00000037ffffffff ffff88087ffdac00 0000000000014d40 ffff88105cd6b4e0 ffff88105cd6ba70 ffff88105b8ebfd8 ffff88105cd6ba78 0000000000014d40 ffff88105b8ea010 0000000000014d40 Call Trace: [] ? sync_page+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_page+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? read_swap_cache_async+0x4f/0x140 [] wait_on_page_bit+0x73/0x80 [] ? wake_bit_function+0x0/0x50 [] __lock_page_or_retry+0x3a/0x60 [] handle_pte_fault+0xac3/0xb20 [] handle_mm_fault+0x1b1/0x320 [] do_page_fault+0x135/0x460 [] ? sys_futex+0x7b/0x180 [] page_fault+0x25/0x30 INFO: task master:6753 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. master D ffff88085d4a65d0 0 6753 1 0x00000080 ffff88085da87c18 0000000000000082 0000003700000000 ffffffff8113edf6 0000000000014d40 ffff88085d4a6040 ffff88085d4a65d0 ffff88085da87fd8 ffff88085d4a65d8 0000000000014d40 ffff88085da86010 0000000000014d40 Call Trace: [] ? transfer_objects+0x56/0x80 [] ? sync_page+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_page+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? read_swap_cache_async+0x4f/0x140 [] wait_on_page_bit+0x73/0x80 [] ? wake_bit_function+0x0/0x50 [] __lock_page_or_retry+0x3a/0x60 [] handle_pte_fault+0xac3/0xb20 [] handle_mm_fault+0x1b1/0x320 [] do_page_fault+0x135/0x460 [] ? sys_epoll_wait+0xa0/0x450 [] ? default_wake_function+0x0/0x20 [] page_fault+0x25/0x30 INFO: task qmgr:6760 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. qmgr D ffff88105bb5baf0 0 6760 6753 0x00000084 ffff88105ce93c18 0000000000000082 0000003700000000 ffff88047ffdac00 0000000000014d40 ffff88105bb5b560 ffff88105bb5baf0 ffff88105ce93fd8 ffff88105bb5baf8 0000000000014d40 ffff88105ce92010 0000000000014d40 Call Trace: [] ? sync_page+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_page+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? read_swap_cache_async+0x4f/0x140 [] wait_on_page_bit+0x73/0x80 [] ? wake_bit_function+0x0/0x50 [] __lock_page_or_retry+0x3a/0x60 [] handle_pte_fault+0xac3/0xb20 [] handle_mm_fault+0x1b1/0x320 [] do_page_fault+0x135/0x460 [] ? sys_epoll_wait+0xa0/0x450 [] ? default_wake_function+0x0/0x20 [] page_fault+0x25/0x30 [1] http://ltp.git.sourceforge.net/git/gitweb.cgi?p=ltp/ltp.git;a=blob;f=testcases/kernel/mem/oom/oom01.c -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: email@kvack.org