From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail190.messagelabs.com (mail190.messagelabs.com [216.82.249.51]) by kanga.kvack.org (Postfix) with ESMTP id 72DF98D0039 for ; Mon, 31 Jan 2011 20:43:30 -0500 (EST) Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by e31.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id p111SpkL021327 for ; Mon, 31 Jan 2011 18:28:51 -0700 Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay04.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p111hLVQ127186 for ; Mon, 31 Jan 2011 18:43:21 -0700 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p111hK2X002363 for ; Mon, 31 Jan 2011 18:43:20 -0700 Subject: Re: kswapd hung tasks in 2.6.38-rc1 From: Dave Hansen In-Reply-To: <20110131231301.GP16981@random.random> References: <1150342867.83404.1295513748640.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> <1296507528.7797.4609.camel@nimitz> <1296513616.7797.4929.camel@nimitz> <20110131231301.GP16981@random.random> Content-Type: text/plain; charset="ANSI_X3.4-1968" Date: Mon, 31 Jan 2011 17:43:19 -0800 Message-ID: <1296524599.27022.439.camel@nimitz> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: Andrea Arcangeli Cc: CAI Qian , linux-mm , linux-kernel@vger.kernel.org List-ID: On Tue, 2011-02-01 at 00:13 +0100, Andrea Arcangeli wrote: > On Mon, Jan 31, 2011 at 02:40:16PM -0800, Dave Hansen wrote: > > Still not a very good data point, but I ran a heavy swap load for an > > hour or so without reproducing this. But, it happened again after I > > enabled transparent huge pages. I managed to get a sysrq-t dump out of > > it: > > > > http://sr71.net/~dave/ibm/2.6.38-rc2-hang-0.txt > > > > khugepaged is one of the three running tasks. Note, I set both its > > sleep timeouts to zero to stress it out a bit. > > sysrq+l? sysrq+t doesn't provide interesting info for running tasks. Got one again. Took ~3 hours to reproduce this time, but I don't think I was hitting swap very hard most of the time. Here's sysrq+l pasted. sysrq-t is here: http://sr71.net/~dave/ibm/2.6.38-rc2-hang-1.txt [ 9834.262374] SysRq : Show backtrace of all active CPUs [ 9834.267437] CPU0: [ 9834.269360] CPU 0 [ 9834.271197] Modules linked in: [ 9834.274455] [ 9834.275948] Pid: 0, comm: swapper Not tainted 2.6.38-rc2-dirty #119 49Y6512 /System x3650 M2 -[7947AC1]- [ 9834.285748] RIP: 0010:[] [] acpi_idle_enter_bm+0x22e/0x265 [ 9834.294443] RSP: 0000:ffffffff818c9ec8 EFLAGS: 00000246 [ 9834.299742] RAX: 00000000000009db RBX: ffffffff8109dbce RCX: 00000000000003e8 [ 9834.306857] RDX: 0000000000000119 RSI: 0000000000000001 RDI: 0000000000000000 [ 9834.313971] RBP: ffffffff818c9ef8 R08: 0000000000000001 R09: 0000000000000020 [ 9834.321086] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff815f1ece [ 9834.328201] R13: ffff88083e786800 R14: ffff88083e786820 R15: ffff88083e78680c [ 9834.335317] FS: 0000000000000000(0000) GS:ffff88007a800000(0000) knlGS:0000000000000000 [ 9834.343386] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 9834.349116] CR2: 00007f9b2eaf5720 CR3: 000000000191b000 CR4: 00000000000006f0 [ 9834.356231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9834.363346] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9834.370461] Process swapper (pid: 0, threadinfo ffffffff818c8000, task ffffffff81923020) [ 9834.378530] Stack: [ 9834.380540] ffffffff818c9ef8 00000000814f76b6 ffff88083e786820 ffff88083e7868f0 [ 9834.387992] 0000000000000000 0000000000000002 ffffffff818c9f38 ffffffff814f68a9 [ 9834.395442] ffffffff818c9f18 ffffffff819aaf80 6db6db6db6db6db7 000000000000001f [ 9834.402893] Call Trace: [ 9834.405338] [] cpuidle_idle_call+0xb9/0x210 [ 9834.411159] [] cpu_idle+0x47/0x80 [ 9834.416113] [] rest_init+0x72/0x80 [ 9834.421152] [] start_kernel+0x2ff/0x3e0 [ 9834.426626] [] x86_64_start_reservations+0x81/0xd0 [ 9834.433049] [] x86_64_start_kernel+0xdd/0x100 [ 9834.439041] Code: 89 d5 ff 48 89 c6 ba e8 03 00 00 48 29 de 48 89 d1 31 d2 48 89 f0 48 89 f7 48 f7 f1 48 89 c3 e8 ba 5e d5 ff fb 41 80 7c 24 08 01 <74> 10 65 48 8b 04 25 08 b7 00 00 83 88 3c e0 ff ff 04 41 ff 44 [ 9834.459137] Call Trace: [ 9834.461580] [] cpuidle_idle_call+0xb9/0x210 [ 9834.467399] [] cpu_idle+0x47/0x80 [ 9834.472353] [] rest_init+0x72/0x80 [ 9834.477391] [] start_kernel+0x2ff/0x3e0 [ 9834.482864] [] x86_64_start_reservations+0x81/0xd0 [ 9834.489287] [] x86_64_start_kernel+0xdd/0x100 [ 9834.495337] CPU4: [ 9834.497266] ffff88107fc03ef8 0000000000000046 0000000000000004 ffffffff8135edc0 [ 9834.504721] ffff88087fffc2d8 ffff88083e067bd0 ffff88107fc03f38 ffffffff8103e74a [ 9834.512175] ffff88107fc03f58 ffffffff8135ee0d ffff88107fc03f68 ffff88007a812580 [ 9834.519633] Call Trace: [ 9834.522077] [] ? showacpu+0x0/0x70 [ 9834.527747] [] ? show_stack+0x1a/0x20 [ 9834.533049] [] ? showacpu+0x4d/0x70 [ 9834.538179] [] ? generic_smp_call_function_interrupt+0x7b/0x170 [ 9834.545733] [] ? smp_call_function_interrupt+0x27/0x40 [ 9834.552505] [] ? call_function_interrupt+0x13/0x20 [ 9834.558927] [] ? shrink_zone+0x16d/0x4c0 [ 9834.565113] [] ? do_try_to_free_pages+0x9b/0x300 [ 9834.571365] [] ? try_to_free_pages+0x90/0x110 [ 9834.577360] [] ? __alloc_pages_nodemask+0x355/0x730 [ 9834.583873] [] ? schedule_timeout+0x11e/0x250 [ 9834.589866] [] ? alloc_pages_vma+0x82/0x130 [ 9834.595687] [] ? khugepaged+0x8c0/0x10c0 [ 9834.601249] [] ? autoremove_wake_function+0x0/0x40 [ 9834.607673] [] ? khugepaged+0x0/0x10c0 [ 9834.613059] [] ? kthread+0x97/0xa0 [ 9834.618099] [] ? kernel_thread_helper+0x4/0x10 [ 9834.624176] [] ? kthread+0x0/0xa0 [ 9834.629132] [] ? kernel_thread_helper+0x0/0x10 And the hung task messages: [ 9687.185858] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 9687.192291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.200102] kswapd0 D 000000010022f817 0 706 2 0x00000000 [ 9687.207173] ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40 [ 9687.214627] ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001 [ 9687.222080] ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308 [ 9687.229532] Call Trace: [ 9687.231986] [] ? mempool_alloc+0x47/0x120 [ 9687.237634] [] ? cfq_may_queue+0x56/0xe0 [ 9687.243201] [] io_schedule+0x44/0x60 [ 9687.248421] [] get_request_wait+0xdb/0x170 [ 9687.254154] [] ? autoremove_wake_function+0x0/0x40 [ 9687.260581] [] ? elv_merge+0x47/0x1d0 [ 9687.265882] [] __make_request+0x6e/0x480 [ 9687.271446] [] generic_make_request+0x282/0x3f0 [ 9687.277612] [] ? bio_init+0x1d/0x40 [ 9687.282741] [] submit_bio+0x67/0xe0 [ 9687.287869] [] ? test_set_page_writeback+0xc5/0x180 [ 9687.294386] [] swap_writepage+0xa0/0xc0 [ 9687.299860] [] shrink_page_list+0x45c/0x800 [ 9687.305682] [] shrink_inactive_list+0x11e/0x3c0 [ 9687.311849] [] shrink_zone+0x291/0x4c0 [ 9687.317239] [] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9687.323579] [] kswapd+0x605/0x8a0 [ 9687.328534] [] ? kswapd+0x0/0x8a0 [ 9687.333488] [] kthread+0x97/0xa0 [ 9687.338362] [] kernel_thread_helper+0x4/0x10 [ 9687.344269] [] ? kthread+0x0/0xa0 [ 9687.349225] [] ? kernel_thread_helper+0x0/0x10 [ 9687.355303] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 9687.361725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.369537] kswapd1 D 000000010022f812 0 707 2 0x00000000 [ 9687.376609] ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40 [ 9687.384062] ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c [ 9687.391518] ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308 [ 9687.398971] Call Trace: [ 9687.401414] [] ? mempool_alloc+0x47/0x120 [ 9687.407062] [] ? cfq_may_queue+0x56/0xe0 [ 9687.412625] [] io_schedule+0x44/0x60 [ 9687.417842] [] get_request_wait+0xdb/0x170 [ 9687.423576] [] ? autoremove_wake_function+0x0/0x40 [ 9687.430002] [] ? elv_merge+0x17b/0x1d0 [ 9687.435392] [] __make_request+0x6e/0x480 [ 9687.440952] [] generic_make_request+0x282/0x3f0 [ 9687.447121] [] ? bio_init+0x1d/0x40 [ 9687.452251] [] submit_bio+0x67/0xe0 [ 9687.457377] [] ? test_set_page_writeback+0xc5/0x180 [ 9687.463889] [] swap_writepage+0xa0/0xc0 [ 9687.469364] [] shrink_page_list+0x45c/0x800 [ 9687.475183] [] shrink_inactive_list+0x11e/0x3c0 [ 9687.481349] [] shrink_zone+0x291/0x4c0 [ 9687.486741] [] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9687.493082] [] kswapd+0x605/0x8a0 [ 9687.498038] [] ? kswapd+0x0/0x8a0 [ 9687.502992] [] kthread+0x97/0xa0 [ 9687.507862] [] kernel_thread_helper+0x4/0x10 [ 9687.513768] [] ? kthread+0x0/0xa0 [ 9687.518723] [] ? kernel_thread_helper+0x0/0x10 [ 9687.524805] INFO: task memknobs:2741 blocked for more than 120 seconds. [ 9687.531403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.539211] memknobs D 000000010022f7f5 0 2741 2674 0x00000000 [ 9687.546282] ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9687.553735] ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e [ 9687.561189] 0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2 [ 9687.568644] Call Trace: [ 9687.571089] [] ? generic_make_request+0x282/0x3f0 [ 9687.577429] [] ? bio_init+0x1d/0x40 [ 9687.582556] [] io_schedule+0x44/0x60 [ 9687.587769] [] sync_page+0x45/0x60 [ 9687.592811] [] __wait_on_bit+0x5e/0x90 [ 9687.598197] [] ? sync_page+0x0/0x60 [ 9687.603324] [] wait_on_page_bit+0x74/0x80 [ 9687.608973] [] ? wake_bit_function+0x0/0x40 [ 9687.614795] [] ? radix_tree_lookup_slot+0xe/0x10 [ 9687.621048] [] __lock_page_or_retry+0x4c/0x60 [ 9687.627043] [] handle_pte_fault+0x79f/0x7c0 [ 9687.632864] [] ? __free_pages+0x27/0x30 [ 9687.638340] [] ? __pte_alloc+0xb1/0x110 [ 9687.643813] [] handle_mm_fault+0x162/0x270 [ 9687.649547] [] do_page_fault+0x161/0x480 [ 9687.655108] [] ? do_sigaction+0x103/0x1a0 [ 9687.660758] [] ? sys_rt_sigaction+0x89/0xc0 [ 9687.666578] [] page_fault+0x1f/0x30 [ 9687.671708] INFO: task memknobs:2794 blocked for more than 120 seconds. [ 9687.678308] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9687.686117] memknobs D 000000010022f90a 0 2794 2724 0x00000000 [ 9687.693187] ffff880c66627c68 0000000000000082 ffff88083faa8f40 0000000000011c40 [ 9687.700639] ffff88103e26cae0 ffff88103e26c780 ffffffff81923020 ffff880800000000 [ 9687.708095] 0000000000000008 0000000000821ba7 ffff880c66627c38 ffffffff812db5d2 [ 9687.715551] Call Trace: [ 9687.717998] [] ? generic_make_request+0x282/0x3f0 [ 9687.724338] [] io_schedule+0x44/0x60 [ 9687.729555] [] sync_page+0x45/0x60 [ 9687.734597] [] __wait_on_bit+0x5e/0x90 [ 9687.739984] [] ? sync_page+0x0/0x60 [ 9687.745112] [] wait_on_page_bit+0x74/0x80 [ 9687.750759] [] ? wake_bit_function+0x0/0x40 [ 9687.756578] [] ? radix_tree_lookup_slot+0xe/0x10 [ 9687.762830] [] __lock_page_or_retry+0x4c/0x60 [ 9687.768824] [] handle_pte_fault+0x79f/0x7c0 [ 9687.774645] [] ? __free_pages+0x27/0x30 [ 9687.780118] [] ? __pte_alloc+0xb1/0x110 [ 9687.785591] [] handle_mm_fault+0x162/0x270 [ 9687.791324] [] do_page_fault+0x161/0x480 [ 9687.796884] [] ? do_sigaction+0x103/0x1a0 [ 9687.802530] [] ? sys_rt_sigaction+0x89/0xc0 [ 9687.808351] [] page_fault+0x1f/0x30 [ 9807.612419] INFO: task kswapd0:706 blocked for more than 120 seconds. [ 9807.618845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.626656] kswapd0 D 000000010022f817 0 706 2 0x00000000 [ 9807.633728] ffff88083e75f920 0000000000000046 0000000000000001 0000000000011c40 [ 9807.641187] ffff88083f998b60 ffff88083f998800 ffff880840309880 ffffffff00000001 [ 9807.648640] ffff88083de8c9f0 000000103e75f880 ffff8805724a3880 ffff88083e7e3308 [ 9807.656097] Call Trace: [ 9807.658544] [] ? mempool_alloc+0x47/0x120 [ 9807.664190] [] ? cfq_may_queue+0x56/0xe0 [ 9807.669754] [] io_schedule+0x44/0x60 [ 9807.674969] [] get_request_wait+0xdb/0x170 [ 9807.680705] [] ? autoremove_wake_function+0x0/0x40 [ 9807.687131] [] ? elv_merge+0x47/0x1d0 [ 9807.692432] [] __make_request+0x6e/0x480 [ 9807.697995] [] generic_make_request+0x282/0x3f0 [ 9807.704160] [] ? bio_init+0x1d/0x40 [ 9807.709288] [] submit_bio+0x67/0xe0 [ 9807.714417] [] ? test_set_page_writeback+0xc5/0x180 [ 9807.720932] [] swap_writepage+0xa0/0xc0 [ 9807.726406] [] shrink_page_list+0x45c/0x800 [ 9807.732231] [] shrink_inactive_list+0x11e/0x3c0 [ 9807.738396] [] shrink_zone+0x291/0x4c0 [ 9807.743788] [] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9807.750127] [] kswapd+0x605/0x8a0 [ 9807.755081] [] ? kswapd+0x0/0x8a0 [ 9807.760037] [] kthread+0x97/0xa0 [ 9807.764908] [] kernel_thread_helper+0x4/0x10 [ 9807.770814] [] ? kthread+0x0/0xa0 [ 9807.775770] [] ? kernel_thread_helper+0x0/0x10 [ 9807.781851] INFO: task kswapd1:707 blocked for more than 120 seconds. [ 9807.788277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.796084] kswapd1 D 000000010022f812 0 707 2 0x00000000 [ 9807.803158] ffff88083e705920 0000000000000046 0000000000000001 0000000000011c40 [ 9807.810615] ffff88083f998460 ffff88083f998100 ffff88083f80b8c0 ffffffff0000000c [ 9807.818070] ffff88083de8c9f0 000000103e705880 ffff880ee402b880 ffff88083e7e3308 [ 9807.825524] Call Trace: [ 9807.827967] [] ? mempool_alloc+0x47/0x120 [ 9807.833614] [] ? cfq_may_queue+0x56/0xe0 [ 9807.839176] [] io_schedule+0x44/0x60 [ 9807.844393] [] get_request_wait+0xdb/0x170 [ 9807.850126] [] ? autoremove_wake_function+0x0/0x40 [ 9807.856551] [] ? elv_merge+0x17b/0x1d0 [ 9807.861940] [] __make_request+0x6e/0x480 [ 9807.867503] [] generic_make_request+0x282/0x3f0 [ 9807.873669] [] ? bio_init+0x1d/0x40 [ 9807.878796] [] submit_bio+0x67/0xe0 [ 9807.883926] [] ? test_set_page_writeback+0xc5/0x180 [ 9807.890439] [] swap_writepage+0xa0/0xc0 [ 9807.895915] [] shrink_page_list+0x45c/0x800 [ 9807.901737] [] shrink_inactive_list+0x11e/0x3c0 [ 9807.907905] [] shrink_zone+0x291/0x4c0 [ 9807.913335] [] ? zone_watermark_ok_safe+0xb6/0xd0 [ 9807.919675] [] kswapd+0x605/0x8a0 [ 9807.924632] [] ? kswapd+0x0/0x8a0 [ 9807.929588] [] kthread+0x97/0xa0 [ 9807.934457] [] kernel_thread_helper+0x4/0x10 [ 9807.940365] [] ? kthread+0x0/0xa0 [ 9807.945321] [] ? kernel_thread_helper+0x0/0x10 [ 9807.951404] INFO: task avahi-daemon:2353 blocked for more than 120 seconds. [ 9807.958346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9807.966155] avahi-daemon D 00000001002332a4 0 2353 1 0x00000000 [ 9807.973227] ffff88103f5437a8 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9807.980682] ffff88103eb764e0 ffff88103eb76180 ffffffff81923020 ffff880800000000 [ 9807.988134] 0000000000000008 00000000007e89ff ffff88103f543778 ffffffff812db5d2 [ 9807.995587] Call Trace: [ 9807.998034] [] ? generic_make_request+0x282/0x3f0 [ 9808.004374] [] ? bio_init+0x1d/0x40 [ 9808.009501] [] io_schedule+0x44/0x60 [ 9808.014716] [] sync_page+0x45/0x60 [ 9808.019759] [] __wait_on_bit+0x5e/0x90 [ 9808.025146] [] ? sync_page+0x0/0x60 [ 9808.030273] [] wait_on_page_bit+0x74/0x80 [ 9808.035922] [] ? wake_bit_function+0x0/0x40 [ 9808.041745] [] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.047999] [] __lock_page_or_retry+0x4c/0x60 [ 9808.053994] [] handle_pte_fault+0x79f/0x7c0 [ 9808.059814] [] ? __free_pages+0x27/0x30 [ 9808.065290] [] ? __pte_alloc+0xb1/0x110 [ 9808.070766] [] handle_mm_fault+0x162/0x270 [ 9808.076500] [] do_page_fault+0x161/0x480 [ 9808.082063] [] ? rb_insert_color+0x8a/0xf0 [ 9808.087800] [] ? timerqueue_add+0x60/0xb0 [ 9808.093449] [] ? lock_hrtimer_base+0x31/0x60 [ 9808.099356] [] ? hrtimer_try_to_cancel+0x51/0xd0 [ 9808.105610] [] page_fault+0x1f/0x30 [ 9808.110741] [] ? do_sys_poll+0x3ac/0x460 [ 9808.116301] [] ? do_sys_poll+0x370/0x460 [ 9808.121863] [] ? __pollwait+0x0/0x100 [ 9808.127164] [] ? pollwake+0x0/0x70 [ 9808.132206] [] ? pollwake+0x0/0x70 [ 9808.137248] [] ? pollwake+0x0/0x70 [ 9808.142291] [] ? pollwake+0x0/0x70 [ 9808.147331] [] ? pollwake+0x0/0x70 [ 9808.152373] [] ? pollwake+0x0/0x70 [ 9808.157416] [] ? pollwake+0x0/0x70 [ 9808.162457] [] ? pollwake+0x0/0x70 [ 9808.167530] [] ? do_sync_read+0xd5/0x120 [ 9808.173093] [] ? timespec_add_safe+0x39/0x70 [ 9808.179000] [] ? poll_select_set_timeout+0x84/0xa0 [ 9808.185426] [] sys_poll+0x7c/0x100 [ 9808.190469] [] system_call_fastpath+0x16/0x1b [ 9808.196463] INFO: task rsyslogd:2523 blocked for more than 120 seconds. [ 9808.203061] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.210872] rsyslogd D 0000000100237de3 0 2523 1 0x00000000 [ 9808.217944] ffff88083ec11a88 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.225403] ffff88083f93a460 ffff88083f93a100 ffffffff81923020 ffff880800000000 [ 9808.232858] 0000000000000008 00000000006f52cf ffff88083ec11a58 ffffffff812db5d2 [ 9808.240311] Call Trace: [ 9808.242754] [] ? generic_make_request+0x282/0x3f0 [ 9808.249092] [] ? bio_init+0x1d/0x40 [ 9808.254220] [] io_schedule+0x44/0x60 [ 9808.259571] [] sync_page+0x45/0x60 [ 9808.264616] [] __wait_on_bit+0x5e/0x90 [ 9808.270002] [] ? sync_page+0x0/0x60 [ 9808.275131] [] wait_on_page_bit+0x74/0x80 [ 9808.280780] [] ? wake_bit_function+0x0/0x40 [ 9808.286600] [] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.292855] [] __lock_page_or_retry+0x4c/0x60 [ 9808.298849] [] handle_pte_fault+0x79f/0x7c0 [ 9808.304668] [] ? __free_pages+0x27/0x30 [ 9808.310142] [] ? __pte_alloc+0xb1/0x110 [ 9808.315618] [] handle_mm_fault+0x162/0x270 [ 9808.321352] [] do_page_fault+0x161/0x480 [ 9808.326913] [] ? __switch_to+0x1e6/0x2a0 [ 9808.332472] [] ? schedule+0x3f8/0xaf0 [ 9808.337775] [] page_fault+0x1f/0x30 [ 9808.342904] [] ? do_syslog+0x347/0x4e0 [ 9808.348292] [] ? do_syslog+0x302/0x4e0 [ 9808.353681] [] ? autoremove_wake_function+0x0/0x40 [ 9808.360107] [] kmsg_read+0x36/0x70 [ 9808.365149] [] proc_reg_read+0x74/0xb0 [ 9808.370538] [] vfs_read+0xcb/0x170 [ 9808.375581] [] sys_read+0x55/0x90 [ 9808.380539] [] system_call_fastpath+0x16/0x1b [ 9808.386532] INFO: task memknobs:2741 blocked for more than 120 seconds. [ 9808.393133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.400942] memknobs D 000000010022f7f5 0 2741 2674 0x00000000 [ 9808.408014] ffff88103e3ffc68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.415471] ffff88103eb024a0 ffff88103eb02140 ffff88083f818800 ffff88080000000e [ 9808.422926] 0000000000000008 000000000081f4c7 ffff88103e3ffc38 ffffffff812db5d2 [ 9808.430377] Call Trace: [ 9808.432824] [] ? generic_make_request+0x282/0x3f0 [ 9808.439165] [] ? bio_init+0x1d/0x40 [ 9808.444295] [] io_schedule+0x44/0x60 [ 9808.449511] [] sync_page+0x45/0x60 [ 9808.454553] [] __wait_on_bit+0x5e/0x90 [ 9808.459943] [] ? sync_page+0x0/0x60 [ 9808.465070] [] wait_on_page_bit+0x74/0x80 [ 9808.470715] [] ? wake_bit_function+0x0/0x40 [ 9808.476535] [] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.482788] [] __lock_page_or_retry+0x4c/0x60 [ 9808.488781] [] handle_pte_fault+0x79f/0x7c0 [ 9808.494604] [] ? __free_pages+0x27/0x30 [ 9808.500077] [] ? __pte_alloc+0xb1/0x110 [ 9808.505549] [] handle_mm_fault+0x162/0x270 [ 9808.511284] [] do_page_fault+0x161/0x480 [ 9808.516848] [] ? do_sigaction+0x103/0x1a0 [ 9808.522498] [] ? sys_rt_sigaction+0x89/0xc0 [ 9808.528318] [] page_fault+0x1f/0x30 [ 9808.533449] INFO: task sshd:2755 blocked for more than 120 seconds. [ 9808.539700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9808.547511] sshd D 00000001002310ff 0 2755 2510 0x00000000 [ 9808.554582] ffff880079c27c68 0000000000000086 ffff88083faa8f40 0000000000011c40 [ 9808.562039] ffff88083e2f1aa0 ffff88083e2f1740 ffffffff81923020 ffff880800000000 [ 9808.569493] 0000000000000008 00000000007de1b7 ffff880079c27c38 ffffffff812db5d2 [ 9808.576948] Call Trace: [ 9808.579393] [] ? generic_make_request+0x282/0x3f0 [ 9808.585731] [] ? bio_init+0x1d/0x40 [ 9808.590859] [] io_schedule+0x44/0x60 [ 9808.596075] [] sync_page+0x45/0x60 [ 9808.601115] [] __wait_on_bit+0x5e/0x90 [ 9808.606502] [] ? sync_page+0x0/0x60 [ 9808.611630] [] wait_on_page_bit+0x74/0x80 [ 9808.617277] [] ? wake_bit_function+0x0/0x40 [ 9808.623096] [] ? radix_tree_lookup_slot+0xe/0x10 [ 9808.629350] [] __lock_page_or_retry+0x4c/0x60 [ 9808.635345] [] handle_pte_fault+0x79f/0x7c0 [ 9808.641166] [] ? __free_pages+0x27/0x30 [ 9808.646643] [] ? __pte_alloc+0xb1/0x110 [ 9808.652118] [] handle_mm_fault+0x162/0x270 [ 9808.657851] [] do_page_fault+0x161/0x480 [ 9808.663413] [] ? vfs_read+0x12e/0x170 [ 9808.668713] [] page_fault+0x1f/0x30 -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: email@kvack.org