* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer [not found] <bug-198497-27@https.bugzilla.kernel.org/> @ 2018-01-18 21:55 ` Andrew Morton 2018-01-18 22:18 ` Laura Abbott 0 siblings, 1 reply; 16+ messages in thread From: Andrew Morton @ 2018-01-18 21:55 UTC (permalink / raw) To: linux-mm; +Cc: bugzilla-daemon, Matthew Wilcox, peter (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Thu, 18 Jan 2018 01:21:56 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=198497 > > Bug ID: 198497 > Summary: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot > Null pointer > Product: Memory Management > Version: 2.5 > Kernel Version: Linux app1.vpsgate.com > 4.14.13-rh10-20180115190010.xenU.i386 #1 SMP Mon Jan > 15 19:04:55 UTC 2018 i686 GNU/Linux > Hardware: Intel > OS: Linux > Tree: Mainline > Status: NEW > Severity: high > Priority: P1 > Component: Page Allocator > Assignee: akpm@linux-foundation.org > Reporter: peter@rimuhosting.com > Regression: No Does this look familiar to anyone? > On a Xen VM running as pvh > > [ 3.499843] Adding 131068k swap on /dev/xvda9. Priority:-2 extents:1 > across:131068k SSFS > [ 3.547312] EXT4-fs (xvda1): re-mounted. Opts: (null) > [ 3.988606] EXT4-fs (xvda1): re-mounted. Opts: errors=remount-ro > [ 24.647744] BUG: unable to handle kernel NULL pointer dereference at > 00000008 > [ 24.647801] IP: __radix_tree_lookup+0x14/0xa0 > [ 24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 > [ 24.647828] Oops: 0000 [#1] SMP > [ 24.647842] CPU: 5 PID: 3600 Comm: java Not tainted > 4.14.13-rh10-20180115190010.xenU.i386 #1 > [ 24.647855] task: e52518c0 task.stack: e4e7a000 > [ 24.647866] EIP: __radix_tree_lookup+0x14/0xa0 > [ 24.647876] EFLAGS: 00010286 CPU: 5 > [ 24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 > [ 24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0 > [ 24.647904] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 > [ 24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660 > [ 24.647930] Call Trace: > [ 24.647942] radix_tree_lookup_slot+0x13/0x30 > [ 24.647955] find_get_entry+0x1d/0x120 > [ 24.647963] pagecache_get_page+0x1f/0x230 > [ 24.647975] lookup_swap_cache+0x42/0x140 > [ 24.647983] swap_readahead_detect+0x66/0x2e0 > [ 24.647993] do_swap_page+0x1fa/0x860 > [ 24.648010] ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10 > [ 24.648026] ? xen_pmd_val+0x10/0x20 > [ 24.648035] handle_mm_fault+0x6f8/0x1020 > [ 24.648046] __do_page_fault+0x18a/0x450 > [ 24.648055] ? vmalloc_sync_all+0x250/0x250 > [ 24.648063] do_page_fault+0x21/0x30 > [ 24.648074] common_exception+0x45/0x4a > [ 24.648082] EIP: 0xb76d873e > [ 24.648088] EFLAGS: 00010206 CPU: 5 > [ 24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006 > [ 24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8 > [ 24.648115] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b > [ 24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff > ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 > 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 > [ 24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0 > [ 24.648205] CR2: 0000000000000008 > [ 24.648273] ---[ end trace ed356e59f215ce07 ]--- > [ 28.890326] BUG: unable to handle kernel NULL pointer dereference at > 00000008 > [ 28.890372] IP: __radix_tree_lookup+0x14/0xa0 > [ 28.890382] *pdpt = 0000000025488027 *pde = 0000000000000000 > [ 28.890396] Oops: 0000 [#2] SMP > [ 28.890408] CPU: 7 PID: 3542 Comm: java Tainted: G D > 4.14.13-rh10-20180115190010.xenU.i386 #1 > [ 28.890423] task: e8691080 task.stack: e52a6000 > [ 28.890433] EIP: __radix_tree_lookup+0x14/0xa0 > [ 28.890442] EFLAGS: 00010286 CPU: 7 > [ 28.890449] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 > [ 28.890459] ESI: 00000000 EDI: 00000000 EBP: e52a7db8 ESP: e52a7da0 > [ 28.890469] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 > [ 28.890484] CR0: 80050033 CR2: 00000008 CR3: 25161000 CR4: 00002660 > [ 28.890498] Call Trace: > [ 28.890510] radix_tree_lookup_slot+0x13/0x30 > [ 28.890522] find_get_entry+0x1d/0x120 > [ 28.890531] pagecache_get_page+0x1f/0x230 > [ 28.890541] lookup_swap_cache+0x42/0x140 > [ 28.890550] swap_readahead_detect+0x66/0x2e0 > [ 28.890559] do_swap_page+0x1fa/0x860 > [ 28.890573] ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10 > [ 28.890588] ? xen_pmd_val+0x10/0x20 > [ 28.890597] handle_mm_fault+0x6f8/0x1020 > [ 28.890607] __do_page_fault+0x18a/0x450 > [ 28.890616] ? vmalloc_sync_all+0x250/0x250 > [ 28.890681] do_page_fault+0x21/0x30 > [ 28.890707] common_exception+0x45/0x4a > [ 28.890715] EIP: 0xb779774f > [ 28.890722] EFLAGS: 00010202 CPU: 7 > [ 28.890730] EAX: 00000000 EBX: 66dd9d6c ECX: 02000000 EDX: 00000001 > [ 28.890740] ESI: 02000000 EDI: 00000000 EBP: 674fe068 ESP: 674fe058 > [ 28.890751] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b > [ 28.890759] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff > ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 > 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 > [ 28.890830] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e52a7da0 > [ 28.890841] CR2: 0000000000000008 > [ 28.890886] ---[ end trace ed356e59f215ce08 ]--- > > # java -version > java version "1.7.0_51" > Java(TM) SE Runtime Environment (build 1.7.0_51-b13) > Java HotSpot(TM) Server VM (build 24.51-b03, mixed mode) > > ~# free -m > total used free shared buffers cached > Mem: 5418 572 4846 0 25 245 > -/+ buffers/cache: 301 5117 > Swap: 127 0 127 > > # uptime > 01:21:08 up 2:02, 3 users, load average: 13.47, 13.65, 13.63 > > -- > You are receiving this mail because: > You are the assignee for the bug. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-18 21:55 ` [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer Andrew Morton @ 2018-01-18 22:18 ` Laura Abbott 2018-01-19 3:04 ` Matthew Wilcox 2018-01-19 13:33 ` Matthew Wilcox 0 siblings, 2 replies; 16+ messages in thread From: Laura Abbott @ 2018-01-18 22:18 UTC (permalink / raw) To: Andrew Morton, linux-mm; +Cc: bugzilla-daemon, Matthew Wilcox, peter On 01/18/2018 01:55 PM, Andrew Morton wrote: > > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > > On Thu, 18 Jan 2018 01:21:56 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > >> https://bugzilla.kernel.org/show_bug.cgi?id=198497 >> >> Bug ID: 198497 >> Summary: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot >> Null pointer >> Product: Memory Management >> Version: 2.5 >> Kernel Version: Linux app1.vpsgate.com >> 4.14.13-rh10-20180115190010.xenU.i386 #1 SMP Mon Jan >> 15 19:04:55 UTC 2018 i686 GNU/Linux >> Hardware: Intel >> OS: Linux >> Tree: Mainline >> Status: NEW >> Severity: high >> Priority: P1 >> Component: Page Allocator >> Assignee: akpm@linux-foundation.org >> Reporter: peter@rimuhosting.com >> Regression: No > > Does this look familiar to anyone? > Fedora has been seeing similar reports https://bugzilla.redhat.com/show_bug.cgi?id=1531779 Multiple reporters, one in XEN, another on actual hardware >> On a Xen VM running as pvh >> >> [ 3.499843] Adding 131068k swap on /dev/xvda9. Priority:-2 extents:1 >> across:131068k SSFS >> [ 3.547312] EXT4-fs (xvda1): re-mounted. Opts: (null) >> [ 3.988606] EXT4-fs (xvda1): re-mounted. Opts: errors=remount-ro >> [ 24.647744] BUG: unable to handle kernel NULL pointer dereference at >> 00000008 >> [ 24.647801] IP: __radix_tree_lookup+0x14/0xa0 >> [ 24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 >> [ 24.647828] Oops: 0000 [#1] SMP >> [ 24.647842] CPU: 5 PID: 3600 Comm: java Not tainted >> 4.14.13-rh10-20180115190010.xenU.i386 #1 >> [ 24.647855] task: e52518c0 task.stack: e4e7a000 >> [ 24.647866] EIP: __radix_tree_lookup+0x14/0xa0 >> [ 24.647876] EFLAGS: 00010286 CPU: 5 >> [ 24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 >> [ 24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0 >> [ 24.647904] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 >> [ 24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660 >> [ 24.647930] Call Trace: >> [ 24.647942] radix_tree_lookup_slot+0x13/0x30 >> [ 24.647955] find_get_entry+0x1d/0x120 >> [ 24.647963] pagecache_get_page+0x1f/0x230 >> [ 24.647975] lookup_swap_cache+0x42/0x140 >> [ 24.647983] swap_readahead_detect+0x66/0x2e0 >> [ 24.647993] do_swap_page+0x1fa/0x860 >> [ 24.648010] ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10 >> [ 24.648026] ? xen_pmd_val+0x10/0x20 >> [ 24.648035] handle_mm_fault+0x6f8/0x1020 >> [ 24.648046] __do_page_fault+0x18a/0x450 >> [ 24.648055] ? vmalloc_sync_all+0x250/0x250 >> [ 24.648063] do_page_fault+0x21/0x30 >> [ 24.648074] common_exception+0x45/0x4a >> [ 24.648082] EIP: 0xb76d873e >> [ 24.648088] EFLAGS: 00010206 CPU: 5 >> [ 24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006 >> [ 24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8 >> [ 24.648115] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b >> [ 24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff >> ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 >> 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 >> [ 24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0 >> [ 24.648205] CR2: 0000000000000008 >> [ 24.648273] ---[ end trace ed356e59f215ce07 ]--- >> [ 28.890326] BUG: unable to handle kernel NULL pointer dereference at >> 00000008 >> [ 28.890372] IP: __radix_tree_lookup+0x14/0xa0 >> [ 28.890382] *pdpt = 0000000025488027 *pde = 0000000000000000 >> [ 28.890396] Oops: 0000 [#2] SMP >> [ 28.890408] CPU: 7 PID: 3542 Comm: java Tainted: G D >> 4.14.13-rh10-20180115190010.xenU.i386 #1 >> [ 28.890423] task: e8691080 task.stack: e52a6000 >> [ 28.890433] EIP: __radix_tree_lookup+0x14/0xa0 >> [ 28.890442] EFLAGS: 00010286 CPU: 7 >> [ 28.890449] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 >> [ 28.890459] ESI: 00000000 EDI: 00000000 EBP: e52a7db8 ESP: e52a7da0 >> [ 28.890469] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 >> [ 28.890484] CR0: 80050033 CR2: 00000008 CR3: 25161000 CR4: 00002660 >> [ 28.890498] Call Trace: >> [ 28.890510] radix_tree_lookup_slot+0x13/0x30 >> [ 28.890522] find_get_entry+0x1d/0x120 >> [ 28.890531] pagecache_get_page+0x1f/0x230 >> [ 28.890541] lookup_swap_cache+0x42/0x140 >> [ 28.890550] swap_readahead_detect+0x66/0x2e0 >> [ 28.890559] do_swap_page+0x1fa/0x860 >> [ 28.890573] ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10 >> [ 28.890588] ? xen_pmd_val+0x10/0x20 >> [ 28.890597] handle_mm_fault+0x6f8/0x1020 >> [ 28.890607] __do_page_fault+0x18a/0x450 >> [ 28.890616] ? vmalloc_sync_all+0x250/0x250 >> [ 28.890681] do_page_fault+0x21/0x30 >> [ 28.890707] common_exception+0x45/0x4a >> [ 28.890715] EIP: 0xb779774f >> [ 28.890722] EFLAGS: 00010202 CPU: 7 >> [ 28.890730] EAX: 00000000 EBX: 66dd9d6c ECX: 02000000 EDX: 00000001 >> [ 28.890740] ESI: 02000000 EDI: 00000000 EBP: 674fe068 ESP: 674fe058 >> [ 28.890751] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b >> [ 28.890759] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff >> ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 >> 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 >> [ 28.890830] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e52a7da0 >> [ 28.890841] CR2: 0000000000000008 >> [ 28.890886] ---[ end trace ed356e59f215ce08 ]--- >> >> # java -version >> java version "1.7.0_51" >> Java(TM) SE Runtime Environment (build 1.7.0_51-b13) >> Java HotSpot(TM) Server VM (build 24.51-b03, mixed mode) >> >> ~# free -m >> total used free shared buffers cached >> Mem: 5418 572 4846 0 25 245 >> -/+ buffers/cache: 301 5117 >> Swap: 127 0 127 >> >> # uptime >> 01:21:08 up 2:02, 3 users, load average: 13.47, 13.65, 13.63 >> >> -- >> You are receiving this mail because: >> You are the assignee for the bug. > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-18 22:18 ` Laura Abbott @ 2018-01-19 3:04 ` Matthew Wilcox 2018-01-19 3:14 ` xen 2018-01-19 13:33 ` Matthew Wilcox 1 sibling, 1 reply; 16+ messages in thread From: Matthew Wilcox @ 2018-01-19 3:04 UTC (permalink / raw) To: Laura Abbott; +Cc: Andrew Morton, linux-mm, bugzilla-daemon, peter On Thu, Jan 18, 2018 at 02:18:20PM -0800, Laura Abbott wrote: > On 01/18/2018 01:55 PM, Andrew Morton wrote: > > > [ 24.647744] BUG: unable to handle kernel NULL pointer dereference at > > > 00000008 > > > [ 24.647801] IP: __radix_tree_lookup+0x14/0xa0 > > > [ 24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 > > > [ 24.647828] Oops: 0000 [#1] SMP > > > [ 24.647842] CPU: 5 PID: 3600 Comm: java Not tainted > > > 4.14.13-rh10-20180115190010.xenU.i386 #1 > > > [ 24.647855] task: e52518c0 task.stack: e4e7a000 > > > [ 24.647866] EIP: __radix_tree_lookup+0x14/0xa0 > > > [ 24.647876] EFLAGS: 00010286 CPU: 5 > > > [ 24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 > > > [ 24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0 > > > [ 24.647904] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 > > > [ 24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660 > > > [ 24.647930] Call Trace: > > > [ 24.647942] radix_tree_lookup_slot+0x13/0x30 > > > [ 24.647955] find_get_entry+0x1d/0x120 > > > [ 24.647963] pagecache_get_page+0x1f/0x230 > > > [ 24.647975] lookup_swap_cache+0x42/0x140 > > > [ 24.647983] swap_readahead_detect+0x66/0x2e0 > > > [ 24.647993] do_swap_page+0x1fa/0x860 > > > [ 24.648010] ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10 > > > [ 24.648026] ? xen_pmd_val+0x10/0x20 > > > [ 24.648035] handle_mm_fault+0x6f8/0x1020 > > > [ 24.648046] __do_page_fault+0x18a/0x450 > > > [ 24.648055] ? vmalloc_sync_all+0x250/0x250 > > > [ 24.648063] do_page_fault+0x21/0x30 > > > [ 24.648074] common_exception+0x45/0x4a > > > [ 24.648082] EIP: 0xb76d873e > > > [ 24.648088] EFLAGS: 00010206 CPU: 5 > > > [ 24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006 > > > [ 24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8 > > > [ 24.648115] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b > > > [ 24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff > > > ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 > > > 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 > > > [ 24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0 > > > [ 24.648205] CR2: 0000000000000008 > > > [ 24.648273] ---[ end trace ed356e59f215ce07 ]--- Running that code through decodecode, I get: 0: 55 push %ebp 1: 89 e5 mov %esp,%ebp 3: 57 push %edi 4: 89 d7 mov %edx,%edi 6: 56 push %esi 7: 53 push %ebx 8: 83 ec 0c sub $0xc,%esp b: 89 45 ec mov %eax,-0x14(%ebp) e: 89 4d e8 mov %ecx,-0x18(%ebp) 11: 8b 45 ec mov -0x14(%ebp),%eax 14:* 8b 58 04 mov 0x4(%eax),%ebx <-- trapping instruction 17: 89 d8 mov %ebx,%eax 19: 83 e0 03 and $0x3,%eax Which I think means it's looking at offset 4 from whichever argument the x86 calling convention puts in register %eax. Which I think is argument 0? Which is the radix tree root. And that makes sense; we're loading the root node from the radix tree root at offset 4. The problem is that %eax has the value 4 in it. That would match with 'page_tree' being at offset 4 from the start of address_space. So find_get_page() got called with a NULL mapping, so pagecache_get_page() got called with a NULL mapping. Which means I've tracked it back to: page = find_get_page(swap_address_space(entry), swp_offset(entry)); and swap_address_space() is returning NULL. Has this machine run swapoff recently, perhaps? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-19 3:04 ` Matthew Wilcox @ 2018-01-19 3:14 ` xen 2018-01-19 13:21 ` Matthew Wilcox 0 siblings, 1 reply; 16+ messages in thread From: xen @ 2018-01-19 3:14 UTC (permalink / raw) To: Matthew Wilcox, Laura Abbott; +Cc: Andrew Morton, linux-mm, bugzilla-daemon [-- Attachment #1: Type: text/plain, Size: 4368 bytes --] On 19/01/18 4:04 PM, Matthew Wilcox wrote: > On Thu, Jan 18, 2018 at 02:18:20PM -0800, Laura Abbott wrote: >> On 01/18/2018 01:55 PM, Andrew Morton wrote: >>>> [ 24.647744] BUG: unable to handle kernel NULL pointer dereference at >>>> 00000008 >>>> [ 24.647801] IP: __radix_tree_lookup+0x14/0xa0 >>>> [ 24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 >>>> [ 24.647828] Oops: 0000 [#1] SMP >>>> [ 24.647842] CPU: 5 PID: 3600 Comm: java Not tainted >>>> 4.14.13-rh10-20180115190010.xenU.i386 #1 >>>> [ 24.647855] task: e52518c0 task.stack: e4e7a000 >>>> [ 24.647866] EIP: __radix_tree_lookup+0x14/0xa0 >>>> [ 24.647876] EFLAGS: 00010286 CPU: 5 >>>> [ 24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 >>>> [ 24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0 >>>> [ 24.647904] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 >>>> [ 24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660 >>>> [ 24.647930] Call Trace: >>>> [ 24.647942] radix_tree_lookup_slot+0x13/0x30 >>>> [ 24.647955] find_get_entry+0x1d/0x120 >>>> [ 24.647963] pagecache_get_page+0x1f/0x230 >>>> [ 24.647975] lookup_swap_cache+0x42/0x140 >>>> [ 24.647983] swap_readahead_detect+0x66/0x2e0 >>>> [ 24.647993] do_swap_page+0x1fa/0x860 >>>> [ 24.648010] ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10 >>>> [ 24.648026] ? xen_pmd_val+0x10/0x20 >>>> [ 24.648035] handle_mm_fault+0x6f8/0x1020 >>>> [ 24.648046] __do_page_fault+0x18a/0x450 >>>> [ 24.648055] ? vmalloc_sync_all+0x250/0x250 >>>> [ 24.648063] do_page_fault+0x21/0x30 >>>> [ 24.648074] common_exception+0x45/0x4a >>>> [ 24.648082] EIP: 0xb76d873e >>>> [ 24.648088] EFLAGS: 00010206 CPU: 5 >>>> [ 24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006 >>>> [ 24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8 >>>> [ 24.648115] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b >>>> [ 24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff >>>> ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 >>>> 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 >>>> [ 24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0 >>>> [ 24.648205] CR2: 0000000000000008 >>>> [ 24.648273] ---[ end trace ed356e59f215ce07 ]--- > Running that code through decodecode, I get: > > 0: 55 push %ebp > 1: 89 e5 mov %esp,%ebp > 3: 57 push %edi > 4: 89 d7 mov %edx,%edi > 6: 56 push %esi > 7: 53 push %ebx > 8: 83 ec 0c sub $0xc,%esp > b: 89 45 ec mov %eax,-0x14(%ebp) > e: 89 4d e8 mov %ecx,-0x18(%ebp) > 11: 8b 45 ec mov -0x14(%ebp),%eax > 14:* 8b 58 04 mov 0x4(%eax),%ebx <-- trapping instruction > 17: 89 d8 mov %ebx,%eax > 19: 83 e0 03 and $0x3,%eax > > Which I think means it's looking at offset 4 from whichever argument > the x86 calling convention puts in register %eax. Which I think is > argument 0? Which is the radix tree root. And that makes sense; we're > loading the root node from the radix tree root at offset 4. The problem > is that %eax has the value 4 in it. That would match with 'page_tree' > being at offset 4 from the start of address_space. So find_get_page() > got called with a NULL mapping, so pagecache_get_page() got called > with a NULL mapping. > > Which means I've tracked it back to: > > page = find_get_page(swap_address_space(entry), swp_offset(entry)); > > and swap_address_space() is returning NULL. Has this machine run swapoff > recently, perhaps? Swap was on.A Swap is small (127MB).A Swap had not been dipped into. A A A A A A A A A A A A totalA A A A A A usedA A A A A A freeA A A A sharedA A A buffers cached Swap:A A A A A A A A A 127A A A A A A A A A 0A A A A A A A 127 PS: cannot recall seeing this issue on x86_64, just 32 bit.A PPS: reminder this is on a Xen VM which per https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options has "out of sync pagetables" if that is relevant (we do not set that option, I am unsure what default is used). [-- Attachment #2: Type: text/html, Size: 5404 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-19 3:14 ` xen @ 2018-01-19 13:21 ` Matthew Wilcox 2018-01-19 17:30 ` Laura Abbott 0 siblings, 1 reply; 16+ messages in thread From: Matthew Wilcox @ 2018-01-19 13:21 UTC (permalink / raw) To: xen; +Cc: Laura Abbott, Andrew Morton, linux-mm, bugzilla-daemon On Fri, Jan 19, 2018 at 04:14:42PM +1300, xen@randomwebstuff.com wrote: > > On 19/01/18 4:04 PM, Matthew Wilcox wrote: > > On Thu, Jan 18, 2018 at 02:18:20PM -0800, Laura Abbott wrote: > > > On 01/18/2018 01:55 PM, Andrew Morton wrote: > > > > > [ 24.647744] BUG: unable to handle kernel NULL pointer dereference at > > > > > 00000008 > > > > > [ 24.647801] IP: __radix_tree_lookup+0x14/0xa0 > > > > > [ 24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 > > > > > [ 24.647828] Oops: 0000 [#1] SMP > > > > > [ 24.647842] CPU: 5 PID: 3600 Comm: java Not tainted > > > > > 4.14.13-rh10-20180115190010.xenU.i386 #1 > > > > > [ 24.647855] task: e52518c0 task.stack: e4e7a000 > > > > > [ 24.647866] EIP: __radix_tree_lookup+0x14/0xa0 > > > > > [ 24.647876] EFLAGS: 00010286 CPU: 5 > > > > > [ 24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 If my understanding is right, EDX contains the index we're looking up. Which is zero. So the swp_entry we got is one bit away from being NULL. Hmm. Have you run memtest86 or some other memory tester on the system recently? > PS: cannot recall seeing this issue on x86_64, just 32 bit. Laura has 64-bit instances of this. PPS: reminder > this is on a Xen VM which per https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options > has "out of sync pagetables" if that is relevant (we do not set that option, > I am unsure what default is used). Laura also has non-Xen instances of this. They may not all be the same bug, of course. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-19 13:21 ` Matthew Wilcox @ 2018-01-19 17:30 ` Laura Abbott 2018-01-26 6:54 ` xen 0 siblings, 1 reply; 16+ messages in thread From: Laura Abbott @ 2018-01-19 17:30 UTC (permalink / raw) To: Matthew Wilcox, xen; +Cc: Andrew Morton, linux-mm, bugzilla-daemon On 01/19/2018 05:21 AM, Matthew Wilcox wrote: > On Fri, Jan 19, 2018 at 04:14:42PM +1300, xen@randomwebstuff.com wrote: >> >> On 19/01/18 4:04 PM, Matthew Wilcox wrote: >>> On Thu, Jan 18, 2018 at 02:18:20PM -0800, Laura Abbott wrote: >>>> On 01/18/2018 01:55 PM, Andrew Morton wrote: >>>>>> [ 24.647744] BUG: unable to handle kernel NULL pointer dereference at >>>>>> 00000008 >>>>>> [ 24.647801] IP: __radix_tree_lookup+0x14/0xa0 >>>>>> [ 24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 >>>>>> [ 24.647828] Oops: 0000 [#1] SMP >>>>>> [ 24.647842] CPU: 5 PID: 3600 Comm: java Not tainted >>>>>> 4.14.13-rh10-20180115190010.xenU.i386 #1 >>>>>> [ 24.647855] task: e52518c0 task.stack: e4e7a000 >>>>>> [ 24.647866] EIP: __radix_tree_lookup+0x14/0xa0 >>>>>> [ 24.647876] EFLAGS: 00010286 CPU: 5 >>>>>> [ 24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000 > > If my understanding is right, EDX contains the index we're looking up. > Which is zero. So the swp_entry we got is one bit away from being NULL. > Hmm. Have you run memtest86 or some other memory tester on the system > recently? > >> PS: cannot recall seeing this issue on x86_64, just 32 bit. > > Laura has 64-bit instances of this. > The 64-bit backtraces reported in the bugzilla looked different, I would consider it a different issue. > PPS: reminder >> this is on a Xen VM which per https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options >> has "out of sync pagetables" if that is relevant (we do not set that option, >> I am unsure what default is used). > > Laura also has non-Xen instances of this. They may not all be the same > bug, of course. > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-19 17:30 ` Laura Abbott @ 2018-01-26 6:54 ` xen 2018-01-26 19:40 ` Matthew Wilcox 0 siblings, 1 reply; 16+ messages in thread From: xen @ 2018-01-26 6:54 UTC (permalink / raw) To: Laura Abbott, Matthew Wilcox; +Cc: Andrew Morton, linux-mm, bugzilla-daemon [-- Attachment #1: Type: text/plain, Size: 5030 bytes --] On 20/01/18 6:30 AM, Laura Abbott wrote: > On 01/19/2018 05:21 AM, Matthew Wilcox wrote: >> On Fri, Jan 19, 2018 at 04:14:42PM +1300, xen@randomwebstuff.com wrote: >>> >>> On 19/01/18 4:04 PM, Matthew Wilcox wrote: >>>> On Thu, Jan 18, 2018 at 02:18:20PM -0800, Laura Abbott wrote: >>>>> On 01/18/2018 01:55 PM, Andrew Morton wrote: >>>>>>> [A A 24.647744] BUG: unable to handle kernel NULL pointer >>>>>>> dereference at >>>>>>> 00000008 >>>>>>> [A A 24.647801] IP: __radix_tree_lookup+0x14/0xa0 >>>>>>> [A A 24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 >>>>>>> [A A 24.647828] Oops: 0000 [#1] SMP >>>>>>> [A A 24.647842] CPU: 5 PID: 3600 Comm: java Not tainted >>>>>>> 4.14.13-rh10-20180115190010.xenU.i386 #1 >>>>>>> [A A 24.647855] task: e52518c0 task.stack: e4e7a000 >>>>>>> [A A 24.647866] EIP: __radix_tree_lookup+0x14/0xa0 >>>>>>> [A A 24.647876] EFLAGS: 00010286 CPU: 5 >>>>>>> [A A 24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: >>>>>>> 00000000 >> >> If my understanding is right, EDX contains the index we're looking up. >> Which is zero.A So the swp_entry we got is one bit away from being NULL. >> Hmm.A Have you run memtest86 or some other memory tester on the system >> recently? >> >>> PS: cannot recall seeing this issue on x86_64, just 32 bit. >> >> Laura has 64-bit instances of this. >> > > The 64-bit backtraces reported in the bugzilla looked different, > I would consider it a different issue. > >> PPS: reminder >>> this is on a Xen VM which per >>> https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options >>> has "out of sync pagetables" if that is relevant (we do not set that >>> option, >>> I am unsure what default is used). >> >> Laura also has non-Xen instances of this.A They may not all be the same >> bug, of course. >> Re-tried with the current latest 4.14 (4.14.15).A Received the following: [2018-01-24 19:26:57] Ubuntu 14.04.5 LTS dev hvc0 [2018-01-24 19:26:57] [2018-01-24 19:26:57] dev login: [44501.106868] BUG: unable to handle kernel NULL pointer dereference at 00000008 [2018-01-25 07:47:50] [44501.106897] IP: __radix_tree_lookup+0x14/0xa0 [2018-01-25 07:47:50] [44501.106905] *pdpt = 000000001fe82027 *pde = 0000000000000000 [2018-01-25 07:47:50] [44501.106916] Oops: 0000 [#1] SMP [2018-01-25 07:47:50] [44501.106924] CPU: 0 PID: 3344 Comm: PassengerAgent Not tainted 4.14.15-rh13-20180123235331.xenU.i386 #1 [2018-01-25 07:47:50] [44501.106935] task: dfee39c0 task.stack: dff12000 [2018-01-25 07:47:50] [44501.106943] EIP: __radix_tree_lookup+0x14/0xa0 [2018-01-25 07:47:50] [44501.106950] EFLAGS: 00210286 CPU: 0 [2018-01-25 07:47:50] [44501.106955] EAX: 00000004 EBX: 00000001 ECX: 00000000 EDX: 00000000 [2018-01-25 07:47:50] [44501.106963] ESI: 00000000 EDI: 00000000 EBP: dff13db8 ESP: dff13da0 [2018-01-25 07:47:50] [44501.106971] A DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 [2018-01-25 07:47:50] [44501.106979] CR0: 80050033 CR2: 00000008 CR3: 1fdb1000 CR4: 00002660 [2018-01-25 07:47:50] [44501.106989] Call Trace: [2018-01-25 07:47:50] [44501.106995] A radix_tree_lookup_slot+0x13/0x30 [2018-01-25 07:47:50] [44501.107004] A find_get_entry+0x1d/0x120 [2018-01-25 07:47:50] [44501.107011] A pagecache_get_page+0x1f/0x230 [2018-01-25 07:47:50] [44501.107018] A lookup_swap_cache+0x42/0x140 [2018-01-25 07:47:50] [44501.107024] A swap_readahead_detect+0x66/0x2e0 [2018-01-25 07:47:50] [44501.107032] A do_swap_page+0x1fa/0x860 [2018-01-25 07:47:50] [44501.107040] A ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10 [2018-01-25 07:47:50] [44501.107050] A ? xen_pmd_val+0x10/0x20 [2018-01-25 07:47:50] [44501.107057] A handle_mm_fault+0x6f8/0x1020 [2018-01-25 07:47:50] [44501.107065] A ? _raw_spin_unlock_irqrestore+0x13/0x20 [2018-01-25 07:47:50] [44501.107074] A ? pvclock_clocksource_read+0xa6/0x1a0 [2018-01-25 07:47:50] [44501.107081] A __do_page_fault+0x18a/0x450 [2018-01-25 07:47:50] [44501.107089] A ? _copy_to_user+0x28/0x40 [2018-01-25 07:47:50] [44501.107096] A ? vmalloc_sync_all+0x250/0x250 [2018-01-25 07:47:50] [44501.107102] A do_page_fault+0x21/0x30 [2018-01-25 07:47:50] [44501.107109] A common_exception+0x45/0x4a [2018-01-25 07:47:50] [44501.107115] EIP: 0x82c3358 [2018-01-25 07:47:50] [44501.107120] EFLAGS: 00210202 CPU: 0 [2018-01-25 07:47:50] [44501.107126] EAX: b702d0b8 EBX: 081557a9 ECX: 00000000 EDX: 0a4296bc [2018-01-25 07:47:50] [44501.107133] ESI: b467c2cc EDI: 00000000 EBP: b467c138 ESP: b467c110 [2018-01-25 07:47:50] [44501.107141] A DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b [2018-01-25 07:47:50] [44501.107147] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 [2018-01-25 07:47:50] [44501.110296] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:dff13da0 [2018-01-25 07:47:50] [44501.110304] CR2: 0000000000000008 [2018-01-25 07:47:50] [44501.110356] ---[ end trace 89cdd2ba8e7323a8 ]--- [-- Attachment #2: Type: text/html, Size: 43307 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-26 6:54 ` xen @ 2018-01-26 19:40 ` Matthew Wilcox 2018-01-29 22:26 ` xen 0 siblings, 1 reply; 16+ messages in thread From: Matthew Wilcox @ 2018-01-26 19:40 UTC (permalink / raw) To: xen; +Cc: Laura Abbott, Andrew Morton, linux-mm, bugzilla-daemon On Fri, Jan 26, 2018 at 07:54:06PM +1300, xen@randomwebstuff.com wrote: > Re-tried with the current latest 4.14 (4.14.15). Received the following: > > [2018-01-24 19:26:57] dev login: [44501.106868] BUG: unable to handle kernel > NULL pointer dereference at 00000008 > [2018-01-25 07:47:50] [44501.106897] IP: __radix_tree_lookup+0x14/0xa0 Please try including this patch: https://bugzilla.kernel.org/show_bug.cgi?id=198497#c7 And have you had the chance to run memtest86 yet? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-26 19:40 ` Matthew Wilcox @ 2018-01-29 22:26 ` xen 2018-01-31 10:54 ` Matthew Wilcox 0 siblings, 1 reply; 16+ messages in thread From: xen @ 2018-01-29 22:26 UTC (permalink / raw) To: Matthew Wilcox; +Cc: Laura Abbott, Andrew Morton, linux-mm, bugzilla-daemon [-- Attachment #1: Type: text/plain, Size: 2881 bytes --] On 27/01/18 8:40 AM, Matthew Wilcox wrote: > On Fri, Jan 26, 2018 at 07:54:06PM +1300, xen@randomwebstuff.com wrote: >> Re-tried with the current latest 4.14 (4.14.15).A Received the following: >> >> [2018-01-24 19:26:57] dev login: [44501.106868] BUG: unable to handle kernel >> NULL pointer dereference at 00000008 >> [2018-01-25 07:47:50] [44501.106897] IP: __radix_tree_lookup+0x14/0xa0 > Please try including this patch: > > https://bugzilla.kernel.org/show_bug.cgi?id=198497#c7 > > And have you had the chance to run memtest86 yet? Added the patch at https://bugzilla.kernel.org/show_bug.cgi?id=198497#c7 After, received this stack. Have not tried memtest86.A These are production hosts.A This has occurred on multiple hosts.A I can only recall this occurring on 32 bit kernels.A I cannot recall issues with other VMs not running that kernel on the same hosts. [ A 125.329163] Bad swp_entry: e000000 [ A 125.329202] ------------[ cut here ]------------ [ A 125.329219] WARNING: CPU: 0 PID: 4175 at mm/swap_state.c:339 lookup_swap_cache+0x140/0x160 [ A 125.329233] CPU: 0 PID: 4175 Comm: apt-show-versio Not tainted 4.14.15-rh14-20180126233810.xenU.i386-00001-g6ba70cb #1 [ A 125.329245] task: ead9a940 task.stack: e7c8c000 [ A 125.329253] EIP: lookup_swap_cache+0x140/0x160 [ A 125.329260] EFLAGS: 00010282 CPU: 0 [ A 125.329267] EAX: 00000016 EBX: 00000000 ECX: ec5289c4 EDX: 0100016d [ A 125.329275] ESI: b6312000 EDI: e7d94ea0 EBP: e7c8de24 ESP: e7c8de0c [ A 125.329284] A DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069 [ A 125.329295] CR0: 80050033 CR2: b63124b0 CR3: 2718c000 CR4: 00002660 [ A 125.329308] Call Trace: [ A 125.329323] A ? percpu_counter_add_batch+0x91/0xb0 [ A 125.329332] A swap_readahead_detect+0x66/0x2e0 [ A 125.329343] A ? radix_tree_tag_set+0x7a/0xe0 [ A 125.329352] A do_swap_page+0x1fa/0x860 [ A 125.329361] A ? __set_page_dirty_buffers+0xb1/0xe0 [ A 125.329372] A ? ext4_set_page_dirty+0x22/0x60 [ A 125.329383] A ? fault_dirty_shared_page.isra.90+0x3e/0xa0 [ A 125.329396] A ? xen_pmd_val+0x10/0x20 [ A 125.329403] A handle_mm_fault+0x6f8/0x1020 [ A 125.329414] A ? handle_irq_event_percpu+0x3c/0x50 [ A 125.329424] A __do_page_fault+0x18a/0x450 [ A 125.329432] A ? vmalloc_sync_all+0x250/0x250 [ A 125.329439] A do_page_fault+0x21/0x30 [ A 125.329449] A common_exception+0x45/0x4a [ A 125.329456] EIP: 0xb7ce397b [ A 125.329462] EFLAGS: 00010202 CPU: 0 [ A 125.329469] EAX: 0000052a EBX: b7d77ff4 ECX: 000004fa EDX: b6311000 [ A 125.329477] ESI: bf90eae0 EDI: b6ed4b20 EBP: bf90ea60 ESP: bf90ea20 [ A 125.329486] A DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b [ A 125.329493] Code: 18 1f 14 c2 85 ff 0f 85 41 ff ff ff f0 ff 05 38 fb 02 c2 e9 35 ff ff ff 8d 76 00 89 44 24 04 c7 04 24 55 93 f3 c1 e8 8c e7 f5 ff <0f> ff 8b 5d f4 31 c0 8b 75 f8 8b 7d fc 89 ec 5d c3 64 ff 05 18 [ A 125.329558] ---[ end trace dd2704ca649b44ba ]--- [-- Attachment #2: Type: text/html, Size: 34424 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-29 22:26 ` xen @ 2018-01-31 10:54 ` Matthew Wilcox 2018-01-31 23:02 ` Tetsuo Handa 2018-02-09 14:47 ` Matthew Wilcox 0 siblings, 2 replies; 16+ messages in thread From: Matthew Wilcox @ 2018-01-31 10:54 UTC (permalink / raw) To: xen; +Cc: Laura Abbott, Andrew Morton, linux-mm, bugzilla-daemon On Tue, Jan 30, 2018 at 11:26:42AM +1300, xen@randonwebstuff.com wrote: > After, received this stack. > > Have not tried memtest86. These are production hosts. This has occurred on > multiple hosts. I can only recall this occurring on 32 bit kernels. I > cannot recall issues with other VMs not running that kernel on the same > hosts. > > [ 125.329163] Bad swp_entry: e000000 Mixed news here then ... 'e' is 8 | 4 | 2, so it's not a single bitflip. So no point in running memtest86. I should have made the printk produce leading zeroes, because that's 0x0e00'0000. ptes use the top 5 bits to encode the swapfile, so this swap entry is decoded as swapfile 1, page number 0x0600'0000. That's clearly ludicrous because you don't have a swapfile 1, and if you did, it wouldn't be so large as a terabyte. I think the next step in debugging this is printing the PTE which gave us this swp_entry. If you can drop the patch I asked you to try, and apply this patch instead, we'll have more idea about what's going on. Thanks! diff --git a/mm/memory.c b/mm/memory.c index 403934297a3d..8caaddb07747 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2892,6 +2892,10 @@ int do_swap_page(struct vm_fault *vmf) if (!page) page = lookup_swap_cache(entry, vma_readahead ? vma : NULL, vmf->address); + if (IS_ERR(page)) { + pte_ERROR(vmf->orig_pte); + page = NULL; + } if (!page) { struct swap_info_struct *si = swp_swap_info(entry); diff --git a/mm/shmem.c b/mm/shmem.c index 7fbe67be86fa..905fa34e022a 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1651,6 +1651,10 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (swap.val) { /* Look it up and read it in.. */ page = lookup_swap_cache(swap, NULL, 0); + if (IS_ERR(page)) { + pte_ERROR(vmf->orig_pte); + page = NULL; + } if (!page) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { diff --git a/mm/swap_state.c b/mm/swap_state.c index 39ae7cfad90f..7ee594c8eadd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -334,8 +334,14 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, struct page *page; unsigned long ra_info; int win, hits, readahead; + struct address_space *swapper_space = swap_address_space(entry); + + if (!swapper_space) { + pr_err("Bad swp_entry: %lx\n", entry.val); + return ERR_PTR(-EFAULT); + } - page = find_get_page(swap_address_space(entry), swp_offset(entry)); + page = find_get_page(swapper_space, swp_offset(entry)); INC_CACHE_INFO(find_total); if (page) { @@ -676,6 +682,10 @@ struct page *swap_readahead_detect(struct vm_fault *vmf, if ((unlikely(non_swap_entry(entry)))) return NULL; page = lookup_swap_cache(entry, vma, faddr); + if (IS_ERR(page)) { + pte_ERROR(vmf->orig_pte); + page = NULL; + } if (page) return page; -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-31 10:54 ` Matthew Wilcox @ 2018-01-31 23:02 ` Tetsuo Handa 2018-02-01 9:48 ` Matthew Wilcox 2018-02-09 14:47 ` Matthew Wilcox 1 sibling, 1 reply; 16+ messages in thread From: Tetsuo Handa @ 2018-01-31 23:02 UTC (permalink / raw) To: Matthew Wilcox, xen Cc: Laura Abbott, Andrew Morton, linux-mm, bugzilla-daemon https://bugzilla.redhat.com/show_bug.cgi?id=1531779 It might be something related that "x86/mm: Found insecure W+X mapping at address" message is printed at boot. Are you seeing "x86/mm: Found insecure W+X mapping at address" before hitting "BUG: unable to handle kernel NULL pointer dereference" ? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-31 23:02 ` Tetsuo Handa @ 2018-02-01 9:48 ` Matthew Wilcox 0 siblings, 0 replies; 16+ messages in thread From: Matthew Wilcox @ 2018-02-01 9:48 UTC (permalink / raw) To: Tetsuo Handa; +Cc: xen, Laura Abbott, Andrew Morton, linux-mm, bugzilla-daemon On Thu, Feb 01, 2018 at 08:02:43AM +0900, Tetsuo Handa wrote: > https://bugzilla.redhat.com/show_bug.cgi?id=1531779 > > It might be something related that > "x86/mm: Found insecure W+X mapping at address" message is printed at boot. > > Are you seeing "x86/mm: Found insecure W+X mapping at address" before > hitting "BUG: unable to handle kernel NULL pointer dereference" ? There are about eight different bugs in that thread; the only commonality I see between them is that there's a null pointer dereference somewhere in the kernel. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-31 10:54 ` Matthew Wilcox 2018-01-31 23:02 ` Tetsuo Handa @ 2018-02-09 14:47 ` Matthew Wilcox 2018-04-12 17:12 ` Andrew Morton 1 sibling, 1 reply; 16+ messages in thread From: Matthew Wilcox @ 2018-02-09 14:47 UTC (permalink / raw) To: xen; +Cc: Laura Abbott, Andrew Morton, linux-mm, bugzilla-daemon ping? On Wed, Jan 31, 2018 at 02:54:56AM -0800, Matthew Wilcox wrote: > On Tue, Jan 30, 2018 at 11:26:42AM +1300, xen@randonwebstuff.com wrote: > > After, received this stack. > > > > Have not tried memtest86. These are production hosts. This has occurred on > > multiple hosts. I can only recall this occurring on 32 bit kernels. I > > cannot recall issues with other VMs not running that kernel on the same > > hosts. > > > > [ 125.329163] Bad swp_entry: e000000 > > Mixed news here then ... 'e' is 8 | 4 | 2, so it's not a single bitflip. > So no point in running memtest86. > > I should have made the printk produce leading zeroes, because that's > 0x0e00'0000. ptes use the top 5 bits to encode the swapfile, so > this swap entry is decoded as swapfile 1, page number 0x0600'0000. > That's clearly ludicrous because you don't have a swapfile 1, and if > you did, it wouldn't be so large as a terabyte. > > I think the next step in debugging this is printing the PTE which gave > us this swp_entry. If you can drop the patch I asked you to try, and > apply this patch instead, we'll have more idea about what's going on. > > Thanks! > > diff --git a/mm/memory.c b/mm/memory.c > index 403934297a3d..8caaddb07747 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -2892,6 +2892,10 @@ int do_swap_page(struct vm_fault *vmf) > if (!page) > page = lookup_swap_cache(entry, vma_readahead ? vma : NULL, > vmf->address); > + if (IS_ERR(page)) { > + pte_ERROR(vmf->orig_pte); > + page = NULL; > + } > if (!page) { > struct swap_info_struct *si = swp_swap_info(entry); > > diff --git a/mm/shmem.c b/mm/shmem.c > index 7fbe67be86fa..905fa34e022a 100644 > --- a/mm/shmem.c > +++ b/mm/shmem.c > @@ -1651,6 +1651,10 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, > if (swap.val) { > /* Look it up and read it in.. */ > page = lookup_swap_cache(swap, NULL, 0); > + if (IS_ERR(page)) { > + pte_ERROR(vmf->orig_pte); > + page = NULL; > + } > if (!page) { > /* Or update major stats only when swapin succeeds?? */ > if (fault_type) { > diff --git a/mm/swap_state.c b/mm/swap_state.c > index 39ae7cfad90f..7ee594c8eadd 100644 > --- a/mm/swap_state.c > +++ b/mm/swap_state.c > @@ -334,8 +334,14 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, > struct page *page; > unsigned long ra_info; > int win, hits, readahead; > + struct address_space *swapper_space = swap_address_space(entry); > + > + if (!swapper_space) { > + pr_err("Bad swp_entry: %lx\n", entry.val); > + return ERR_PTR(-EFAULT); > + } > > - page = find_get_page(swap_address_space(entry), swp_offset(entry)); > + page = find_get_page(swapper_space, swp_offset(entry)); > > INC_CACHE_INFO(find_total); > if (page) { > @@ -676,6 +682,10 @@ struct page *swap_readahead_detect(struct vm_fault *vmf, > if ((unlikely(non_swap_entry(entry)))) > return NULL; > page = lookup_swap_cache(entry, vma, faddr); > + if (IS_ERR(page)) { > + pte_ERROR(vmf->orig_pte); > + page = NULL; > + } > if (page) > return page; > > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-02-09 14:47 ` Matthew Wilcox @ 2018-04-12 17:12 ` Andrew Morton 2018-04-12 17:28 ` Matthew Wilcox 0 siblings, 1 reply; 16+ messages in thread From: Andrew Morton @ 2018-04-12 17:12 UTC (permalink / raw) To: Matthew Wilcox; +Cc: xen, Laura Abbott, linux-mm, bugzilla-daemon On Fri, 9 Feb 2018 06:47:26 -0800 Matthew Wilcox <willy@infradead.org> wrote: > > ping? > There have been a bunch of updates to this issue in bugzilla (https://bugzilla.kernel.org/show_bug.cgi?id=198497). Sigh, I don't know what to do about this - maybe there's some way of getting bugzilla to echo everything to linux-mm or something. Anyway, please take a look - we appear to have a bug here. Perhaps this bug is sufficiently gnarly for you to prepare a debugging patch which we can add to the mainline kernel so we get (much) more debugging info when people hit it? ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-04-12 17:12 ` Andrew Morton @ 2018-04-12 17:28 ` Matthew Wilcox 0 siblings, 0 replies; 16+ messages in thread From: Matthew Wilcox @ 2018-04-12 17:28 UTC (permalink / raw) To: Andrew Morton; +Cc: xen, Laura Abbott, linux-mm, bugzilla-daemon On Thu, Apr 12, 2018 at 10:12:09AM -0700, Andrew Morton wrote: > On Fri, 9 Feb 2018 06:47:26 -0800 Matthew Wilcox <willy@infradead.org> wrote: > > > > > ping? > > > > There have been a bunch of updates to this issue in bugzilla > (https://bugzilla.kernel.org/show_bug.cgi?id=198497). Sigh, I don't > know what to do about this - maybe there's some way of getting bugzilla > to echo everything to linux-mm or something. > > Anyway, please take a look - we appear to have a bug here. Perhaps > this bug is sufficiently gnarly for you to prepare a debugging patch > which we can add to the mainline kernel so we get (much) more debugging > info when people hit it? I have a few thoughts ... - The debugging patch I prepared appears to be doing its job well. People get the message and their machine stays working. - The commonality appears to be Xen running 32-bit kernels. Maybe we can kick the problem over to them to solve? - If we are seeing corruption purely in the lower bits, *we'll never know*. The radix tree lookup will simply not find anything, and all will be well. That said, the bad PTE values reported in that bug have the NX bit and one other bit set; generally bit 32, 33 or 34. I have an idea for adding a parity bit, but haven't had time to implement it. Anyone have an intern who wants an interesting kernel project to work on? Given that this is happening on Xen, I wonder if Xen is using some of the bits in the page table for its own purposes. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer 2018-01-18 22:18 ` Laura Abbott 2018-01-19 3:04 ` Matthew Wilcox @ 2018-01-19 13:33 ` Matthew Wilcox 1 sibling, 0 replies; 16+ messages in thread From: Matthew Wilcox @ 2018-01-19 13:33 UTC (permalink / raw) To: Laura Abbott; +Cc: Andrew Morton, linux-mm, bugzilla-daemon, peter On Thu, Jan 18, 2018 at 02:18:20PM -0800, Laura Abbott wrote: > Fedora has been seeing similar reports > https://bugzilla.redhat.com/show_bug.cgi?id=1531779 > > Multiple reporters, one in XEN, another on actual hardware Can you chuck this patch into Fedora? Should make it easier to see if it's a "stuck bit" kind of a problem. --- From: Matthew Wilcox <mawilcox@microsoft.com> Subject: Detect bad swap entries in lookup If we have a stuck bit in a PTE, we can end up looking for an entry in a NULL mapping, which oopses fairly quickly. Print a warning to help us debug, and return NULL which will help the machine survive a little longer. Although if it has a permanently stuck bit in a PTE, there's only a 50% chance it'll surive the insertion of a real PTE into that entry. Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com> diff --git a/mm/swap_state.c b/mm/swap_state.c index 39ae7cfad90f..5a928e0191a1 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -334,8 +334,12 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, struct page *page; unsigned long ra_info; int win, hits, readahead; + struct address_space *swapper_space = swap_address_space(entry); + + if (WARN(!swapper_space, "Bad swp_entry: %lx\n", entry.val)) + return NULL; - page = find_get_page(swap_address_space(entry), swp_offset(entry)); + page = find_get_page(swapper_space, swp_offset(entry)); INC_CACHE_INFO(find_total); if (page) { -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 16+ messages in thread
end of thread, other threads:[~2018-04-12 17:28 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <bug-198497-27@https.bugzilla.kernel.org/>
2018-01-18 21:55 ` [Bug 198497] New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer Andrew Morton
2018-01-18 22:18 ` Laura Abbott
2018-01-19 3:04 ` Matthew Wilcox
2018-01-19 3:14 ` xen
2018-01-19 13:21 ` Matthew Wilcox
2018-01-19 17:30 ` Laura Abbott
2018-01-26 6:54 ` xen
2018-01-26 19:40 ` Matthew Wilcox
2018-01-29 22:26 ` xen
2018-01-31 10:54 ` Matthew Wilcox
2018-01-31 23:02 ` Tetsuo Handa
2018-02-01 9:48 ` Matthew Wilcox
2018-02-09 14:47 ` Matthew Wilcox
2018-04-12 17:12 ` Andrew Morton
2018-04-12 17:28 ` Matthew Wilcox
2018-01-19 13:33 ` Matthew Wilcox
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).