From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D630EB64D7 for ; Wed, 21 Jun 2023 03:52:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF6D48D0002; Tue, 20 Jun 2023 23:52:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA52F8D0001; Tue, 20 Jun 2023 23:52:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A94A68D0002; Tue, 20 Jun 2023 23:52:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9B0D68D0001 for ; Tue, 20 Jun 2023 23:52:10 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 65B8C160308 for ; Wed, 21 Jun 2023 03:52:10 +0000 (UTC) X-FDA: 80925382020.23.D503409 Received: from gandalf.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by imf25.hostedemail.com (Postfix) with ESMTP id E39E0A0005 for ; Wed, 21 Jun 2023 03:52:07 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=ellerman.id.au header.s=201909 header.b=qkWxn4h+; spf=pass (imf25.hostedemail.com: domain of mpe@ellerman.id.au designates 150.107.74.76 as permitted sender) smtp.mailfrom=mpe@ellerman.id.au; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687319528; a=rsa-sha256; cv=none; b=Yy4y0XNcOXg/qYbnbg0BV04/zfse6Kb+9Jmru0ebvP+rPaDmRTryQJEPvEdK+hCkLvg/Cw 1a4/1eT2jAyX1KdkNgn/ZBAE6e9yD2VDMR/B7H3qe3UH5uC+9drqPdz3NEEBx5X1qp3Kzu 41Rl/WSRCSap097xUHeeQD/q4XOp9Kc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=ellerman.id.au header.s=201909 header.b=qkWxn4h+; spf=pass (imf25.hostedemail.com: domain of mpe@ellerman.id.au designates 150.107.74.76 as permitted sender) smtp.mailfrom=mpe@ellerman.id.au; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687319528; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ev4/x1519eJIoPsG7DY94Nr3djNYODadAwI3LCl7mnY=; b=qz5Pz3O3VdpjtZePxxfmb5YIMKgsm56HCIFydG6bCIt6BwWMvEkeU3vWZFtkgSkGccXHcg 1HEHfHycDhfGJmyUamQjZmA2/l+9GMHOdwvUE8N5sgBfsYYrDoVSmzip1BXXJPgi3yP+if bj7WeDvq2idJVNTtvpwIB9A9Vn1LBu8= Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4Qm8hM0kYcz4wgC; Wed, 21 Jun 2023 13:52:03 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ellerman.id.au; s=201909; t=1687319523; bh=ev4/x1519eJIoPsG7DY94Nr3djNYODadAwI3LCl7mnY=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=qkWxn4h+GOxx8szqHvQtWqL57cAwiLczYmT0qwTjzWrKitU9khT9PvY9G4/hwaNPL 0qlWlUaI6+pesgFdx26taK/fIdg3RhE41C7GAbGinmZllbmjD9X/7q3KpqObpVITNA +Sc5sK4GSE2MIpVwNQpeX2RVAKrRdc4ScAy20ZZXTbHAiQSeoZZHfVfNoIfMYyer8H sIaacfMGx+p2aSDOojTlvMjKGgUCxGYZIu3t9xKPScqP8vjtjmSfQ/JUYYrkQRMI5h fbKEmI08E61F4YCUAZWHjODIMwfsFzLXHwgf6vsGaS39jxjwkioTFLYpSMaWMh9tgM 2EQcscNvY2fTA== From: Michael Ellerman To: Sachin Sant , linux-mm@kvack.org Cc: linuxppc-dev Subject: Re: [6.4.0-rc7-next-20230620] Boot failure on IBM Power LPAR In-Reply-To: References: Date: Wed, 21 Jun 2023 13:52:01 +1000 Message-ID: <87edm5pj4u.fsf@mail.lhotse> MIME-Version: 1.0 Content-Type: text/plain X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E39E0A0005 X-Stat-Signature: n3jqu65nyq4y4tauxzc4x31xdt56346b X-Rspam-User: X-HE-Tag: 1687319527-377676 X-HE-Meta: U2FsdGVkX1+rcP8mayqSHGYJWEjkFs78WxcOAdcbcAe3Ym/pkRTOVn2EWVroiMiUaxPI3fBTEDxVADmnlKVj7thqz4WmX88RqSMk3JBOs+OZ/foSKYEKDw/TzqoWPjsFQNCSlgmdYa6O62nfoJ/C15GbLXTd/gQHW0iAjy2C/TLLEXUiLS1dNzb4eJqksvb7IAVpRIW2NshFaqVIpVCZyZDrp5NpFH0blFsfSqtyApl9T3DpQ62g5WGgGsVCkc3VQBKr9dajCyKmvej6VTVVWxf6IGoOLZRK2pfV+K+xPBNdNFcMyTMIhQ4T49DGtj+F1TExw5lTsMjSV8VW9lz3XN5BnhIwPPTwjOsdJwm2/FquELnyMuTqBQbhCzscyxM/Rnyj8MMF2d901nIVnjmevPzVspeBvrToU3v1dGm7uL6+GagKq1ECPVOnMwMEOs8qtpfvira8FxaKvdVBCV/2srBGGtaPnzYhM+ddlqaxcVGmwTgbZ1D8p5bwJk/++gQcIfqZIE40ea5yCkjlAkqWQcRGrBR9mdYPUX1rvF28Vkgdf/zfKGhxOuJtxGCdoU0KFU9NO0bID+WIo7ytEiERENjtsa9qN/lSVMHxvlcgDHrnSHE6lD01i3kLP6wZzWuWDKk5/2A1YJA18sp2503l9Uf/RsuLFp9M9D5jyjy/aUpavIzSZosNRZ7sFkHbDhcvdr9/egIl/JVjurETawREJD406rlF6awo2GnEPJ4l18whinrSttYIz4UN8ZvaAI4iDDSnwcrs5RakTPOx32IE2Qio5dICICibHFjOpS1iGa3cPcLidYRxgu7CATcIEYpREXzJMLB4k8W+3nJi3x6dPZ4dsm4RIswe8/hZmN5wU6DJ4lkIoIl/q9kDvCJ+YQ4+ixJU1yrW66neSRs5rsg45hcBslIoZntdeLu9ISJD+PbsiMukHFKm4Jaloe38Vz+diMf93SzNRZPrItVwCAD yx/wkEc8 1DwHZB8QHb8D5V94Vg2UsLkbllRRHZYOjtLPfPPZ2aUVWSeVH+OKLralJGK4A8uHn7/yPQdrcio2SBF44lZqupcjxF77ZWoHNTlCQLQwaHj+zRYmBFXMfApaCxv0v0At41S4ghJ78rS2uHlGBexQ1RLWE0d0ldtjG0YDZ3/XlBEN6CaYfCJOGCRxfuLmJILeb1aXauqEhesYyUiMgJPUms6gh8CkszkxWLbC0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Sachin Sant writes: > 6.4.0-rc7-next-20230620 fails to boot on IBM Power LPAR with following > > [ 5.548368] BUG: Unable to handle kernel data access at 0x95bdcf954bc34e73 > [ 5.548380] Faulting instruction address: 0xc000000000548090 > [ 5.548384] Oops: Kernel access of bad area, sig: 11 [#1] > [ 5.548387] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries > [ 5.548391] Modules linked in: nf_tables(E) nfnetlink(E) sunrpc(E) binfmt_misc(E) pseries_rng(E) aes_gcm_p10_crypto(E) drm(E) drm_panel_orientation_quirks(E) xfs(E) libcrc32c(E) sd_mod(E) sr_mod(E) t10_pi(E) crc64_rocksoft_generic(E) cdrom(E) crc64_rocksoft(E) crc64(E) sg(E) ibmvscsi(E) scsi_transport_srp(E) ibmveth(E) vmx_crypto(E) fuse(E) > [ 5.548413] CPU: 1 PID: 789 Comm: systemd-udevd Tainted: G E 6.4.0-rc7-next-20230620 #1 > [ 5.548417] Hardware name: IBM,9080-HEX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1030.20 (NH1030_058) hv:phyp pSeries > [ 5.548421] NIP: c000000000548090 LR: c000000000547fbc CTR: c0000000004206f0 > [ 5.548424] REGS: c0000000afb536f0 TRAP: 0380 Tainted: G E (6.4.0-rc7-next-20230620) > [ 5.548427] MSR: 800000000280b033 CR: 88028202 XER: 20040000 > [ 5.548436] CFAR: c000000000547fc4 IRQMASK: 0 > [ 5.548436] GPR00: c000000000547fbc c0000000afb53990 c0000000014b1600 0000000000000000 > [ 5.548436] GPR04: 0000000000000cc0 00000000000034d8 0000000000000e6f ed5e02cab43c21e0 > [ 5.548436] GPR08: 0000000000000e6e 0000000000000058 0000001356ea0000 0000000000002000 > [ 5.548436] GPR12: c0000000004206f0 c0000013fffff300 0000000000000000 0000000000000000 > [ 5.548436] GPR16: 0000000000000000 0000000000000000 0000000000000000 c000000092f43708 > [ 5.548436] GPR20: c000000092f436b0 0000000000000000 fffffffffff7dfff c0000000afa80000 > [ 5.548436] GPR24: c000000002b87aa0 00000000000000b8 c000000000159914 0000000000000cc0 > [ 5.548436] GPR28: 95bdcf954bc34e1b c00000000a1fafc0 0000000000000000 c000000003019800 > [ 5.548473] NIP [c000000000548090] kmem_cache_alloc+0x1a0/0x420 > [ 5.548480] LR [c000000000547fbc] kmem_cache_alloc+0xcc/0x420 > [ 5.548485] Call Trace: > [ 5.548487] [c0000000afb53990] [c000000000547fbc] kmem_cache_alloc+0xcc/0x420 (unreliable) > [ 5.548493] [c0000000afb53a00] [c000000000159914] vm_area_dup+0x44/0xf0 > [ 5.548499] [c0000000afb53a40] [c00000000015a638] dup_mmap+0x298/0x8b0 > [ 5.548504] [c0000000afb53bb0] [c00000000015acd0] dup_mm.constprop.0+0x80/0x180 > [ 5.548509] [c0000000afb53bf0] [c00000000015bdc0] copy_process+0xc00/0x1510 > [ 5.548514] [c0000000afb53cb0] [c00000000015c848] kernel_clone+0xb8/0x5a0 > [ 5.548519] [c0000000afb53d30] [c00000000015ceb8] __do_sys_clone+0x88/0xd0 > [ 5.548524] [c0000000afb53e10] [c000000000033bcc] system_call_exception+0x13c/0x340 > [ 5.548529] [c0000000afb53e50] [c00000000000d05c] system_call_vectored_common+0x15c/0x2ec > [ 5.548534] --- interrupt: 3000 at 0x7fff87f0c178 > [ 5.548538] NIP: 00007fff87f0c178 LR: 0000000000000000 CTR: 0000000000000000 > [ 5.548540] REGS: c0000000afb53e80 TRAP: 3000 Tainted: G E (6.4.0-rc7-next-20230620) > [ 5.548544] MSR: 800000000000f033 CR: 44004204 XER: 00000000 > [ 5.548552] IRQMASK: 0 > [ 5.548552] GPR00: 0000000000000078 00007ffffde8cb80 00007fff88637500 0000000001200011 > [ 5.548552] GPR04: 0000000000000000 0000000000000000 0000000000000000 00007fff888bd490 > [ 5.548552] GPR08: 0000000000000001 0000000000000000 0000000000000000 0000000000000000 > [ 5.548552] GPR12: 0000000000000000 00007fff888c4c00 0000000000000002 00007ffffde95698 > [ 5.548552] GPR16: 00007ffffde95690 00007ffffde95688 00007ffffde956a0 0000000000000028 > [ 5.548552] GPR20: 0000000132bca308 0000000000000001 0000000000000001 0000000000000315 > [ 5.548552] GPR24: 0000000000000003 0000000000000040 0000000000000000 0000000000000003 > [ 5.548552] GPR28: 0000000000000000 0000000000000000 00007ffffde8cf24 0000000000000045 > [ 5.548586] NIP [00007fff87f0c178] 0x7fff87f0c178 > [ 5.548589] LR [0000000000000000] 0x0 > [ 5.548591] --- interrupt: 3000 > [ 5.548593] Code: e93f0000 7ce95214 e9070008 7f89502a e9270010 2e3c0000 41920258 2c290000 41820250 813f0028 e8ff00b8 38c80001 <7fdc482a> 7d3c4a14 79250022 552ac03e > [ 5.548605] ---[ end trace 0000000000000000 ]--- > [ 5.550849] pstore: backend (nvram) writing error (-1) > [ 5.550852] > Starting Network Manager... > [ 5.566384] BUG: Bad rss-counter state mm:00000000dc60f1c1 type:MM_ANONPAGES val:36 > [ 5.568784] BUG: Bad rss-counter state mm:000000008eb9341b type:MM_ANONPAGES val:36 > [ 5.689774] BUG: Bad rss-counter state mm:00000000edbda345 type:MM_ANONPAGES val:36 > [ 5.692187] BUG: Bad rss-counter state mm:000000003f7ec21f type:MM_ANONPAGES val:36 > [ 5.705947] BUG: Bad rss-counter state mm:00000000cdbb7cfd type:MM_ANONPAGES val:36 > [ 6.550855] Kernel panic - not syncing: Fatal exception > [ 6.568226] Rebooting in 10 seconds.. > > The problem was introduced in 6.4.0-rc7-next-20230619. I tried git bisect, but unsure of the > result reported by it. Bisect points to following patch > > # git bisect bad > 70c94cc2eefd4f98d222834cbe7512804977c2d4 is the first bad commit > commit 70c94cc2eefd4f98d222834cbe7512804977c2d4 > Merge: 48f5ee5c48c3 3fe08f7d5e80 > Author: Stephen Rothwell > Date: Tue Jun 20 09:43:25 2023 +1000 > > Merge branch 'mm-everything' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm > # Conflicts: > # mm/mmap.c Usually bisect pointing to a merge means something has gone wrong with the bisect. It's not impossible for a merge to be the cause of a bug, but IME it's rare. In this case though the merge itself has a reasonably large diff, so it's more likely that the merge itself has introduced a bug. commit 70c94cc2eefd4f98d222834cbe7512804977c2d4 Merge: 48f5ee5c48c3 3fe08f7d5e80 Author: Stephen Rothwell AuthorDate: Tue Jun 20 09:43:25 2023 +1000 Commit: Stephen Rothwell CommitDate: Tue Jun 20 09:43:25 2023 +1000 Merge branch 'mm-everything' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm # Conflicts: # mm/mmap.c diff --cc mm/mmap.c index 98cda6f72605,474a0d856622..9a93b054148a --- a/mm/mmap.c +++ b/mm/mmap.c @@@ -2398,15 -2409,27 +2396,29 @@@ do_vmi_align_munmap(struct vma_iterato if (error) goto end_split_failed; } - mas_set(&mas_detach, count); - error = munmap_sidetree(next, &mas_detach); - if (error) - goto munmap_sidetree_failed; + vma_start_write(next); - mas_set_range(&mas_detach, next->vm_start, next->vm_end - 1); + if (mas_store_gfp(&mas_detach, next, GFP_KERNEL)) + goto munmap_gather_failed; + vma_mark_detached(next, true); + if (next->vm_flags & VM_LOCKED) + locked_vm += vma_pages(next); count++; + if (unlikely(uf)) { + /* + * If userfaultfd_unmap_prep returns an error the vmas + * will remain split, but userland will get a + * highly unexpected error anyway. This is no + * different than the case where the first of the two + * __split_vma fails, but we don't undo the first + * split, despite we could. This is unlikely enough + * failure that it's not worth optimizing it for. + */ + error = userfaultfd_unmap_prep(next, start, end, uf); + + if (error) + goto userfaultfd_error; + } #ifdef CONFIG_DEBUG_VM_MAPLE_TREE BUG_ON(next->vm_start < start); BUG_ON(next->vm_start > end); @@@ -2454,14 -2455,18 +2444,20 @@@ BUG_ON(count != test_count); } #endif - /* Point of no return */ + error = -ENOMEM; - vma_iter_set(vmi, start); + while (vma_iter_addr(vmi) > start) + vma_iter_prev_range(vmi); + if (vma_iter_clear_gfp(vmi, start, end, GFP_KERNEL)) - return -ENOMEM; + goto clear_tree_failed; + mm->locked_vm -= locked_vm; mm->map_count -= count; + prev = vma_iter_prev_range(vmi); + next = vma_next(vmi); + if (next) + vma_iter_prev_range(vmi); + /* * Do not downgrade mmap_lock if we are next to VM_GROWSDOWN or * VM_GROWSUP VMA. Such VMAs can change their size under cheers