From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 912F31BF53 for ; Tue, 13 Aug 2024 18:47:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723574871; cv=none; b=qoQe/B1Df3TenJvPqi2DM6Cp9jjCCiTWFolk2lPlFQ9JA/rcIanhjXYP1PegglqW0wlbRkljzjDptyRlcKuZsxK+E07ehLA+uuNxJ+jNMQ0AzyjcJqgT6xqaMuR7qJ8Yil1JII9hba24B9Tv4ubqZx4xqvmJ5wlJWRe/FMnHkvc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723574871; c=relaxed/simple; bh=bhMVoLQQptT4luSr6Op9GcB2mMeYk3DxLMMliBbUbGc=; h=Date:To:From:Subject:Message-Id; b=cSbVMTQOXLKXJ27FoYha5JepNqDmo6bqtQcCFOuseSYl4jfNdxSsWGALG2raQBQlPJtAubz64UeDUKzBrE70+KfOgCXPZUz84C37msnCz3STA9Lh4nFDu5AgNojm8ALbszCtzjR6n5Fswcb0UPX1j5kvjw6gYAZD9UZPmIjUj2k= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=eK5uUnEY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="eK5uUnEY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A206C32782; Tue, 13 Aug 2024 18:47:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1723574871; bh=bhMVoLQQptT4luSr6Op9GcB2mMeYk3DxLMMliBbUbGc=; h=Date:To:From:Subject:From; b=eK5uUnEYB/6Va0kftHfTpA02y9SvT5YDcFiJWg8HpCP1lxctt77kHdsWC8+0tAEUu /leblrznsocsHdrE0mIU4YB9WBV01k7bGxgFh603WO6r9f3WMlVledl5yhWddzVwrS EJbLfKHWFL25L6SYYYURyG7Q99XUzfydyGbqNVxo= Date: Tue, 13 Aug 2024 11:47:50 -0700 To: mm-commits@vger.kernel.org,willy@infradead.org,vbabka@suse.cz,tglx@linutronix.de,seanjc@google.com,rientjes@google.com,riel@surriel.com,rick.p.edgecombe@intel.com,pbonzini@redhat.com,osalvador@suse.de,npiggin@gmail.com,mpe@ellerman.id.au,mingo@redhat.com,kirill@shutemov.name,hughd@google.com,david@redhat.com,dave.jiang@intel.com,dave.hansen@linux.intel.com,dan.j.williams@intel.com,christophe.leroy@csgroup.eu,bp@alien8.de,aneesh.kumar@linux.ibm.com,peterx@redhat.com,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-x86-add-missing-pud-helpers.patch added to mm-unstable branch Message-Id: <20240813184751.5A206C32782@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm/x86: add missing pud helpers has been added to the -mm mm-unstable branch. Its filename is mm-x86-add-missing-pud-helpers.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-x86-add-missing-pud-helpers.patch This patch will later appear in the mm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Peter Xu Subject: mm/x86: add missing pud helpers Date: Mon, 12 Aug 2024 14:12:24 -0400 Some new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate(): this helper invalidates a huge pud before a split happens, so that the invalidated pud entry will make sure no race will happen (either with software, like a concurrent zap, or hardware, like a/d bit lost). - pud_modify(): this helper applies a new pgprot to an existing huge pud mapping. For more information on why we need these two helpers, please refer to the corresponding pmd helpers in the mprotect() code path. When at it, simplify the pud_modify()/pmd_modify() comments on shadow stack pgtable entries to reference pte_modify() to avoid duplicating the whole paragraph three times. Link: https://lkml.kernel.org/r/20240812181225.1360970-7-peterx@redhat.com Signed-off-by: Peter Xu Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Aneesh Kumar K.V Cc: Christophe Leroy Cc: Dan Williams Cc: Dave Jiang Cc: David Hildenbrand Cc: David Rientjes Cc: "Edgecombe, Rick P" Cc: Hugh Dickins Cc: Kirill A. Shutemov Cc: Matthew Wilcox Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Oscar Salvador Cc: Paolo Bonzini Cc: Rik van Riel Cc: Sean Christopherson Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- arch/x86/include/asm/pgtable.h | 57 ++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 12 ++++++ 2 files changed, 61 insertions(+), 8 deletions(-) --- a/arch/x86/include/asm/pgtable.h~mm-x86-add-missing-pud-helpers +++ a/arch/x86/include/asm/pgtable.h @@ -787,6 +787,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -834,14 +840,8 @@ static inline pmd_t pmd_modify(pmd_t pmd pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -851,6 +851,29 @@ static inline pmd_t pmd_modify(pmd_t pmd return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1389,10 +1412,28 @@ static inline pmd_t pmdp_establish(struc } #endif +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} +#endif + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. --- a/arch/x86/mm/pgtable.c~mm-x86-add-missing-pud-helpers +++ a/arch/x86/mm/pgtable.c @@ -641,6 +641,18 @@ pmd_t pmdp_invalidate_ad(struct vm_area_ } #endif +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve _ Patches currently in -mm which might be from peterx@redhat.com are mm-dax-dump-start-address-in-fault-handler.patch mm-mprotect-push-mmu-notifier-to-puds.patch mm-powerpc-add-missing-pud-helpers.patch mm-x86-make-pud_leaf-only-care-about-pse-bit.patch mm-x86-implement-arch_check_zapped_pud.patch mm-x86-add-missing-pud-helpers.patch mm-mprotect-fix-dax-pud-handlings.patch