From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mta1.migadu.com (out-146.mta1.migadu.com [95.215.58.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E5094749CF for ; Tue, 18 Aug 2026 13:12:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.146 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787058739; cv=none; b=PONsN5ydyZo0HIqJ8FFHpQNIIBHSUjQ+CAC/R8O6cQ8yKap6VqlDQFx4c11p+RdCJSFX4Yc5A4xAd66RM4+lEpXKpsP/dBS40eYkTsrW+sgdCxDuV7h9r1Was8UhV63Ezq8LIwpJ2ine+mmvweLp5cNzZVtHiacVshIzQNxrx2Y= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787058739; c=relaxed/simple; bh=Zq+Ctczt2l05WWIcApO8BE7BxqptNyjFszeHYSoNb5Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HR95fQVsdFogGi7TiyEcBIOWbYOdxQRWy49nTmm9vFS0FvBKO/sdU/g0LDtxsYbhFKG6T2DAESk/12S2WGaI7zxvuYwAByq4JvwXgJrnlbOuKPOxCKE/8WG1K4xKk8LY1DsY2SQkwYCGtTRhwJsn6+HnmN63DxLBiveX8jyweww= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=iETa7Yfq; arc=none smtp.client-ip=95.215.58.146 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="iETa7Yfq" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=Zq+Ctczt2l05WWIcApO8BE7BxqptNyjFszeHYSoNb5Y=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1787058735; v=1; x=1787663535; b=iETa7YfqCDf16oR4bL68jzlFBT6sD29y3n9ar2+BVMl22DWq+0bLgBlACwhW40WElExtNBSX sa6j2vts48wAzP2pOB4uzgGQEo6e+JnRIwIBsjVs56HKM2zoGOvTnc6WCQSoALypIrApDAfowH5 RV1xGlf+O4VLvg6AumaCzI9A= X-Envelope-To: linux-kernel@vger.kernel.org Received: from localhost (2a03:2880:10ff:42::) by mta10.migadu.com with ESMTPS id 2117379505cae7b9; Tue, 18 Aug 2026 13:12:14 +0000 X-Migadu-Flow: FLOW_OUT From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com, linux-mm@kvack.org Cc: ying.huang@linux.alibaba.com, Baoquan He , willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, Nico Pache , Liam R. Howlett , ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, yosry@kernel.org, kernel-team@meta.com, Usama Arif Subject: [PATCH v6 02/12] mm: add PMD swap entry detection support Date: Tue, 18 Aug 2026 06:09:43 -0700 Message-ID: <20260818131202.494754-3-usama.arif@linux.dev> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260818131202.494754-1-usama.arif@linux.dev> References: <20260818131202.494754-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Currently when a PMD-mapped THP is swapped out, the PMD is always split into HPAGE_PMD_NR PTE-level swap entries. To preserve huge page information across swap cycles, later patches will install a single PMD-level swap entry instead. Add the infrastructure to detect those entries. Teach the softleaf layer to recognise PMD swap entries: pmd_is_swap_entry() detects them and softleaf_is_valid_pmd_entry() accepts them as a valid non-present type. Because swap entries do not encode a PFN, make pmd_softleaf_to_folio() warn and return NULL for them instead of passing the swap offset to softleaf_to_folio(). Clear the exclusive overlay bit in softleaf_from_pmd() before decoding, matching how soft_dirty and uffd bits are already stripped. Add pmd_swp_mkexclusive(), pmd_swp_exclusive(), and pmd_swp_clear_exclusive() helpers to each architecture that supports PMD softleaf entries (x86, arm64, s390, riscv, loongarch, powerpc), mirroring the existing PTE swap exclusive helpers in each arch's pgtable.h. Provide generic no-op PMD swap exclusive fallbacks for architectures without PMD softleaf support, matching the generic PMD swap soft-dirty fallbacks. Signed-off-by: Usama Arif --- arch/arm64/include/asm/pgtable.h | 6 +++++ arch/loongarch/include/asm/pgtable.h | 19 ++++++++++++++ arch/powerpc/include/asm/book3s/64/pgtable.h | 17 +++++++++++++ arch/riscv/include/asm/pgtable.h | 15 +++++++++++ arch/s390/include/asm/pgtable.h | 17 +++++++++++++ arch/x86/include/asm/pgtable.h | 17 +++++++++++++ include/linux/leafops.h | 26 ++++++++++++++++---- include/linux/pgtable.h | 17 +++++++++++++ 8 files changed, 129 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index a2681d7553584..860f95573d1fb 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -598,6 +598,12 @@ static inline int pmd_protnone(pmd_t pmd) #define pmd_swp_clear_uffd(pmd) \ pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd))) #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +#define pmd_swp_exclusive(pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_mkexclusive(pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_clear_exclusive(pmd) \ + pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) +#endif #define pmd_write(pmd) pte_write(pmd_pte(pmd)) diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h index 1952e34bc8ee0..aa8e1223d3973 100644 --- a/arch/loongarch/include/asm/pgtable.h +++ b/arch/loongarch/include/asm/pgtable.h @@ -357,6 +357,25 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return pte; } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + pmd_val(pmd) |= _PAGE_SWP_EXCLUSIVE; + return pmd; +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_val(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + pmd_val(pmd) &= ~_PAGE_SWP_EXCLUSIVE; + return pmd; +} +#endif + #define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL)) #define pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE)) #define pte_no_exec(pte) (pte_val(pte) & _PAGE_NO_EXEC) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index f4db7d7fbd5c6..6a899d0793b3b 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -699,6 +699,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return __pte_raw(pte_raw(pte) & cpu_to_be64(~_PAGE_SWP_EXCLUSIVE)); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return __pmd_raw(pmd_raw(pmd) | cpu_to_be64(_PAGE_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return !!(pmd_raw(pmd) & cpu_to_be64(_PAGE_SWP_EXCLUSIVE)); +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return __pmd_raw(pmd_raw(pmd) & cpu_to_be64(~_PAGE_SWP_EXCLUSIVE)); +} +#endif + static inline bool check_pte_access(unsigned long access, unsigned long ptev) { /* diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 1225cf05696a2..65b4181c62d9a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -1213,6 +1213,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) } #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pte_swp_exclusive(pmd_pte(pmd)); +} + +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))); +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))); +} + #define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) }) #define __swp_entry_to_pmd(swp) __pmd((swp).val) #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */ diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index e882663a58e77..490e4a3464b19 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -870,6 +870,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return clear_pte_bit(pte, __pgprot(_PAGE_SWP_EXCLUSIVE)); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(_PAGE_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_val(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return clear_pmd_bit(pmd, __pgprot(_PAGE_SWP_EXCLUSIVE)); +} +#endif + static inline int pte_soft_dirty(pte_t pte) { return pte_val(pte) & _PAGE_SOFT_DIRTY; diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 8e0018fadd14e..b5da5447e83d0 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1525,6 +1525,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return pte_clear_flags(pte, _PAGE_SWP_EXCLUSIVE); } +#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline int pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pmd_clear_flags(pmd, _PAGE_SWP_EXCLUSIVE); +} +#endif + #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty(pte_t pte) { diff --git a/include/linux/leafops.h b/include/linux/leafops.h index 7c13c58a5e218..4a6c52974b305 100644 --- a/include/linux/leafops.h +++ b/include/linux/leafops.h @@ -102,6 +102,8 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd) pmd = pmd_swp_clear_soft_dirty(pmd); if (pmd_swp_uffd(pmd)) pmd = pmd_swp_clear_uffd(pmd); + if (pmd_swp_exclusive(pmd)) + pmd = pmd_swp_clear_exclusive(pmd); arch_entry = __pmd_to_swp_entry(pmd); /* Temporary until swp_entry_t eliminated. */ @@ -634,18 +636,30 @@ static inline bool pmd_is_migration_entry(pmd_t pmd) */ static inline bool softleaf_is_valid_pmd_entry(softleaf_t entry) { - /* Only device private, migration entries valid for PMD. */ + /* Device private, migration, and swap entries valid for PMD. */ return softleaf_is_device_private(entry) || - softleaf_is_migration(entry); + softleaf_is_migration(entry) || + softleaf_is_swap(entry); +} + +/** + * pmd_is_swap_entry() - Does this PMD entry encode an actual swap entry? + * @pmd: PMD entry. + * + * Returns: true if the PMD encodes a swap entry, otherwise false. + */ +static inline bool pmd_is_swap_entry(pmd_t pmd) +{ + return softleaf_is_swap(softleaf_from_pmd(pmd)); } /** * pmd_is_valid_softleaf() - Is this PMD entry a valid softleaf entry? * @pmd: PMD entry. * - * PMD leaf entries are valid only if they are device private or migration - * entries. This function asserts that a PMD leaf entry is valid in this - * respect. + * PMD leaf entries are valid only if they are device private, migration, + * or swap entries. This function asserts that a PMD leaf entry is valid + * in this respect. * * Returns: true if the PMD entry is a valid leaf entry, otherwise false. */ @@ -673,6 +687,8 @@ static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd) VM_WARN_ON_ONCE(true); return NULL; } + if (WARN_ON_ONCE(!softleaf_has_pfn(entry))) + return NULL; return softleaf_to_folio(entry); } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8c093c119e5a8..e10a7e91e4260 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1917,6 +1917,23 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) } #endif +#ifndef CONFIG_ARCH_HAS_PMD_SOFTLEAVES +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pmd; +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pmd; +} +#endif + #ifndef __HAVE_PFNMAP_TRACKING /* * Interfaces that can be used by architecture code to keep track of -- 2.53.0-Meta