From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7D9E3B2FEA for ; Mon, 27 Apr 2026 10:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.171 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777284389; cv=none; b=XT2efkOZNPb82kF50L9STwLZcPRvZLHBxAr4SQGeOa/IaxrsSwT1AMRjrFvjRaPKQfILzTrvdxVEU3JS9OHHu6yrS3bVJuf8ISbwi1M+YSQTza2ddIWdBoyoO7R8yK+rK6xNgFFbnQ215hCkSwPn+6y9RMbmulBGZAAr6CPI/Qk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777284389; c=relaxed/simple; bh=3MN45ET580FiWB/iTABCasd/hYMKWH6hZqu8yfAwbfc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bc76nC+V/YFwXfxr6gUsTFZX/JQ5TvEg6eO4hfdAiGmb+ohMbesUVBtew+QSfXuMxaYK1FT7AcuL+zPPEce2/bpI1+Nj7OHbVZ1mWZ5ThF7Pbe9MbxuuHRLlpnUVSMlUTwuN4689sPcJxB7D/Hd8dyV4Oh6ImvLpqpJbi6rOfzw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=BT/toJAJ; arc=none smtp.client-ip=91.218.175.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="BT/toJAJ" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1777284385; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d3DTRvDyseaP44w4E84pI+LnObWi0aHOFCJwUMf+7rE=; b=BT/toJAJdxq0x5Y1sjo5OkL3aclLdcBoalcv/sMZayH8A5NmdC1/x6R3OwuyyeFABDq5ZH NQk3fFVlnGdJsmDbiz0qaFa3nZq8KlpKMIwF1ZhbYghblB7QlhXyAgZLm2w2v/UAA8C0mk quxHF7gN2dTqH70tl4LM0pbxP9Ty7HI= From: Usama Arif To: Andrew Morton , david@kernel.org, chrisl@kernel.org, kasong@tencent.com, ljs@kernel.org, ziy@nvidia.com Cc: bhe@redhat.com, willy@infradead.org, youngjun.park@lge.com, hannes@cmpxchg.org, riel@surriel.com, shakeel.butt@linux.dev, alex@ghiti.fr, kas@kernel.org, baohua@kernel.org, dev.jain@arm.com, baolin.wang@linux.alibaba.com, npache@redhat.com, Liam.Howlett@oracle.com, ryan.roberts@arm.com, Vlastimil Babka , lance.yang@linux.dev, linux-kernel@vger.kernel.org, nphamcs@gmail.com, shikemeng@huaweicloud.com, kernel-team@meta.com, Usama Arif Subject: [PATCH 05/13] mm: add PMD swap entry detection support Date: Mon, 27 Apr 2026 03:01:54 -0700 Message-ID: <20260427100553.2754667-6-usama.arif@linux.dev> In-Reply-To: <20260427100553.2754667-1-usama.arif@linux.dev> References: <20260427100553.2754667-1-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT Currently when a PMD-mapped THP is swapped out, the PMD is always split into 512 PTE-level swap entries. To preserve huge page information across swap cycles, later patches will install a single PMD-level swap entry instead. This patch adds the infrastructure to detect those entries. Teach the softleaf layer to recognise PMD swap entries: pmd_is_swap_entry() detects them and softleaf_is_valid_pmd_entry() accepts them as a valid non-present type. Clear the exclusive overlay bit in softleaf_from_pmd() before decoding, matching how soft_dirty and uffd_wp bits are already stripped. Add pmd_swp_mkexclusive(), pmd_swp_exclusive(), and pmd_swp_clear_exclusive() helpers to each architecture that supports THP migration (x86, arm64, s390, riscv, loongarch), mirroring the existing PTE swap exclusive helpers in each arch's pgtable.h. Signed-off-by: Usama Arif --- arch/arm64/include/asm/pgtable.h | 4 ++++ arch/loongarch/include/asm/pgtable.h | 17 +++++++++++++++++ arch/riscv/include/asm/pgtable.h | 15 +++++++++++++++ arch/s390/include/asm/pgtable.h | 15 +++++++++++++++ arch/x86/include/asm/pgtable.h | 15 +++++++++++++++ include/linux/leafops.h | 18 ++++++++++++++++-- 6 files changed, 82 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9029b81ccbe8..ecb0ef6994cb 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -601,6 +601,10 @@ static inline int pmd_protnone(pmd_t pmd) #define pmd_swp_clear_uffd_wp(pmd) \ pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd))) #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +#define pmd_swp_exclusive(pmd) pte_swp_exclusive(pmd_pte(pmd)) +#define pmd_swp_mkexclusive(pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))) +#define pmd_swp_clear_exclusive(pmd) \ + pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))) #define pmd_write(pmd) pte_write(pmd_pte(pmd)) diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h index 155f70e93460..f8e7761eb54e 100644 --- a/arch/loongarch/include/asm/pgtable.h +++ b/arch/loongarch/include/asm/pgtable.h @@ -345,6 +345,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return pte; } +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + pmd_val(pmd) |= _PAGE_SWP_EXCLUSIVE; + return pmd; +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_val(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + pmd_val(pmd) &= ~_PAGE_SWP_EXCLUSIVE; + return pmd; +} + #define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL)) #define pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE)) #define pte_no_exec(pte) (pte_val(pte) & _PAGE_NO_EXEC) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index a6e0eaba2653..f4cd59ebab58 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -935,6 +935,21 @@ static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd) } #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */ +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pte_swp_exclusive(pmd_pte(pmd)); +} + +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd))); +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd))); +} + #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline bool pmd_soft_dirty(pmd_t pmd) { diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 40a6fb19dd1d..9b05fd3e4df0 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -868,6 +868,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return clear_pte_bit(pte, __pgprot(_PAGE_SWP_EXCLUSIVE)); } +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(_PAGE_SWP_EXCLUSIVE)); +} + +static inline bool pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_val(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return clear_pmd_bit(pmd, __pgprot(_PAGE_SWP_EXCLUSIVE)); +} + static inline int pte_soft_dirty(pte_t pte) { return pte_val(pte) & _PAGE_SOFT_DIRTY; diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 13e3e9a054cb..eb8b7a6f4bb4 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1517,6 +1517,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) return pte_clear_flags(pte, _PAGE_SWP_EXCLUSIVE); } +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SWP_EXCLUSIVE); +} + +static inline int pmd_swp_exclusive(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SWP_EXCLUSIVE; +} + +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd) +{ + return pmd_clear_flags(pmd, _PAGE_SWP_EXCLUSIVE); +} + #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY static inline pte_t pte_swp_mksoft_dirty(pte_t pte) { diff --git a/include/linux/leafops.h b/include/linux/leafops.h index 803d312437df..79e04db45bfb 100644 --- a/include/linux/leafops.h +++ b/include/linux/leafops.h @@ -102,6 +102,8 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd) pmd = pmd_swp_clear_soft_dirty(pmd); if (pmd_swp_uffd_wp(pmd)) pmd = pmd_swp_clear_uffd_wp(pmd); + if (pmd_swp_exclusive(pmd)) + pmd = pmd_swp_clear_exclusive(pmd); arch_entry = __pmd_to_swp_entry(pmd); /* Temporary until swp_entry_t eliminated. */ @@ -634,9 +636,21 @@ static inline bool pmd_is_migration_entry(pmd_t pmd) */ static inline bool softleaf_is_valid_pmd_entry(softleaf_t entry) { - /* Only device private, migration entries valid for PMD. */ + /* Device private, migration, and swap entries valid for PMD. */ return softleaf_is_device_private(entry) || - softleaf_is_migration(entry); + softleaf_is_migration(entry) || + softleaf_is_swap(entry); +} + +/** + * pmd_is_swap_entry() - Does this PMD entry encode an actual swap entry? + * @pmd: PMD entry. + * + * Returns: true if the PMD encodes a swap entry, otherwise false. + */ +static inline bool pmd_is_swap_entry(pmd_t pmd) +{ + return softleaf_is_swap(softleaf_from_pmd(pmd)); } /** -- 2.52.0