[PATCH v9 06/32] s390: mm: add PMD swap-exclusive helpers

[PATCH v9 06/32] s390: mm: add PMD swap-exclusive helpers

[Usama Arif]

Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the
swap round-trip, which makes PG_anon_exclusive have to survive in a swap
PMD and not just in a swap PTE.

s390 is the one architecture where a swap PMD is not a swap PTE in
disguise: it is an RSTE with its own layout, converted to a fake PTE swap
entry for the common code. Give it its own exclusive bit rather than
borrowing the PTE-format macro, and document it in the RSTE swap layout,
where bit 52 was listed as unused.

Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: linux-s390@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/s390/include/asm/pgtable.h | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 2d5c2ab06de98..2c6b5205ee3bc 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -333,6 +333,7 @@ void setup_protection_map(void);
 /* Common bits in region and segment table entries, for swap entries */
 #define _RST_ENTRY_COMM		0x0010	/* Common-Region/Segment, marks swap entry */
 #define _RST_ENTRY_INVALID	0x0020	/* invalid region/segment table entry */
+#define _RST_ENTRY_SWP_EXCLUSIVE 0x0800	/* SW exclusive swap bit, see mk_swap_rste() */
 
 #define _CRST_ENTRIES	2048	/* number of region/segment table entries */
 #define _PAGE_ENTRIES	256	/* number of page table entries	*/
@@ -893,6 +894,21 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)
 }
 
 #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+static inline bool pmd_swp_exclusive(pmd_t pmd)
+{
+	return pmd_val(pmd) & _RST_ENTRY_SWP_EXCLUSIVE;
+}
+
+static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
+{
+	return set_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE));
+}
+
+static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
+{
+	return clear_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE));
+}
+
 #define pmd_swp_soft_dirty(pmd)		pmd_soft_dirty(pmd)
 #define pmd_swp_mksoft_dirty(pmd)	pmd_mksoft_dirty(pmd)
 #define pmd_swp_clear_soft_dirty(pmd)	pmd_clear_soft_dirty(pmd)
@@ -1900,15 +1916,15 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)
  * Bits 59 and 63 are used to indicate the swap entry. Bit 58 marks the rste
  * as invalid.
  * A swap entry is indicated by bit pattern (rste & 0x011) == 0x010
- * |			  offset			|Xtype |11TT|S0|
+ * |			  offset			|Etype |11TT|S0|
  * |0000000000111111111122222222223333333333444444444455|555555|5566|66|
  * |0123456789012345678901234567890123456789012345678901|234567|8901|23|
  *
  * Bits 0-51 store the offset.
+ * Bit 52 (E) is used to remember PG_anon_exclusive (_RST_ENTRY_SWP_EXCLUSIVE).
  * Bits 53-57 store the type.
  * Bit 62 (S) is used for softdirty tracking.
  * Bits 60-61 (TT) indicate the table type: 0x01 for REGION3 and 0x00 for SEGMENT.
- * Bit 52 (X) is unused.
  */
 
 #define __SWP_OFFSET_MASK_RSTE	((1UL << 52) - 1)
-- 
2.53.0-Meta

Re: [PATCH v9 06/32] s390: mm: add PMD swap-exclusive helpers

[sashiko-bot]

> Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the
> swap round-trip, which makes PG_anon_exclusive have to survive in a swap
> PMD and not just in a swap PTE.
> 
> s390 is the one architecture where a swap PMD is not a swap PTE in
> disguise: it is an RSTE with its own layout, converted to a fake PTE swap
> entry for the common code. Give it its own exclusive bit rather than
> borrowing the PTE-format macro, and document it in the RSTE swap layout,
> where bit 52 was listed as unused.
> 
> Cc: Alexander Gordeev <agordeev@linux.ibm.com>
> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Vasily Gorbik <gor@linux.ibm.com>
> Cc: linux-s390@vger.kernel.org
> Signed-off-by: Usama Arif <usama.arif@linux.dev>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261105183940.2236717-1-usama.arif@linux.dev?part=6

Re: [PATCH v9 06/32] s390: mm: add PMD swap-exclusive helpers

[Gerald Schaefer]

On Thu,  5 Nov 2026 10:38:50 -0800
Usama Arif <usama.arif@linux.dev> wrote:

> Prepare for keeping a PMD-mapped anonymous THP mapped by a PMD across the
> swap round-trip, which makes PG_anon_exclusive have to survive in a swap
> PMD and not just in a swap PTE.
> 
> s390 is the one architecture where a swap PMD is not a swap PTE in
> disguise: it is an RSTE with its own layout, converted to a fake PTE swap
> entry for the common code. Give it its own exclusive bit rather than
> borrowing the PTE-format macro, and document it in the RSTE swap layout,
> where bit 52 was listed as unused.
> 
> Cc: Alexander Gordeev <agordeev@linux.ibm.com>
> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Vasily Gorbik <gor@linux.ibm.com>
> Cc: linux-s390@vger.kernel.org
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  arch/s390/include/asm/pgtable.h | 20 ++++++++++++++++++--
>  1 file changed, 18 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
> index 2d5c2ab06de98..2c6b5205ee3bc 100644
> --- a/arch/s390/include/asm/pgtable.h
> +++ b/arch/s390/include/asm/pgtable.h
> @@ -333,6 +333,7 @@ void setup_protection_map(void);
>  /* Common bits in region and segment table entries, for swap entries */
>  #define _RST_ENTRY_COMM		0x0010	/* Common-Region/Segment, marks swap entry */
>  #define _RST_ENTRY_INVALID	0x0020	/* invalid region/segment table entry */
> +#define _RST_ENTRY_SWP_EXCLUSIVE 0x0800	/* SW exclusive swap bit, see mk_swap_rste() */
>  
>  #define _CRST_ENTRIES	2048	/* number of region/segment table entries */
>  #define _PAGE_ENTRIES	256	/* number of page table entries	*/
> @@ -893,6 +894,21 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)
>  }
>  
>  #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
> +static inline bool pmd_swp_exclusive(pmd_t pmd)
> +{
> +	return pmd_val(pmd) & _RST_ENTRY_SWP_EXCLUSIVE;
> +}
> +
> +static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
> +{
> +	return set_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE));
> +}
> +
> +static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
> +{
> +	return clear_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE));
> +}
> +
>  #define pmd_swp_soft_dirty(pmd)		pmd_soft_dirty(pmd)
>  #define pmd_swp_mksoft_dirty(pmd)	pmd_mksoft_dirty(pmd)
>  #define pmd_swp_clear_soft_dirty(pmd)	pmd_clear_soft_dirty(pmd)
> @@ -1900,15 +1916,15 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)
>   * Bits 59 and 63 are used to indicate the swap entry. Bit 58 marks the rste
>   * as invalid.
>   * A swap entry is indicated by bit pattern (rste & 0x011) == 0x010
> - * |			  offset			|Xtype |11TT|S0|
> + * |			  offset			|Etype |11TT|S0|
>   * |0000000000111111111122222222223333333333444444444455|555555|5566|66|
>   * |0123456789012345678901234567890123456789012345678901|234567|8901|23|
>   *
>   * Bits 0-51 store the offset.
> + * Bit 52 (E) is used to remember PG_anon_exclusive (_RST_ENTRY_SWP_EXCLUSIVE).
>   * Bits 53-57 store the type.
>   * Bit 62 (S) is used for softdirty tracking.
>   * Bits 60-61 (TT) indicate the table type: 0x01 for REGION3 and 0x00 for SEGMENT.
> - * Bit 52 (X) is unused.
>   */
>  
>  #define __SWP_OFFSET_MASK_RSTE	((1UL << 52) - 1)

Nice, you found the last free bit in s390 RST swap entry, and made good
use of it. Bit 52 would also be free in regular RST entries, but as far
as I remember from PTE SWP_EXCLUSIVE, this will only ever be checked or
set for swap entries, right? Otherwise we should also document and
reserve for regular RST entries, but it should be safe for now in any
case.

Also did some basic tests with your series on s390. Couldn't see an
increase in throughput yet, but my swap device was way too slow an a
bottleneck itself. Verified that thp_swpout_pmd counter was increased,
system runs stable, found no problems so far.

I think this series could give s390 some considerable benefit, besides
swap throughput, by saving the 256 IPTEs (PTE invalidations) for each
1MB THP that doesn't need to get split before unmap and swap out. This
could have a positive system-wide impact, not just for one guest.

Great job, thanks for the effort!

Reviewed-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Tested-by: Gerald Schaefer <gerald.schaefer@linux.ibm.com>