* [PATCH 1/7] shmem: provide a shmem_write_folio wrapper
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
@ 2026-07-06 13:21 ` Christoph Hellwig
2026-07-07 2:40 ` Baolin Wang
2026-07-06 13:21 ` [PATCH 2/7] mm/swap: introduce struct swap_io_ctx Christoph Hellwig
` (6 subsequent siblings)
7 siblings, 1 reply; 11+ messages in thread
From: Christoph Hellwig @ 2026-07-06 13:21 UTC (permalink / raw)
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
Provide a wrapper for the shmem abuses in drm to prepare for swap I/O
refactoring by keeping swap_iocb handling entirely contained in mm/.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Acked-by: Chris Li <chrisl@kernel.org>
---
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 2 +-
drivers/gpu/drm/ttm/ttm_backup.c | 2 +-
include/linux/shmem_fs.h | 5 +----
mm/shmem.c | 7 ++++++-
mm/swap.h | 4 ++++
5 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
index 06543ae60706..ef9440166295 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
@@ -325,7 +325,7 @@ void __shmem_writeback(size_t size, struct address_space *mapping)
if (folio_mapped(folio))
folio_redirty_for_writepage(&wbc, folio);
else
- error = shmem_writeout(folio, NULL, NULL);
+ error = shmem_write_folio(folio);
}
}
diff --git a/drivers/gpu/drm/ttm/ttm_backup.c b/drivers/gpu/drm/ttm/ttm_backup.c
index 81df4cb5606b..c5b813a563e7 100644
--- a/drivers/gpu/drm/ttm/ttm_backup.c
+++ b/drivers/gpu/drm/ttm/ttm_backup.c
@@ -117,7 +117,7 @@ ttm_backup_backup_page(struct file *backup, struct page *page,
if (writeback && !folio_mapped(to_folio) &&
folio_clear_dirty_for_io(to_folio)) {
folio_set_reclaim(to_folio);
- ret = shmem_writeout(to_folio, NULL, NULL);
+ ret = shmem_write_folio(to_folio);
if (!folio_test_writeback(to_folio))
folio_clear_reclaim(to_folio);
/*
diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index e729b9b0e38d..5663dff53186 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -12,8 +12,6 @@
#include <linux/userfaultfd_k.h>
#include <linux/bits.h>
-struct swap_iocb;
-
/* inode in-kernel data */
#ifdef CONFIG_TMPFS_QUOTA
@@ -123,8 +121,7 @@ static inline bool shmem_mapping(const struct address_space *mapping)
void shmem_unlock_mapping(struct address_space *mapping);
struct page *shmem_read_mapping_page_gfp(struct address_space *mapping,
pgoff_t index, gfp_t gfp_mask);
-int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
- struct list_head *folio_list);
+int shmem_write_folio(struct folio *folio);
void shmem_truncate_range(struct inode *inode, loff_t start, uoff_t end);
int shmem_unuse(unsigned int type);
diff --git a/mm/shmem.c b/mm/shmem.c
index b51f83c970bb..6011a7f729ae 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1736,7 +1736,12 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
folio_mark_dirty(folio);
return AOP_WRITEPAGE_ACTIVATE; /* Return with folio locked */
}
-EXPORT_SYMBOL_GPL(shmem_writeout);
+
+int shmem_write_folio(struct folio *folio)
+{
+ return shmem_writeout(folio, NULL, NULL);
+}
+EXPORT_SYMBOL_GPL(shmem_write_folio);
#if defined(CONFIG_NUMA) && defined(CONFIG_TMPFS)
static void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol)
diff --git a/mm/swap.h b/mm/swap.h
index 77d2d14eda42..4f86ef338a60 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -473,4 +473,8 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
}
#endif /* CONFIG_SWAP */
+
+int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
+ struct list_head *folio_list);
+
#endif /* _MM_SWAP_H */
--
2.53.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH 1/7] shmem: provide a shmem_write_folio wrapper
2026-07-06 13:21 ` [PATCH 1/7] shmem: provide a shmem_write_folio wrapper Christoph Hellwig
@ 2026-07-07 2:40 ` Baolin Wang
0 siblings, 0 replies; 11+ messages in thread
From: Baolin Wang @ 2026-07-07 2:40 UTC (permalink / raw)
To: Christoph Hellwig
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
On 7/6/26 9:21 PM, Christoph Hellwig wrote:
> Provide a wrapper for the shmem abuses in drm to prepare for swap I/O
> refactoring by keeping swap_iocb handling entirely contained in mm/.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Baoquan He <baoquan.he@linux.dev>
> Reviewed-by: Nhat Pham <nphamcs@gmail.com>
> Acked-by: Chris Li <chrisl@kernel.org>
> ---
LGTM.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 2/7] mm/swap: introduce struct swap_io_ctx
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
2026-07-06 13:21 ` [PATCH 1/7] shmem: provide a shmem_write_folio wrapper Christoph Hellwig
@ 2026-07-06 13:21 ` Christoph Hellwig
2026-07-06 13:21 ` [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O Christoph Hellwig
` (5 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2026-07-06 13:21 UTC (permalink / raw)
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
Generalize the context currently provided by double pointers to struct
swap_iocb to an on-stack context. This cleans up the code and prepares
for adding more fields and supporting batching multiple folios into a
single bio for block-based swap as well.
This new swap_io_ctx is required for all functions using it, the old
way of allowing a NULL iocb for some callers is removed to keep the
interface consistent. To reduce code duplication caused by this, a
new swap_cache_read_folio_sync helper is added to consolidate the
code to call swap_cache_read_folio with a local swap_io_ctx.
The unpug helpers are renamed to use the submit wording as they
are generalized.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
---
mm/madvise.c | 16 ++++++-------
mm/page_io.c | 60 ++++++++++++++++++++++++++-----------------------
mm/shmem.c | 13 +++++++----
mm/swap.h | 36 +++++++++++++----------------
mm/swap_state.c | 53 +++++++++++++++++++++++++------------------
mm/vmscan.c | 15 ++++++-------
mm/zswap.c | 4 +++-
7 files changed, 106 insertions(+), 91 deletions(-)
diff --git a/mm/madvise.c b/mm/madvise.c
index cd9bb077072c..cd84e993190a 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -188,7 +188,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,
unsigned long end, struct mm_walk *walk)
{
struct vm_area_struct *vma = walk->private;
- struct swap_iocb *splug = NULL;
+ struct swap_io_ctx ctx = {};
pte_t *ptep = NULL;
spinlock_t *ptl;
unsigned long addr;
@@ -212,15 +212,15 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,
pte_unmap_unlock(ptep, ptl);
ptep = NULL;
- folio = read_swap_cache_async(entry, GFP_HIGHUSER_MOVABLE,
- vma, addr, &splug);
+ folio = read_swap_cache_async(&ctx, entry, GFP_HIGHUSER_MOVABLE,
+ vma, addr);
if (folio)
folio_put(folio);
}
if (ptep)
pte_unmap_unlock(ptep, ptl);
- swap_read_unplug(splug);
+ swap_read_submit(&ctx);
cond_resched();
return 0;
@@ -238,7 +238,7 @@ static void shmem_swapin_range(struct vm_area_struct *vma,
XA_STATE(xas, &mapping->i_pages, linear_page_index(vma, start));
pgoff_t end_index = linear_page_index(vma, end) - 1;
struct folio *folio;
- struct swap_iocb *splug = NULL;
+ struct swap_io_ctx ctx = {};
rcu_read_lock();
xas_for_each(&xas, folio, end_index) {
@@ -257,15 +257,15 @@ static void shmem_swapin_range(struct vm_area_struct *vma,
xas_pause(&xas);
rcu_read_unlock();
- folio = read_swap_cache_async(entry, mapping_gfp_mask(mapping),
- vma, addr, &splug);
+ folio = read_swap_cache_async(&ctx, entry,
+ mapping_gfp_mask(mapping), vma, addr);
if (folio)
folio_put(folio);
rcu_read_lock();
}
rcu_read_unlock();
- swap_read_unplug(splug);
+ swap_read_submit(&ctx);
}
#endif /* CONFIG_SWAP */
diff --git a/mm/page_io.c b/mm/page_io.c
index b23f494fcc83..fe24a49e034c 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -248,7 +248,7 @@ static void swap_zeromap_folio_clear(struct folio *folio)
* We may have stale swap cache pages in memory: notice
* them here and get rid of the unnecessary final write.
*/
-int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
+int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
{
int ret = 0;
@@ -295,7 +295,7 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
}
rcu_read_unlock();
- __swap_writepage(folio, swap_plug);
+ __swap_writepage(ctx, folio);
return 0;
out_unlock:
folio_unlock(folio);
@@ -390,9 +390,9 @@ static void sio_write_complete(struct kiocb *iocb, long ret)
mempool_free(sio, sio_pool);
}
-static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug)
+static void swap_writepage_fs(struct swap_io_ctx *ctx, struct folio *folio)
{
- struct swap_iocb *sio = swap_plug ? *swap_plug : NULL;
+ struct swap_iocb *sio = ctx->sio;
struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
struct file *swap_file = sis->swap_file;
loff_t pos = swap_dev_pos(folio->swap);
@@ -403,7 +403,7 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug)
if (sio) {
if (sio->iocb.ki_filp != swap_file ||
sio->iocb.ki_pos + sio->len != pos) {
- swap_write_unplug(sio);
+ swap_write_submit(ctx);
sio = NULL;
}
}
@@ -418,12 +418,11 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug)
bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
sio->len += folio_size(folio);
sio->nr_bvecs += 1;
- if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !swap_plug) {
- swap_write_unplug(sio);
+ if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) {
+ swap_write_submit(ctx);
sio = NULL;
}
- if (swap_plug)
- *swap_plug = sio;
+ ctx->sio = sio;
}
static void swap_writepage_bdev_sync(struct folio *folio,
@@ -463,7 +462,7 @@ static void swap_writepage_bdev_async(struct folio *folio,
submit_bio(bio);
}
-void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug)
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
{
struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
@@ -474,7 +473,7 @@ void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug)
* is safe.
*/
if (data_race(sis->flags & SWP_FS_OPS))
- swap_writepage_fs(folio, swap_plug);
+ swap_writepage_fs(ctx, folio);
/*
* ->flags can be updated non-atomically,
* but that will never affect SWP_SYNCHRONOUS_IO, so the data_race
@@ -486,16 +485,20 @@ void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug)
swap_writepage_bdev_async(folio, sis);
}
-void swap_write_unplug(struct swap_iocb *sio)
+void swap_write_submit(struct swap_io_ctx *ctx)
{
+ struct swap_iocb *sio = ctx->sio;
struct iov_iter from;
- struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
int ret;
+ if (!sio)
+ return;
+
iov_iter_bvec(&from, ITER_SOURCE, sio->bvecs, sio->nr_bvecs, sio->len);
- ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
+ ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from);
if (ret != -EIOCBQUEUED)
sio_write_complete(&sio->iocb, ret);
+ ctx->sio = NULL;
}
static void sio_read_complete(struct kiocb *iocb, long ret)
@@ -587,18 +590,16 @@ static bool swap_read_folio_zeromap(struct folio *folio)
return true;
}
-static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug)
+static void swap_read_folio_fs(struct swap_io_ctx *ctx, struct folio *folio)
{
struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
- struct swap_iocb *sio = NULL;
+ struct swap_iocb *sio = ctx->sio;
loff_t pos = swap_dev_pos(folio->swap);
- if (plug)
- sio = *plug;
if (sio) {
if (sio->iocb.ki_filp != sis->swap_file ||
sio->iocb.ki_pos + sio->len != pos) {
- swap_read_unplug(sio);
+ swap_read_submit(ctx);
sio = NULL;
}
}
@@ -613,12 +614,11 @@ static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug)
bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
sio->len += folio_size(folio);
sio->nr_bvecs += 1;
- if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !plug) {
- swap_read_unplug(sio);
+ if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) {
+ swap_read_submit(ctx);
sio = NULL;
}
- if (plug)
- *plug = sio;
+ ctx->sio = sio;
}
static void swap_read_folio_bdev_sync(struct folio *folio,
@@ -658,7 +658,7 @@ static void swap_read_folio_bdev_async(struct folio *folio,
submit_bio(bio);
}
-void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
+void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
{
struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
bool synchronous = sis->flags & SWP_SYNCHRONOUS_IO;
@@ -693,7 +693,7 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
zswap_folio_swapin(folio);
if (data_race(sis->flags & SWP_FS_OPS)) {
- swap_read_folio_fs(folio, plug);
+ swap_read_folio_fs(ctx, folio);
} else if (synchronous) {
swap_read_folio_bdev_sync(folio, sis);
} else {
@@ -708,14 +708,18 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
delayacct_swapin_end();
}
-void __swap_read_unplug(struct swap_iocb *sio)
+void swap_read_submit(struct swap_io_ctx *ctx)
{
+ struct swap_iocb *sio = ctx->sio;
struct iov_iter from;
- struct address_space *mapping = sio->iocb.ki_filp->f_mapping;
int ret;
+ if (!sio)
+ return;
+
iov_iter_bvec(&from, ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len);
- ret = mapping->a_ops->swap_rw(&sio->iocb, &from);
+ ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from);
if (ret != -EIOCBQUEUED)
sio_read_complete(&sio->iocb, ret);
+ ctx->sio = NULL;
}
diff --git a/mm/shmem.c b/mm/shmem.c
index 6011a7f729ae..ac68876cca08 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1582,13 +1582,13 @@ int shmem_unuse(unsigned int type)
/**
* shmem_writeout - Write the folio to swap
+ * @ctx: swap I/O context
* @folio: The folio to write
- * @plug: swap plug
* @folio_list: list to put back folios on split
*
* Move the folio from the page cache to the swap cache.
*/
-int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
+int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
struct list_head *folio_list)
{
struct address_space *mapping = folio->mapping;
@@ -1700,7 +1700,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
BUG_ON(folio_mapped(folio));
- error = swap_writeout(folio, plug);
+ error = swap_writeout(ctx, folio);
if (error != AOP_WRITEPAGE_ACTIVATE) {
/* folio has been unlocked */
return error;
@@ -1739,7 +1739,12 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
int shmem_write_folio(struct folio *folio)
{
- return shmem_writeout(folio, NULL, NULL);
+ struct swap_io_ctx ctx = {};
+ int err;
+
+ err = shmem_writeout(&ctx, folio, NULL);
+ swap_write_submit(&ctx);
+ return err;
}
EXPORT_SYMBOL_GPL(shmem_write_folio);
diff --git a/mm/swap.h b/mm/swap.h
index 4f86ef338a60..79d66272dfd4 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -4,6 +4,7 @@
#include <linux/atomic.h> /* for atomic_long_t */
#include <linux/mm.h> /* for PAGE_SHIFT */
+
struct mempolicy;
struct swap_iocb;
struct swap_memcg_table;
@@ -78,6 +79,10 @@ enum swap_cluster_flags {
CLUSTER_FLAG_MAX,
};
+struct swap_io_ctx {
+ struct swap_iocb *sio;
+};
+
#ifdef CONFIG_SWAP
#include <linux/swapops.h> /* for swp_offset */
#include <linux/blk_types.h> /* for bio_end_io_t */
@@ -240,17 +245,11 @@ extern void __swap_cluster_free_entries(struct swap_info_struct *si,
/* linux/mm/page_io.c */
int sio_pool_init(void);
-struct swap_iocb;
-void swap_read_folio(struct folio *folio, struct swap_iocb **plug);
-void __swap_read_unplug(struct swap_iocb *plug);
-static inline void swap_read_unplug(struct swap_iocb *plug)
-{
- if (unlikely(plug))
- __swap_read_unplug(plug);
-}
-void swap_write_unplug(struct swap_iocb *sio);
-int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug);
-void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug);
+void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio);
+void swap_read_submit(struct swap_io_ctx *ctx);
+void swap_write_submit(struct swap_io_ctx *ctx);
+int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio);
/* linux/mm/swap_state.c */
extern struct address_space swap_space __read_mostly;
@@ -317,9 +316,8 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
void show_swap_cache_info(void);
void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr);
-struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
- struct vm_area_struct *vma, unsigned long addr,
- struct swap_iocb **plug);
+struct folio *read_swap_cache_async(struct swap_io_ctx *ctx, swp_entry_t entry,
+ gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr);
struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag,
struct mempolicy *mpol, pgoff_t ilx);
struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,
@@ -335,7 +333,6 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
}
#else /* CONFIG_SWAP */
-struct swap_iocb;
static inline struct swap_cluster_info *swap_cluster_lock(
struct swap_info_struct *si, pgoff_t offset, bool irq)
{
@@ -381,11 +378,11 @@ static inline void folio_put_swap(struct folio *folio, struct page *page)
{
}
-static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
+static inline void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
{
}
-static inline void swap_write_unplug(struct swap_iocb *sio)
+static inline void swap_write_submit(struct swap_io_ctx *ctx)
{
}
@@ -427,8 +424,7 @@ static inline void swap_update_readahead(struct folio *folio,
{
}
-static inline int swap_writeout(struct folio *folio,
- struct swap_iocb **swap_plug)
+static inline int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
{
return 0;
}
@@ -474,7 +470,7 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
#endif /* CONFIG_SWAP */
-int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
+int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
struct list_head *folio_list);
#endif /* _MM_SWAP_H */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 9c3a5cf99778..d8045db88057 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -633,9 +633,9 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
}
}
-static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
- struct mempolicy *mpol, pgoff_t ilx,
- struct swap_iocb **plug, bool readahead)
+static struct folio *swap_cache_read_folio(struct swap_io_ctx *ctx,
+ swp_entry_t entry, gfp_t gfp, struct mempolicy *mpol,
+ pgoff_t ilx, bool readahead)
{
struct folio *folio;
@@ -649,7 +649,7 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
if (IS_ERR_OR_NULL(folio))
return NULL;
- swap_read_folio(folio, plug);
+ swap_read_folio(ctx, folio);
if (readahead) {
folio_set_readahead(folio);
count_vm_event(SWAP_RA);
@@ -677,6 +677,7 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
{
+ struct swap_io_ctx ctx = {};
struct folio *folio;
do {
@@ -689,7 +690,8 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
if (IS_ERR(folio))
return folio;
- swap_read_folio(folio, NULL);
+ swap_read_folio(&ctx, folio);
+ swap_read_submit(&ctx);
return folio;
}
@@ -699,9 +701,8 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
* A failure return means that either the page allocation failed or that
* the swap entry is no longer in use.
*/
-struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
- struct vm_area_struct *vma, unsigned long addr,
- struct swap_iocb **plug)
+struct folio *read_swap_cache_async(struct swap_io_ctx *ctx, swp_entry_t entry,
+ gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr)
{
struct swap_info_struct *si;
struct mempolicy *mpol;
@@ -713,13 +714,24 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
return NULL;
mpol = get_vma_policy(vma, addr, 0, &ilx);
- folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx, plug, false);
+ folio = swap_cache_read_folio(ctx, entry, gfp_mask, mpol, ilx, false);
mpol_cond_put(mpol);
put_swap_device(si);
return folio;
}
+static struct folio *swap_cache_read_folio_sync(swp_entry_t entry, gfp_t gfp,
+ struct mempolicy *mpol, pgoff_t ilx)
+{
+ struct swap_io_ctx ctx = {};
+ struct folio *folio;
+
+ folio = swap_cache_read_folio(&ctx, entry, gfp, mpol, ilx, false);
+ swap_read_submit(&ctx);
+ return folio;
+}
+
static unsigned int __swapin_nr_pages(unsigned long prev_offset,
unsigned long offset,
int hits,
@@ -808,8 +820,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
unsigned long start_offset, end_offset;
unsigned long mask;
struct swap_info_struct *si = __swap_entry_to_info(entry);
+ struct swap_io_ctx ctx = {};
struct blk_plug plug;
- struct swap_iocb *splug = NULL;
swp_entry_t ra_entry;
mask = swapin_nr_pages(offset) - 1;
@@ -828,18 +840,17 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
for (offset = start_offset; offset <= end_offset ; offset++) {
/* Ok, do the async read-ahead now */
ra_entry = swp_entry(swp_type(entry), offset);
- folio = swap_cache_read_folio(ra_entry, gfp_mask, mpol, ilx,
- &splug, offset != entry_offset);
+ folio = swap_cache_read_folio(&ctx, ra_entry, gfp_mask, mpol,
+ ilx, offset != entry_offset);
if (!folio)
continue;
folio_put(folio);
}
blk_finish_plug(&plug);
- swap_read_unplug(splug);
+ swap_read_submit(&ctx);
lru_add_drain(); /* Push any new pages onto the LRU now */
skip:
- /* The page was likely read above, so no need for plugging here */
- return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false);
+ return swap_cache_read_folio_sync(entry, gfp_mask, mpol, ilx);
}
static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,
@@ -899,8 +910,8 @@ static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,
static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
struct mempolicy *mpol, pgoff_t targ_ilx, struct vm_fault *vmf)
{
+ struct swap_io_ctx ctx = {};
struct blk_plug plug;
- struct swap_iocb *splug = NULL;
struct folio *folio;
pte_t *pte = NULL, pentry;
int win;
@@ -939,8 +950,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
if (!si)
continue;
}
- folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx,
- &splug, addr != vmf->address);
+ folio = swap_cache_read_folio(&ctx, entry, gfp_mask, mpol, ilx,
+ addr != vmf->address);
if (si)
put_swap_device(si);
if (!folio)
@@ -950,13 +961,11 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
if (pte)
pte_unmap(pte);
blk_finish_plug(&plug);
- swap_read_unplug(splug);
+ swap_read_submit(&ctx);
lru_add_drain();
skip:
/* The folio was likely read above, so no need for plugging here */
- folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
- NULL, false);
- return folio;
+ return swap_cache_read_folio_sync(targ_entry, gfp_mask, mpol, targ_ilx);
}
/**
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 35c3bb15ae96..2747ee68d327 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -615,8 +615,8 @@ typedef enum {
/*
* pageout is called by shrink_folio_list() for each dirty folio.
*/
-static pageout_t pageout(struct folio *folio, struct address_space *mapping,
- struct swap_iocb **plug, struct list_head *folio_list)
+static pageout_t pageout(struct swap_io_ctx *ctx, struct address_space *mapping,
+ struct folio *folio, struct list_head *folio_list)
{
int res;
@@ -652,9 +652,9 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping,
* the split out folios get added back to folio_list.
*/
if (shmem_mapping(mapping))
- res = shmem_writeout(folio, plug, folio_list);
+ res = shmem_writeout(ctx, folio, folio_list);
else
- res = swap_writeout(folio, plug);
+ res = swap_writeout(ctx, folio);
if (res < 0)
handle_write_error(mapping, folio, res);
@@ -1063,7 +1063,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
unsigned int nr_reclaimed = 0, nr_demoted = 0;
unsigned int pgactivate = 0;
bool do_demote_pass;
- struct swap_iocb *plug = NULL;
+ struct swap_io_ctx ctx = {};
folio_batch_init(&free_folios);
memset(stat, 0, sizeof(*stat));
@@ -1394,7 +1394,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
* starts and then write it out here.
*/
try_to_unmap_flush_dirty();
- switch (pageout(folio, mapping, &plug, folio_list)) {
+ switch (pageout(&ctx, mapping, folio, folio_list)) {
case PAGE_KEEP:
goto keep_locked;
case PAGE_ACTIVATE:
@@ -1584,8 +1584,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
list_splice(&ret_folios, folio_list);
count_vm_events(PGACTIVATE, pgactivate);
- if (plug)
- swap_write_unplug(plug);
+ swap_write_submit(&ctx);
return nr_reclaimed;
}
diff --git a/mm/zswap.c b/mm/zswap.c
index 761cd699e0a3..feed2557f6ed 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -992,6 +992,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
struct folio *folio;
struct mempolicy *mpol;
struct swap_info_struct *si;
+ struct swap_io_ctx ctx = {};
int ret = 0;
/* try to allocate swap cache folio */
@@ -1049,7 +1050,8 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
folio_set_reclaim(folio);
/* start writeback */
- __swap_writepage(folio, NULL);
+ __swap_writepage(&ctx, folio);
+ swap_write_submit(&ctx);
out:
if (ret) {
--
2.53.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
2026-07-06 13:21 ` [PATCH 1/7] shmem: provide a shmem_write_folio wrapper Christoph Hellwig
2026-07-06 13:21 ` [PATCH 2/7] mm/swap: introduce struct swap_io_ctx Christoph Hellwig
@ 2026-07-06 13:21 ` Christoph Hellwig
2026-07-06 13:21 ` [PATCH 4/7] mm/swap: remove count_swpout_vm_event Christoph Hellwig
` (4 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2026-07-06 13:21 UTC (permalink / raw)
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
Block I/O benefits from batching just as much as remote file systems.
Extend struct swap_iocb to support building a bio on the fly as well,
and rewrite the block based swap code for it. This especially benefits
submit_bio based drivers that do not have the block plugging available,
but also saves allocating extra bios for blk-mq drivers.
Add a pre-allocated bio to struct swap_iocb in a union with kiocb
used for file system based swap so that struct swap_iocb can be used
for all swap I/O, and initialize the pool for it unconditionally.
Various low-level bdev and fs functions are now replaced with a unified
can_merge/add/submit scheme.
Note that the block based swap code now uses the same memcg-based
check previously added for file system based swap as well.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
mm/page_io.c | 525 ++++++++++++++++++++++++--------------------------
mm/swap.h | 1 +
mm/swapfile.c | 9 +-
3 files changed, 253 insertions(+), 282 deletions(-)
diff --git a/mm/page_io.c b/mm/page_io.c
index fe24a49e034c..f49e667c2b3a 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -28,54 +28,6 @@
#include "swap.h"
#include "swap_table.h"
-static void __end_swap_bio_write(struct bio *bio)
-{
- struct folio *folio = bio_first_folio_all(bio);
-
- if (bio->bi_status) {
- /*
- * We failed to write the page out to swap-space.
- * Re-dirty the page in order to avoid it being reclaimed.
- * Also print a dire warning that things will go BAD (tm)
- * very quickly.
- *
- * Also clear PG_reclaim to avoid folio_rotate_reclaimable()
- */
- folio_mark_dirty(folio);
- pr_alert_ratelimited("Write-error on swap-device (%u:%u:%llu)\n",
- MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
- (unsigned long long)bio->bi_iter.bi_sector);
- folio_clear_reclaim(folio);
- }
- folio_end_writeback(folio);
-}
-
-static void end_swap_bio_write(struct bio *bio)
-{
- __end_swap_bio_write(bio);
- bio_put(bio);
-}
-
-static void __end_swap_bio_read(struct bio *bio)
-{
- struct folio *folio = bio_first_folio_all(bio);
-
- if (bio->bi_status) {
- pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n",
- MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
- (unsigned long long)bio->bi_iter.bi_sector);
- } else {
- folio_mark_uptodate(folio);
- }
- folio_unlock(folio);
-}
-
-static void end_swap_bio_read(struct bio *bio)
-{
- __end_swap_bio_read(bio);
- bio_put(bio);
-}
-
int generic_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file,
sector_t *span)
@@ -316,18 +268,35 @@ static inline void count_swpout_vm_event(struct folio *folio)
}
#if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
+static struct cgroup_subsys_state *folio_memcg_blkg_css(struct folio *folio)
+{
+ return cgroup_e_css(folio_memcg(folio)->css.cgroup, &io_cgrp_subsys);
+}
+
+static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio)
+{
+ if (!folio_memcg_charged(folio) || !folio_memcg_charged(prev_folio))
+ return true;
+
+ rcu_read_lock();
+ if (folio_memcg_blkg_css(folio) != folio_memcg_blkg_css(prev_folio)) {
+ rcu_read_unlock();
+ return false;
+ }
+ rcu_read_unlock();
+
+ return true;
+}
+
static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
{
struct cgroup_subsys_state *css;
- struct mem_cgroup *memcg;
if (!folio_memcg_charged(folio))
return;
-
rcu_read_lock();
- memcg = folio_memcg(folio);
- css = cgroup_e_css(memcg->css.cgroup, &io_cgrp_subsys);
- if (!css || !css_tryget(css))
+ css = folio_memcg_blkg_css(folio);
+ if (css && !css_tryget(css))
css = NULL;
rcu_read_unlock();
@@ -336,11 +305,18 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
css_put(css);
}
#else
+static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio)
+{
+ return true;
+}
#define bio_associate_blkg_from_page(bio, folio) do { } while (0)
#endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */
struct swap_iocb {
- struct kiocb iocb;
+ union {
+ struct kiocb iocb;
+ struct bio bio;
+ };
struct bio_vec bvecs[SWAP_CLUSTER_MAX];
int nr_bvecs;
int len;
@@ -360,171 +336,70 @@ int sio_pool_init(void)
return 0;
}
-static void sio_write_complete(struct kiocb *iocb, long ret)
+static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,
+ int rw)
{
- struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
- struct page *page = sio->bvecs[0].bv_page;
- int p;
+ struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1];
+ struct folio *prev_folio = bvec_folio(last_bv);
+ size_t prev_folio_size = folio_size(prev_folio);
- if (ret != sio->len) {
- /*
- * In the case of swap-over-nfs, this can be a
- * temporary failure if the system has limited
- * memory for allocating transmit buffers.
- * Mark the page dirty and avoid
- * folio_rotate_reclaimable but rate-limit the
- * messages.
- */
- pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
- ret, swap_dev_pos(page_swap_entry(page)));
- for (p = 0; p < sio->nr_bvecs; p++) {
- page = sio->bvecs[p].bv_page;
- set_page_dirty(page);
- ClearPageReclaim(page);
- }
- }
+ if (ctx->sis != sis)
+ return false;
- for (p = 0; p < sio->nr_bvecs; p++)
- end_page_writeback(sio->bvecs[p].bv_page);
+ if (sis->flags & SWP_FS_OPS) {
+ if (swap_dev_pos(folio->swap) !=
+ swap_dev_pos(prev_folio->swap) + prev_folio_size)
+ return false;
+ } else {
+ if (swap_folio_sector(folio) !=
+ swap_folio_sector(prev_folio) +
+ (prev_folio_size >> SECTOR_SHIFT))
+ return false;
+ if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
+ return false;
+ }
- mempool_free(sio, sio_pool);
+ return true;
}
-static void swap_writepage_fs(struct swap_io_ctx *ctx, struct folio *folio)
+static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
{
- struct swap_iocb *sio = ctx->sio;
struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
- struct file *swap_file = sis->swap_file;
- loff_t pos = swap_dev_pos(folio->swap);
+ struct swap_iocb *sio = ctx->sio;
- count_swpout_vm_event(folio);
- folio_start_writeback(folio);
- folio_unlock(folio);
- if (sio) {
- if (sio->iocb.ki_filp != swap_file ||
- sio->iocb.ki_pos + sio->len != pos) {
+ if (sio && !swap_can_merge(ctx, folio, rw)) {
+ if (rw == WRITE)
swap_write_submit(ctx);
- sio = NULL;
- }
+ else
+ swap_read_submit(ctx);
+ sio = ctx->sio;
}
+
if (!sio) {
- sio = mempool_alloc(sio_pool, GFP_NOIO);
- init_sync_kiocb(&sio->iocb, swap_file);
- sio->iocb.ki_complete = sio_write_complete;
- sio->iocb.ki_pos = pos;
+ ctx->sis = sis;
+ ctx->sio = sio = mempool_alloc(sio_pool, GFP_NOIO);
sio->nr_bvecs = 0;
sio->len = 0;
}
bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
sio->len += folio_size(folio);
- sio->nr_bvecs += 1;
- if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) {
- swap_write_submit(ctx);
- sio = NULL;
+ if (++sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) {
+ if (rw == WRITE)
+ swap_write_submit(ctx);
+ else
+ swap_read_submit(ctx);
}
- ctx->sio = sio;
-}
-
-static void swap_writepage_bdev_sync(struct folio *folio,
- struct swap_info_struct *sis)
-{
- struct bio_vec bv;
- struct bio bio;
-
- bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_WRITE | REQ_SWAP);
- bio.bi_iter.bi_sector = swap_folio_sector(folio);
- bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
-
- bio_associate_blkg_from_page(&bio, folio);
- count_swpout_vm_event(folio);
-
- folio_start_writeback(folio);
- folio_unlock(folio);
-
- submit_bio_wait(&bio);
- __end_swap_bio_write(&bio);
}
-static void swap_writepage_bdev_async(struct folio *folio,
- struct swap_info_struct *sis)
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
{
- struct bio *bio;
-
- bio = bio_alloc(sis->bdev, 1, REQ_OP_WRITE | REQ_SWAP, GFP_NOIO);
- bio->bi_iter.bi_sector = swap_folio_sector(folio);
- bio->bi_end_io = end_swap_bio_write;
- bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
+ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
- bio_associate_blkg_from_page(bio, folio);
count_swpout_vm_event(folio);
folio_start_writeback(folio);
folio_unlock(folio);
- submit_bio(bio);
-}
-
-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
-{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
-
- VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
- /*
- * ->flags can be updated non-atomically,
- * but that will never affect SWP_FS_OPS, so the data_race
- * is safe.
- */
- if (data_race(sis->flags & SWP_FS_OPS))
- swap_writepage_fs(ctx, folio);
- /*
- * ->flags can be updated non-atomically,
- * but that will never affect SWP_SYNCHRONOUS_IO, so the data_race
- * is safe.
- */
- else if (data_race(sis->flags & SWP_SYNCHRONOUS_IO))
- swap_writepage_bdev_sync(folio, sis);
- else
- swap_writepage_bdev_async(folio, sis);
-}
-
-void swap_write_submit(struct swap_io_ctx *ctx)
-{
- struct swap_iocb *sio = ctx->sio;
- struct iov_iter from;
- int ret;
-
- if (!sio)
- return;
-
- iov_iter_bvec(&from, ITER_SOURCE, sio->bvecs, sio->nr_bvecs, sio->len);
- ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from);
- if (ret != -EIOCBQUEUED)
- sio_write_complete(&sio->iocb, ret);
- ctx->sio = NULL;
-}
-
-static void sio_read_complete(struct kiocb *iocb, long ret)
-{
- struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
- int p;
-
- if (ret == sio->len) {
- for (p = 0; p < sio->nr_bvecs; p++) {
- struct folio *folio = bvec_folio(&sio->bvecs[p]);
-
- count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
- count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
- folio_mark_uptodate(folio);
- folio_unlock(folio);
- }
- count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);
- } else {
- for (p = 0; p < sio->nr_bvecs; p++) {
- struct folio *folio = bvec_folio(&sio->bvecs[p]);
-
- folio_unlock(folio);
- }
- pr_alert_ratelimited("Read-error on swap-device\n");
- }
- mempool_free(sio, sio_pool);
+ swap_add_folio(ctx, folio, WRITE);
}
/*
@@ -590,74 +465,6 @@ static bool swap_read_folio_zeromap(struct folio *folio)
return true;
}
-static void swap_read_folio_fs(struct swap_io_ctx *ctx, struct folio *folio)
-{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
- struct swap_iocb *sio = ctx->sio;
- loff_t pos = swap_dev_pos(folio->swap);
-
- if (sio) {
- if (sio->iocb.ki_filp != sis->swap_file ||
- sio->iocb.ki_pos + sio->len != pos) {
- swap_read_submit(ctx);
- sio = NULL;
- }
- }
- if (!sio) {
- sio = mempool_alloc(sio_pool, GFP_KERNEL);
- init_sync_kiocb(&sio->iocb, sis->swap_file);
- sio->iocb.ki_pos = pos;
- sio->iocb.ki_complete = sio_read_complete;
- sio->nr_bvecs = 0;
- sio->len = 0;
- }
- bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
- sio->len += folio_size(folio);
- sio->nr_bvecs += 1;
- if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) {
- swap_read_submit(ctx);
- sio = NULL;
- }
- ctx->sio = sio;
-}
-
-static void swap_read_folio_bdev_sync(struct folio *folio,
- struct swap_info_struct *sis)
-{
- struct bio_vec bv;
- struct bio bio;
-
- bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_READ);
- bio.bi_iter.bi_sector = swap_folio_sector(folio);
- bio_add_folio_nofail(&bio, folio, folio_size(folio), 0);
- /*
- * Keep this task valid during swap readpage because the oom killer may
- * attempt to access it in the page fault retry time check.
- */
- get_task_struct(current);
- count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
- count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
- count_vm_events(PSWPIN, folio_nr_pages(folio));
- submit_bio_wait(&bio);
- __end_swap_bio_read(&bio);
- put_task_struct(current);
-}
-
-static void swap_read_folio_bdev_async(struct folio *folio,
- struct swap_info_struct *sis)
-{
- struct bio *bio;
-
- bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL);
- bio->bi_iter.bi_sector = swap_folio_sector(folio);
- bio->bi_end_io = end_swap_bio_read;
- bio_add_folio_nofail(bio, folio, folio_size(folio), 0);
- count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
- count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
- count_vm_events(PSWPIN, folio_nr_pages(folio));
- submit_bio(bio);
-}
-
void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
{
struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
@@ -691,14 +498,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
-
- if (data_race(sis->flags & SWP_FS_OPS)) {
- swap_read_folio_fs(ctx, folio);
- } else if (synchronous) {
- swap_read_folio_bdev_sync(folio, sis);
- } else {
- swap_read_folio_bdev_async(folio, sis);
- }
+ swap_add_folio(ctx, folio, READ);
finish:
if (workingset) {
@@ -708,18 +508,189 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
delayacct_swapin_end();
}
-void swap_read_submit(struct swap_io_ctx *ctx)
+static void swap_write_end(struct swap_iocb *sio, bool failed)
+{
+ int p;
+
+ for (p = 0; p < sio->nr_bvecs; p++) {
+ struct page *page = sio->bvecs[p].bv_page;
+
+ if (failed) {
+ set_page_dirty(page);
+ ClearPageReclaim(page);
+ }
+ end_page_writeback(page);
+ }
+ mempool_free(sio, sio_pool);
+}
+
+static void swap_fs_write_complete(struct kiocb *iocb, long ret)
+{
+ struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
+ bool failed = ret != sio->len;
+
+ if (failed) {
+ struct page *page = sio->bvecs[0].bv_page;
+
+ /*
+ * In the case of swap-over-nfs, this can be a temporary failure
+ * if the system has limited memory for allocating transmit
+ * buffers. Mark the page dirty and avoid
+ * folio_rotate_reclaimable but rate-limit the messages.
+ */
+ pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
+ ret, swap_dev_pos(page_swap_entry(page)));
+ }
+
+ swap_write_end(sio, failed);
+}
+
+static void end_swap_bio_write(struct bio *bio)
+{
+ struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio);
+ bool failed = !!bio->bi_status;
+
+ if (failed)
+ pr_alert_ratelimited("Write-error on swap-device (%u:%u:%llu)\n",
+ MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
+ (unsigned long long)bio->bi_iter.bi_sector);
+ bio_uninit(bio);
+ swap_write_end(sio, failed);
+}
+
+static void swap_read_end(struct swap_iocb *sio, bool failed)
+{
+ int p;
+
+ for (p = 0; p < sio->nr_bvecs; p++) {
+ struct folio *folio = bvec_folio(&sio->bvecs[p]);
+
+ if (!failed) {
+ count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
+ count_memcg_folio_events(folio, PSWPIN,
+ folio_nr_pages(folio));
+ folio_mark_uptodate(folio);
+ }
+ folio_unlock(folio);
+ }
+
+ if (!failed)
+ count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT);
+
+ mempool_free(sio, sio_pool);
+}
+
+static void swap_fs_read_complete(struct kiocb *iocb, long ret)
+{
+ struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb);
+ bool failed = ret != sio->len;
+
+ if (failed)
+ pr_alert_ratelimited("Read-error on swap-device\n");
+ swap_read_end(sio, failed);
+}
+
+static void swap_bio_read_end_io(struct bio *bio)
+{
+ struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio);
+ bool failed = !!bio->bi_status;
+
+ if (failed)
+ pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n",
+ MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)),
+ (unsigned long long)bio->bi_iter.bi_sector);
+ bio_uninit(bio);
+ swap_read_end(sio, failed);
+}
+
+static void swap_bdev_submit_write(struct swap_io_ctx *ctx)
{
struct swap_iocb *sio = ctx->sio;
- struct iov_iter from;
+ struct bio *bio = &sio->bio;
+
+ bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
+ REQ_OP_WRITE | REQ_SWAP);
+ bio->bi_iter.bi_size = sio->len;
+ bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio_associate_blkg_from_page(bio, bio_first_folio_all(bio));
+
+ if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
+ submit_bio_wait(bio);
+ end_swap_bio_write(bio);
+ } else {
+ bio->bi_end_io = end_swap_bio_write;
+ submit_bio(bio);
+ }
+}
+
+static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
+{
+ struct swap_iocb *sio = ctx->sio;
+ struct bio *bio = &sio->bio;
+
+ bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
+ REQ_OP_READ);
+ bio->bi_iter.bi_size = sio->len;
+ bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+
+ if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
+ /*
+ * Keep this task valid during swap readpage because the oom
+ * killer may attempt to access it in the page fault retry
+ * time check.
+ */
+ get_task_struct(current);
+ submit_bio_wait(bio);
+ swap_bio_read_end_io(bio);
+ put_task_struct(current);
+ } else {
+ bio->bi_end_io = swap_bio_read_end_io;
+ submit_bio(bio);
+ }
+}
+
+static void swap_fs_submit(struct swap_io_ctx *ctx, int rw)
+{
+ struct swap_iocb *sio = ctx->sio;
+ struct iov_iter iter;
int ret;
- if (!sio)
- return;
+ init_sync_kiocb(&sio->iocb, ctx->sis->swap_file);
+ sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap);
+ if (rw == WRITE)
+ sio->iocb.ki_complete = swap_fs_write_complete;
+ else
+ sio->iocb.ki_complete = swap_fs_read_complete;
- iov_iter_bvec(&from, ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len);
- ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from);
+ iov_iter_bvec(&iter, rw == WRITE ? ITER_SOURCE : ITER_DEST,
+ sio->bvecs, sio->nr_bvecs, sio->len);
+ ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &iter);
if (ret != -EIOCBQUEUED)
- sio_read_complete(&sio->iocb, ret);
+ sio->iocb.ki_complete(&sio->iocb, ret);
+}
+
+void swap_write_submit(struct swap_io_ctx *ctx)
+{
+ if (!ctx->sio)
+ return;
+
+ if (ctx->sis->flags & SWP_FS_OPS)
+ swap_fs_submit(ctx, WRITE);
+ else
+ swap_bdev_submit_write(ctx);
+ ctx->sio = NULL;
+ ctx->sis = NULL;
+}
+
+void swap_read_submit(struct swap_io_ctx *ctx)
+{
+ if (!ctx->sio)
+ return;
+
+ if (ctx->sis->flags & SWP_FS_OPS)
+ swap_fs_submit(ctx, READ);
+ else
+ swap_bdev_submit_read(ctx);
ctx->sio = NULL;
+ ctx->sis = NULL;
}
diff --git a/mm/swap.h b/mm/swap.h
index 79d66272dfd4..b6ba80c2afb0 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -81,6 +81,7 @@ enum swap_cluster_flags {
struct swap_io_ctx {
struct swap_iocb *sio;
+ struct swap_info_struct *sis;
};
#ifdef CONFIG_SWAP
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 78b49b0658ad..1887df22bd1e 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2941,6 +2941,10 @@ static int setup_swap_extents(struct swap_info_struct *sis,
struct inode *inode = mapping->host;
int ret;
+ ret = sio_pool_init();
+ if (ret)
+ return ret;
+
if (S_ISBLK(inode->i_mode)) {
ret = add_swap_extent(sis, 0, sis->max, 0);
*span = sis->pages;
@@ -2952,11 +2956,6 @@ static int setup_swap_extents(struct swap_info_struct *sis,
if (ret < 0)
return ret;
sis->flags |= SWP_ACTIVATED;
- if ((sis->flags & SWP_FS_OPS) &&
- sio_pool_init() != 0) {
- destroy_swap_extents(sis, swap_file);
- return -ENOMEM;
- }
return ret;
}
--
2.53.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 4/7] mm/swap: remove count_swpout_vm_event
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
` (2 preceding siblings ...)
2026-07-06 13:21 ` [PATCH 3/7] mm/swap: also use struct swap_iocb for block I/O Christoph Hellwig
@ 2026-07-06 13:21 ` Christoph Hellwig
2026-07-06 13:21 ` [PATCH 5/7] mm/swap: use swap_ops to register swap device's methods Christoph Hellwig
` (3 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2026-07-06 13:21 UTC (permalink / raw)
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
There is only one caller left, so merge it into that.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
---
mm/page_io.c | 24 ++++++++++--------------
1 file changed, 10 insertions(+), 14 deletions(-)
diff --git a/mm/page_io.c b/mm/page_io.c
index f49e667c2b3a..6009f6991b73 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -254,19 +254,6 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
return ret;
}
-static inline void count_swpout_vm_event(struct folio *folio)
-{
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- if (unlikely(folio_test_pmd_mappable(folio))) {
- count_memcg_folio_events(folio, THP_SWPOUT, 1);
- count_vm_event(THP_SWPOUT);
- }
-#endif
- count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT);
- count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio));
- count_vm_events(PSWPOUT, folio_nr_pages(folio));
-}
-
#if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
static struct cgroup_subsys_state *folio_memcg_blkg_css(struct folio *folio)
{
@@ -396,7 +383,16 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
{
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
- count_swpout_vm_event(folio);
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ if (unlikely(folio_test_pmd_mappable(folio))) {
+ count_memcg_folio_events(folio, THP_SWPOUT, 1);
+ count_vm_event(THP_SWPOUT);
+ }
+#endif
+ count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT);
+ count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio));
+ count_vm_events(PSWPOUT, folio_nr_pages(folio));
+
folio_start_writeback(folio);
folio_unlock(folio);
swap_add_folio(ctx, folio, WRITE);
--
2.53.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 5/7] mm/swap: use swap_ops to register swap device's methods
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
` (3 preceding siblings ...)
2026-07-06 13:21 ` [PATCH 4/7] mm/swap: remove count_swpout_vm_event Christoph Hellwig
@ 2026-07-06 13:21 ` Christoph Hellwig
2026-07-06 13:21 ` [PATCH 6/7] mm/swap: remove SWP_FS_OPS Christoph Hellwig
` (2 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2026-07-06 13:21 UTC (permalink / raw)
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
This simplifies codes and makes logic clearer. And also makes later any
new swap device type being added easier to handle.
Currently there are two types of swap devices: fs and bdev.
Suggested-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Baoquan He <baoquan.he@linux.dev>
[hch: updated for the new submit and can_merge abstraction]
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 1 +
mm/page_io.c | 68 ++++++++++++++++++++++++++++----------------
mm/swap.h | 10 +++++++
mm/swapfile.c | 4 +++
4 files changed, 58 insertions(+), 25 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8f0f68e245ba..74ceb7a88fd5 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -282,6 +282,7 @@ struct swap_info_struct {
struct work_struct reclaim_work; /* reclaim worker */
struct list_head discard_clusters; /* discard clusters list */
struct plist_node avail_list; /* entry in swap_avail_head */
+ const struct swap_ops *ops;
};
static inline swp_entry_t page_swap_entry(struct page *page)
diff --git a/mm/page_io.c b/mm/page_io.c
index 6009f6991b73..d2b11db1a57f 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -333,21 +333,7 @@ static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,
if (ctx->sis != sis)
return false;
-
- if (sis->flags & SWP_FS_OPS) {
- if (swap_dev_pos(folio->swap) !=
- swap_dev_pos(prev_folio->swap) + prev_folio_size)
- return false;
- } else {
- if (swap_folio_sector(folio) !=
- swap_folio_sector(prev_folio) +
- (prev_folio_size >> SECTOR_SHIFT))
- return false;
- if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
- return false;
- }
-
- return true;
+ return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw);
}
static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
@@ -645,6 +631,23 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
}
}
+static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio,
+ size_t prev_folio_size, int rw)
+{
+ if (swap_folio_sector(folio) !=
+ swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT))
+ return false;
+ if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
+ return false;
+ return true;
+}
+
+const struct swap_ops swap_bdev_ops = {
+ .submit_write = swap_bdev_submit_write,
+ .submit_read = swap_bdev_submit_read,
+ .can_merge = swap_bdev_can_merge,
+};
+
static void swap_fs_submit(struct swap_io_ctx *ctx, int rw)
{
struct swap_iocb *sio = ctx->sio;
@@ -665,15 +668,34 @@ static void swap_fs_submit(struct swap_io_ctx *ctx, int rw)
sio->iocb.ki_complete(&sio->iocb, ret);
}
+static void swap_fs_submit_write(struct swap_io_ctx *ctx)
+{
+ swap_fs_submit(ctx, WRITE);
+}
+
+static void swap_fs_submit_read(struct swap_io_ctx *ctx)
+{
+ swap_fs_submit(ctx, READ);
+}
+
+static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
+ size_t prev_folio_size, int rw)
+{
+ return swap_dev_pos(folio->swap) ==
+ swap_dev_pos(prev_folio->swap) + prev_folio_size;
+}
+
+const struct swap_ops swap_fs_ops = {
+ .submit_write = swap_fs_submit_write,
+ .submit_read = swap_fs_submit_read,
+ .can_merge = swap_fs_can_merge,
+};
+
void swap_write_submit(struct swap_io_ctx *ctx)
{
if (!ctx->sio)
return;
-
- if (ctx->sis->flags & SWP_FS_OPS)
- swap_fs_submit(ctx, WRITE);
- else
- swap_bdev_submit_write(ctx);
+ ctx->sis->ops->submit_write(ctx);
ctx->sio = NULL;
ctx->sis = NULL;
}
@@ -682,11 +704,7 @@ void swap_read_submit(struct swap_io_ctx *ctx)
{
if (!ctx->sio)
return;
-
- if (ctx->sis->flags & SWP_FS_OPS)
- swap_fs_submit(ctx, READ);
- else
- swap_bdev_submit_read(ctx);
+ ctx->sis->ops->submit_read(ctx);
ctx->sio = NULL;
ctx->sis = NULL;
}
diff --git a/mm/swap.h b/mm/swap.h
index b6ba80c2afb0..5119feff0e93 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -84,6 +84,13 @@ struct swap_io_ctx {
struct swap_info_struct *sis;
};
+struct swap_ops {
+ bool (*can_merge)(struct folio *folio, struct folio *prev_folio,
+ size_t prev_folio_size, int rw);
+ void (*submit_write)(struct swap_io_ctx *ctx);
+ void (*submit_read)(struct swap_io_ctx *ctx);
+};
+
#ifdef CONFIG_SWAP
#include <linux/swapops.h> /* for swp_offset */
#include <linux/blk_types.h> /* for bio_end_io_t */
@@ -471,6 +478,9 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
#endif /* CONFIG_SWAP */
+extern const struct swap_ops swap_bdev_ops;
+extern const struct swap_ops swap_fs_ops;
+
int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
struct list_head *folio_list);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 1887df22bd1e..b291510813a6 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2945,6 +2945,8 @@ static int setup_swap_extents(struct swap_info_struct *sis,
if (ret)
return ret;
+ sis->ops = &swap_bdev_ops;
+
if (S_ISBLK(inode->i_mode)) {
ret = add_swap_extent(sis, 0, sis->max, 0);
*span = sis->pages;
@@ -2955,6 +2957,8 @@ static int setup_swap_extents(struct swap_info_struct *sis,
ret = mapping->a_ops->swap_activate(sis, swap_file, span);
if (ret < 0)
return ret;
+ if (sis->flags & SWP_FS_OPS)
+ sis->ops = &swap_fs_ops;
sis->flags |= SWP_ACTIVATED;
return ret;
}
--
2.53.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 6/7] mm/swap: remove SWP_FS_OPS
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
` (4 preceding siblings ...)
2026-07-06 13:21 ` [PATCH 5/7] mm/swap: use swap_ops to register swap device's methods Christoph Hellwig
@ 2026-07-06 13:21 ` Christoph Hellwig
2026-07-06 13:21 ` [PATCH 7/7] mm/vmstat: add NRSWP{IN,OUT} counters Christoph Hellwig
2026-07-06 17:21 ` better block swap batching and a different take on swap_ops v3 Andrew Morton
7 siblings, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2026-07-06 13:21 UTC (permalink / raw)
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
Provide a swap_fs_activate helper that directly sets up swap_fs_ops,
and a flag in struct swap_ops to indicate of NOFS swapping is allowed.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
Documentation/filesystems/locking.rst | 5 +++--
Documentation/filesystems/vfs.rst | 4 ++--
fs/nfs/file.c | 4 +---
fs/smb/client/file.c | 4 +---
include/linux/swap.h | 6 +++++-
mm/page_io.c | 10 +++++++++-
mm/swap.h | 22 ++++++++++------------
mm/swapfile.c | 2 --
mm/vmscan.c | 15 +++++++--------
9 files changed, 38 insertions(+), 34 deletions(-)
diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst
index 08d01bc62c31..1a50d41a39a1 100644
--- a/Documentation/filesystems/locking.rst
+++ b/Documentation/filesystems/locking.rst
@@ -355,13 +355,14 @@ should perform any validation and preparation necessary to ensure that
writes can be performed with minimal memory allocation. It should call
add_swap_extent(), or the helper iomap_swapfile_activate(), and return
the number of extents added. If IO should be submitted through
-->swap_rw(), it should set SWP_FS_OPS, otherwise IO will be submitted
+->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted
directly to the block device ``sis->bdev``.
->swap_deactivate() will be called in the sys_swapoff()
path after ->swap_activate() returned success.
-->swap_rw will be called for swap IO if SWP_FS_OPS was set by ->swap_activate().
+->swap_rw will be called for swap IO if swap_fs_activate was called by
+->swap_activate().
file_lock_operations
====================
diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst
index 7c753148af88..e7677423a20f 100644
--- a/Documentation/filesystems/vfs.rst
+++ b/Documentation/filesystems/vfs.rst
@@ -977,7 +977,7 @@ cache in your filesystem. The following members are defined:
can be performed with minimal memory allocation. It should call
add_swap_extent(), or the helper iomap_swapfile_activate(), and
return the number of extents added. If IO should be submitted
- through ->swap_rw(), it should set SWP_FS_OPS, otherwise IO will
+ through ->swap_rw(), it should call swap_fs_activate, otherwise IO will
be submitted directly to the block device ``sis->bdev``.
``swap_deactivate``
@@ -985,7 +985,7 @@ cache in your filesystem. The following members are defined:
successful.
``swap_rw``
- Called to read or write swap pages when SWP_FS_OPS is set.
+ Called to read or write swap pages when swap_fs_activate was called.
The File Object
===============
diff --git a/fs/nfs/file.c b/fs/nfs/file.c
index a0d8f1c1cf10..851d93a09988 100644
--- a/fs/nfs/file.c
+++ b/fs/nfs/file.c
@@ -597,7 +597,7 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file,
ret = rpc_clnt_swap_activate(clnt);
if (ret)
return ret;
- ret = add_swap_extent(sis, 0, sis->max, 0);
+ ret = swap_fs_activate(sis);
if (ret < 0) {
rpc_clnt_swap_deactivate(clnt);
return ret;
@@ -607,8 +607,6 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file,
if (cl->rpc_ops->enable_swap)
cl->rpc_ops->enable_swap(inode);
-
- sis->flags |= SWP_FS_OPS;
return ret;
}
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index 5a25635bc62a..4b1bc16b8f3f 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -3387,9 +3387,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis,
* but we could add call to grab a byte range lock to prevent others
* from reading or writing the file
*/
-
- sis->flags |= SWP_FS_OPS;
- return add_swap_extent(sis, 0, sis->max, 0);
+ return swap_fs_activate(sis);
}
static void cifs_swap_deactivate(struct file *file)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 74ceb7a88fd5..bb53601e05b3 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -208,7 +208,6 @@ enum {
SWP_SOLIDSTATE = (1 << 4), /* blkdev seeks are cheap */
SWP_BLKDEV = (1 << 6), /* its a block device */
SWP_ACTIVATED = (1 << 7), /* set after swap_activate success */
- SWP_FS_OPS = (1 << 8), /* swapfile operations go through fs */
SWP_AREA_DISCARD = (1 << 9), /* single-time swap area discards */
SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */
SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
@@ -404,6 +403,7 @@ extern void __meminit kswapd_stop(int nid);
#ifdef CONFIG_SWAP
+int swap_fs_activate(struct swap_info_struct *sis);
int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page,
unsigned long nr_pages, sector_t start_block);
int generic_swapfile_activate(struct swap_info_struct *, struct file *,
@@ -530,6 +530,10 @@ static inline bool folio_free_swap(struct folio *folio)
return false;
}
+static inline int swap_fs_activate(struct swap_info_struct *sis)
+{
+ return -EINVAL;
+}
static inline int add_swap_extent(struct swap_info_struct *sis,
unsigned long start_page,
unsigned long nr_pages, sector_t start_block)
diff --git a/mm/page_io.c b/mm/page_io.c
index d2b11db1a57f..1e29b36cec07 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -685,12 +685,20 @@ static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
swap_dev_pos(prev_folio->swap) + prev_folio_size;
}
-const struct swap_ops swap_fs_ops = {
+static const struct swap_ops swap_fs_ops = {
+ .flags = SWAP_OPS_F_REQUIRE_NOFS,
.submit_write = swap_fs_submit_write,
.submit_read = swap_fs_submit_read,
.can_merge = swap_fs_can_merge,
};
+int swap_fs_activate(struct swap_info_struct *sis)
+{
+ sis->ops = &swap_fs_ops;
+ return add_swap_extent(sis, 0, sis->max, 0);
+}
+EXPORT_SYMBOL_GPL(swap_fs_activate);
+
void swap_write_submit(struct swap_io_ctx *ctx)
{
if (!ctx->sio)
diff --git a/mm/swap.h b/mm/swap.h
index 5119feff0e93..0ff2f82eced3 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -84,7 +84,17 @@ struct swap_io_ctx {
struct swap_info_struct *sis;
};
+/*
+ * SWAP_OPS_F_REQUIRE_NOFS:
+ * When set, all reclaim operations must operated as GFS_NOFS and not
+ * just GFP_NOIO, as GFP_NOIO allocations could recourse into the
+ * file system backing this swap file.
+ */
+#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0)
+
struct swap_ops {
+ unsigned int flags;
+
bool (*can_merge)(struct folio *folio, struct folio *prev_folio,
size_t prev_folio_size, int rw);
void (*submit_write)(struct swap_io_ctx *ctx);
@@ -335,11 +345,6 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders,
void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
unsigned long addr);
-static inline unsigned int folio_swap_flags(struct folio *folio)
-{
- return __swap_entry_to_info(folio->swap)->flags;
-}
-
#else /* CONFIG_SWAP */
static inline struct swap_cluster_info *swap_cluster_lock(
struct swap_info_struct *si, pgoff_t offset, bool irq)
@@ -470,16 +475,9 @@ static inline void __swap_cache_replace_folio(struct swap_cluster_info *ci,
struct folio *old, struct folio *new)
{
}
-
-static inline unsigned int folio_swap_flags(struct folio *folio)
-{
- return 0;
-}
-
#endif /* CONFIG_SWAP */
extern const struct swap_ops swap_bdev_ops;
-extern const struct swap_ops swap_fs_ops;
int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
struct list_head *folio_list);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index b291510813a6..b0b843372f67 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2957,8 +2957,6 @@ static int setup_swap_extents(struct swap_info_struct *sis,
ret = mapping->a_ops->swap_activate(sis, swap_file, span);
if (ret < 0)
return ret;
- if (sis->flags & SWP_FS_OPS)
- sis->ops = &swap_fs_ops;
sis->flags |= SWP_ACTIVATED;
return ret;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 2747ee68d327..ef280a24454f 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1037,16 +1037,15 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask)
{
if (gfp_mask & __GFP_FS)
return true;
- if (!folio_test_swapcache(folio) || !(gfp_mask & __GFP_IO))
- return false;
/*
- * We can "enter_fs" for swap-cache with only __GFP_IO
- * providing this isn't SWP_FS_OPS.
- * ->flags can be updated non-atomically,
- * but that will never affect SWP_FS_OPS, so the data_race
- * is safe.
+ * We can "enter_fs" for swap-cache with only __GFP_IO unless backed by
+ * a swapfile that requires GFP_NOFS I/O.
*/
- return !data_race(folio_swap_flags(folio) & SWP_FS_OPS);
+ if (folio_test_swapcache(folio) && (gfp_mask & __GFP_IO) &&
+ !(__swap_entry_to_info(folio->swap)->ops->flags &
+ SWAP_OPS_F_REQUIRE_NOFS))
+ return true;
+ return false;
}
/*
--
2.53.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 7/7] mm/vmstat: add NRSWP{IN,OUT} counters
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
` (5 preceding siblings ...)
2026-07-06 13:21 ` [PATCH 6/7] mm/swap: remove SWP_FS_OPS Christoph Hellwig
@ 2026-07-06 13:21 ` Christoph Hellwig
2026-07-06 17:21 ` better block swap batching and a different take on swap_ops v3 Andrew Morton
7 siblings, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2026-07-06 13:21 UTC (permalink / raw)
Cc: baoquan.he, akpm, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm
Count how many swap I/Os we cause. Due to batching this can be different
than the current counter number of pages written/read, and tracking this
information is useful to see how efficient the batching is.
The counters are added at the end of enum vm_event_item and the
vmstat_text array under the assumption that the order of fields in
/proc/vmstat is an ABI. If that is not the case, they could be grouped
with the other swap counters.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/vm_event_item.h | 4 ++++
mm/page_io.c | 2 ++
mm/vmstat.c | 6 +++++-
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 03fe95f5a020..2628ccda076a 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -175,6 +175,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
KSTACK_REST,
#endif
#endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_SWAP
+ NRSWPIN,
+ NRSWPOUT,
+#endif /* CONFIG_SWAP */
NR_VM_EVENT_ITEMS
};
diff --git a/mm/page_io.c b/mm/page_io.c
index 1e29b36cec07..c7681dbcfc76 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -703,6 +703,7 @@ void swap_write_submit(struct swap_io_ctx *ctx)
{
if (!ctx->sio)
return;
+ count_vm_events(NRSWPOUT, 1);
ctx->sis->ops->submit_write(ctx);
ctx->sio = NULL;
ctx->sis = NULL;
@@ -712,6 +713,7 @@ void swap_read_submit(struct swap_io_ctx *ctx)
{
if (!ctx->sio)
return;
+ count_vm_events(NRSWPIN, 1);
ctx->sis->ops->submit_read(ctx);
ctx->sio = NULL;
ctx->sis = NULL;
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..9559f3c95735 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1488,7 +1488,11 @@ const char * const vmstat_text[] = {
#if THREAD_SIZE > 65536
[I(KSTACK_REST)] = "kstack_rest",
#endif
-#endif
+#endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_SWAP
+ [I(NRSWPIN)] = "nrswpin",
+ [I(NRSWPOUT)] = "nrswpout",
+#endif /* CONFIG_SWAP */
#undef I
#endif /* CONFIG_VM_EVENT_COUNTERS */
};
--
2.53.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: better block swap batching and a different take on swap_ops v3
2026-07-06 13:21 better block swap batching and a different take on swap_ops v3 Christoph Hellwig
` (6 preceding siblings ...)
2026-07-06 13:21 ` [PATCH 7/7] mm/vmstat: add NRSWP{IN,OUT} counters Christoph Hellwig
@ 2026-07-06 17:21 ` Andrew Morton
7 siblings, 0 replies; 11+ messages in thread
From: Andrew Morton @ 2026-07-06 17:21 UTC (permalink / raw)
To: Christoph Hellwig
Cc: baoquan.he, chrisl, usama.arif, kasong, nphamcs, shikemeng,
youngjun.park, linux-mm, dri-devel
On Mon, 6 Jul 2026 15:21:52 +0200 Christoph Hellwig <hch@lst.de> wrote:
> this series makes use of the swap_iocb for block as well so that it
> doesn't do inefficient single-bio I/O, and then rebases the swap_ops from
> Baoquan on top of the now very different method structure.
>
> When running doing kernels builds, which is a workload that doesn't
> really do much THP anonymous memory it still gets 2x clustering for
> writeout and 1.2x for reading back swap in. The overall times do
> not actually change, though.
Thanks.
AI review might have found some pre-existing bugs in
drivers/gpu/drm/i915/gem/i915_gem_shmem.c
drivers/gpu/drm/ttm/ttm_backup.c
https://sashiko.dev/#/patchset/20260706132206.935272-1-hch@lst.de
^ permalink raw reply [flat|nested] 11+ messages in thread