[PATCH v3 00/20] iomap: convert to in-iter iomap_next() model

[PATCH v3 00/20] iomap: convert to in-iter iomap_next() model

[Joanne Koong]

This series implements a suggestion by Christoph for finishing the conversion
of iomap to an iterator model. This revives Matthew's previous RFC [1], which
had the same intention.

Every iomap operation currently drives its iteration through a struct
iomap_ops, which contains two callbacks, ->iomap_begin() and ->iomap_end().
iomap_iter() only ever sees these as pointers, so every step of every
iteration is an indirect call, including on the hottest paths. 

This series replaces the begin/end pair with a single ->iomap_next() callback
that finishes the previous mapping (if any) and produces the next one.
Collapsing to one callback lets a performance-critical caller inline its
iteration loop and pass its ->iomap_next() function as a compile-time
constant, where the compiler can devirtualize the callback into a direct and
inlineable call rather than an indirect one. It also allows future callers
more flexibility in expressing custom logic in the IO path for driving the
iteration forward.

This series has no functional changes intended. The patches are broken down as
follows:

1) Patch 1: refactors existing iomap_iter() logic into an iomap_iter_next()
function. Sets up DEFINE_IOMAP_ITER_NEXT/DEFINE_IOMAP_ITER_NEXT_END macro.

2) Patch 2: has iomap_iter() and iomap_dio_simple() paths support
->iomap_next() in addition to ->iomap_begin()/->iomap_end() callbacks

3) Patch 3: (from Fengnan) optimizes dio simple path for callers who will use
->iomap_next() who do not need an iomap_end() callback

4) Patches 4 to 17: converts each filesystem to ->iomap_next() model

5) Patch 18: Removes the legacy ->iomap_begin()/->iomap_end() path

6) Patch 19: At this point, struct iomap_ops only has one item in it, the
->iomap_next() callback. Gets rid of struct iomap_ops and passes
iomap_iter_next_fn directly.

7) Patch 20: Updates the iomap documentation to match.

This series is submitted against the vfs tree on top of the vfs-7.3.iomap
branch (head commit e1b77fb85836 "Merge patch series "iomap: add simple dio
path..."). The changes can also be found in this github link [2]. This series
was run through an ai review system for sanity-checking.

As discussed in v2 about the merge logistics [3], the plan is for patches 1 to
17 to be merged in the v7.3-rc1 cycle and for patches 18 to 20 to be merged
after v7.3-rc1 has been tagged, in order to minimize disruption to filesystems
that are currently landing iomap conversions during the 7.3 merge window.
Patches 18 to 20 are included in this series for upstream review, so that
hopefully they are all approved and ready to go when the 7.3 merge window
closes.

Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/20200728173216.7184-1-willy@infradead.org/T/#u
[2] https://github.com/joannekoong/linux/tree/iomap_iter_next_v3
[3] https://lore.kernel.org/linux-fsdevel/20260703-nachrangig-gegeben-befestigen-8219a53648c7@brauner/ 

Changelog
---------
v2: https://lore.kernel.org/linux-fsdevel/20260701000949.1666714-1-joannelkoong@gmail.com/
v2 -> v3:
* Rename iomap_next_fn to iomap_iter_next_fn and iomap_process() to
  iomap_iter_next() (Darrick)
* Add integration with dio simple path, add Fengnan's patch
* Reconstruct patch that adds iomap_iter_next() logic as a refactoring of
  existing code (Christoph)
* Add DEFINE_IOMAP_ITER_NEXT{_END} macro, which nicely simplifies things
  (Christoph)
* Make documentation wording changes and a rename from dops -> next (Darrick)
* Update tracepoint in patch 19 to reflect taking iomap_iter_next_fn instead
  of ops. iomap tracepoints are explicitly called out as non-stable ABI so I
  kept Christoph's reviewed-by for this, but if that should be revoked, please
  let me know
* Add reviewed-bys

v1: https://lore.kernel.org/linux-fsdevel/20260625024723.1611000-1-joannelkoong@gmail.com/
v1 -> v2:
* Implement conversion for all callers

Fengnan Chang (1):
  iomap: skip the finishing iomap_iter() on the simple DIO read fast
    path

Joanne Koong (19):
  iomap: split iomap_iter() logic into iomap_iter_next()
  iomap: add ->iomap_next()
  xfs: convert iomap ops to ->iomap_next()
  btrfs: convert iomap ops to ->iomap_next()
  ntfs3: convert iomap ops to ->iomap_next()
  ntfs: convert iomap ops to ->iomap_next()
  ext4: convert iomap ops to ->iomap_next()
  erofs: convert iomap ops to ->iomap_next()
  zonefs: convert iomap ops to ->iomap_next()
  ext2: convert iomap ops to ->iomap_next()
  block: convert iomap ops to ->iomap_next()
  f2fs: convert iomap ops to ->iomap_next()
  gfs2: convert iomap ops to ->iomap_next()
  hpfs: convert iomap ops to ->iomap_next()
  fuse: convert iomap ops to ->iomap_next()
  exfat: convert iomap ops to ->iomap_next()
  iomap: remove ->iomap_begin()/->iomap_end() legacy path
  iomap: pass iomap_iter_next_fn directly instead of struct iomap_ops
  Documentation: iomap: update docs to reflect iomap_iter_next model

 Documentation/filesystems/iomap/design.rst    | 140 +++++++++++----
 .../filesystems/iomap/operations.rst          |  68 ++++----
 Documentation/filesystems/iomap/porting.rst   |  16 +-
 block/fops.c                                  |  10 +-
 fs/btrfs/direct-io.c                          |  10 +-
 fs/dax.c                                      |  48 +++---
 fs/erofs/data.c                               |  30 ++--
 fs/erofs/internal.h                           |   3 +-
 fs/erofs/zmap.c                               |   4 +-
 fs/exfat/file.c                               |  20 +--
 fs/exfat/inode.c                              |   6 +-
 fs/exfat/iomap.c                              |  12 +-
 fs/exfat/iomap.h                              |   6 +-
 fs/ext2/ext2.h                                |   3 +-
 fs/ext2/file.c                                |   4 +-
 fs/ext2/inode.c                               |   7 +-
 fs/ext4/ext4.h                                |   6 +-
 fs/ext4/extents.c                             |   8 +-
 fs/ext4/file.c                                |  15 +-
 fs/ext4/inode.c                               |  14 +-
 fs/f2fs/data.c                                |   4 +-
 fs/f2fs/f2fs.h                                |   3 +-
 fs/f2fs/file.c                                |   4 +-
 fs/fuse/dax.c                                 |  12 +-
 fs/fuse/file.c                                |  10 +-
 fs/fuse/virtio_fs.c                           |   3 +-
 fs/gfs2/aops.c                                |   6 +-
 fs/gfs2/bmap.c                                |   7 +-
 fs/gfs2/bmap.h                                |   3 +-
 fs/gfs2/file.c                                |   6 +-
 fs/gfs2/inode.c                               |   6 +-
 fs/hpfs/file.c                                |   6 +-
 fs/internal.h                                 |   1 -
 fs/iomap/buffered-io.c                        |  40 ++---
 fs/iomap/direct-io.c                          |  57 +++---
 fs/iomap/fiemap.c                             |   8 +-
 fs/iomap/iter.c                               | 122 ++++++-------
 fs/iomap/seek.c                               |   8 +-
 fs/iomap/swapfile.c                           |   4 +-
 fs/iomap/trace.h                              |  15 +-
 fs/ntfs/aops.c                                |   6 +-
 fs/ntfs/file.c                                |  26 +--
 fs/ntfs/inode.c                               |   2 +-
 fs/ntfs/iomap.c                               |  34 ++--
 fs/ntfs/iomap.h                               |  15 +-
 fs/ntfs3/file.c                               |  16 +-
 fs/ntfs3/inode.c                              |  11 +-
 fs/ntfs3/ntfs_fs.h                            |   3 +-
 fs/remap_range.c                              |   6 +-
 fs/xfs/xfs_aops.c                             |   8 +-
 fs/xfs/xfs_file.c                             |  48 +++---
 fs/xfs/xfs_iomap.c                            |  48 ++----
 fs/xfs/xfs_iomap.h                            |  24 ++-
 fs/xfs/xfs_iops.c                             |   4 +-
 fs/xfs/xfs_reflink.c                          |   6 +-
 fs/zonefs/file.c                              |  23 ++-
 include/linux/dax.h                           |  18 +-
 include/linux/fs.h                            |   7 +-
 include/linux/iomap.h                         | 163 +++++++++++++-----
 59 files changed, 680 insertions(+), 543 deletions(-)

-- 
2.52.0

[PATCH v3 01/20] iomap: split iomap_iter() logic into iomap_iter_next()

[Joanne Koong]

In preparation for changing iomap to use an in-iter (->iomap_next())
model, move the iomap_iter() logic out into the new iomap_iter_next()
helper function.

iomap_iter_next() is added as an inlined helper so it can be called
directly by ->iomap_next() implementations where the begin()/end()
callbacks can be direct calls.

The DEFINE_IOMAP_ITER_NEXT() and DEFINE_IOMAP_ITER_NEXT_END() macros are
also provided to generate the boilerplate ->iomap_next() wrapper
functions that simply forward to iomap_iter_next() with the appropriate
begin/end callbacks. DEFINE_IOMAP_ITER_NEXT() is for the common case
where there is no end() callback. DEFINE_IOMAP_ITER_NEXT_END() is for
the case where there is an explicit end() callback.

No functional change intended. The only code-level difference is that on
the iomap_end() error path (ret < 0 && !advanced), the old code returned
with iter.status left as the caller's last value whereas the new code
zeroes it, but this is not observable in practice as there are no in-tree
callers that read iter.status after the iteration loop.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/iomap/iter.c       | 123 +++++++++++++++++++++---------------------
 include/linux/iomap.h | 102 +++++++++++++++++++++++++++++------
 2 files changed, 147 insertions(+), 78 deletions(-)

diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index e4a29829591a..66ccb87441ab 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -6,15 +6,6 @@
 #include <linux/iomap.h>
 #include "trace.h"
 
-static inline void iomap_iter_clean_fbatch(struct iomap_iter *iter)
-{
-	if (iter->iomap.flags & IOMAP_F_FOLIO_BATCH) {
-		folio_batch_release(iter->fbatch);
-		folio_batch_reinit(iter->fbatch);
-		iter->iomap.flags &= ~IOMAP_F_FOLIO_BATCH;
-	}
-}
-
 /* Advance the current iterator position and decrement the remaining length */
 int iomap_iter_advance(struct iomap_iter *iter, u64 count)
 {
@@ -40,51 +31,28 @@ static inline void iomap_iter_done(struct iomap_iter *iter)
 }
 
 /**
- * iomap_iter - iterate over a ranges in a file
- * @iter: iteration structue
- * @ops: iomap ops provided by the file system
+ * iomap_iter_continue - decide whether iteration should continue
+ * @iter: iteration structure
+ * @iomap: the mapping that was just processed
+ * @srcmap: the source mapping that was just processed
  *
- * Iterate over filesystem-provided space mappings for the provided file range.
+ * Helper normally called via iomap_iter_next(). Called after the previous
+ * mapping has been finished to determine whether there is more of the file
+ * range left to process.
  *
- * This function handles cleanup of resources acquired for iteration when the
- * filesystem indicates there are no more space mappings, which means that this
- * function must be called in a loop that continues as long it returns a
- * positive value.  If 0 or a negative value is returned, the caller must not
- * return to the loop body.  Within a loop body, there are two ways to break out
- * of the loop body:  leave @iter.status unchanged, or set it to a negative
- * errno.
+ * Returns 1 if there is more work to do, in which case @iomap and @srcmap are
+ * cleared so the caller can produce the next mapping; zero if the range is
+ * fully consumed; or a negative errno on error. Any folio batch attached to
+ * the mapping is released before returning.
  */
-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
+int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap, int ret)
 {
-	bool stale = iter->iomap.flags & IOMAP_F_STALE;
-	ssize_t advanced;
-	u64 olen;
-	int ret;
-
-	trace_iomap_iter(iter, ops, _RET_IP_);
-
-	if (!iter->iomap.length)
-		goto begin;
-
-	/*
-	 * Calculate how far the iter was advanced and the original length bytes
-	 * for ->iomap_end().
-	 */
-	advanced = iter->pos - iter->iter_start_pos;
-	olen = iter->len + advanced;
-
-	if (ops->iomap_end) {
-		ret = ops->iomap_end(iter->inode, iter->iter_start_pos,
-				iomap_length_trim(iter, iter->iter_start_pos,
-						  olen),
-				advanced, iter->flags, &iter->iomap);
-		if (ret < 0 && !advanced)
-			return ret;
-	}
+	const bool stale = iomap->flags & IOMAP_F_STALE;
+	const ssize_t advanced = iter->pos - iter->iter_start_pos;
 
-	/* detect old return semantics where this would advance */
-	if (WARN_ON_ONCE(iter->status > 0))
-		iter->status = -EIO;
+	if (ret < 0 && !advanced)
+		return ret;
 
 	/*
 	 * Use iter->len to determine whether to continue onto the next mapping.
@@ -92,25 +60,58 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
 	 * advanced at all (i.e. no work was done for some reason) unless the
 	 * mapping has been marked stale and needs to be reprocessed.
 	 */
-	if (iter->status < 0)
+	if (WARN_ON_ONCE(iter->status > 0))
+		/* detect old return semantics where this would advance */
+		ret = -EIO;
+	else if (iter->status < 0)
 		ret = iter->status;
 	else if (iter->len == 0 || (!advanced && !stale))
 		ret = 0;
 	else
 		ret = 1;
-	iomap_iter_clean_fbatch(iter);
-	iter->status = 0;
+
+	if (iomap->flags & IOMAP_F_FOLIO_BATCH) {
+		folio_batch_release(iter->fbatch);
+		folio_batch_reinit(iter->fbatch);
+		iomap->flags &= ~IOMAP_F_FOLIO_BATCH;
+	}
+
 	if (ret <= 0)
 		return ret;
 
-	memset(&iter->iomap, 0, sizeof(iter->iomap));
-	memset(&iter->srcmap, 0, sizeof(iter->srcmap));
+	memset(iomap, 0, sizeof(*iomap));
+	memset(srcmap, 0, sizeof(*srcmap));
 
-begin:
-	ret = ops->iomap_begin(iter->inode, iter->pos, iter->len, iter->flags,
-			       &iter->iomap, &iter->srcmap);
-	if (ret < 0)
-		return ret;
-	iomap_iter_done(iter);
-	return 1;
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iomap_iter_continue);
+
+/**
+ * iomap_iter - iterate over ranges in a file
+ * @iter: iteration structure
+ * @ops: iomap ops provided by the filesystem
+ *
+ * Iterate over filesystem-provided space mappings for the provided file range.
+ *
+ * This function handles cleanup of resources acquired for iteration when the
+ * filesystem indicates there are no more space mappings, which means that this
+ * function must be called in a loop that continues as long it returns a
+ * positive value.  If 0 or a negative value is returned, the caller must not
+ * return to the loop body.  Within a loop body, there are two ways to break out
+ * of the loop body:  leave @iter.status unchanged, or set it to a negative
+ * errno.
+ */
+int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
+{
+	int ret;
+
+	trace_iomap_iter(iter, ops, _RET_IP_);
+
+	ret = iomap_iter_next(iter, &iter->iomap, &iter->srcmap,
+			ops->iomap_begin, ops->iomap_end);
+	iter->status = 0;
+	if (ret > 0)
+		iomap_iter_done(iter);
+
+	return ret;
 }
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 3582ed1fe236..3df851d0f76b 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -212,24 +212,27 @@ struct iomap_write_ops {
 #define IOMAP_ATOMIC		(1 << 9) /* torn-write protection */
 #define IOMAP_DONTCACHE		(1 << 10)
 
-struct iomap_ops {
-	/*
-	 * Return the existing mapping at pos, or reserve space starting at
-	 * pos for up to length, as long as we can do it as a single mapping.
-	 * The actual length is returned in iomap->length.
-	 */
-	int (*iomap_begin)(struct inode *inode, loff_t pos, loff_t length,
-			unsigned flags, struct iomap *iomap,
-			struct iomap *srcmap);
+/*
+ * Return the existing mapping at pos, or reserve space starting at pos for up
+ * to length, as long as we can do it as a single mapping.
+ * The actual length is returned in iomap->length.
+ */
+typedef int (*iomap_iter_begin_fn)(struct inode *inode, loff_t pos,
+		loff_t length, unsigned flags, struct iomap *iomap,
+		struct iomap *srcmap);
 
-	/*
-	 * Commit and/or unreserve space previous allocated using iomap_begin.
-	 * Written indicates the length of the successful write operation which
-	 * needs to be commited, while the rest needs to be unreserved.
-	 * Written might be zero if no data was written.
-	 */
-	int (*iomap_end)(struct inode *inode, loff_t pos, loff_t length,
-			ssize_t written, unsigned flags, struct iomap *iomap);
+/*
+ * Commit and/or unreserve space previously allocated by iomap_iter_begin_fn.
+ * Written indicates the length of the successful write operation which needs
+ * to be committed, while the rest needs to be unreserved.
+ * Written might be zero if no data was written.
+ */
+typedef int (*iomap_iter_end_fn)(struct inode *inode, loff_t pos, loff_t length,
+		ssize_t written, unsigned flags, struct iomap *iomap);
+
+struct iomap_ops {
+	iomap_iter_begin_fn iomap_begin;
+	iomap_iter_end_fn iomap_end;
 };
 
 /**
@@ -317,6 +320,71 @@ static inline const struct iomap *iomap_iter_srcmap(const struct iomap_iter *i)
 	return &i->iomap;
 }
 
+int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap, int ret);
+
+/**
+ * iomap_iter_next - finish the previous mapping and produce the next one
+ * @iter: iteration structure
+ * @iomap: mapping to finish and then repopulate
+ * @srcmap: source mapping to finish and then repopulate
+ * @begin: callback that produces a mapping for the current position
+ * @end: optional callback that finishes the previous mapping, or NULL
+ *
+ * Inline helper that implements the common body of an ->iomap_next()
+ * callback: it finishes the previous mapping via @end (if present), decides
+ * via iomap_iter_continue() whether to keep going, and obtains the next
+ * mapping via @begin.
+ *
+ * This helper is marked __always_inline so that when a caller passes
+ * compile-time-constant @begin and @end callbacks, the compiler can call them
+ * directly, avoiding the indirect-call overhead.
+ *
+ * Returns 1 to continue iterating, 0 once the range is fully consumed, or a
+ * negative errno on error.
+ */
+static __always_inline int iomap_iter_next(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap,
+		iomap_iter_begin_fn begin, iomap_iter_end_fn end)
+{
+	int ret = 0;
+
+	if (iomap->length) {
+		if (end) {
+			/*
+			 * Calculate how far the iter was advanced and the
+			 * original length bytes for end().
+			 */
+			ssize_t advanced = iter->pos - iter->iter_start_pos;
+			loff_t len;
+
+			len = iomap_length_trim(iter, iter->iter_start_pos,
+					iter->len + advanced);
+
+			ret = end(iter->inode, iter->iter_start_pos, len,
+					advanced, iter->flags, iomap);
+		}
+		ret = iomap_iter_continue(iter, iomap, srcmap, ret);
+		if (ret <= 0)
+			return ret;
+	}
+
+	ret = begin(iter->inode, iter->pos, iter->len, iter->flags, iomap,
+			srcmap);
+
+	return ret < 0 ? ret : 1;
+}
+
+#define DEFINE_IOMAP_ITER_NEXT_END(name, begin_fn, end_fn)		\
+int name(const struct iomap_iter *iter, struct iomap *iomap,		\
+		struct iomap *srcmap)					\
+{									\
+	return iomap_iter_next(iter, iomap, srcmap, begin_fn, end_fn);	\
+}
+
+#define DEFINE_IOMAP_ITER_NEXT(name, begin_fn)				\
+	DEFINE_IOMAP_ITER_NEXT_END(name, begin_fn, NULL)
+
 /*
  * Return the file offset for the first unchanged block after a short write.
  *
-- 
2.52.0

[PATCH v3 02/20] iomap: add ->iomap_next()

[Joanne Koong]

Have one ->iomap_next() callback instead of ->iomap_begin() and
->iomap_end(). ->iomap_next() finishes the previous mapping if needed,
and produces the next mapping.

Collapsing to a single callback lets a performance-critical caller
inline its iteration loop and pass its ->iomap_next() function as a
compile-time constant, so the compiler can devirtualize that callback
into a direct call instead of an indirect call through a function
pointer.

iomap_iter() uses ->iomap_next() when the filesystem provides that
callback and otherwise falls back to the ->iomap_begin()/->iomap_end()
path, so filesystems can be converted one at a time.

Suggested-by: Christoph Hellwig <hch@lst.de>
Suggested-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/iomap/direct-io.c  | 49 ++++++++++++++++++++++++++++++++-----------
 fs/iomap/iter.c       |  8 +++++--
 include/linux/iomap.h |  9 ++++++++
 3 files changed, 52 insertions(+), 14 deletions(-)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index ca790239e5eb..a89c7c697c63 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -1000,6 +1000,21 @@ iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
 	return true;
 }
 
+static inline void
+iomap_dio_simple_finish(struct iomap_iter *iomi, const struct iomap_ops *ops,
+		size_t len, size_t written)
+{
+	if (ops->iomap_next) {
+		iomi->iter_start_pos = iomi->pos;
+		iomi->pos += written;
+		iomi->len -= written;
+		ops->iomap_next(iomi, &iomi->iomap, &iomi->srcmap);
+	} else if (ops->iomap_end) {
+		ops->iomap_end(iomi->inode, iomi->pos, len, written,
+			       iomi->flags, &iomi->iomap);
+	}
+}
+
 /*
  * Fast path for small, block-aligned direct I/Os that map to a single
  * contiguous on-disk extent.
@@ -1062,11 +1077,25 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 
 	inode_dio_begin(inode);
 
-	ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
-			       &iomi.iomap, &iomi.srcmap);
-	if (ret) {
-		inode_dio_end(inode);
-		return ret;
+	if (ops->iomap_next) {
+		ret = ops->iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
+		if (ret <= 0) {
+			inode_dio_end(inode);
+			/*
+			 * the first iteration must yield a mapping (>0) or an
+			 * error
+			 */
+			if (WARN_ON_ONCE(!ret))
+				return -EFAULT;
+			return ret;
+		}
+	} else {
+		ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
+				       &iomi.iomap, &iomi.srcmap);
+		if (ret) {
+			inode_dio_end(inode);
+			return ret;
+		}
 	}
 
 	if (iomi.iomap.type != IOMAP_MAPPED ||
@@ -1125,14 +1154,12 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 		WRITE_ONCE(iocb->private, bio);
 	}
 
-	if (ops->iomap_end)
-		ops->iomap_end(inode, iomi.pos, count, count, iomi.flags,
-			       &iomi.iomap);
+	iomap_dio_simple_finish(&iomi, ops, count, count);
 
 	if (!wait_for_completion) {
 		bio->bi_end_io = iomap_dio_simple_end_io;
 		submit_bio(bio);
-		trace_iomap_dio_rw_queued(inode, iomi.pos, count);
+		trace_iomap_dio_rw_queued(inode, iocb->ki_pos, count);
 		return -EIOCBQUEUED;
 	}
 
@@ -1144,9 +1171,7 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 out_bio_put:
 	bio_put(bio);
 out_iomap_end:
-	if (ops->iomap_end)
-		ops->iomap_end(inode, iomi.pos, count, 0, iomi.flags,
-			       &iomi.iomap);
+	iomap_dio_simple_finish(&iomi, ops, count, 0);
 	inode_dio_end(inode);
 	return ret;
 }
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index 66ccb87441ab..e05574602946 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -107,8 +107,12 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
 
 	trace_iomap_iter(iter, ops, _RET_IP_);
 
-	ret = iomap_iter_next(iter, &iter->iomap, &iter->srcmap,
-			ops->iomap_begin, ops->iomap_end);
+	if (ops->iomap_next)
+		ret = ops->iomap_next(iter, &iter->iomap, &iter->srcmap);
+	else
+		ret = iomap_iter_next(iter, &iter->iomap, &iter->srcmap,
+				ops->iomap_begin, ops->iomap_end);
+
 	iter->status = 0;
 	if (ret > 0)
 		iomap_iter_done(iter);
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 3df851d0f76b..74d79b963533 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -230,9 +230,18 @@ typedef int (*iomap_iter_begin_fn)(struct inode *inode, loff_t pos,
 typedef int (*iomap_iter_end_fn)(struct inode *inode, loff_t pos, loff_t length,
 		ssize_t written, unsigned flags, struct iomap *iomap);
 
+/*
+ * Produce the next mapping (finishing the previous one if needed).
+ * Return 1 to continue iterating, 0 if the range is fully consumed, or a
+ * negative error on failure.
+ */
+typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap);
+
 struct iomap_ops {
 	iomap_iter_begin_fn iomap_begin;
 	iomap_iter_end_fn iomap_end;
+	iomap_iter_next_fn iomap_next;
 };
 
 /**
-- 
2.52.0

[PATCH v3 03/20] iomap: skip the finishing iomap_iter() on the simple DIO read fast path

[Joanne Koong]

From: Fengnan Chang <changfengnan@bytedance.com>

With the ->iomap_next() model, iomap_dio_simple() will run a second,
full iomap_iter() just to finish the range.  For DIO reads on
filesystems with no ->iomap_end() work (ext4, xfs, exfat, ntfs) this is
a no-op, but still costs an extra indirect ->iomap_next() call and the
iomap_iter() frames on every I/O.

Add IOMAP_DIO_NO_IOMAP_END so such callers can promise there is no
->iomap_end() work and no folio batch, letting iomap_dio_simple() skip
the finishing iomap_iter() on both the success and error-exit paths. Opt
in the ext4, xfs, exfat and ntfs direct read paths; filesystems with
real read-path ->iomap_end() work keep the existing behaviour.

Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/exfat/file.c       |  4 ++--
 fs/ext4/file.c        |  3 ++-
 fs/iomap/direct-io.c  | 16 +++++++++++++---
 fs/iomap/trace.h      |  3 ++-
 fs/ntfs/file.c        |  4 ++--
 fs/xfs/xfs_file.c     |  2 +-
 include/linux/iomap.h |  8 ++++++++
 7 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index 5fc13378d35f..3a278c95239e 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -809,8 +809,8 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
 	if (iocb->ki_flags & IOCB_DIRECT) {
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, iter, &exfat_iomap_ops, NULL, 0,
-				NULL, 0);
+		ret = iomap_dio_rw(iocb, iter, &exfat_iomap_ops, NULL,
+				IOMAP_DIO_NO_IOMAP_END, NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, iter);
 	}
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index eb1a323962b1..7bd5de67ce78 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,8 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		return generic_file_read_iter(iocb, to);
 	}
 
-	ret = iomap_dio_rw(iocb, to, &ext4_iomap_ops, NULL, 0, NULL, 0);
+	ret = iomap_dio_rw(iocb, to, &ext4_iomap_ops, NULL,
+			   IOMAP_DIO_NO_IOMAP_END, NULL, 0);
 	inode_unlock_shared(inode);
 
 	file_accessed(iocb->ki_filp);
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index a89c7c697c63..fcdae3ef6dd0 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -1040,7 +1040,9 @@ iomap_dio_simple_finish(struct iomap_iter *iomi, const struct iomap_ops *ops,
  * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
  * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
  * I/O, or bouncing.  The range must also stay within i_size and encrypted
- * inodes must use the generic iomap direct I/O path.
+ * inodes must use the generic iomap direct I/O path. IOMAP_DIO_NO_IOMAP_END is
+ * a fast-path optimization the caller can set if there is no work that needs to
+ * be finished after a mapping.
  *
  * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
  * decides the request does not fit the fast path.  In that case we proceed to
@@ -1154,7 +1156,14 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 		WRITE_ONCE(iocb->private, bio);
 	}
 
-	iomap_dio_simple_finish(&iomi, ops, count, count);
+	/*
+	 * The finishing iomap_iter() only runs the filesystem's iomap_end()
+	 * work and releases any attached folio batch.  When the caller promised
+	 * neither applies (IOMAP_DIO_NO_IOMAP_END), skip it to avoid an extra
+	 * iomap_next() call on the hot read path.
+	 */
+	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
+		iomap_dio_simple_finish(&iomi, ops, count, count);
 
 	if (!wait_for_completion) {
 		bio->bi_end_io = iomap_dio_simple_end_io;
@@ -1171,7 +1180,8 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 out_bio_put:
 	bio_put(bio);
 out_iomap_end:
-	iomap_dio_simple_finish(&iomi, ops, count, 0);
+	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
+		iomap_dio_simple_finish(&iomi, ops, count, 0);
 	inode_dio_end(inode);
 	return ret;
 }
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47..e43ca419b13a 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -127,7 +127,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
 	{IOMAP_DIO_FORCE_WAIT,		"DIO_FORCE_WAIT" }, \
 	{IOMAP_DIO_OVERWRITE_ONLY,	"DIO_OVERWRITE_ONLY" }, \
 	{IOMAP_DIO_PARTIAL,		"DIO_PARTIAL" }, \
-	{IOMAP_DIO_FSBLOCK_ALIGNED,	"DIO_FSBLOCK_ALIGNED" }
+	{IOMAP_DIO_FSBLOCK_ALIGNED,	"DIO_FSBLOCK_ALIGNED" }, \
+	{IOMAP_DIO_NO_IOMAP_END,	"DIO_NO_IOMAP_END" }
 
 DECLARE_EVENT_CLASS(iomap_class,
 	TP_PROTO(struct inode *inode, struct iomap *iomap),
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index 6a7b638e523d..2e72069c593a 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -458,8 +458,8 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		}
 
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, to, &ntfs_read_iomap_ops, NULL, 0,
-				NULL, 0);
+		ret = iomap_dio_rw(iocb, to, &ntfs_read_iomap_ops, NULL,
+				IOMAP_DIO_NO_IOMAP_END, NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, to);
 	}
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 845a97c9b063..8b8de8064e31 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -251,7 +251,7 @@ xfs_file_dio_read(
 	struct iov_iter		*to)
 {
 	struct xfs_inode	*ip = XFS_I(file_inode(iocb->ki_filp));
-	unsigned int		dio_flags = 0;
+	unsigned int		dio_flags = IOMAP_DIO_NO_IOMAP_END;
 	const struct iomap_dio_ops *dio_ops = NULL;
 	ssize_t			ret;
 
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 74d79b963533..8662bd528e9e 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -674,6 +674,14 @@ struct iomap_dio_ops {
  */
 #define IOMAP_DIO_BOUNCE		(1 << 4)
 
+/*
+ * The caller's ->iomap_next() has no ->iomap_end() work to run and this I/O
+ * does not use a folio batch, so the iomap_dio_simple() fast path may skip the
+ * finishing iomap_iter() call entirely.  Only meaningful on the simple read
+ * fast path; ignored by the generic __iomap_dio_rw() slow path.
+ */
+#define IOMAP_DIO_NO_IOMAP_END		(1 << 5)
+
 ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags, void *private, size_t done_before);
-- 
2.52.0

[PATCH v3 04/20] xfs: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert xfs iomap_ops to the new ->iomap_next() callback. Each callback is
generated with the DEFINE_IOMAP_ITER_NEXT()/DEFINE_IOMAP_ITER_NEXT_END()
macros, which wrap the iomap_iter_next() helper to finish the previous
mapping if needed and produce the next one. No functional changes are
intended.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/xfs/xfs_file.c  |  4 ++--
 fs/xfs/xfs_iomap.c | 39 +++++++++++++++++++++++++++++----------
 2 files changed, 31 insertions(+), 12 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 8b8de8064e31..3f7cbe117fb6 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -857,9 +857,9 @@ xfs_file_dio_write_atomic(
 			NULL, 0);
 
 	/*
-	 * The retry mechanism is based on the ->iomap_begin method returning
+	 * The retry mechanism is based on the ->iomap_next method returning
 	 * -ENOPROTOOPT, which would be when the REQ_ATOMIC-based write is not
-	 * possible. The REQ_ATOMIC-based method typically not be possible if
+	 * possible. The REQ_ATOMIC-based method is typically not possible if
 	 * the write spans multiple extents or the disk blocks are misaligned.
 	 */
 	if (ret == -ENOPROTOOPT && dops == &xfs_direct_write_iomap_ops) {
diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
index 225c3de88d03..d3e28202db05 100644
--- a/fs/xfs/xfs_iomap.c
+++ b/fs/xfs/xfs_iomap.c
@@ -1037,8 +1037,11 @@ xfs_direct_write_iomap_begin(
 	return error;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(xfs_direct_write_iomap_next,
+		xfs_direct_write_iomap_begin);
+
 const struct iomap_ops xfs_direct_write_iomap_ops = {
-	.iomap_begin		= xfs_direct_write_iomap_begin,
+	.iomap_next		= xfs_direct_write_iomap_next,
 };
 
 #ifdef CONFIG_XFS_RT
@@ -1089,8 +1092,11 @@ xfs_zoned_direct_write_iomap_begin(
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(xfs_zoned_direct_write_iomap_next,
+		xfs_zoned_direct_write_iomap_begin);
+
 const struct iomap_ops xfs_zoned_direct_write_iomap_ops = {
-	.iomap_begin		= xfs_zoned_direct_write_iomap_begin,
+	.iomap_next		= xfs_zoned_direct_write_iomap_next,
 };
 #endif /* CONFIG_XFS_RT */
 
@@ -1274,8 +1280,11 @@ xfs_atomic_write_cow_iomap_begin(
 	return error;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(xfs_atomic_write_cow_iomap_next,
+		xfs_atomic_write_cow_iomap_begin);
+
 const struct iomap_ops xfs_atomic_write_cow_iomap_ops = {
-	.iomap_begin		= xfs_atomic_write_cow_iomap_begin,
+	.iomap_next		= xfs_atomic_write_cow_iomap_next,
 };
 
 static int
@@ -1298,9 +1307,11 @@ xfs_dax_write_iomap_end(
 	return xfs_reflink_end_cow(ip, pos, written);
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(xfs_dax_write_iomap_next,
+		xfs_direct_write_iomap_begin, xfs_dax_write_iomap_end);
+
 const struct iomap_ops xfs_dax_write_iomap_ops = {
-	.iomap_begin	= xfs_direct_write_iomap_begin,
-	.iomap_end	= xfs_dax_write_iomap_end,
+	.iomap_next	= xfs_dax_write_iomap_next,
 };
 
 /*
@@ -2168,9 +2179,11 @@ xfs_buffered_write_iomap_end(
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(xfs_buffered_write_iomap_next,
+		xfs_buffered_write_iomap_begin, xfs_buffered_write_iomap_end);
+
 const struct iomap_ops xfs_buffered_write_iomap_ops = {
-	.iomap_begin		= xfs_buffered_write_iomap_begin,
-	.iomap_end		= xfs_buffered_write_iomap_end,
+	.iomap_next		= xfs_buffered_write_iomap_next,
 };
 
 static int
@@ -2214,8 +2227,10 @@ xfs_read_iomap_begin(
 				 shared ? IOMAP_F_SHARED : 0, seq);
 }
 
+static DEFINE_IOMAP_ITER_NEXT(xfs_read_iomap_next, xfs_read_iomap_begin);
+
 const struct iomap_ops xfs_read_iomap_ops = {
-	.iomap_begin		= xfs_read_iomap_begin,
+	.iomap_next		= xfs_read_iomap_next,
 };
 
 static int
@@ -2302,8 +2317,10 @@ xfs_seek_iomap_begin(
 	return error;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(xfs_seek_iomap_next, xfs_seek_iomap_begin);
+
 const struct iomap_ops xfs_seek_iomap_ops = {
-	.iomap_begin		= xfs_seek_iomap_begin,
+	.iomap_next		= xfs_seek_iomap_next,
 };
 
 static int
@@ -2349,8 +2366,10 @@ xfs_xattr_iomap_begin(
 	return xfs_bmbt_to_iomap(ip, iomap, &imap, flags, IOMAP_F_XATTR, seq);
 }
 
+static DEFINE_IOMAP_ITER_NEXT(xfs_xattr_iomap_next, xfs_xattr_iomap_begin);
+
 const struct iomap_ops xfs_xattr_iomap_ops = {
-	.iomap_begin		= xfs_xattr_iomap_begin,
+	.iomap_next		= xfs_xattr_iomap_next,
 };
 
 int
-- 
2.52.0

[PATCH v3 05/20] btrfs: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert btrfs iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT_END() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/btrfs/direct-io.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/direct-io.c b/fs/btrfs/direct-io.c
index 460326d34143..d5439b06cdc9 100644
--- a/fs/btrfs/direct-io.c
+++ b/fs/btrfs/direct-io.c
@@ -798,9 +798,11 @@ static void btrfs_dio_submit_io(const struct iomap_iter *iter, struct bio *bio,
 	btrfs_submit_bbio(bbio, 0);
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(btrfs_dio_iomap_next, btrfs_dio_iomap_begin,
+				  btrfs_dio_iomap_end);
+
 static const struct iomap_ops btrfs_dio_iomap_ops = {
-	.iomap_begin            = btrfs_dio_iomap_begin,
-	.iomap_end              = btrfs_dio_iomap_end,
+	.iomap_next             = btrfs_dio_iomap_next,
 };
 
 static const struct iomap_dio_ops btrfs_dio_ops = {
-- 
2.52.0

[PATCH v3 06/20] ntfs3: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert ntfs3 iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT_END() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/ntfs3/inode.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index c43101cc064d..53031e71c8fc 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -2101,9 +2101,11 @@ const struct address_space_operations ntfs_aops_cmpr = {
 	.invalidate_folio = iomap_invalidate_folio,
 };
 
+static DEFINE_IOMAP_ITER_NEXT_END(ntfs_iomap_next, ntfs_iomap_begin,
+				  ntfs_iomap_end);
+
 const struct iomap_ops ntfs_iomap_ops = {
-	.iomap_begin	= ntfs_iomap_begin,
-	.iomap_end	= ntfs_iomap_end,
+	.iomap_next	= ntfs_iomap_next,
 };
 
 const struct iomap_write_ops ntfs_iomap_folio_ops = {
-- 
2.52.0

[PATCH v3 07/20] ntfs: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert ntfs iomap_ops to the new ->iomap_next() callback. Each callback is
generated with the DEFINE_IOMAP_ITER_NEXT()/DEFINE_IOMAP_ITER_NEXT_END()
macros, which wrap the iomap_iter_next() helper to finish the previous
mapping if needed and produce the next one. No functional changes are
intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/ntfs/iomap.c | 32 ++++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/fs/ntfs/iomap.c b/fs/ntfs/iomap.c
index 52eecf5cb256..d0964ac840d9 100644
--- a/fs/ntfs/iomap.c
+++ b/fs/ntfs/iomap.c
@@ -277,8 +277,10 @@ static int ntfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t leng
 			srcmap, true);
 }
 
+static DEFINE_IOMAP_ITER_NEXT(ntfs_read_iomap_next, ntfs_read_iomap_begin);
+
 const struct iomap_ops ntfs_read_iomap_ops = {
-	.iomap_begin = ntfs_read_iomap_begin,
+	.iomap_next = ntfs_read_iomap_next,
 };
 
 /*
@@ -329,13 +331,17 @@ static int ntfs_zero_read_iomap_end(struct inode *inode, loff_t pos, loff_t leng
 	return written;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(ntfs_zero_read_iomap_next,
+		ntfs_seek_iomap_begin, ntfs_zero_read_iomap_end);
+
 static const struct iomap_ops ntfs_zero_read_iomap_ops = {
-	.iomap_begin = ntfs_seek_iomap_begin,
-	.iomap_end = ntfs_zero_read_iomap_end,
+	.iomap_next = ntfs_zero_read_iomap_next,
 };
 
+static DEFINE_IOMAP_ITER_NEXT(ntfs_seek_iomap_next, ntfs_seek_iomap_begin);
+
 const struct iomap_ops ntfs_seek_iomap_ops = {
-	.iomap_begin = ntfs_seek_iomap_begin,
+	.iomap_next = ntfs_seek_iomap_next,
 };
 
 int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length)
@@ -764,9 +770,11 @@ static int ntfs_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return written;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(ntfs_write_iomap_next,
+		ntfs_write_iomap_begin, ntfs_write_iomap_end);
+
 const struct iomap_ops ntfs_write_iomap_ops = {
-	.iomap_begin		= ntfs_write_iomap_begin,
-	.iomap_end		= ntfs_write_iomap_end,
+	.iomap_next		= ntfs_write_iomap_next,
 };
 
 static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
@@ -777,9 +785,11 @@ static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
 			NTFS_IOMAP_FLAGS_MKWRITE);
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(ntfs_page_mkwrite_iomap_next,
+		ntfs_page_mkwrite_iomap_begin, ntfs_write_iomap_end);
+
 const struct iomap_ops ntfs_page_mkwrite_iomap_ops = {
-	.iomap_begin		= ntfs_page_mkwrite_iomap_begin,
-	.iomap_end		= ntfs_write_iomap_end,
+	.iomap_next		= ntfs_page_mkwrite_iomap_next,
 };
 
 static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
@@ -790,9 +800,11 @@ static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
 			NTFS_IOMAP_FLAGS_DIO);
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(ntfs_dio_iomap_next,
+		ntfs_dio_iomap_begin, ntfs_write_iomap_end);
+
 const struct iomap_ops ntfs_dio_iomap_ops = {
-	.iomap_begin		= ntfs_dio_iomap_begin,
-	.iomap_end		= ntfs_write_iomap_end,
+	.iomap_next		= ntfs_dio_iomap_next,
 };
 
 static ssize_t ntfs_writeback_range(struct iomap_writepage_ctx *wpc,
-- 
2.52.0

[PATCH v3 08/20] ext4: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert ext4 iomap_ops to the new ->iomap_next() callback. Each callback is
generated with the DEFINE_IOMAP_ITER_NEXT() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/ext4/extents.c | 4 +++-
 fs/ext4/inode.c   | 8 ++++++--
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 91c97af64b31..15972410d460 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -5171,8 +5171,10 @@ static int ext4_iomap_xattr_begin(struct inode *inode, loff_t offset,
 	return error;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_xattr_next, ext4_iomap_xattr_begin);
+
 static const struct iomap_ops ext4_iomap_xattr_ops = {
-	.iomap_begin		= ext4_iomap_xattr_begin,
+	.iomap_next		= ext4_iomap_xattr_next,
 };
 
 static int ext4_fiemap_check_ranges(struct inode *inode, u64 start, u64 *len)
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index ce99807c5f5b..278a032b300c 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3850,8 +3850,10 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next, ext4_iomap_begin);
+
 const struct iomap_ops ext4_iomap_ops = {
-	.iomap_begin		= ext4_iomap_begin,
+	.iomap_next		= ext4_iomap_next,
 };
 
 static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
@@ -3905,8 +3907,10 @@ static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next_report, ext4_iomap_begin_report);
+
 const struct iomap_ops ext4_iomap_report_ops = {
-	.iomap_begin = ext4_iomap_begin_report,
+	.iomap_next = ext4_iomap_next_report,
 };
 
 /*
-- 
2.52.0

[PATCH v3 09/20] erofs: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert erofs iomap_ops to the new ->iomap_next() callback. Each
callback is generated with the DEFINE_IOMAP_ITER_NEXT() and
DEFINE_IOMAP_ITER_NEXT_END() macros, which wrap the iomap_iter_next()
helper to finish the previous mapping if needed and produce the next
one. No functional changes are intended.

Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/erofs/data.c | 6 ++++--
 fs/erofs/zmap.c | 5 ++++-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 9aa48c8d67d1..d2f01245ee79 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -380,9 +380,11 @@ static int erofs_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return written;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(erofs_iomap_next, erofs_iomap_begin,
+				  erofs_iomap_end);
+
 static const struct iomap_ops erofs_iomap_ops = {
-	.iomap_begin = erofs_iomap_begin,
-	.iomap_end = erofs_iomap_end,
+	.iomap_next = erofs_iomap_next,
 };
 
 int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c
index bab521613552..3966b62a7051 100644
--- a/fs/erofs/zmap.c
+++ b/fs/erofs/zmap.c
@@ -821,6 +821,9 @@ static int z_erofs_iomap_begin_report(struct inode *inode, loff_t offset,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(z_erofs_iomap_next_report,
+			      z_erofs_iomap_begin_report);
+
 const struct iomap_ops z_erofs_iomap_report_ops = {
-	.iomap_begin = z_erofs_iomap_begin_report,
+	.iomap_next = z_erofs_iomap_next_report,
 };
-- 
2.52.0

[PATCH v3 10/20] zonefs: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert zonefs iomap_ops to the new ->iomap_next() callback. Each
callback is generated with the DEFINE_IOMAP_ITER_NEXT() macro, which
wraps the iomap_iter_next() helper to finish the previous mapping if
needed and produce the next one. No functional changes are intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/zonefs/file.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/fs/zonefs/file.c b/fs/zonefs/file.c
index 5ada33f70bb4..5b34849be7a2 100644
--- a/fs/zonefs/file.c
+++ b/fs/zonefs/file.c
@@ -57,8 +57,10 @@ static int zonefs_read_iomap_begin(struct inode *inode, loff_t offset,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(zonefs_read_iomap_next, zonefs_read_iomap_begin);
+
 static const struct iomap_ops zonefs_read_iomap_ops = {
-	.iomap_begin	= zonefs_read_iomap_begin,
+	.iomap_next	= zonefs_read_iomap_next,
 };
 
 static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,
@@ -106,8 +108,11 @@ static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(zonefs_write_iomap_next,
+			      zonefs_write_iomap_begin);
+
 static const struct iomap_ops zonefs_write_iomap_ops = {
-	.iomap_begin	= zonefs_write_iomap_begin,
+	.iomap_next	= zonefs_write_iomap_next,
 };
 
 static int zonefs_read_folio(struct file *unused, struct folio *folio)
-- 
2.52.0

[PATCH v3 11/20] ext2: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert ext2 iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT_END() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/ext2/inode.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 29808629cce5..7e0fa9c454e1 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -860,9 +860,11 @@ ext2_iomap_end(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(ext2_iomap_next, ext2_iomap_begin,
+				  ext2_iomap_end);
+
 const struct iomap_ops ext2_iomap_ops = {
-	.iomap_begin		= ext2_iomap_begin,
-	.iomap_end		= ext2_iomap_end,
+	.iomap_next		= ext2_iomap_next,
 };
 
 int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
-- 
2.52.0

[PATCH v3 12/20] block: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert block iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Reviewed-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 block/fops.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/block/fops.c b/block/fops.c
index 15783a6180de..4cff76e9eb71 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -453,8 +453,10 @@ static int blkdev_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(blkdev_iomap_next, blkdev_iomap_begin);
+
 static const struct iomap_ops blkdev_iomap_ops = {
-	.iomap_begin		= blkdev_iomap_begin,
+	.iomap_next		= blkdev_iomap_next,
 };
 
 #ifdef CONFIG_BUFFER_HEAD
-- 
2.52.0

[PATCH v3 13/20] f2fs: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert f2fs iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/f2fs/data.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index a765fda71536..8977ad379f50 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -4653,6 +4653,8 @@ static int f2fs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(f2fs_iomap_next, f2fs_iomap_begin);
+
 const struct iomap_ops f2fs_iomap_ops = {
-	.iomap_begin	= f2fs_iomap_begin,
+	.iomap_next	= f2fs_iomap_next,
 };
-- 
2.52.0

[PATCH v3 14/20] gfs2: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert gfs2 iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT_END() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/gfs2/bmap.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index 51ac1fd44f78..73c626971163 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -1200,9 +1200,11 @@ static int gfs2_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(gfs2_iomap_next, gfs2_iomap_begin,
+				  gfs2_iomap_end);
+
 const struct iomap_ops gfs2_iomap_ops = {
-	.iomap_begin = gfs2_iomap_begin,
-	.iomap_end = gfs2_iomap_end,
+	.iomap_next = gfs2_iomap_next,
 };
 
 /**
-- 
2.52.0

[PATCH v3 15/20] hpfs: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert hpfs iomap_ops to the new ->iomap_next() callback. The callback is
generated with the DEFINE_IOMAP_ITER_NEXT() macro, which wraps the
iomap_iter_next() helper to finish the previous mapping if needed and
produce the next one. No functional changes are intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/hpfs/file.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/fs/hpfs/file.c b/fs/hpfs/file.c
index 29e876705369..6a629ab956fc 100644
--- a/fs/hpfs/file.c
+++ b/fs/hpfs/file.c
@@ -156,8 +156,10 @@ static int hpfs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(hpfs_iomap_next, hpfs_iomap_begin);
+
 static const struct iomap_ops hpfs_iomap_ops = {
-	.iomap_begin		= hpfs_iomap_begin,
+	.iomap_next		= hpfs_iomap_next,
 };
 
 static int hpfs_read_folio(struct file *file, struct folio *folio)
-- 
2.52.0

[PATCH v3 16/20] fuse: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert fuse iomap_ops to the new ->iomap_next() callback. Each callback is
generated with the DEFINE_IOMAP_ITER_NEXT()/DEFINE_IOMAP_ITER_NEXT_END()
macros, which wrap the iomap_iter_next() helper to finish the previous
mapping if needed and produce the next one. No functional changes are
intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/fuse/dax.c       | 6 ++++--
 fs/fuse/file.c      | 4 +++-
 fs/fuse/virtio_fs.c | 3 +--
 3 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c
index 8b53625ac7ab..85cdf0199bc0 100644
--- a/fs/fuse/dax.c
+++ b/fs/fuse/dax.c
@@ -653,9 +653,11 @@ static int fuse_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(fuse_iomap_next, fuse_iomap_begin,
+				  fuse_iomap_end);
+
 static const struct iomap_ops fuse_iomap_ops = {
-	.iomap_begin = fuse_iomap_begin,
-	.iomap_end = fuse_iomap_end,
+	.iomap_next = fuse_iomap_next,
 };
 
 static void fuse_wait_dax_page(struct inode *inode)
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index e052a0d44dee..403decd4bb52 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -890,8 +890,10 @@ static int fuse_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
+static DEFINE_IOMAP_ITER_NEXT(fuse_iomap_next, fuse_iomap_begin);
+
 static const struct iomap_ops fuse_iomap_ops = {
-	.iomap_begin	= fuse_iomap_begin,
+	.iomap_next	= fuse_iomap_next,
 };
 
 struct fuse_fill_read_data {
diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c
index df25d4faca41..f15e516ebcb5 100644
--- a/fs/fuse/virtio_fs.c
+++ b/fs/fuse/virtio_fs.c
@@ -1024,8 +1024,7 @@ static void virtio_fs_cleanup_vqs(struct virtio_device *vdev)
 }
 
 /* Map a window offset to a page frame number.  The window offset will have
- * been produced by .iomap_begin(), which maps a file offset to a window
- * offset.
+ * been produced by .iomap_next(), which maps a file offset to a window offset.
  */
 static long virtio_fs_direct_access(struct dax_device *dax_dev, pgoff_t pgoff,
 				    long nr_pages, enum dax_access_mode mode,
-- 
2.52.0

[PATCH v3 17/20] exfat: convert iomap ops to ->iomap_next()

[Joanne Koong]

Convert exfat iomap_ops to the new ->iomap_next() callback. Each
callback is generated with the DEFINE_IOMAP_ITER_NEXT() and
DEFINE_IOMAP_ITER_NEXT_END() macros, which wrap the iomap_iter_next()
helper to finish the previous mapping if needed and produce the next
one. No functional changes are intended.

Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/exfat/iomap.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/fs/exfat/iomap.c b/fs/exfat/iomap.c
index 1aac38e63fe6..b6dd396aa60a 100644
--- a/fs/exfat/iomap.c
+++ b/fs/exfat/iomap.c
@@ -151,8 +151,10 @@ static int exfat_write_iomap_begin(struct inode *inode, loff_t offset, loff_t le
 	return __exfat_iomap_begin(inode, offset, length, flags, iomap, true);
 }
 
+static DEFINE_IOMAP_ITER_NEXT(exfat_iomap_next, exfat_iomap_begin);
+
 const struct iomap_ops exfat_iomap_ops = {
-	.iomap_begin = exfat_iomap_begin,
+	.iomap_next = exfat_iomap_next,
 };
 
 /*
@@ -186,9 +188,11 @@ static int exfat_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return written;
 }
 
+static DEFINE_IOMAP_ITER_NEXT_END(exfat_write_iomap_next,
+		exfat_write_iomap_begin, exfat_write_iomap_end);
+
 const struct iomap_ops exfat_write_iomap_ops = {
-	.iomap_begin	= exfat_write_iomap_begin,
-	.iomap_end	= exfat_write_iomap_end,
+	.iomap_next	= exfat_write_iomap_next,
 };
 
 /*
-- 
2.52.0

[PATCH v3 18/20] iomap: remove ->iomap_begin()/->iomap_end() legacy path

[Joanne Koong]

With all in-tree filesystems implementing ->iomap_next(), drop the
->iomap_begin()/->iomap_end() path.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 fs/iomap/direct-io.c  | 47 ++++++++++++++-----------------------------
 fs/iomap/iter.c       |  6 +-----
 include/linux/iomap.h |  6 ++----
 3 files changed, 18 insertions(+), 41 deletions(-)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index fcdae3ef6dd0..a47369e6b4e1 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -1001,18 +1001,13 @@ iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
 }
 
 static inline void
-iomap_dio_simple_finish(struct iomap_iter *iomi, const struct iomap_ops *ops,
-		size_t len, size_t written)
+iomap_dio_simple_finish(struct iomap_iter *iomi, iomap_iter_next_fn next,
+		size_t written)
 {
-	if (ops->iomap_next) {
-		iomi->iter_start_pos = iomi->pos;
-		iomi->pos += written;
-		iomi->len -= written;
-		ops->iomap_next(iomi, &iomi->iomap, &iomi->srcmap);
-	} else if (ops->iomap_end) {
-		ops->iomap_end(iomi->inode, iomi->pos, len, written,
-			       iomi->flags, &iomi->iomap);
-	}
+	iomi->iter_start_pos = iomi->pos;
+	iomi->pos += written;
+	iomi->len -= written;
+	next(iomi, &iomi->iomap, &iomi->srcmap);
 }
 
 /*
@@ -1079,25 +1074,13 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 
 	inode_dio_begin(inode);
 
-	if (ops->iomap_next) {
-		ret = ops->iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
-		if (ret <= 0) {
-			inode_dio_end(inode);
-			/*
-			 * the first iteration must yield a mapping (>0) or an
-			 * error
-			 */
-			if (WARN_ON_ONCE(!ret))
-				return -EFAULT;
-			return ret;
-		}
-	} else {
-		ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
-				       &iomi.iomap, &iomi.srcmap);
-		if (ret) {
-			inode_dio_end(inode);
-			return ret;
-		}
+	ret = ops->iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
+	if (ret <= 0) {
+		inode_dio_end(inode);
+		/* the first iteration must yield a mapping (>0) or an error */
+		if (WARN_ON_ONCE(!ret))
+			return -EFAULT;
+		return ret;
 	}
 
 	if (iomi.iomap.type != IOMAP_MAPPED ||
@@ -1163,7 +1146,7 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 	 * iomap_next() call on the hot read path.
 	 */
 	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
-		iomap_dio_simple_finish(&iomi, ops, count, count);
+		iomap_dio_simple_finish(&iomi, ops->iomap_next, count);
 
 	if (!wait_for_completion) {
 		bio->bi_end_io = iomap_dio_simple_end_io;
@@ -1181,7 +1164,7 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 	bio_put(bio);
 out_iomap_end:
 	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
-		iomap_dio_simple_finish(&iomi, ops, count, 0);
+		iomap_dio_simple_finish(&iomi, ops->iomap_next, 0);
 	inode_dio_end(inode);
 	return ret;
 }
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index e05574602946..f7c4d5cf56be 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -107,11 +107,7 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
 
 	trace_iomap_iter(iter, ops, _RET_IP_);
 
-	if (ops->iomap_next)
-		ret = ops->iomap_next(iter, &iter->iomap, &iter->srcmap);
-	else
-		ret = iomap_iter_next(iter, &iter->iomap, &iter->srcmap,
-				ops->iomap_begin, ops->iomap_end);
+	ret = ops->iomap_next(iter, &iter->iomap, &iter->srcmap);
 
 	iter->status = 0;
 	if (ret > 0)
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 8662bd528e9e..66e2f7186299 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -194,7 +194,7 @@ struct iomap_write_ops {
 };
 
 /*
- * Flags for iomap_begin / iomap_end.  No flag implies a read.
+ * Flags for iomap_next.  No flag implies a read.
  */
 #define IOMAP_WRITE		(1 << 0) /* writing, must allocate blocks */
 #define IOMAP_ZERO		(1 << 1) /* zeroing operation, may skip holes */
@@ -239,8 +239,6 @@ typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,
 		struct iomap *iomap, struct iomap *srcmap);
 
 struct iomap_ops {
-	iomap_iter_begin_fn iomap_begin;
-	iomap_iter_end_fn iomap_end;
 	iomap_iter_next_fn iomap_next;
 };
 
@@ -255,7 +253,7 @@ struct iomap_ops {
  *	incremental iter advance.
  * @status: Status of the most recent iteration. Zero on success or a negative
  *	errno on error.
- * @flags: Zero or more of the iomap_begin flags above.
+ * @flags: Zero or more of the iomap_next flags above.
  * @iomap: Map describing the I/O iteration
  * @srcmap: Source map for COW operations
  */
-- 
2.52.0

[PATCH v3 19/20] iomap: pass iomap_iter_next_fn directly instead of struct iomap_ops

[Joanne Koong]

Now that all filesystems implement ->iomap_next() and the legacy
->iomap_begin()/->iomap_end() fallback is gone, struct iomap_ops only
wraps a single iomap_next function pointer. Drop the struct entirely and
pass the iomap_iter_next_fn directly to iomap_iter() and all the
iomap/dax entry points. Filesystems pass their iomap_iter_next_fn
callback directly instead of an ops struct. Several iomap_iter_next_fn
callbacks become non-static and gain header declarations since callers
now reference them directly.

The iomap_iter tracepoint's ops field is renamed to next to match. These
tracepoints are explicitly not stable ABI (see the note in iomap/trace.h).

No functional change intended other than the tracepoint field rename
noted above.

Acked-by: Jan Kara <jack@suse.cz>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 block/fops.c           | 10 +++-----
 fs/btrfs/direct-io.c   |  8 ++----
 fs/dax.c               | 48 ++++++++++++++++++------------------
 fs/erofs/data.c        | 28 +++++++++------------
 fs/erofs/internal.h    |  3 ++-
 fs/erofs/zmap.c        |  7 +-----
 fs/exfat/file.c        | 18 +++++++-------
 fs/exfat/inode.c       |  6 ++---
 fs/exfat/iomap.c       | 16 +++---------
 fs/exfat/iomap.h       |  6 +++--
 fs/ext2/ext2.h         |  3 ++-
 fs/ext2/file.c         |  4 +--
 fs/ext2/inode.c        |  9 ++-----
 fs/ext4/ext4.h         |  6 +++--
 fs/ext4/extents.c      |  8 ++----
 fs/ext4/file.c         | 14 +++++------
 fs/ext4/inode.c        | 18 ++++----------
 fs/f2fs/data.c         |  6 +----
 fs/f2fs/f2fs.h         |  3 ++-
 fs/f2fs/file.c         |  4 +--
 fs/fuse/dax.c          | 10 +++-----
 fs/fuse/file.c         | 10 +++-----
 fs/gfs2/aops.c         |  6 ++---
 fs/gfs2/bmap.c         |  9 ++-----
 fs/gfs2/bmap.h         |  3 ++-
 fs/gfs2/file.c         |  6 ++---
 fs/gfs2/inode.c        |  6 ++---
 fs/hpfs/file.c         |  6 +----
 fs/internal.h          |  1 -
 fs/iomap/buffered-io.c | 40 +++++++++++++++---------------
 fs/iomap/direct-io.c   | 23 +++++++++---------
 fs/iomap/fiemap.c      |  8 +++---
 fs/iomap/iter.c        |  9 +++----
 fs/iomap/seek.c        |  8 +++---
 fs/iomap/swapfile.c    |  4 +--
 fs/iomap/trace.h       | 12 ++++-----
 fs/ntfs/aops.c         |  6 ++---
 fs/ntfs/file.c         | 24 +++++++++---------
 fs/ntfs/inode.c        |  2 +-
 fs/ntfs/iomap.c        | 40 ++++++------------------------
 fs/ntfs/iomap.h        | 15 ++++++++----
 fs/ntfs3/file.c        | 16 ++++++------
 fs/ntfs3/inode.c       | 13 +++-------
 fs/ntfs3/ntfs_fs.h     |  3 ++-
 fs/remap_range.c       |  6 ++---
 fs/xfs/xfs_aops.c      |  8 +++---
 fs/xfs/xfs_file.c      | 42 ++++++++++++++++----------------
 fs/xfs/xfs_iomap.c     | 55 +++++++++---------------------------------
 fs/xfs/xfs_iomap.h     | 24 ++++++++++++------
 fs/xfs/xfs_iops.c      |  4 +--
 fs/xfs/xfs_reflink.c   |  6 ++---
 fs/zonefs/file.c       | 22 ++++++-----------
 include/linux/dax.h    | 18 ++++++--------
 include/linux/fs.h     |  7 ++++--
 include/linux/iomap.h  | 52 ++++++++++++++++++---------------------
 55 files changed, 318 insertions(+), 431 deletions(-)

diff --git a/block/fops.c b/block/fops.c
index 4cff76e9eb71..53fde467f6ee 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -455,10 +455,6 @@ static int blkdev_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 
 static DEFINE_IOMAP_ITER_NEXT(blkdev_iomap_next, blkdev_iomap_begin);
 
-static const struct iomap_ops blkdev_iomap_ops = {
-	.iomap_next		= blkdev_iomap_next,
-};
-
 #ifdef CONFIG_BUFFER_HEAD
 static int blkdev_get_block(struct inode *inode, sector_t iblock,
 		struct buffer_head *bh, int create)
@@ -512,13 +508,13 @@ const struct address_space_operations def_blk_aops = {
 #else /* CONFIG_BUFFER_HEAD */
 static int blkdev_read_folio(struct file *file, struct folio *folio)
 {
-	iomap_bio_read_folio(folio, &blkdev_iomap_ops);
+	iomap_bio_read_folio(folio, blkdev_iomap_next);
 	return 0;
 }
 
 static void blkdev_readahead(struct readahead_control *rac)
 {
-	iomap_bio_readahead(rac, &blkdev_iomap_ops);
+	iomap_bio_readahead(rac, blkdev_iomap_next);
 }
 
 static ssize_t blkdev_writeback_range(struct iomap_writepage_ctx *wpc,
@@ -709,7 +705,7 @@ blkdev_direct_write(struct kiocb *iocb, struct iov_iter *from)
 
 static ssize_t blkdev_buffered_write(struct kiocb *iocb, struct iov_iter *from)
 {
-	return iomap_file_buffered_write(iocb, from, &blkdev_iomap_ops, NULL,
+	return iomap_file_buffered_write(iocb, from, blkdev_iomap_next, NULL,
 			NULL);
 }
 
diff --git a/fs/btrfs/direct-io.c b/fs/btrfs/direct-io.c
index d5439b06cdc9..b9ff7f9e59f0 100644
--- a/fs/btrfs/direct-io.c
+++ b/fs/btrfs/direct-io.c
@@ -801,10 +801,6 @@ static void btrfs_dio_submit_io(const struct iomap_iter *iter, struct bio *bio,
 static DEFINE_IOMAP_ITER_NEXT_END(btrfs_dio_iomap_next, btrfs_dio_iomap_begin,
 				  btrfs_dio_iomap_end);
 
-static const struct iomap_ops btrfs_dio_iomap_ops = {
-	.iomap_next             = btrfs_dio_iomap_next,
-};
-
 static const struct iomap_dio_ops btrfs_dio_ops = {
 	.submit_io		= btrfs_dio_submit_io,
 	.bio_set		= &btrfs_dio_bioset,
@@ -815,7 +811,7 @@ static ssize_t btrfs_dio_read(struct kiocb *iocb, struct iov_iter *iter,
 {
 	struct btrfs_dio_data data = { 0 };
 
-	return iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
+	return iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, &btrfs_dio_ops,
 			    IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, &data, done_before);
 }
 
@@ -824,7 +820,7 @@ static struct iomap_dio *btrfs_dio_write(struct kiocb *iocb, struct iov_iter *it
 {
 	struct btrfs_dio_data data = { 0 };
 
-	return __iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
+	return __iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, &btrfs_dio_ops,
 			    IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, &data, done_before);
 }
 
diff --git a/fs/dax.c b/fs/dax.c
index 6d175cd47a99..19e5bbe31b69 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -1492,7 +1492,7 @@ static int dax_unshare_iter(struct iomap_iter *iter)
 }
 
 int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct iomap_iter iter = {
 		.inode		= inode,
@@ -1506,7 +1506,7 @@ int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
 		return 0;
 
 	iter.len = min(len, size - pos);
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = dax_unshare_iter(&iter);
 	return ret;
 }
@@ -1584,7 +1584,7 @@ static int dax_zero_iter(struct iomap_iter *iter, bool *did_zero)
 }
 
 int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct iomap_iter iter = {
 		.inode		= inode,
@@ -1594,14 +1594,14 @@ int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
 	};
 	int ret;
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = dax_zero_iter(&iter, did_zero);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(dax_zero_range);
 
 int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	unsigned int blocksize = i_blocksize(inode);
 	unsigned int off = pos & (blocksize - 1);
@@ -1609,7 +1609,7 @@ int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
 	/* Block boundary? Nothing to do */
 	if (!off)
 		return 0;
-	return dax_zero_range(inode, pos, blocksize - off, did_zero, ops);
+	return dax_zero_range(inode, pos, blocksize - off, did_zero, next);
 }
 EXPORT_SYMBOL_GPL(dax_truncate_page);
 
@@ -1734,7 +1734,8 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)
  * dax_iomap_rw - Perform I/O to a DAX file
  * @iocb:	The control block for this I/O
  * @iter:	The addresses to do I/O from or to
- * @ops:	iomap ops passed from the file system
+ * @next:	Callback passed from the file system for advancing to next
+ *		iteration
  *
  * This function performs read and write operations to directly mapped
  * persistent memory.  The callers needs to take care of read/write exclusion
@@ -1742,7 +1743,7 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)
  */
 ssize_t
 dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct iomap_iter iomi = {
 		.inode		= iocb->ki_filp->f_mapping->host,
@@ -1769,7 +1770,7 @@ dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
 	if (iocb->ki_flags & IOCB_NOWAIT)
 		iomi.flags |= IOMAP_NOWAIT;
 
-	while ((ret = iomap_iter(&iomi, ops)) > 0)
+	while ((ret = iomap_iter(&iomi, next)) > 0)
 		iomi.status = dax_iomap_iter(&iomi, iter);
 
 	done = iomi.pos - iocb->ki_pos;
@@ -1897,7 +1898,7 @@ static vm_fault_t dax_fault_iter(struct vm_fault *vmf,
 }
 
 static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,
-			       int *iomap_errp, const struct iomap_ops *ops)
+			       int *iomap_errp, iomap_iter_next_fn next)
 {
 	struct address_space *mapping = vmf->vma->vm_file->f_mapping;
 	XA_STATE(xas, &mapping->i_pages, vmf->pgoff);
@@ -1942,7 +1943,7 @@ static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,
 		goto unlock_entry;
 	}
 
-	while ((error = iomap_iter(&iter, ops)) > 0) {
+	while ((error = iomap_iter(&iter, next)) > 0) {
 		if (WARN_ON_ONCE(iomap_length(&iter) < PAGE_SIZE)) {
 			iter.status = -EIO;	/* fs corruption? */
 			continue;
@@ -2007,7 +2008,7 @@ static bool dax_fault_check_fallback(struct vm_fault *vmf, struct xa_state *xas,
 }
 
 static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
-			       const struct iomap_ops *ops)
+			       iomap_iter_next_fn next)
 {
 	struct address_space *mapping = vmf->vma->vm_file->f_mapping;
 	XA_STATE_ORDER(xas, &mapping->i_pages, vmf->pgoff, PMD_ORDER);
@@ -2064,7 +2065,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
 	}
 
 	iter.pos = (loff_t)xas.xa_index << PAGE_SHIFT;
-	while (iomap_iter(&iter, ops) > 0) {
+	while (iomap_iter(&iter, next) > 0) {
 		if (iomap_length(&iter) < PMD_SIZE)
 			continue; /* actually breaks out of the loop */
 
@@ -2086,7 +2087,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
 }
 #else
 static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
-			       const struct iomap_ops *ops)
+			       iomap_iter_next_fn next)
 {
 	return VM_FAULT_FALLBACK;
 }
@@ -2098,7 +2099,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
  * @order: Order of the page to fault in
  * @pfnp: PFN to insert for synchronous faults if fsync is required
  * @iomap_errp: Storage for detailed error code in case of error
- * @ops: Iomap ops passed from the file system
+ * @next: The iomap iteration function for the filesystem
  *
  * When a page fault occurs, filesystems may call this helper in
  * their fault handler for DAX files. dax_iomap_fault() assumes the caller
@@ -2107,12 +2108,12 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
  */
 vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,
 			unsigned long *pfnp, int *iomap_errp,
-			const struct iomap_ops *ops)
+			iomap_iter_next_fn next)
 {
 	if (order == 0)
-		return dax_iomap_pte_fault(vmf, pfnp, iomap_errp, ops);
+		return dax_iomap_pte_fault(vmf, pfnp, iomap_errp, next);
 	else if (order == PMD_ORDER)
-		return dax_iomap_pmd_fault(vmf, pfnp, ops);
+		return dax_iomap_pmd_fault(vmf, pfnp, next);
 	else
 		return VM_FAULT_FALLBACK;
 }
@@ -2240,7 +2241,7 @@ static int dax_range_compare_iter(struct iomap_iter *it_src,
 
 int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
 		struct inode *dst, loff_t dstoff, loff_t len, bool *same,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct iomap_iter src_iter = {
 		.inode		= src,
@@ -2256,8 +2257,8 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
 	};
 	int ret, status;
 
-	while ((ret = iomap_iter(&src_iter, ops)) > 0 &&
-	       (ret = iomap_iter(&dst_iter, ops)) > 0) {
+	while ((ret = iomap_iter(&src_iter, next)) > 0 &&
+	       (ret = iomap_iter(&dst_iter, next)) > 0) {
 		status = dax_range_compare_iter(&src_iter, &dst_iter,
 				min(src_iter.len, dst_iter.len), same);
 		if (status < 0)
@@ -2270,9 +2271,10 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
 int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,
 			      struct file *file_out, loff_t pos_out,
 			      loff_t *len, unsigned int remap_flags,
-			      const struct iomap_ops *ops)
+			      iomap_iter_next_fn next)
 {
 	return __generic_remap_file_range_prep(file_in, pos_in, file_out,
-					       pos_out, len, remap_flags, ops);
+					       pos_out, len, remap_flags,
+					       next);
 }
 EXPORT_SYMBOL_GPL(dax_remap_file_range_prep);
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index d2f01245ee79..d965ed015214 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -383,10 +383,6 @@ static int erofs_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 static DEFINE_IOMAP_ITER_NEXT_END(erofs_iomap_next, erofs_iomap_begin,
 				  erofs_iomap_end);
 
-static const struct iomap_ops erofs_iomap_ops = {
-	.iomap_next = erofs_iomap_next,
-};
-
 int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 		 u64 start, u64 len)
 {
@@ -394,9 +390,9 @@ int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 		if (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))
 			return -EOPNOTSUPP;
 		return iomap_fiemap(inode, fieinfo, start, len,
-				    &z_erofs_iomap_report_ops);
+				    z_erofs_iomap_next_report);
 	}
-	return iomap_fiemap(inode, fieinfo, start, len, &erofs_iomap_ops);
+	return iomap_fiemap(inode, fieinfo, start, len, erofs_iomap_next);
 }
 
 /*
@@ -415,7 +411,7 @@ static int erofs_read_folio(struct file *file, struct folio *folio)
 	};
 
 	trace_erofs_read_folio(iter_ctx.realinode, folio, true);
-	iomap_read_folio(&erofs_iomap_ops, &read_ctx, &iter_ctx);
+	iomap_read_folio(erofs_iomap_next, &read_ctx, &iter_ctx);
 	if (need_iput)
 		iput(iter_ctx.realinode);
 	return 0;
@@ -434,14 +430,14 @@ static void erofs_readahead(struct readahead_control *rac)
 
 	trace_erofs_readahead(iter_ctx.realinode, readahead_index(rac),
 			      readahead_count(rac), true);
-	iomap_readahead(&erofs_iomap_ops, &read_ctx, &iter_ctx);
+	iomap_readahead(erofs_iomap_next, &read_ctx, &iter_ctx);
 	if (need_iput)
 		iput(iter_ctx.realinode);
 }
 
 static sector_t erofs_bmap(struct address_space *mapping, sector_t block)
 {
-	return iomap_bmap(mapping, block, &erofs_iomap_ops);
+	return iomap_bmap(mapping, block, erofs_iomap_next);
 }
 
 static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
@@ -453,14 +449,14 @@ static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		return 0;
 
 	if (IS_ENABLED(CONFIG_FS_DAX) && IS_DAX(inode))
-		return dax_iomap_rw(iocb, to, &erofs_iomap_ops);
+		return dax_iomap_rw(iocb, to, erofs_iomap_next);
 
 	if ((iocb->ki_flags & IOCB_DIRECT) && inode->i_sb->s_bdev) {
 		struct erofs_iomap_iter_ctx iter_ctx = {
 			.realinode = inode,
 		};
 
-		return iomap_dio_rw(iocb, to, &erofs_iomap_ops,
+		return iomap_dio_rw(iocb, to, erofs_iomap_next,
 				    NULL, 0, &iter_ctx, 0);
 	}
 	return filemap_read(iocb, to, 0);
@@ -480,7 +476,7 @@ const struct address_space_operations erofs_aops = {
 static vm_fault_t erofs_dax_huge_fault(struct vm_fault *vmf,
 		unsigned int order)
 {
-	return dax_iomap_fault(vmf, order, NULL, NULL, &erofs_iomap_ops);
+	return dax_iomap_fault(vmf, order, NULL, NULL, erofs_iomap_next);
 }
 
 static vm_fault_t erofs_dax_fault(struct vm_fault *vmf)
@@ -512,18 +508,18 @@ static int erofs_file_mmap_prepare(struct vm_area_desc *desc)
 static loff_t erofs_file_llseek(struct file *file, loff_t offset, int whence)
 {
 	struct inode *inode = file->f_mapping->host;
-	const struct iomap_ops *ops = &erofs_iomap_ops;
+	iomap_iter_next_fn next = erofs_iomap_next;
 
 	if (erofs_inode_is_data_compressed(EROFS_I(inode)->datalayout)) {
 		if (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))
 			return generic_file_llseek(file, offset, whence);
-		ops = &z_erofs_iomap_report_ops;
+		next = z_erofs_iomap_next_report;
 	}
 
 	if (whence == SEEK_HOLE)
-		offset = iomap_seek_hole(inode, offset, ops);
+		offset = iomap_seek_hole(inode, offset, next);
 	else if (whence == SEEK_DATA)
-		offset = iomap_seek_data(inode, offset, ops);
+		offset = iomap_seek_data(inode, offset, next);
 	else
 		return generic_file_llseek(file, offset, whence);
 
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 580f8d9f14e7..72ccd6f335b8 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -397,7 +397,8 @@ extern const struct file_operations erofs_file_fops;
 extern const struct file_operations erofs_dir_fops;
 extern const struct file_operations erofs_ishare_fops;
 
-extern const struct iomap_ops z_erofs_iomap_report_ops;
+int z_erofs_iomap_next_report(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap);
 
 void *erofs_read_metadata(struct super_block *sb, struct erofs_buf *buf,
 			  erofs_off_t *offset, int *lengthp);
diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c
index 3966b62a7051..98968a9d942e 100644
--- a/fs/erofs/zmap.c
+++ b/fs/erofs/zmap.c
@@ -821,9 +821,4 @@ static int z_erofs_iomap_begin_report(struct inode *inode, loff_t offset,
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(z_erofs_iomap_next_report,
-			      z_erofs_iomap_begin_report);
-
-const struct iomap_ops z_erofs_iomap_report_ops = {
-	.iomap_next = z_erofs_iomap_next_report,
-};
+DEFINE_IOMAP_ITER_NEXT(z_erofs_iomap_next_report, z_erofs_iomap_begin_report);
diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index 3a278c95239e..adda6342de5b 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -668,7 +668,7 @@ static int exfat_extend_valid_size(struct inode *inode, loff_t new_valid_size)
 
 		ret = iomap_zero_range(inode, old_valid_size,
 				new_valid_size - old_valid_size, NULL,
-				&exfat_write_iomap_ops, NULL, NULL);
+				exfat_write_iomap_next, NULL, NULL);
 		if (ret) {
 			truncate_setsize(inode, old_valid_size);
 			exfat_truncate(inode);
@@ -687,7 +687,7 @@ static ssize_t exfat_fallback_buffered_write(struct kiocb *iocb,
 
 	iocb->ki_flags &= ~IOCB_DIRECT;
 
-	written = iomap_file_buffered_write(iocb, from, &exfat_write_iomap_ops,
+	written = iomap_file_buffered_write(iocb, from, exfat_write_iomap_next,
 			NULL, NULL);
 	if (written < 0)
 		return written;
@@ -709,7 +709,7 @@ static ssize_t exfat_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 {
 	ssize_t ret;
 
-	ret = iomap_dio_rw(iocb, from, &exfat_write_iomap_ops,
+	ret = iomap_dio_rw(iocb, from, exfat_write_iomap_next,
 			&exfat_write_dio_ops, 0, NULL, 0);
 	if (ret == -ENOTBLK)
 		ret = 0;
@@ -773,7 +773,7 @@ static ssize_t exfat_file_write_iter(struct kiocb *iocb, struct iov_iter *iter)
 		ret = exfat_dio_write_iter(iocb, iter);
 	else
 		ret = iomap_file_buffered_write(iocb, iter,
-				&exfat_write_iomap_ops, NULL, NULL);
+				exfat_write_iomap_next, NULL, NULL);
 	if (ret < 0)
 		goto unlock;
 
@@ -809,7 +809,7 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
 	if (iocb->ki_flags & IOCB_DIRECT) {
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, iter, &exfat_iomap_ops, NULL,
+		ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL,
 				IOMAP_DIO_NO_IOMAP_END, NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, iter);
@@ -850,7 +850,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)
 			 */
 			err = iomap_zero_range(inode, ei->zeroed_size,
 					mmap_valid_size - ei->zeroed_size, NULL,
-					&exfat_iomap_ops, NULL, NULL);
+					exfat_iomap_next, NULL, NULL);
 			if (err < 0) {
 				inode_unlock(inode);
 				return vmf_fs_error(err);
@@ -866,7 +866,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)
 	file_update_time(vmf->vma->vm_file);
 
 	filemap_invalidate_lock_shared(inode->i_mapping);
-	ret = iomap_page_mkwrite(vmf, &exfat_write_iomap_ops, NULL);
+	ret = iomap_page_mkwrite(vmf, exfat_write_iomap_next, NULL);
 	filemap_invalidate_unlock_shared(inode->i_mapping);
 	sb_end_pagefault(inode->i_sb);
 	inode_unlock(inode);
@@ -939,12 +939,12 @@ static loff_t exfat_file_llseek(struct file *file, loff_t offset, int whence)
 	switch (whence) {
 	case SEEK_HOLE:
 		inode_lock_shared(inode);
-		offset = iomap_seek_hole(inode, offset, &exfat_iomap_ops);
+		offset = iomap_seek_hole(inode, offset, exfat_iomap_next);
 		inode_unlock_shared(inode);
 		break;
 	case SEEK_DATA:
 		inode_lock_shared(inode);
-		offset = iomap_seek_data(inode, offset, &exfat_iomap_ops);
+		offset = iomap_seek_data(inode, offset, exfat_iomap_next);
 		inode_unlock_shared(inode);
 		break;
 	default:
diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c
index 89826aea5e1e..a6b9aa2ad792 100644
--- a/fs/exfat/inode.c
+++ b/fs/exfat/inode.c
@@ -248,7 +248,7 @@ static int exfat_read_folio(struct file *file, struct folio *folio)
 		.ops = &exfat_iomap_bio_read_ops,
 	};
 
-	iomap_read_folio(&exfat_iomap_ops, &ctx, NULL);
+	iomap_read_folio(exfat_iomap_next, &ctx, NULL);
 	return 0;
 }
 
@@ -269,7 +269,7 @@ static void exfat_readahead(struct readahead_control *rac)
 	    ei->valid_size < pos + readahead_length(rac))
 		return;
 
-	iomap_readahead(&exfat_iomap_ops, &ctx, NULL);
+	iomap_readahead(exfat_iomap_next, &ctx, NULL);
 }
 
 static int exfat_writepages(struct address_space *mapping,
@@ -293,7 +293,7 @@ static sector_t exfat_aop_bmap(struct address_space *mapping, sector_t block)
 
 	/* exfat_get_cluster() assumes the requested blocknr isn't truncated. */
 	down_read(&EXFAT_I(mapping->host)->truncate_lock);
-	blocknr = iomap_bmap(mapping, block, &exfat_iomap_ops);
+	blocknr = iomap_bmap(mapping, block, exfat_iomap_next);
 	up_read(&EXFAT_I(mapping->host)->truncate_lock);
 	return blocknr;
 }
diff --git a/fs/exfat/iomap.c b/fs/exfat/iomap.c
index b6dd396aa60a..631de19559d5 100644
--- a/fs/exfat/iomap.c
+++ b/fs/exfat/iomap.c
@@ -151,11 +151,7 @@ static int exfat_write_iomap_begin(struct inode *inode, loff_t offset, loff_t le
 	return __exfat_iomap_begin(inode, offset, length, flags, iomap, true);
 }
 
-static DEFINE_IOMAP_ITER_NEXT(exfat_iomap_next, exfat_iomap_begin);
-
-const struct iomap_ops exfat_iomap_ops = {
-	.iomap_next = exfat_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(exfat_iomap_next, exfat_iomap_begin);
 
 /*
  * exfat_write_iomap_end - Update the state after write
@@ -188,12 +184,8 @@ static int exfat_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return written;
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(exfat_write_iomap_next,
-		exfat_write_iomap_begin, exfat_write_iomap_end);
-
-const struct iomap_ops exfat_write_iomap_ops = {
-	.iomap_next	= exfat_write_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT_END(exfat_write_iomap_next, exfat_write_iomap_begin,
+		exfat_write_iomap_end);
 
 /*
  * exfat_writeback_range - Map folio during writeback
@@ -271,5 +263,5 @@ const struct iomap_read_ops exfat_iomap_bio_read_ops = {
 int exfat_iomap_swap_activate(struct swap_info_struct *sis,
 			       struct file *file, sector_t *span)
 {
-	return iomap_swapfile_activate(sis, file, span, &exfat_iomap_ops);
+	return iomap_swapfile_activate(sis, file, span, exfat_iomap_next);
 }
diff --git a/fs/exfat/iomap.h b/fs/exfat/iomap.h
index fd8a913f7794..47d7b753735e 100644
--- a/fs/exfat/iomap.h
+++ b/fs/exfat/iomap.h
@@ -7,8 +7,10 @@
 #define _LINUX_EXFAT_IOMAP_H
 
 extern const struct iomap_dio_ops exfat_write_dio_ops;
-extern const struct iomap_ops exfat_iomap_ops;
-extern const struct iomap_ops exfat_write_iomap_ops;
+int exfat_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int exfat_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
 extern const struct iomap_writeback_ops exfat_writeback_ops;
 extern const struct iomap_read_ops exfat_iomap_bio_read_ops;
 
diff --git a/fs/ext2/ext2.h b/fs/ext2/ext2.h
index 79f7b395258c..59ef8b898940 100644
--- a/fs/ext2/ext2.h
+++ b/fs/ext2/ext2.h
@@ -780,7 +780,8 @@ extern const struct file_operations ext2_file_operations;
 /* inode.c */
 extern void ext2_set_file_ops(struct inode *inode);
 extern const struct address_space_operations ext2_aops;
-extern const struct iomap_ops ext2_iomap_ops;
+int ext2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
 
 /* namei.c */
 extern const struct inode_operations ext2_dir_inode_operations;
diff --git a/fs/ext2/file.c b/fs/ext2/file.c
index 8dca9ec4cacd..1fc00ad77517 100644
--- a/fs/ext2/file.c
+++ b/fs/ext2/file.c
@@ -70,7 +70,7 @@ static ssize_t ext2_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
 
 	trace_ext2_dio_read_begin(iocb, to, 0);
 	inode_lock_shared(inode);
-	ret = iomap_dio_rw(iocb, to, &ext2_iomap_ops, NULL, 0, NULL, 0);
+	ret = iomap_dio_rw(iocb, to, ext2_iomap_next, NULL, 0, NULL, 0);
 	inode_unlock_shared(inode);
 	trace_ext2_dio_read_end(iocb, to, ret);
 
@@ -134,7 +134,7 @@ static ssize_t ext2_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	   (!IS_ALIGNED(iocb->ki_pos | iov_iter_alignment(from), blocksize)))
 		flags |= IOMAP_DIO_FORCE_WAIT;
 
-	ret = iomap_dio_rw(iocb, from, &ext2_iomap_ops, &ext2_dio_write_ops,
+	ret = iomap_dio_rw(iocb, from, ext2_iomap_next, &ext2_dio_write_ops,
 			   flags, NULL, 0);
 
 	/* ENOTBLK is magic return value for fallback to buffered-io */
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 7e0fa9c454e1..873ad1d90e0d 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -860,12 +860,7 @@ ext2_iomap_end(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(ext2_iomap_next, ext2_iomap_begin,
-				  ext2_iomap_end);
-
-const struct iomap_ops ext2_iomap_ops = {
-	.iomap_next		= ext2_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ext2_iomap_next, ext2_iomap_begin, ext2_iomap_end);
 
 int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 		u64 start, u64 len)
@@ -884,7 +879,7 @@ int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 	if (i_size == 0)
 		i_size = 1;
 	len = min_t(u64, len, i_size);
-	ret = iomap_fiemap(inode, fieinfo, start, len, &ext2_iomap_ops);
+	ret = iomap_fiemap(inode, fieinfo, start, len, ext2_iomap_next);
 	inode_unlock(inode);
 
 	return ret;
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index b37c136ea3ab..755fde1baf03 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -4004,8 +4004,10 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end)
 		io_end->flag &= ~EXT4_IO_END_UNWRITTEN;
 }
 
-extern const struct iomap_ops ext4_iomap_ops;
-extern const struct iomap_ops ext4_iomap_report_ops;
+int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
 
 static inline int ext4_buffer_uptodate(struct buffer_head *bh)
 {
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 15972410d460..71bbf8e77261 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -5173,10 +5173,6 @@ static int ext4_iomap_xattr_begin(struct inode *inode, loff_t offset,
 
 static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_xattr_next, ext4_iomap_xattr_begin);
 
-static const struct iomap_ops ext4_iomap_xattr_ops = {
-	.iomap_next		= ext4_iomap_xattr_next,
-};
-
 static int ext4_fiemap_check_ranges(struct inode *inode, u64 start, u64 *len)
 {
 	u64 maxbytes = ext4_get_maxbytes(inode);
@@ -5219,10 +5215,10 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 	if (fieinfo->fi_flags & FIEMAP_FLAG_XATTR) {
 		fieinfo->fi_flags &= ~FIEMAP_FLAG_XATTR;
 		error = iomap_fiemap(inode, fieinfo, start, len,
-				     &ext4_iomap_xattr_ops);
+				     ext4_iomap_xattr_next);
 	} else {
 		error = iomap_fiemap(inode, fieinfo, start, len,
-				     &ext4_iomap_report_ops);
+				     ext4_iomap_next_report);
 	}
 unlock:
 	inode_unlock_shared(inode);
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index 7bd5de67ce78..bf0c18cf1017 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,7 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		return generic_file_read_iter(iocb, to);
 	}
 
-	ret = iomap_dio_rw(iocb, to, &ext4_iomap_ops, NULL,
+	ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL,
 			   IOMAP_DIO_NO_IOMAP_END, NULL, 0);
 	inode_unlock_shared(inode);
 
@@ -120,7 +120,7 @@ static ssize_t ext4_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		/* Fallback to buffered IO in case we cannot support DAX */
 		return generic_file_read_iter(iocb, to);
 	}
-	ret = dax_iomap_rw(iocb, to, &ext4_iomap_ops);
+	ret = dax_iomap_rw(iocb, to, ext4_iomap_next);
 	inode_unlock_shared(inode);
 
 	file_accessed(iocb->ki_filp);
@@ -590,7 +590,7 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 			goto out;
 	}
 
-	ret = iomap_dio_rw(iocb, from, &ext4_iomap_ops, &ext4_dio_write_ops,
+	ret = iomap_dio_rw(iocb, from, ext4_iomap_next, &ext4_dio_write_ops,
 			   dio_flags, NULL, 0);
 	if (ret == -ENOTBLK)
 		ret = 0;
@@ -689,7 +689,7 @@ ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		ext4_journal_stop(handle);
 	}
 
-	ret = dax_iomap_rw(iocb, from, &ext4_iomap_ops);
+	ret = dax_iomap_rw(iocb, from, ext4_iomap_next);
 
 	if (extend) {
 		ret = ext4_handle_inode_extension(inode, offset, ret, count);
@@ -777,7 +777,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, unsigned int order)
 	} else {
 		filemap_invalidate_lock_shared(mapping);
 	}
-	result = dax_iomap_fault(vmf, order, &pfn, &error, &ext4_iomap_ops);
+	result = dax_iomap_fault(vmf, order, &pfn, &error, ext4_iomap_next);
 	if (write) {
 		ext4_journal_stop(handle);
 
@@ -956,13 +956,13 @@ loff_t ext4_llseek(struct file *file, loff_t offset, int whence)
 	case SEEK_HOLE:
 		inode_lock_shared(inode);
 		offset = iomap_seek_hole(inode, offset,
-					 &ext4_iomap_report_ops);
+					 ext4_iomap_next_report);
 		inode_unlock_shared(inode);
 		break;
 	case SEEK_DATA:
 		inode_lock_shared(inode);
 		offset = iomap_seek_data(inode, offset,
-					 &ext4_iomap_report_ops);
+					 ext4_iomap_next_report);
 		inode_unlock_shared(inode);
 		break;
 	}
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 278a032b300c..f455708a8c31 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3391,7 +3391,7 @@ static sector_t ext4_bmap(struct address_space *mapping, sector_t block)
 		filemap_write_and_wait(mapping);
 	}
 
-	ret = iomap_bmap(mapping, block, &ext4_iomap_ops);
+	ret = iomap_bmap(mapping, block, ext4_iomap_next);
 
 out:
 	inode_unlock_shared(inode);
@@ -3850,11 +3850,7 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next, ext4_iomap_begin);
-
-const struct iomap_ops ext4_iomap_ops = {
-	.iomap_next		= ext4_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next, ext4_iomap_begin);
 
 static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
 				   loff_t length, unsigned int flags,
@@ -3907,11 +3903,7 @@ static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next_report, ext4_iomap_begin_report);
-
-const struct iomap_ops ext4_iomap_report_ops = {
-	.iomap_next = ext4_iomap_next_report,
-};
+DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next_report, ext4_iomap_begin_report);
 
 /*
  * For data=journal mode, folio should be marked dirty only when it was
@@ -3948,7 +3940,7 @@ static int ext4_iomap_swap_activate(struct swap_info_struct *sis,
 				    struct file *file, sector_t *span)
 {
 	return iomap_swapfile_activate(sis, file, span,
-				       &ext4_iomap_report_ops);
+				       ext4_iomap_next_report);
 }
 
 static const struct address_space_operations ext4_aops = {
@@ -4195,7 +4187,7 @@ static int ext4_block_zero_range(struct inode *inode,
 
 	if (IS_DAX(inode)) {
 		return dax_zero_range(inode, from, length, did_zero,
-				      &ext4_iomap_ops);
+				      ext4_iomap_next);
 	} else if (ext4_should_journal_data(inode)) {
 		return ext4_block_journalled_zero_range(inode, from, length,
 							did_zero);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 8977ad379f50..1cb993244b1c 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -4653,8 +4653,4 @@ static int f2fs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(f2fs_iomap_next, f2fs_iomap_begin);
-
-const struct iomap_ops f2fs_iomap_ops = {
-	.iomap_next	= f2fs_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(f2fs_iomap_next, f2fs_iomap_begin);
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 8f3e632f315c..946a91834aec 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -4216,7 +4216,8 @@ int f2fs_init_post_read_processing(void);
 void f2fs_destroy_post_read_processing(void);
 int f2fs_init_wq(struct f2fs_sb_info *sbi);
 void f2fs_destroy_wq(struct f2fs_sb_info *sbi);
-extern const struct iomap_ops f2fs_iomap_ops;
+int f2fs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		    struct iomap *srcmap);
 
 /*
  * gc.c
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 4b52c56d71f0..74514b117257 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -4884,7 +4884,7 @@ static ssize_t f2fs_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	 * F2FS_DIO_READ counter will be decremented correctly in all cases.
 	 */
 	inc_page_count(sbi, F2FS_DIO_READ);
-	dio = __iomap_dio_rw(iocb, to, &f2fs_iomap_ops,
+	dio = __iomap_dio_rw(iocb, to, f2fs_iomap_next,
 			     &f2fs_iomap_dio_read_ops, 0, NULL, 0);
 	if (IS_ERR_OR_NULL(dio)) {
 		ret = PTR_ERR_OR_ZERO(dio);
@@ -5220,7 +5220,7 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 	dio_flags = 0;
 	if (pos + count > inode->i_size)
 		dio_flags |= IOMAP_DIO_FORCE_WAIT;
-	dio = __iomap_dio_rw(iocb, from, &f2fs_iomap_ops,
+	dio = __iomap_dio_rw(iocb, from, f2fs_iomap_next,
 			     &f2fs_iomap_dio_write_ops, dio_flags, iocb, 0);
 	if (IS_ERR_OR_NULL(dio)) {
 		ret = PTR_ERR_OR_ZERO(dio);
diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c
index 85cdf0199bc0..480cfd2d1fa5 100644
--- a/fs/fuse/dax.c
+++ b/fs/fuse/dax.c
@@ -656,10 +656,6 @@ static int fuse_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 static DEFINE_IOMAP_ITER_NEXT_END(fuse_iomap_next, fuse_iomap_begin,
 				  fuse_iomap_end);
 
-static const struct iomap_ops fuse_iomap_ops = {
-	.iomap_next = fuse_iomap_next,
-};
-
 static void fuse_wait_dax_page(struct inode *inode)
 {
 	filemap_invalidate_unlock(inode->i_mapping);
@@ -687,7 +683,7 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		inode_lock_shared(inode);
 	}
 
-	ret = dax_iomap_rw(iocb, to, &fuse_iomap_ops);
+	ret = dax_iomap_rw(iocb, to, fuse_iomap_next);
 	inode_unlock_shared(inode);
 
 	/* TODO file_accessed(iocb->f_filp) */
@@ -742,7 +738,7 @@ ssize_t fuse_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (file_extending_write(iocb, from))
 		ret = fuse_dax_direct_write(iocb, from);
 	else
-		ret = dax_iomap_rw(iocb, from, &fuse_iomap_ops);
+		ret = dax_iomap_rw(iocb, from, fuse_iomap_next);
 
 out:
 	inode_unlock(inode);
@@ -777,7 +773,7 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, unsigned int order,
 	 * to populate page cache or access memory we are trying to free.
 	 */
 	filemap_invalidate_lock_shared(inode->i_mapping);
-	ret = dax_iomap_fault(vmf, order, &pfn, &error, &fuse_iomap_ops);
+	ret = dax_iomap_fault(vmf, order, &pfn, &error, fuse_iomap_next);
 	if ((ret & VM_FAULT_ERROR) && error == -EAGAIN) {
 		error = 0;
 		retry = true;
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 403decd4bb52..fc5d5f6013d5 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -892,10 +892,6 @@ static int fuse_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 
 static DEFINE_IOMAP_ITER_NEXT(fuse_iomap_next, fuse_iomap_begin);
 
-static const struct iomap_ops fuse_iomap_ops = {
-	.iomap_next	= fuse_iomap_next,
-};
-
 struct fuse_fill_read_data {
 	struct file *file;
 
@@ -1016,7 +1012,7 @@ static int fuse_read_folio(struct file *file, struct folio *folio)
 		return -EIO;
 	}
 
-	iomap_read_folio(&fuse_iomap_ops, &ctx, NULL);
+	iomap_read_folio(fuse_iomap_next, &ctx, NULL);
 	fuse_invalidate_atime(inode);
 	return 0;
 }
@@ -1117,7 +1113,7 @@ static void fuse_readahead(struct readahead_control *rac)
 	if (fuse_is_bad(inode))
 		return;
 
-	iomap_readahead(&fuse_iomap_ops, &ctx, NULL);
+	iomap_readahead(fuse_iomap_next, &ctx, NULL);
 }
 
 static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to)
@@ -1549,7 +1545,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		 * and granular dirty tracking for large folios.
 		 */
 		written = iomap_file_buffered_write(iocb, from,
-						    &fuse_iomap_ops,
+						    fuse_iomap_next,
 						    &fuse_iomap_write_ops,
 						    file);
 	} else {
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index 0a7b8076af3a..66bc19c011cc 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -425,7 +425,7 @@ static int gfs2_read_folio(struct file *file, struct folio *folio)
 
 	if (!gfs2_is_jdata(ip) ||
 	    (i_blocksize(inode) == PAGE_SIZE && !folio_buffers(folio))) {
-		iomap_bio_read_folio(folio, &gfs2_iomap_ops);
+		iomap_bio_read_folio(folio, gfs2_iomap_next);
 	} else if (gfs2_is_stuffed(ip)) {
 		error = stuffed_read_folio(ip, folio);
 	} else {
@@ -500,7 +500,7 @@ static void gfs2_readahead(struct readahead_control *rac)
 	else if (gfs2_is_jdata(ip))
 		mpage_readahead(rac, gfs2_block_map);
 	else
-		iomap_bio_readahead(rac, &gfs2_iomap_ops);
+		iomap_bio_readahead(rac, gfs2_iomap_next);
 }
 
 /**
@@ -571,7 +571,7 @@ static sector_t gfs2_bmap(struct address_space *mapping, sector_t lblock)
 		return 0;
 
 	if (!gfs2_is_stuffed(ip))
-		dblock = iomap_bmap(mapping, lblock, &gfs2_iomap_ops);
+		dblock = iomap_bmap(mapping, lblock, gfs2_iomap_next);
 
 	gfs2_glock_dq_uninit(&i_gh);
 
diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index 73c626971163..f835bd343b6e 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -1200,12 +1200,7 @@ static int gfs2_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(gfs2_iomap_next, gfs2_iomap_begin,
-				  gfs2_iomap_end);
-
-const struct iomap_ops gfs2_iomap_ops = {
-	.iomap_next = gfs2_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT_END(gfs2_iomap_next, gfs2_iomap_begin, gfs2_iomap_end);
 
 /**
  * gfs2_block_map - Map one or more blocks of an inode to a disk block
@@ -1320,7 +1315,7 @@ static int gfs2_block_zero_range(struct inode *inode, loff_t from, loff_t length
 	if (from >= inode->i_size)
 		return 0;
 	length = min(length, inode->i_size - from);
-	return iomap_zero_range(inode, from, length, NULL, &gfs2_iomap_ops,
+	return iomap_zero_range(inode, from, length, NULL, gfs2_iomap_next,
 			&gfs2_iomap_write_ops, NULL);
 }
 
diff --git a/fs/gfs2/bmap.h b/fs/gfs2/bmap.h
index e3d6efdfd890..2c2b7ab39259 100644
--- a/fs/gfs2/bmap.h
+++ b/fs/gfs2/bmap.h
@@ -43,7 +43,8 @@ static inline void gfs2_write_calc_reserv(const struct gfs2_inode *ip,
 	}
 }
 
-extern const struct iomap_ops gfs2_iomap_ops;
+int gfs2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
 extern const struct iomap_write_ops gfs2_iomap_write_ops;
 extern const struct iomap_writeback_ops gfs2_writeback_ops;
 
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index b8c10de113ba..ef5f521a46c0 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -844,7 +844,7 @@ static ssize_t gfs2_file_direct_read(struct kiocb *iocb, struct iov_iter *to,
 		goto out_uninit;
 	pagefault_disable();
 	to->nofault = true;
-	ret = iomap_dio_rw(iocb, to, &gfs2_iomap_ops, NULL,
+	ret = iomap_dio_rw(iocb, to, gfs2_iomap_next, NULL,
 			   IOMAP_DIO_PARTIAL, NULL, read);
 	to->nofault = false;
 	pagefault_enable();
@@ -910,7 +910,7 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from,
 		goto out_unlock;
 
 	from->nofault = true;
-	ret = iomap_dio_rw(iocb, from, &gfs2_iomap_ops, NULL,
+	ret = iomap_dio_rw(iocb, from, gfs2_iomap_next, NULL,
 			   IOMAP_DIO_PARTIAL, NULL, written);
 	from->nofault = false;
 	if (ret <= 0) {
@@ -1062,7 +1062,7 @@ static ssize_t gfs2_file_buffered_write(struct kiocb *iocb,
 		goto out_unlock;
 
 	pagefault_disable();
-	ret = iomap_file_buffered_write(iocb, from, &gfs2_iomap_ops,
+	ret = iomap_file_buffered_write(iocb, from, gfs2_iomap_next,
 			&gfs2_iomap_write_ops, NULL);
 	pagefault_enable();
 	if (ret > 0)
diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c
index 8a77794bbd4a..737a3b6c5268 100644
--- a/fs/gfs2/inode.c
+++ b/fs/gfs2/inode.c
@@ -2217,7 +2217,7 @@ static int gfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 		goto out;
 
 	pagefault_disable();
-	ret = iomap_fiemap(inode, fieinfo, start, len, &gfs2_iomap_ops);
+	ret = iomap_fiemap(inode, fieinfo, start, len, gfs2_iomap_next);
 	pagefault_enable();
 
 	gfs2_glock_dq_uninit(&gh);
@@ -2242,7 +2242,7 @@ loff_t gfs2_seek_data(struct file *file, loff_t offset)
 	inode_lock_shared(inode);
 	ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, 0, &gh);
 	if (!ret)
-		ret = iomap_seek_data(inode, offset, &gfs2_iomap_ops);
+		ret = iomap_seek_data(inode, offset, gfs2_iomap_next);
 	gfs2_glock_dq_uninit(&gh);
 	inode_unlock_shared(inode);
 
@@ -2261,7 +2261,7 @@ loff_t gfs2_seek_hole(struct file *file, loff_t offset)
 	inode_lock_shared(inode);
 	ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, 0, &gh);
 	if (!ret)
-		ret = iomap_seek_hole(inode, offset, &gfs2_iomap_ops);
+		ret = iomap_seek_hole(inode, offset, gfs2_iomap_next);
 	gfs2_glock_dq_uninit(&gh);
 	inode_unlock_shared(inode);
 
diff --git a/fs/hpfs/file.c b/fs/hpfs/file.c
index 6a629ab956fc..ef7eb601be09 100644
--- a/fs/hpfs/file.c
+++ b/fs/hpfs/file.c
@@ -158,10 +158,6 @@ static int hpfs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 
 static DEFINE_IOMAP_ITER_NEXT(hpfs_iomap_next, hpfs_iomap_begin);
 
-static const struct iomap_ops hpfs_iomap_ops = {
-	.iomap_next		= hpfs_iomap_next,
-};
-
 static int hpfs_read_folio(struct file *file, struct folio *folio)
 {
 	return mpage_read_folio(folio, hpfs_get_block);
@@ -238,7 +234,7 @@ static int hpfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 
 	inode_lock(inode);
 	len = min_t(u64, len, i_size_read(inode));
-	ret = iomap_fiemap(inode, fieinfo, start, len, &hpfs_iomap_ops);
+	ret = iomap_fiemap(inode, fieinfo, start, len, hpfs_iomap_next);
 	inode_unlock(inode);
 
 	return ret;
diff --git a/fs/internal.h b/fs/internal.h
index 355d93f92208..19601f8406dc 100644
--- a/fs/internal.h
+++ b/fs/internal.h
@@ -8,7 +8,6 @@
 struct super_block;
 struct file_system_type;
 struct iomap;
-struct iomap_ops;
 struct linux_binprm;
 struct path;
 struct mount;
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 27bc2455a98d..07d1882d4480 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -618,8 +618,8 @@ static int iomap_read_folio_iter(struct iomap_iter *iter,
 	return 0;
 }
 
-void iomap_read_folio(const struct iomap_ops *ops,
-		struct iomap_read_folio_ctx *ctx, void *private)
+void iomap_read_folio(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+		void *private)
 {
 	struct folio *folio = ctx->cur_folio;
 	struct iomap_iter iter = {
@@ -642,7 +642,7 @@ void iomap_read_folio(const struct iomap_ops *ops,
 		fsverity_readahead(ctx->vi, folio->index,
 				   folio_nr_pages(folio));
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_read_folio_iter(&iter, ctx,
 				&bytes_submitted);
 
@@ -681,22 +681,22 @@ static int iomap_readahead_iter(struct iomap_iter *iter,
 
 /**
  * iomap_readahead - Attempt to read pages from a file.
- * @ops: The operations vector for the filesystem.
+ * @next: The iomap iteration function for the filesystem.
  * @ctx: The ctx used for issuing readahead.
  * @private: The filesystem-specific information for issuing iomap_iter.
  *
  * This function is for filesystems to call to implement their readahead
  * address_space operation.
  *
- * Context: The @ops callbacks may submit I/O (eg to read the addresses of
+ * Context: The @next callback may submit I/O (eg to read the addresses of
  * blocks from disc), and may wait for it.  The caller may be trying to
  * access a different page, and so sleeping excessively should be avoided.
  * It may allocate memory, but should avoid costly allocations.  This
  * function is called with memalloc_nofs set, so allocations will not cause
  * the filesystem to be reentered.
  */
-void iomap_readahead(const struct iomap_ops *ops,
-		struct iomap_read_folio_ctx *ctx, void *private)
+void iomap_readahead(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+		void *private)
 {
 	struct readahead_control *rac = ctx->rac;
 	struct iomap_iter iter = {
@@ -718,7 +718,7 @@ void iomap_readahead(const struct iomap_ops *ops,
 		fsverity_readahead(ctx->vi, readahead_index(rac),
 				readahead_count(rac));
 
-	while (iomap_iter(&iter, ops) > 0)
+	while (iomap_iter(&iter, next) > 0)
 		iter.status = iomap_readahead_iter(&iter, ctx,
 					&cur_bytes_submitted);
 
@@ -1262,7 +1262,7 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
 
 ssize_t
 iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
-		const struct iomap_ops *ops,
+		iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops, void *private)
 {
 	struct iomap_iter iter = {
@@ -1279,7 +1279,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 	if (iocb->ki_flags & IOCB_DONTCACHE)
 		iter.flags |= IOMAP_DONTCACHE;
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_write_iter(&iter, i, write_ops);
 
 	if (unlikely(iter.pos == iocb->ki_pos))
@@ -1291,7 +1291,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
 EXPORT_SYMBOL_GPL(iomap_file_buffered_write);
 
 int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
-		const void *buf, const struct iomap_ops *ops,
+		const void *buf, iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops)
 {
 	int			ret;
@@ -1308,7 +1308,7 @@ int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
 
 	iov_iter_kvec(&iiter, WRITE, &kvec, 1, length);
 
-	ret = iomap_file_buffered_write(&iocb, &iiter, ops, write_ops, NULL);
+	ret = iomap_file_buffered_write(&iocb, &iiter, next, write_ops, NULL);
 	if (ret < 0)
 		return ret;
 	return ret == length ? 0 : -EIO;
@@ -1580,7 +1580,7 @@ static int iomap_unshare_iter(struct iomap_iter *iter,
 
 int
 iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
-		const struct iomap_ops *ops,
+		iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops)
 {
 	struct iomap_iter iter = {
@@ -1595,7 +1595,7 @@ iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
 		return 0;
 
 	iter.len = min(len, size - pos);
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_unshare_iter(&iter, write_ops);
 	return ret;
 }
@@ -1704,7 +1704,7 @@ EXPORT_SYMBOL_GPL(iomap_fill_dirty_folios);
 
 int
 iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
-		const struct iomap_ops *ops,
+		iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops, void *private)
 {
 	struct folio_batch fbatch;
@@ -1729,7 +1729,7 @@ iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
 	 */
 	range_dirty = filemap_range_needs_writeback(mapping, iter.pos,
 					iter.pos + iter.len - 1);
-	while ((ret = iomap_iter(&iter, ops)) > 0) {
+	while ((ret = iomap_iter(&iter, next)) > 0) {
 		const struct iomap *srcmap = iomap_iter_srcmap(&iter);
 
 		if (!(iter.iomap.flags & IOMAP_F_FOLIO_BATCH) &&
@@ -1755,7 +1755,7 @@ EXPORT_SYMBOL_GPL(iomap_zero_range);
 
 int
 iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
-		const struct iomap_ops *ops,
+		iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops, void *private)
 {
 	unsigned int blocksize = i_blocksize(inode);
@@ -1764,7 +1764,7 @@ iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
 	/* Block boundary? Nothing to do */
 	if (!off)
 		return 0;
-	return iomap_zero_range(inode, pos, blocksize - off, did_zero, ops,
+	return iomap_zero_range(inode, pos, blocksize - off, did_zero, next,
 			write_ops, private);
 }
 EXPORT_SYMBOL_GPL(iomap_truncate_page);
@@ -1789,7 +1789,7 @@ static int iomap_folio_mkwrite_iter(struct iomap_iter *iter,
 	return iomap_iter_advance(iter, length);
 }
 
-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_iter_next_fn next,
 		void *private)
 {
 	struct iomap_iter iter = {
@@ -1806,7 +1806,7 @@ vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
 		goto out_unlock;
 	iter.pos = folio_pos(folio);
 	iter.len = ret;
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_folio_mkwrite_iter(&iter, folio);
 
 	if (ret < 0)
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index a47369e6b4e1..9c50adc93cfa 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -682,7 +682,7 @@ static int iomap_dio_iter(struct iomap_iter *iter, struct iomap_dio *dio)
  */
 struct iomap_dio *
 __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
-		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+		iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags, void *private, size_t done_before)
 {
 	struct inode *inode = file_inode(iocb->ki_filp);
@@ -806,7 +806,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	inode_dio_begin(inode);
 
 	blk_start_plug(&plug);
-	while ((ret = iomap_iter(&iomi, ops)) > 0) {
+	while ((ret = iomap_iter(&iomi, next)) > 0) {
 		iomi.status = iomap_dio_iter(&iomi, dio);
 
 		/*
@@ -1047,8 +1047,7 @@ iomap_dio_simple_finish(struct iomap_iter *iomi, iomap_iter_next_fn next,
  */
 static ssize_t
 iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
-		 const struct iomap_ops *ops, void *private,
-		 unsigned int dio_flags)
+		 iomap_iter_next_fn next, void *private, unsigned int dio_flags)
 {
 	struct inode *inode = file_inode(iocb->ki_filp);
 	size_t count = iov_iter_count(iter);
@@ -1074,7 +1073,7 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 
 	inode_dio_begin(inode);
 
-	ret = ops->iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
+	ret = next(&iomi, &iomi.iomap, &iomi.srcmap);
 	if (ret <= 0) {
 		inode_dio_end(inode);
 		/* the first iteration must yield a mapping (>0) or an error */
@@ -1146,7 +1145,7 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 	 * iomap_next() call on the hot read path.
 	 */
 	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
-		iomap_dio_simple_finish(&iomi, ops->iomap_next, count);
+		iomap_dio_simple_finish(&iomi, next, count);
 
 	if (!wait_for_completion) {
 		bio->bi_end_io = iomap_dio_simple_end_io;
@@ -1164,27 +1163,27 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 	bio_put(bio);
 out_iomap_end:
 	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
-		iomap_dio_simple_finish(&iomi, ops->iomap_next, 0);
+		iomap_dio_simple_finish(&iomi, next, 0);
 	inode_dio_end(inode);
 	return ret;
 }
 
 ssize_t
-iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
-		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
-		unsigned int dio_flags, void *private, size_t done_before)
+iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_iter_next_fn next,
+		const struct iomap_dio_ops *dops, unsigned int dio_flags,
+		void *private, size_t done_before)
 {
 	struct iomap_dio *dio;
 	ssize_t ret;
 
 	if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
 				       done_before)) {
-		ret = iomap_dio_simple(iocb, iter, ops, private, dio_flags);
+		ret = iomap_dio_simple(iocb, iter, next, private, dio_flags);
 		if (ret != -ENOTBLK)
 			return ret;
 	}
 
-	dio = __iomap_dio_rw(iocb, iter, ops, dops, dio_flags, private,
+	dio = __iomap_dio_rw(iocb, iter, next, dops, dio_flags, private,
 			     done_before);
 	if (IS_ERR_OR_NULL(dio))
 		return PTR_ERR_OR_ZERO(dio);
diff --git a/fs/iomap/fiemap.c b/fs/iomap/fiemap.c
index d11dadff8286..c79c13cc617b 100644
--- a/fs/iomap/fiemap.c
+++ b/fs/iomap/fiemap.c
@@ -56,7 +56,7 @@ static int iomap_fiemap_iter(struct iomap_iter *iter,
 }
 
 int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,
-		u64 start, u64 len, const struct iomap_ops *ops)
+		u64 start, u64 len, iomap_iter_next_fn next)
 {
 	struct iomap_iter iter = {
 		.inode		= inode,
@@ -73,7 +73,7 @@ int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,
 	if (ret)
 		return ret;
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_fiemap_iter(&iter, fi, &prev);
 
 	if (prev.type != IOMAP_HOLE) {
@@ -92,7 +92,7 @@ EXPORT_SYMBOL_GPL(iomap_fiemap);
 /* legacy ->bmap interface.  0 is the error return (!) */
 sector_t
 iomap_bmap(struct address_space *mapping, sector_t bno,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct iomap_iter iter = {
 		.inode	= mapping->host,
@@ -107,7 +107,7 @@ iomap_bmap(struct address_space *mapping, sector_t bno,
 		return 0;
 
 	bno = 0;
-	while ((ret = iomap_iter(&iter, ops)) > 0) {
+	while ((ret = iomap_iter(&iter, next)) > 0) {
 		if (iter.iomap.type == IOMAP_MAPPED)
 			bno = iomap_sector(&iter.iomap, iter.pos) >> blkshift;
 		/* leave iter.status unset to abort loop */
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index f7c4d5cf56be..89071b0a8b9c 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -89,7 +89,7 @@ EXPORT_SYMBOL_GPL(iomap_iter_continue);
 /**
  * iomap_iter - iterate over ranges in a file
  * @iter: iteration structure
- * @ops: iomap ops provided by the filesystem
+ * @next: the iomap iteration function for the filesystem
  *
  * Iterate over filesystem-provided space mappings for the provided file range.
  *
@@ -101,14 +101,13 @@ EXPORT_SYMBOL_GPL(iomap_iter_continue);
  * of the loop body:  leave @iter.status unchanged, or set it to a negative
  * errno.
  */
-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
+int iomap_iter(struct iomap_iter *iter, iomap_iter_next_fn next)
 {
 	int ret;
 
-	trace_iomap_iter(iter, ops, _RET_IP_);
-
-	ret = ops->iomap_next(iter, &iter->iomap, &iter->srcmap);
+	trace_iomap_iter(iter, next, _RET_IP_);
 
+	ret = next(iter, &iter->iomap, &iter->srcmap);
 	iter->status = 0;
 	if (ret > 0)
 		iomap_iter_done(iter);
diff --git a/fs/iomap/seek.c b/fs/iomap/seek.c
index 6cbc587c93da..6fd3173d25f5 100644
--- a/fs/iomap/seek.c
+++ b/fs/iomap/seek.c
@@ -27,7 +27,7 @@ static int iomap_seek_hole_iter(struct iomap_iter *iter,
 }
 
 loff_t
-iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
+iomap_seek_hole(struct inode *inode, loff_t pos, iomap_iter_next_fn next)
 {
 	loff_t size = i_size_read(inode);
 	struct iomap_iter iter = {
@@ -42,7 +42,7 @@ iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
 		return -ENXIO;
 
 	iter.len = size - pos;
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_seek_hole_iter(&iter, &pos);
 	if (ret < 0)
 		return ret;
@@ -73,7 +73,7 @@ static int iomap_seek_data_iter(struct iomap_iter *iter,
 }
 
 loff_t
-iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
+iomap_seek_data(struct inode *inode, loff_t pos, iomap_iter_next_fn next)
 {
 	loff_t size = i_size_read(inode);
 	struct iomap_iter iter = {
@@ -88,7 +88,7 @@ iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
 		return -ENXIO;
 
 	iter.len = size - pos;
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_seek_data_iter(&iter, &pos);
 	if (ret < 0)
 		return ret;
diff --git a/fs/iomap/swapfile.c b/fs/iomap/swapfile.c
index 0db77c449467..c59f67a7de09 100644
--- a/fs/iomap/swapfile.c
+++ b/fs/iomap/swapfile.c
@@ -139,7 +139,7 @@ static int iomap_swapfile_iter(struct iomap_iter *iter,
  */
 int iomap_swapfile_activate(struct swap_info_struct *sis,
 		struct file *swap_file, sector_t *pagespan,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct inode *inode = swap_file->f_mapping->host;
 	struct iomap_iter iter = {
@@ -163,7 +163,7 @@ int iomap_swapfile_activate(struct swap_info_struct *sis,
 	if (ret)
 		return ret;
 
-	while ((ret = iomap_iter(&iter, ops)) > 0)
+	while ((ret = iomap_iter(&iter, next)) > 0)
 		iter.status = iomap_swapfile_iter(&iter, &iter.iomap, &isi);
 	if (ret < 0)
 		return ret;
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e43ca419b13a..012c073c9b01 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -219,9 +219,9 @@ TRACE_EVENT(iomap_add_to_ioend,
 );
 
 TRACE_EVENT(iomap_iter,
-	TP_PROTO(struct iomap_iter *iter, const void *ops,
+	TP_PROTO(struct iomap_iter *iter, const void *next,
 		 unsigned long caller),
-	TP_ARGS(iter, ops, caller),
+	TP_ARGS(iter, next, caller),
 	TP_STRUCT__entry(
 		__field(dev_t, dev)
 		__field(u64, ino)
@@ -229,7 +229,7 @@ TRACE_EVENT(iomap_iter,
 		__field(u64, length)
 		__field(int, status)
 		__field(unsigned int, flags)
-		__field(const void *, ops)
+		__field(const void *, next)
 		__field(unsigned long, caller)
 	),
 	TP_fast_assign(
@@ -239,10 +239,10 @@ TRACE_EVENT(iomap_iter,
 		__entry->length = iomap_length(iter);
 		__entry->status = iter->status;
 		__entry->flags = iter->flags;
-		__entry->ops = ops;
+		__entry->next = next;
 		__entry->caller = caller;
 	),
-	TP_printk("dev %d:%d ino 0x%llx pos 0x%llx length 0x%llx status %d flags %s (0x%x) ops %ps caller %pS",
+	TP_printk("dev %d:%d ino 0x%llx pos 0x%llx length 0x%llx status %d flags %s (0x%x) next %ps caller %pS",
 		  MAJOR(__entry->dev), MINOR(__entry->dev),
 		   __entry->ino,
 		   __entry->pos,
@@ -250,7 +250,7 @@ TRACE_EVENT(iomap_iter,
 		   __entry->status,
 		   __print_flags(__entry->flags, "|", IOMAP_FLAGS_STRINGS),
 		   __entry->flags,
-		   __entry->ops,
+		   __entry->next,
 		   (void *)__entry->caller)
 );
 
diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c
index 1fbf832ad165..43ad597ed491 100644
--- a/fs/ntfs/aops.c
+++ b/fs/ntfs/aops.c
@@ -97,7 +97,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)
 			return ntfs_read_compressed_block(folio);
 	}
 
-	iomap_read_folio(&ntfs_read_iomap_ops, &ctx, NULL);
+	iomap_read_folio(ntfs_read_iomap_next, &ctx, NULL);
 	return 0;
 }
 
@@ -238,7 +238,7 @@ static void ntfs_readahead(struct readahead_control *rac)
 	 */
 	if (!NInoNonResident(ni) || NInoCompressed(ni))
 		return;
-	iomap_readahead(&ntfs_read_iomap_ops, &ctx, NULL);
+	iomap_readahead(ntfs_read_iomap_next, &ctx, NULL);
 }
 
 static int ntfs_writepages(struct address_space *mapping,
@@ -274,7 +274,7 @@ static int ntfs_swap_activate(struct swap_info_struct *sis,
 		struct file *swap_file, sector_t *span)
 {
 	return iomap_swapfile_activate(sis, swap_file, span,
-			&ntfs_read_iomap_ops);
+			ntfs_read_iomap_next);
 }
 
 const struct address_space_operations ntfs_aops = {
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index 2e72069c593a..24c621ac984a 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -281,7 +281,7 @@ static int ntfs_setattr_size(struct inode *vi, struct iattr *attr)
 				round_up(old_size, PAGE_SIZE) - old_size,
 				attr->ia_size - old_size);
 		err = iomap_zero_range(vi, old_size, len,
-				NULL, &ntfs_seek_iomap_ops,
+				NULL, ntfs_seek_iomap_next,
 				&ntfs_iomap_folio_ops, NULL);
 	}
 
@@ -417,12 +417,12 @@ static loff_t ntfs_file_llseek(struct file *file, loff_t offset, int whence)
 	switch (whence) {
 	case SEEK_HOLE:
 		inode_lock_shared(inode);
-		offset = iomap_seek_hole(inode, offset, &ntfs_seek_iomap_ops);
+		offset = iomap_seek_hole(inode, offset, ntfs_seek_iomap_next);
 		inode_unlock_shared(inode);
 		break;
 	case SEEK_DATA:
 		inode_lock_shared(inode);
-		offset = iomap_seek_data(inode, offset, &ntfs_seek_iomap_ops);
+		offset = iomap_seek_data(inode, offset, ntfs_seek_iomap_next);
 		inode_unlock_shared(inode);
 		break;
 	default:
@@ -458,7 +458,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		}
 
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, to, &ntfs_read_iomap_ops, NULL,
+		ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL,
 				IOMAP_DIO_NO_IOMAP_END, NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, to);
@@ -496,7 +496,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 {
 	ssize_t ret;
 
-	ret = iomap_dio_rw(iocb, from, &ntfs_dio_iomap_ops,
+	ret = iomap_dio_rw(iocb, from, ntfs_dio_iomap_next,
 			&ntfs_write_dio_ops, 0, NULL, 0);
 	if (ret == -ENOTBLK)
 		ret = 0;
@@ -511,7 +511,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		offset = iocb->ki_pos;
 		iocb->ki_flags &= ~IOCB_DIRECT;
 		written = iomap_file_buffered_write(iocb, from,
-				&ntfs_write_iomap_ops, &ntfs_iomap_folio_ops,
+				ntfs_write_iomap_next, &ntfs_iomap_folio_ops,
 				NULL);
 		if (written < 0) {
 			ret = written;
@@ -594,7 +594,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (NInoNonResident(ni) && iocb->ki_flags & IOCB_DIRECT)
 		ret = ntfs_dio_write_iter(iocb, from);
 	else
-		ret = iomap_file_buffered_write(iocb, from, &ntfs_write_iomap_ops,
+		ret = iomap_file_buffered_write(iocb, from, ntfs_write_iomap_next,
 				&ntfs_iomap_folio_ops, NULL);
 out:
 	if (ret < 0 && ret != -EIOCBQUEUED) {
@@ -623,7 +623,7 @@ static vm_fault_t ntfs_filemap_page_mkwrite(struct vm_fault *vmf)
 	sb_start_pagefault(inode->i_sb);
 	file_update_time(vmf->vma->vm_file);
 
-	ret = iomap_page_mkwrite(vmf, &ntfs_page_mkwrite_iomap_ops, NULL);
+	ret = iomap_page_mkwrite(vmf, ntfs_page_mkwrite_iomap_next, NULL);
 	sb_end_pagefault(inode->i_sb);
 	return ret;
 }
@@ -670,7 +670,7 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc)
 static int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 		u64 start, u64 len)
 {
-	return iomap_fiemap(inode, fieinfo, start, len, &ntfs_read_iomap_ops);
+	return iomap_fiemap(inode, fieinfo, start, len, ntfs_read_iomap_next);
 }
 
 static const char *ntfs_get_link(struct dentry *dentry, struct inode *inode,
@@ -911,7 +911,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,
 				   ntfs_cluster_to_bytes(vol, start_vcn + 1),
 				   end_offset);
 			err = iomap_zero_range(vi, offset, to - offset,
-					       NULL, &ntfs_seek_iomap_ops,
+					       NULL, ntfs_seek_iomap_next,
 					       &ntfs_iomap_folio_ops, NULL);
 			if (err < 0)
 				goto out;
@@ -927,7 +927,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,
 		from = ntfs_cluster_to_bytes(vol, end_vcn - 1);
 		if (from < ni->initialized_size) {
 			err = iomap_zero_range(vi, from, end_offset - from,
-					       NULL, &ntfs_seek_iomap_ops,
+					       NULL, ntfs_seek_iomap_next,
 					       &ntfs_iomap_folio_ops, NULL);
 			if (err < 0)
 				goto out;
@@ -1131,7 +1131,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t offset, loff_t le
 					   round_up(old_size, PAGE_SIZE) - old_size,
 					   offset - old_size);
 			err = iomap_zero_range(vi, old_size, len, NULL,
-					       &ntfs_seek_iomap_ops,
+					       ntfs_seek_iomap_next,
 					       &ntfs_iomap_folio_ops, NULL);
 		}
 		NInoSetFileNameDirty(ni);
diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c
index c2715521e562..05132d92e87b 100644
--- a/fs/ntfs/inode.c
+++ b/fs/ntfs/inode.c
@@ -2415,7 +2415,7 @@ int ntfs_extend_initialized_size(struct inode *vi, const loff_t offset,
 	if (!NInoCompressed(ni) && old_init_size < offset) {
 		err = iomap_zero_range(vi, old_init_size,
 				       offset - old_init_size,
-				       NULL, &ntfs_seek_iomap_ops,
+				       NULL, ntfs_seek_iomap_next,
 				       &ntfs_iomap_folio_ops, NULL);
 		if (err)
 			return err;
diff --git a/fs/ntfs/iomap.c b/fs/ntfs/iomap.c
index d0964ac840d9..271902e69200 100644
--- a/fs/ntfs/iomap.c
+++ b/fs/ntfs/iomap.c
@@ -277,11 +277,7 @@ static int ntfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t leng
 			srcmap, true);
 }
 
-static DEFINE_IOMAP_ITER_NEXT(ntfs_read_iomap_next, ntfs_read_iomap_begin);
-
-const struct iomap_ops ntfs_read_iomap_ops = {
-	.iomap_next = ntfs_read_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(ntfs_read_iomap_next, ntfs_read_iomap_begin);
 
 /*
  * Check that the cached iomap still matches the NTFS runlist before
@@ -334,15 +330,7 @@ static int ntfs_zero_read_iomap_end(struct inode *inode, loff_t pos, loff_t leng
 static DEFINE_IOMAP_ITER_NEXT_END(ntfs_zero_read_iomap_next,
 		ntfs_seek_iomap_begin, ntfs_zero_read_iomap_end);
 
-static const struct iomap_ops ntfs_zero_read_iomap_ops = {
-	.iomap_next = ntfs_zero_read_iomap_next,
-};
-
-static DEFINE_IOMAP_ITER_NEXT(ntfs_seek_iomap_next, ntfs_seek_iomap_begin);
-
-const struct iomap_ops ntfs_seek_iomap_ops = {
-	.iomap_next = ntfs_seek_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(ntfs_seek_iomap_next, ntfs_seek_iomap_begin);
 
 int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length)
 {
@@ -361,7 +349,7 @@ static int ntfs_zero_range(struct inode *inode, loff_t offset, loff_t length)
 	return iomap_zero_range(inode,
 				offset, length,
 				NULL,
-				&ntfs_zero_read_iomap_ops,
+				ntfs_zero_read_iomap_next,
 				&ntfs_zero_iomap_folio_ops,
 				NULL);
 }
@@ -770,12 +758,8 @@ static int ntfs_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
 	return written;
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(ntfs_write_iomap_next,
-		ntfs_write_iomap_begin, ntfs_write_iomap_end);
-
-const struct iomap_ops ntfs_write_iomap_ops = {
-	.iomap_next		= ntfs_write_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_write_iomap_next, ntfs_write_iomap_begin,
+		ntfs_write_iomap_end);
 
 static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
 				  loff_t length, unsigned int flags,
@@ -785,13 +769,9 @@ static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
 			NTFS_IOMAP_FLAGS_MKWRITE);
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(ntfs_page_mkwrite_iomap_next,
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_page_mkwrite_iomap_next,
 		ntfs_page_mkwrite_iomap_begin, ntfs_write_iomap_end);
 
-const struct iomap_ops ntfs_page_mkwrite_iomap_ops = {
-	.iomap_next		= ntfs_page_mkwrite_iomap_next,
-};
-
 static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
 				  loff_t length, unsigned int flags,
 				  struct iomap *iomap, struct iomap *srcmap)
@@ -800,12 +780,8 @@ static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
 			NTFS_IOMAP_FLAGS_DIO);
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(ntfs_dio_iomap_next,
-		ntfs_dio_iomap_begin, ntfs_write_iomap_end);
-
-const struct iomap_ops ntfs_dio_iomap_ops = {
-	.iomap_next		= ntfs_dio_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_dio_iomap_next, ntfs_dio_iomap_begin,
+		ntfs_write_iomap_end);
 
 static ssize_t ntfs_writeback_range(struct iomap_writepage_ctx *wpc,
 		struct folio *folio, u64 offset, unsigned int len, u64 end_pos)
diff --git a/fs/ntfs/iomap.h b/fs/ntfs/iomap.h
index 3abc1d493e91..69443de1fefd 100644
--- a/fs/ntfs/iomap.h
+++ b/fs/ntfs/iomap.h
@@ -12,11 +12,16 @@
 #include "volume.h"
 #include "inode.h"
 
-extern const struct iomap_ops ntfs_write_iomap_ops;
-extern const struct iomap_ops ntfs_read_iomap_ops;
-extern const struct iomap_ops ntfs_seek_iomap_ops;
-extern const struct iomap_ops ntfs_page_mkwrite_iomap_ops;
-extern const struct iomap_ops ntfs_dio_iomap_ops;
+int ntfs_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int ntfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int ntfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int ntfs_page_mkwrite_iomap_next(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap);
+int ntfs_dio_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
 extern const struct iomap_writeback_ops ntfs_writeback_ops;
 extern const struct iomap_write_ops ntfs_iomap_folio_ops;
 extern int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length);
diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c
index d601f088618c..55844b42920a 100644
--- a/fs/ntfs3/file.c
+++ b/fs/ntfs3/file.c
@@ -315,7 +315,7 @@ static int ntfs_extend_initialized_size(struct file *file,
 	}
 
 	err = iomap_zero_range(inode, valid, new_valid - valid, NULL,
-			       &ntfs_iomap_ops, &ntfs_iomap_folio_ops, NULL);
+			       ntfs_iomap_next, &ntfs_iomap_folio_ops, NULL);
 	if (err) {
 		ni->i_valid = valid;
 		ntfs_inode_warn(inode,
@@ -554,7 +554,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)
 		/* Zero head of punch. */
 		if (tmp > from) {
 			err = iomap_zero_range(inode, from, tmp - from, NULL,
-					       &ntfs_iomap_ops,
+					       ntfs_iomap_next,
 					       &ntfs_iomap_folio_ops, NULL);
 			if (err)
 				goto out;
@@ -572,7 +572,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)
 		/* Zero tail of punch. */
 		if (vbo < end_a && end_a < end) {
 			err = iomap_zero_range(inode, end_a, end - end_a, NULL,
-					       &ntfs_iomap_ops,
+					       ntfs_iomap_next,
 					       &ntfs_iomap_folio_ops, NULL);
 			if (err)
 				goto out;
@@ -872,7 +872,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 				goto out;
 		}
 
-		err = iomap_dio_rw(iocb, iter, &ntfs_iomap_ops, NULL, dio_flags,
+		err = iomap_dio_rw(iocb, iter, ntfs_iomap_next, NULL, dio_flags,
 				   NULL, 0);
 
 		if (err <= 0)
@@ -1286,7 +1286,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	    !ntfs_should_use_dio(iocb, from)) {
 		iocb->ki_flags &= ~IOCB_DIRECT;
 
-		ret = iomap_file_buffered_write(iocb, from, &ntfs_iomap_ops,
+		ret = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,
 						&ntfs_iomap_folio_ops, NULL);
 		inode_unlock(inode);
 
@@ -1303,7 +1303,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 			goto out;
 	}
 
-	ret = iomap_dio_rw(iocb, from, &ntfs_iomap_ops, NULL,
+	ret = iomap_dio_rw(iocb, from, ntfs_iomap_next, NULL,
 			   IOMAP_DIO_FORCE_WAIT, NULL, 0);
 
 	if (ret == -ENOTBLK) {
@@ -1316,7 +1316,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		vbo = iocb->ki_pos;
 
 		iocb->ki_flags &= ~IOCB_DIRECT;
-		err = iomap_file_buffered_write(iocb, from, &ntfs_iomap_ops,
+		err = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,
 						&ntfs_iomap_folio_ops, NULL);
 		if (err < 0) {
 			ret = err;
@@ -1465,7 +1465,7 @@ int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 
 	inode_lock_shared(inode);
 
-	err = iomap_fiemap(inode, fieinfo, start, len, &ntfs_iomap_ops);
+	err = iomap_fiemap(inode, fieinfo, start, len, ntfs_iomap_next);
 
 	inode_unlock_shared(inode);
 	return err;
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index 53031e71c8fc..edd4fa499a66 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -576,7 +576,7 @@ static sector_t ntfs_bmap(struct address_space *mapping, sector_t block)
 		ni_allocate_da_blocks(ni);
 	}
 
-	return iomap_bmap(mapping, block, &ntfs_iomap_ops);
+	return iomap_bmap(mapping, block, ntfs_iomap_next);
 }
 
 static void ntfs_iomap_read_end_io(struct bio *bio)
@@ -649,7 +649,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)
 		return err;
 	}
 
-	iomap_read_folio(&ntfs_iomap_ops, &ctx, NULL);
+	iomap_read_folio(ntfs_iomap_next, &ctx, NULL);
 	return 0;
 }
 
@@ -673,7 +673,7 @@ static void ntfs_readahead(struct readahead_control *rac)
 		return;
 	}
 
-	iomap_readahead(&ntfs_iomap_ops, &ctx, NULL);
+	iomap_readahead(ntfs_iomap_next, &ctx, NULL);
 }
 
 int ntfs_set_size(struct inode *inode, u64 new_size)
@@ -2101,12 +2101,7 @@ const struct address_space_operations ntfs_aops_cmpr = {
 	.invalidate_folio = iomap_invalidate_folio,
 };
 
-static DEFINE_IOMAP_ITER_NEXT_END(ntfs_iomap_next, ntfs_iomap_begin,
-				  ntfs_iomap_end);
-
-const struct iomap_ops ntfs_iomap_ops = {
-	.iomap_next	= ntfs_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT_END(ntfs_iomap_next, ntfs_iomap_begin, ntfs_iomap_end);
 
 const struct iomap_write_ops ntfs_iomap_folio_ops = {
 	.put_folio = ntfs_iomap_put_folio,
diff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h
index d98d7e474476..e00dae3ce700 100644
--- a/fs/ntfs3/ntfs_fs.h
+++ b/fs/ntfs3/ntfs_fs.h
@@ -785,7 +785,8 @@ int ntfs_create_inode(struct mnt_idmap *idmap, struct inode *dir,
 int ntfs_link_inode(struct inode *inode, struct dentry *dentry);
 int ntfs_unlink_inode(struct inode *dir, const struct dentry *dentry);
 void ntfs_evict_inode(struct inode *inode);
-extern const struct iomap_ops ntfs_iomap_ops;
+int ntfs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		    struct iomap *srcmap);
 extern const struct iomap_write_ops ntfs_iomap_folio_ops;
 extern const struct inode_operations ntfs_link_inode_operations;
 extern const struct address_space_operations ntfs_aops;
diff --git a/fs/remap_range.c b/fs/remap_range.c
index 26afbbbfb10c..eab94d42654d 100644
--- a/fs/remap_range.c
+++ b/fs/remap_range.c
@@ -277,7 +277,7 @@ int
 __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
 				struct file *file_out, loff_t pos_out,
 				loff_t *len, unsigned int remap_flags,
-				const struct iomap_ops *dax_read_ops)
+				iomap_iter_next_fn dax_read_next)
 {
 	struct inode *inode_in = file_inode(file_in);
 	struct inode *inode_out = file_inode(file_out);
@@ -340,10 +340,10 @@ __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
 		if (!IS_DAX(inode_in))
 			ret = vfs_dedupe_file_range_compare(file_in, pos_in,
 					file_out, pos_out, *len, &is_same);
-		else if (dax_read_ops)
+		else if (dax_read_next)
 			ret = dax_dedupe_file_range_compare(inode_in, pos_in,
 					inode_out, pos_out, *len, &is_same,
-					dax_read_ops);
+					dax_read_next);
 		else
 			return -EINVAL;
 		if (ret)
diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 2a0c54256e93..91480cb6a4d8 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -752,7 +752,7 @@ xfs_vm_bmap(
 	 */
 	if (xfs_is_cow_inode(ip) || XFS_IS_REALTIME_INODE(ip))
 		return 0;
-	return iomap_bmap(mapping, block, &xfs_read_iomap_ops);
+	return iomap_bmap(mapping, block, xfs_read_iomap_next);
 }
 
 static void
@@ -793,7 +793,7 @@ xfs_vm_read_folio(
 	struct iomap_read_folio_ctx	ctx = { .cur_folio = folio };
 
 	ctx.ops = xfs_get_iomap_read_ops(folio->mapping);
-	iomap_read_folio(&xfs_read_iomap_ops, &ctx, NULL);
+	iomap_read_folio(xfs_read_iomap_next, &ctx, NULL);
 	return 0;
 }
 
@@ -804,7 +804,7 @@ xfs_vm_readahead(
 	struct iomap_read_folio_ctx	ctx = { .rac = rac };
 
 	ctx.ops = xfs_get_iomap_read_ops(rac->mapping),
-	iomap_readahead(&xfs_read_iomap_ops, &ctx, NULL);
+	iomap_readahead(xfs_read_iomap_next, &ctx, NULL);
 }
 
 static int
@@ -850,7 +850,7 @@ xfs_vm_swap_activate(
 	sis->bdev = xfs_inode_buftarg(ip)->bt_bdev;
 
 	return iomap_swapfile_activate(sis, swap_file, span,
-			&xfs_read_iomap_ops);
+			xfs_read_iomap_next);
 }
 
 const struct address_space_operations xfs_address_space_operations = {
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 3f7cbe117fb6..ea709e4c9607 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -269,7 +269,7 @@ xfs_file_dio_read(
 		dio_ops = &xfs_dio_read_bounce_ops;
 		dio_flags |= IOMAP_DIO_BOUNCE;
 	}
-	ret = iomap_dio_rw(iocb, to, &xfs_read_iomap_ops, dio_ops, dio_flags,
+	ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, dio_ops, dio_flags,
 			NULL, 0);
 	xfs_iunlock(ip, XFS_IOLOCK_SHARED);
 
@@ -292,7 +292,7 @@ xfs_file_dax_read(
 	ret = xfs_ilock_iocb(iocb, XFS_IOLOCK_SHARED);
 	if (ret)
 		return ret;
-	ret = dax_iomap_rw(iocb, to, &xfs_read_iomap_ops);
+	ret = dax_iomap_rw(iocb, to, xfs_read_iomap_next);
 	xfs_iunlock(ip, XFS_IOLOCK_SHARED);
 
 	file_accessed(iocb->ki_filp);
@@ -742,7 +742,7 @@ xfs_file_dio_write_aligned(
 	struct xfs_inode	*ip,
 	struct kiocb		*iocb,
 	struct iov_iter		*from,
-	const struct iomap_ops	*ops,
+	iomap_iter_next_fn	next,
 	const struct iomap_dio_ops *dops,
 	struct xfs_zone_alloc_ctx *ac)
 {
@@ -777,7 +777,7 @@ xfs_file_dio_write_aligned(
 	if (mapping_stable_writes(iocb->ki_filp->f_mapping))
 		dio_flags |= IOMAP_DIO_BOUNCE;
 	trace_xfs_file_direct_write(iocb, from);
-	ret = iomap_dio_rw(iocb, from, ops, dops, dio_flags, ac, 0);
+	ret = iomap_dio_rw(iocb, from, next, dops, dio_flags, ac, 0);
 out_unlock:
 	xfs_iunlock(ip, iolock);
 	return ret;
@@ -799,7 +799,7 @@ xfs_file_dio_write_zoned(
 	if (ret < 0)
 		return ret;
 	ret = xfs_file_dio_write_aligned(ip, iocb, from,
-			&xfs_zoned_direct_write_iomap_ops,
+			xfs_zoned_direct_write_iomap_next,
 			&xfs_dio_zoned_write_ops, &ac);
 	xfs_zoned_space_unreserve(ip->i_mount, &ac);
 	return ret;
@@ -824,16 +824,16 @@ xfs_file_dio_write_atomic(
 	unsigned int		iolock = XFS_IOLOCK_SHARED;
 	ssize_t			ret, ocount = iov_iter_count(from);
 	unsigned int		dio_flags = 0;
-	const struct iomap_ops	*dops;
+	iomap_iter_next_fn	next;
 
 	/*
 	 * HW offload should be faster, so try that first if it is already
 	 * known that the write length is not too large.
 	 */
 	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
-		dops = &xfs_atomic_write_cow_iomap_ops;
+		next = xfs_atomic_write_cow_iomap_next;
 	else
-		dops = &xfs_direct_write_iomap_ops;
+		next = xfs_direct_write_iomap_next;
 
 retry:
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
@@ -853,7 +853,7 @@ xfs_file_dio_write_atomic(
 	trace_xfs_file_direct_write(iocb, from);
 	if (mapping_stable_writes(iocb->ki_filp->f_mapping))
 		dio_flags |= IOMAP_DIO_BOUNCE;
-	ret = iomap_dio_rw(iocb, from, dops, &xfs_dio_write_ops, dio_flags,
+	ret = iomap_dio_rw(iocb, from, next, &xfs_dio_write_ops, dio_flags,
 			NULL, 0);
 
 	/*
@@ -862,9 +862,9 @@ xfs_file_dio_write_atomic(
 	 * possible. The REQ_ATOMIC-based method is typically not possible if
 	 * the write spans multiple extents or the disk blocks are misaligned.
 	 */
-	if (ret == -ENOPROTOOPT && dops == &xfs_direct_write_iomap_ops) {
+	if (ret == -ENOPROTOOPT && next == xfs_direct_write_iomap_next) {
 		xfs_iunlock(ip, iolock);
-		dops = &xfs_atomic_write_cow_iomap_ops;
+		next = xfs_atomic_write_cow_iomap_next;
 		goto retry;
 	}
 
@@ -947,7 +947,7 @@ xfs_file_dio_write_unaligned(
 		flags |= IOMAP_DIO_BOUNCE;
 
 	trace_xfs_file_direct_write(iocb, from);
-	ret = iomap_dio_rw(iocb, from, &xfs_direct_write_iomap_ops,
+	ret = iomap_dio_rw(iocb, from, xfs_direct_write_iomap_next,
 			   &xfs_dio_write_ops, flags, NULL, 0);
 
 	/*
@@ -987,7 +987,7 @@ xfs_file_dio_write(
 	if (iocb->ki_flags & IOCB_ATOMIC)
 		return xfs_file_dio_write_atomic(ip, iocb, from);
 	return xfs_file_dio_write_aligned(ip, iocb, from,
-			&xfs_direct_write_iomap_ops, &xfs_dio_write_ops, NULL);
+			xfs_direct_write_iomap_next, &xfs_dio_write_ops, NULL);
 }
 
 static noinline ssize_t
@@ -1011,7 +1011,7 @@ xfs_file_dax_write(
 	pos = iocb->ki_pos;
 
 	trace_xfs_file_dax_write(iocb, from);
-	ret = dax_iomap_rw(iocb, from, &xfs_dax_write_iomap_ops);
+	ret = dax_iomap_rw(iocb, from, xfs_dax_write_iomap_next);
 	if (ret > 0 && iocb->ki_pos > i_size_read(inode)) {
 		i_size_write(inode, iocb->ki_pos);
 		error = xfs_setfilesize(ip, pos, ret);
@@ -1054,7 +1054,7 @@ xfs_file_buffered_write(
 
 	trace_xfs_file_buffered_write(iocb, from);
 	ret = iomap_file_buffered_write(iocb, from,
-			&xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+			xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
 			NULL);
 
 	/*
@@ -1135,7 +1135,7 @@ xfs_file_buffered_write_zoned(
 retry:
 	trace_xfs_file_buffered_write(iocb, from);
 	ret = iomap_file_buffered_write(iocb, from,
-			&xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+			xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
 			&ac);
 	if (ret == -ENOSPC && !cleared_space) {
 		/*
@@ -1856,10 +1856,10 @@ xfs_file_llseek(
 	default:
 		return generic_file_llseek(file, offset, whence);
 	case SEEK_HOLE:
-		offset = iomap_seek_hole(inode, offset, &xfs_seek_iomap_ops);
+		offset = iomap_seek_hole(inode, offset, xfs_seek_iomap_next);
 		break;
 	case SEEK_DATA:
-		offset = iomap_seek_data(inode, offset, &xfs_seek_iomap_ops);
+		offset = iomap_seek_data(inode, offset, xfs_seek_iomap_next);
 		break;
 	}
 
@@ -1883,8 +1883,8 @@ xfs_dax_fault_locked(
 	}
 	ret = dax_iomap_fault(vmf, order, &pfn, NULL,
 			(write_fault && !vmf->cow_page) ?
-				&xfs_dax_write_iomap_ops :
-				&xfs_read_iomap_ops);
+				xfs_dax_write_iomap_next :
+				xfs_read_iomap_next);
 	if (ret & VM_FAULT_NEEDDSYNC)
 		ret = dax_finish_sync_fault(vmf, order, pfn);
 	return ret;
@@ -1948,7 +1948,7 @@ __xfs_write_fault(
 	if (IS_DAX(inode))
 		ret = xfs_dax_fault_locked(vmf, order, true);
 	else
-		ret = iomap_page_mkwrite(vmf, &xfs_buffered_write_iomap_ops,
+		ret = iomap_page_mkwrite(vmf, xfs_buffered_write_iomap_next,
 				ac);
 	xfs_iunlock(ip, lock_mode);
 
diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
index d3e28202db05..1314c95f7416 100644
--- a/fs/xfs/xfs_iomap.c
+++ b/fs/xfs/xfs_iomap.c
@@ -1037,13 +1037,9 @@ xfs_direct_write_iomap_begin(
 	return error;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(xfs_direct_write_iomap_next,
+DEFINE_IOMAP_ITER_NEXT(xfs_direct_write_iomap_next,
 		xfs_direct_write_iomap_begin);
 
-const struct iomap_ops xfs_direct_write_iomap_ops = {
-	.iomap_next		= xfs_direct_write_iomap_next,
-};
-
 #ifdef CONFIG_XFS_RT
 /*
  * This is really simple.  The space has already been reserved before taking the
@@ -1092,12 +1088,9 @@ xfs_zoned_direct_write_iomap_begin(
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(xfs_zoned_direct_write_iomap_next,
+DEFINE_IOMAP_ITER_NEXT(xfs_zoned_direct_write_iomap_next,
 		xfs_zoned_direct_write_iomap_begin);
 
-const struct iomap_ops xfs_zoned_direct_write_iomap_ops = {
-	.iomap_next		= xfs_zoned_direct_write_iomap_next,
-};
 #endif /* CONFIG_XFS_RT */
 
 #ifdef DEBUG
@@ -1280,13 +1273,9 @@ xfs_atomic_write_cow_iomap_begin(
 	return error;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(xfs_atomic_write_cow_iomap_next,
+DEFINE_IOMAP_ITER_NEXT(xfs_atomic_write_cow_iomap_next,
 		xfs_atomic_write_cow_iomap_begin);
 
-const struct iomap_ops xfs_atomic_write_cow_iomap_ops = {
-	.iomap_next		= xfs_atomic_write_cow_iomap_next,
-};
-
 static int
 xfs_dax_write_iomap_end(
 	struct inode		*inode,
@@ -1307,13 +1296,9 @@ xfs_dax_write_iomap_end(
 	return xfs_reflink_end_cow(ip, pos, written);
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(xfs_dax_write_iomap_next,
+DEFINE_IOMAP_ITER_NEXT_END(xfs_dax_write_iomap_next,
 		xfs_direct_write_iomap_begin, xfs_dax_write_iomap_end);
 
-const struct iomap_ops xfs_dax_write_iomap_ops = {
-	.iomap_next	= xfs_dax_write_iomap_next,
-};
-
 /*
  * Convert a hole to a delayed allocation.
  */
@@ -2179,13 +2164,9 @@ xfs_buffered_write_iomap_end(
 	return 0;
 }
 
-static DEFINE_IOMAP_ITER_NEXT_END(xfs_buffered_write_iomap_next,
+DEFINE_IOMAP_ITER_NEXT_END(xfs_buffered_write_iomap_next,
 		xfs_buffered_write_iomap_begin, xfs_buffered_write_iomap_end);
 
-const struct iomap_ops xfs_buffered_write_iomap_ops = {
-	.iomap_next		= xfs_buffered_write_iomap_next,
-};
-
 static int
 xfs_read_iomap_begin(
 	struct inode		*inode,
@@ -2227,11 +2208,7 @@ xfs_read_iomap_begin(
 				 shared ? IOMAP_F_SHARED : 0, seq);
 }
 
-static DEFINE_IOMAP_ITER_NEXT(xfs_read_iomap_next, xfs_read_iomap_begin);
-
-const struct iomap_ops xfs_read_iomap_ops = {
-	.iomap_next		= xfs_read_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_read_iomap_next, xfs_read_iomap_begin);
 
 static int
 xfs_seek_iomap_begin(
@@ -2317,11 +2294,7 @@ xfs_seek_iomap_begin(
 	return error;
 }
 
-static DEFINE_IOMAP_ITER_NEXT(xfs_seek_iomap_next, xfs_seek_iomap_begin);
-
-const struct iomap_ops xfs_seek_iomap_ops = {
-	.iomap_next		= xfs_seek_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_seek_iomap_next, xfs_seek_iomap_begin);
 
 static int
 xfs_xattr_iomap_begin(
@@ -2366,11 +2339,7 @@ xfs_xattr_iomap_begin(
 	return xfs_bmbt_to_iomap(ip, iomap, &imap, flags, IOMAP_F_XATTR, seq);
 }
 
-static DEFINE_IOMAP_ITER_NEXT(xfs_xattr_iomap_next, xfs_xattr_iomap_begin);
-
-const struct iomap_ops xfs_xattr_iomap_ops = {
-	.iomap_next		= xfs_xattr_iomap_next,
-};
+DEFINE_IOMAP_ITER_NEXT(xfs_xattr_iomap_next, xfs_xattr_iomap_begin);
 
 int
 xfs_zero_range(
@@ -2386,9 +2355,9 @@ xfs_zero_range(
 
 	if (IS_DAX(inode))
 		return dax_zero_range(inode, pos, len, did_zero,
-				      &xfs_dax_write_iomap_ops);
+				      xfs_dax_write_iomap_next);
 	return iomap_zero_range(inode, pos, len, did_zero,
-			&xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+			xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
 			ac);
 }
 
@@ -2403,8 +2372,8 @@ xfs_truncate_page(
 
 	if (IS_DAX(inode))
 		return dax_truncate_page(inode, pos, did_zero,
-					&xfs_dax_write_iomap_ops);
+					xfs_dax_write_iomap_next);
 	return iomap_truncate_page(inode, pos, did_zero,
-			&xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+			xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
 			ac);
 }
diff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h
index ebcce7d49446..01875d20fb66 100644
--- a/fs/xfs/xfs_iomap.h
+++ b/fs/xfs/xfs_iomap.h
@@ -49,14 +49,22 @@ xfs_aligned_fsb_count(
 	return count_fsb;
 }
 
-extern const struct iomap_ops xfs_buffered_write_iomap_ops;
-extern const struct iomap_ops xfs_direct_write_iomap_ops;
-extern const struct iomap_ops xfs_zoned_direct_write_iomap_ops;
-extern const struct iomap_ops xfs_read_iomap_ops;
-extern const struct iomap_ops xfs_seek_iomap_ops;
-extern const struct iomap_ops xfs_xattr_iomap_ops;
-extern const struct iomap_ops xfs_dax_write_iomap_ops;
-extern const struct iomap_ops xfs_atomic_write_cow_iomap_ops;
+int xfs_buffered_write_iomap_next(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap);
+int xfs_direct_write_iomap_next(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap);
+int xfs_zoned_direct_write_iomap_next(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap);
+int xfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int xfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int xfs_xattr_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int xfs_dax_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+		struct iomap *srcmap);
+int xfs_atomic_write_cow_iomap_next(const struct iomap_iter *iter,
+		struct iomap *iomap, struct iomap *srcmap);
 extern const struct iomap_write_ops xfs_iomap_write_ops;
 
 #endif /* __XFS_IOMAP_H__*/
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 6339f4956ecb..5c3d9a365f93 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -1239,10 +1239,10 @@ xfs_vn_fiemap(
 	if (fieinfo->fi_flags & FIEMAP_FLAG_XATTR) {
 		fieinfo->fi_flags &= ~FIEMAP_FLAG_XATTR;
 		error = iomap_fiemap(inode, fieinfo, start, length,
-				&xfs_xattr_iomap_ops);
+				xfs_xattr_iomap_next);
 	} else {
 		error = iomap_fiemap(inode, fieinfo, start, length,
-				&xfs_read_iomap_ops);
+				xfs_read_iomap_next);
 	}
 	xfs_iunlock(XFS_I(inode), XFS_IOLOCK_SHARED);
 
diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c
index a5c188b78138..2b9792626bab 100644
--- a/fs/xfs/xfs_reflink.c
+++ b/fs/xfs/xfs_reflink.c
@@ -1683,7 +1683,7 @@ xfs_reflink_remap_prep(
 				pos_out, len, remap_flags);
 	else
 		ret = dax_remap_file_range_prep(file_in, pos_in, file_out,
-				pos_out, len, remap_flags, &xfs_read_iomap_ops);
+				pos_out, len, remap_flags, xfs_read_iomap_next);
 	if (ret || *len == 0)
 		goto out_unlock;
 
@@ -1878,10 +1878,10 @@ xfs_reflink_unshare(
 
 	if (IS_DAX(inode))
 		error = dax_file_unshare(inode, offset, len,
-				&xfs_dax_write_iomap_ops);
+				xfs_dax_write_iomap_next);
 	else
 		error = iomap_file_unshare(inode, offset, len,
-				&xfs_buffered_write_iomap_ops,
+				xfs_buffered_write_iomap_next,
 				&xfs_iomap_write_ops);
 	if (error)
 		goto out;
diff --git a/fs/zonefs/file.c b/fs/zonefs/file.c
index 5b34849be7a2..319d9502ba56 100644
--- a/fs/zonefs/file.c
+++ b/fs/zonefs/file.c
@@ -59,10 +59,6 @@ static int zonefs_read_iomap_begin(struct inode *inode, loff_t offset,
 
 static DEFINE_IOMAP_ITER_NEXT(zonefs_read_iomap_next, zonefs_read_iomap_begin);
 
-static const struct iomap_ops zonefs_read_iomap_ops = {
-	.iomap_next	= zonefs_read_iomap_next,
-};
-
 static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,
 				    loff_t length, unsigned int flags,
 				    struct iomap *iomap, struct iomap *srcmap)
@@ -111,19 +107,15 @@ static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,
 static DEFINE_IOMAP_ITER_NEXT(zonefs_write_iomap_next,
 			      zonefs_write_iomap_begin);
 
-static const struct iomap_ops zonefs_write_iomap_ops = {
-	.iomap_next	= zonefs_write_iomap_next,
-};
-
 static int zonefs_read_folio(struct file *unused, struct folio *folio)
 {
-	iomap_bio_read_folio(folio, &zonefs_read_iomap_ops);
+	iomap_bio_read_folio(folio, zonefs_read_iomap_next);
 	return 0;
 }
 
 static void zonefs_readahead(struct readahead_control *rac)
 {
-	iomap_bio_readahead(rac, &zonefs_read_iomap_ops);
+	iomap_bio_readahead(rac, zonefs_read_iomap_next);
 }
 
 /*
@@ -184,7 +176,7 @@ static int zonefs_swap_activate(struct swap_info_struct *sis,
 	}
 
 	return iomap_swapfile_activate(sis, swap_file, span,
-				       &zonefs_read_iomap_ops);
+				       zonefs_read_iomap_next);
 }
 
 const struct address_space_operations zonefs_file_aops = {
@@ -314,7 +306,7 @@ static vm_fault_t zonefs_filemap_page_mkwrite(struct vm_fault *vmf)
 
 	/* Serialize against truncates */
 	filemap_invalidate_lock_shared(inode->i_mapping);
-	ret = iomap_page_mkwrite(vmf, &zonefs_write_iomap_ops, NULL);
+	ret = iomap_page_mkwrite(vmf, zonefs_write_iomap_next, NULL);
 	filemap_invalidate_unlock_shared(inode->i_mapping);
 
 	sb_end_pagefault(inode->i_sb);
@@ -530,7 +522,7 @@ static ssize_t zonefs_file_dio_write(struct kiocb *iocb, struct iov_iter *from)
 	 * page invalidation. Overwrite that error code with EBUSY so that
 	 * the user can make sense of the error.
 	 */
-	ret = iomap_dio_rw(iocb, from, &zonefs_write_iomap_ops,
+	ret = iomap_dio_rw(iocb, from, zonefs_write_iomap_next,
 			   &zonefs_write_dio_ops, 0, NULL, 0);
 	if (ret == -ENOTBLK)
 		ret = -EBUSY;
@@ -580,7 +572,7 @@ static ssize_t zonefs_file_buffered_write(struct kiocb *iocb,
 	if (ret <= 0)
 		goto inode_unlock;
 
-	ret = iomap_file_buffered_write(iocb, from, &zonefs_write_iomap_ops,
+	ret = iomap_file_buffered_write(iocb, from, zonefs_write_iomap_next,
 			NULL, NULL);
 	if (ret == -EIO)
 		zonefs_io_error(inode, true);
@@ -675,7 +667,7 @@ static ssize_t zonefs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 			goto inode_unlock;
 		}
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, to, &zonefs_read_iomap_ops,
+		ret = iomap_dio_rw(iocb, to, zonefs_read_iomap_next,
 				   &zonefs_read_dio_ops, 0, NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, to);
diff --git a/include/linux/dax.h b/include/linux/dax.h
index fe6c3ded1b50..e0bc653ffe9f 100644
--- a/include/linux/dax.h
+++ b/include/linux/dax.h
@@ -3,6 +3,7 @@
 #define _LINUX_DAX_H
 
 #include <linux/fs.h>
+#include <linux/iomap.h>
 #include <linux/mm.h>
 #include <linux/radix-tree.h>
 
@@ -10,9 +11,6 @@ typedef unsigned long dax_entry_t;
 
 struct dax_device;
 struct gendisk;
-struct iomap_ops;
-struct iomap_iter;
-struct iomap;
 
 enum dax_access_mode {
 	DAX_ACCESS,
@@ -213,11 +211,11 @@ static inline void dax_unlock_mapping_entry(struct address_space *mapping,
 #endif
 
 int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 
 static inline bool dax_page_is_idle(struct page *page)
 {
@@ -266,10 +264,10 @@ int dax_holder_notify_failure(struct dax_device *dax_dev, u64 off, u64 len,
 void dax_flush(struct dax_device *dax_dev, void *addr, size_t size);
 
 ssize_t dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,
 			unsigned long *pfnp, int *errp,
-			const struct iomap_ops *ops);
+			iomap_iter_next_fn next);
 vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf,
 		unsigned int order, unsigned long pfn);
 int dax_delete_mapping_entry(struct address_space *mapping, pgoff_t index);
@@ -288,11 +286,11 @@ void dax_break_layout_final(struct inode *inode);
 int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
 				  struct inode *dest, loff_t destoff,
 				  loff_t len, bool *is_same,
-				  const struct iomap_ops *ops);
+				  iomap_iter_next_fn next);
 int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,
 			      struct file *file_out, loff_t pos_out,
 			      loff_t *len, unsigned int remap_flags,
-			      const struct iomap_ops *ops);
+			      iomap_iter_next_fn next);
 static inline bool dax_mapping(struct address_space *mapping)
 {
 	return mapping->host && IS_DAX(mapping->host);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index d10897b3a1e3..2eb063438a3b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -70,7 +70,8 @@ struct fsnotify_mark_connector;
 struct fs_context;
 struct fs_parameter_spec;
 struct file_kattr;
-struct iomap_ops;
+struct iomap_iter;
+struct iomap;
 struct delegated_inode;
 
 extern void __init inode_init(void);
@@ -2079,7 +2080,9 @@ int remap_verify_area(struct file *file, loff_t pos, loff_t len, bool write);
 int __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
 				    struct file *file_out, loff_t pos_out,
 				    loff_t *len, unsigned int remap_flags,
-				    const struct iomap_ops *dax_read_ops);
+				    int (*dax_read_next)(const struct iomap_iter *iter,
+							 struct iomap *iomap,
+							 struct iomap *srcmap));
 int generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
 				  struct file *file_out, loff_t pos_out,
 				  loff_t *count, unsigned int remap_flags);
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 66e2f7186299..519613f281aa 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -238,10 +238,6 @@ typedef int (*iomap_iter_end_fn)(struct inode *inode, loff_t pos, loff_t length,
 typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,
 		struct iomap *iomap, struct iomap *srcmap);
 
-struct iomap_ops {
-	iomap_iter_next_fn iomap_next;
-};
-
 /**
  * struct iomap_iter - Iterate through a range of a file
  * @inode: Set at the start of the iteration and should not change.
@@ -253,7 +249,7 @@ struct iomap_ops {
  *	incremental iter advance.
  * @status: Status of the most recent iteration. Zero on success or a negative
  *	errno on error.
- * @flags: Zero or more of the iomap_next flags above.
+ * @flags: Zero or more of the iomap_iter_next flags above.
  * @iomap: Map describing the I/O iteration
  * @srcmap: Source map for COW operations
  */
@@ -270,7 +266,7 @@ struct iomap_iter {
 	void *private;
 };
 
-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops);
+int iomap_iter(struct iomap_iter *iter, iomap_iter_next_fn next);
 int iomap_iter_advance(struct iomap_iter *iter, u64 count);
 
 /**
@@ -338,7 +334,7 @@ int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,
  * @begin: callback that produces a mapping for the current position
  * @end: optional callback that finishes the previous mapping, or NULL
  *
- * Inline helper that implements the common body of an ->iomap_next()
+ * Inline helper that implements the common body of an iomap_iter_next_fn
  * callback: it finishes the previous mapping via @end (if present), decides
  * via iomap_iter_continue() whether to keep going, and obtains the next
  * mapping via @begin.
@@ -426,32 +422,32 @@ static inline bool iomap_want_unshare_iter(const struct iomap_iter *iter)
 }
 
 ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *from,
-		const struct iomap_ops *ops,
+		iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops, void *private);
 int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
-		const void *buf, const struct iomap_ops *ops,
+		const void *buf, iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops);
-void iomap_read_folio(const struct iomap_ops *ops,
-		struct iomap_read_folio_ctx *ctx, void *private);
-void iomap_readahead(const struct iomap_ops *ops,
-		struct iomap_read_folio_ctx *ctx, void *private);
+void iomap_read_folio(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+		void *private);
+void iomap_readahead(iomap_iter_next_fn next, struct iomap_read_folio_ctx *ctx,
+		void *private);
 bool iomap_is_partially_uptodate(struct folio *, size_t from, size_t count);
 struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len);
 bool iomap_release_folio(struct folio *folio, gfp_t gfp_flags);
 void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len);
 bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio);
 int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
-		const struct iomap_ops *ops,
+		iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops);
 unsigned int iomap_fill_dirty_folios(struct iomap_iter *iter, loff_t *start,
 		loff_t end, unsigned int *iomap_flags);
 int iomap_zero_range(struct inode *inode, loff_t pos, loff_t len,
-		bool *did_zero, const struct iomap_ops *ops,
+		bool *did_zero, iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops, void *private);
 int iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
-		const struct iomap_ops *ops,
+		iomap_iter_next_fn next,
 		const struct iomap_write_ops *write_ops, void *private);
-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_iter_next_fn next,
 		void *private);
 typedef void (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length,
 		struct iomap *iomap);
@@ -460,13 +456,13 @@ void iomap_write_delalloc_release(struct inode *inode, loff_t start_byte,
 		iomap_punch_t punch);
 
 int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
-		u64 start, u64 len, const struct iomap_ops *ops);
+		u64 start, u64 len, iomap_iter_next_fn next);
 loff_t iomap_seek_hole(struct inode *inode, loff_t offset,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 loff_t iomap_seek_data(struct inode *inode, loff_t offset,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 sector_t iomap_bmap(struct address_space *mapping, sector_t bno,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 
 /*
  * Flags for iomap_ioend->io_flags.
@@ -681,10 +677,10 @@ struct iomap_dio_ops {
 #define IOMAP_DIO_NO_IOMAP_END		(1 << 5)
 
 ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
-		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+		iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags, void *private, size_t done_before);
 struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
-		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+		iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags, void *private, size_t done_before);
 ssize_t iomap_dio_complete(struct iomap_dio *dio);
 void iomap_dio_bio_end_io(struct bio *bio);
@@ -695,7 +691,7 @@ struct swap_info_struct;
 
 int iomap_swapfile_activate(struct swap_info_struct *sis,
 		struct file *swap_file, sector_t *pagespan,
-		const struct iomap_ops *ops);
+		iomap_iter_next_fn next);
 #else
 # define iomap_swapfile_activate(sis, swapfile, pagespan, ops)	(-EIO)
 #endif /* CONFIG_SWAP */
@@ -709,25 +705,25 @@ int iomap_bio_read_folio_range(const struct iomap_iter *iter,
 extern const struct iomap_read_ops iomap_bio_read_ops;
 
 static inline void iomap_bio_read_folio(struct folio *folio,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct iomap_read_folio_ctx ctx = {
 		.ops		= &iomap_bio_read_ops,
 		.cur_folio	= folio,
 	};
 
-	iomap_read_folio(ops, &ctx, NULL);
+	iomap_read_folio(next, &ctx, NULL);
 }
 
 static inline void iomap_bio_readahead(struct readahead_control *rac,
-		const struct iomap_ops *ops)
+		iomap_iter_next_fn next)
 {
 	struct iomap_read_folio_ctx ctx = {
 		.ops		= &iomap_bio_read_ops,
 		.rac		= rac,
 	};
 
-	iomap_readahead(ops, &ctx, NULL);
+	iomap_readahead(next, &ctx, NULL);
 }
 #endif /* CONFIG_BLOCK */
 
-- 
2.52.0

[PATCH v3 20/20] Documentation: iomap: update docs to reflect iomap_iter_next model

[Joanne Koong]

Filesystems no longer pass a struct iomap_ops with separate
->iomap_begin() and ->iomap_end() callbacks. Instead, every iomap
operation takes a single iomap_iter_next() callback directly.
iomap_iter_next() finishes the previous mapping (if any) and produces
the next one. Most filesystems build it from begin and end helpers via
the iomap_iter_next() helper.

Update the iomap documentation to match this change.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
 Documentation/filesystems/iomap/design.rst    | 140 +++++++++++++-----
 .../filesystems/iomap/operations.rst          |  68 ++++-----
 Documentation/filesystems/iomap/porting.rst   |  16 +-
 3 files changed, 151 insertions(+), 73 deletions(-)

diff --git a/Documentation/filesystems/iomap/design.rst b/Documentation/filesystems/iomap/design.rst
index 0f7672676c0b..bed8a5beac81 100644
--- a/Documentation/filesystems/iomap/design.rst
+++ b/Documentation/filesystems/iomap/design.rst
@@ -75,7 +75,10 @@ At a high level, an iomap operation `looks like this
 
 1. For each byte in the operation range...
 
-   1. Obtain a space mapping via ``->iomap_begin``
+   1. Obtain the next space mapping via the ``iomap_iter_next_fn`` callback.
+      From the second iteration onwards this same callback first finishes
+      the previous mapping (committing or unreserving space as needed)
+      and then produces the next one.
 
    2. For each sub-unit of work...
 
@@ -86,7 +89,13 @@ At a high level, an iomap operation `looks like this
 
    3. Increment operation cursor
 
-   4. Release the mapping via ``->iomap_end``, if necessary
+iomap repeats this until the range is fully consumed.  The ``iomap_iter_next_fn``
+callback returns ``1`` while there is more of the range left to process,
+``0`` once it is fully consumed, or a negative errno on error.
+Filesystems rarely implement this callback by hand. The ``iomap_iter_next``
+helper implements the finish-then-produce sequence in terms of two smaller
+callbacks, ``begin`` and ``end``. See `The Mapping Callback`_ below for more
+info.
 
 Each iomap operation will be covered in more detail below.
 This library was covered previously by an `LWN article
@@ -189,7 +198,7 @@ The fields are as follows:
    * **IOMAP_DELALLOC**: A promise to allocate space at a later time
      ("delayed allocation").
      If the filesystem returns IOMAP_F_NEW here and the write fails, the
-     ``->iomap_end`` function must delete the reservation.
+     ``end`` function must delete the reservation.
      The ``addr`` field must be set to ``IOMAP_NULL_ADDR``.
 
    * **IOMAP_MAPPED**: The file range maps to specific space on the
@@ -208,12 +217,11 @@ The fields are as follows:
 
    * **IOMAP_INLINE**: The file range maps to the memory buffer
      specified by ``inline_data``.
-     For write operation, the ``->iomap_end`` function presumably
-     handles persisting the data.
+     For write operation, the ``end`` function must persist the data.
      The ``addr`` field must be set to ``IOMAP_NULL_ADDR``.
 
  * ``flags`` describe the status of the space mapping.
-   These flags should be set by the filesystem in ``->iomap_begin``:
+   These flags should be set by the filesystem in ``begin``:
 
    * **IOMAP_F_NEW**: The space under the mapping is newly allocated.
      Areas that will not be written to must be zeroed.
@@ -262,15 +270,15 @@ The fields are as follows:
      update.
 
    These flags can be set by iomap itself during file operations.
-   The filesystem should supply an ``->iomap_end`` function if it needs
+   The filesystem should supply an ``end`` function if it needs
    to observe these flags:
 
    * **IOMAP_F_SIZE_CHANGED**: The file size has changed as a result of
      using this mapping.
 
    * **IOMAP_F_STALE**: The mapping was found to be stale.
-     iomap will call ``->iomap_end`` on this mapping and then
-     ``->iomap_begin`` to obtain a new mapping.
+     iomap will call ``end`` on this mapping and then ``begin`` to obtain a
+     new mapping.
 
    Currently, these flags are only set by pagecache operations.
 
@@ -289,41 +297,103 @@ The fields are as follows:
 
  * ``private`` is a pointer to `filesystem-private information
    <https://lore.kernel.org/all/20180619164137.13720-7-hch@lst.de/>`_.
-   This value will be passed unchanged to ``->iomap_end``.
+   This value will be passed unchanged to ``end``.
 
  * ``validity_cookie`` is a magic freshness value set by the filesystem
    that should be used to detect stale mappings.
    For pagecache operations this is critical for correct operation
    because page faults can occur, which implies that filesystem locks
-   should not be held between ``->iomap_begin`` and ``->iomap_end``.
+   should not be held between ``begin`` and ``end``.
    Filesystems with completely static mappings need not set this value.
    Only pagecache operations revalidate mappings; see the section about
    ``iomap_valid`` for details.
 
-``struct iomap_ops``
+The Mapping Callback
 --------------------
 
-Every iomap function requires the filesystem to pass an operations
-structure to obtain a mapping and (optionally) to release the mapping:
+Every iomap operation takes an ``iomap_iter_next_fn`` callback from the
+filesystem. iomap calls it once per iteration of the file range:
 
 .. code-block:: c
 
- struct iomap_ops {
-     int (*iomap_begin)(struct inode *inode, loff_t pos, loff_t length,
-                        unsigned flags, struct iomap *iomap,
-                        struct iomap *srcmap);
+ typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,
+                              struct iomap *iomap, struct iomap *srcmap);
 
-     int (*iomap_end)(struct inode *inode, loff_t pos, loff_t length,
-                      ssize_t written, unsigned flags,
-                      struct iomap *iomap);
- };
+``iomap_iter_next_fn``
+~~~~~~~~~~~~~~~~~~~~~~
+
+This must finish the previous mapping, if any, and then produce the next
+mapping for the current iteration position described by ``iter``.
+The mapping is returned through ``iomap`` (and through ``srcmap`` for
+operations that read from one mapping while writing to another; see
+``begin`` below).
+
+The callback returns ``1`` to continue iterating, ``0`` once the file
+range has been fully consumed, or a negative errno on error.
+
+Most filesystems are not expected to implement all of these behaviors
+in this callback themselves.  They should instead call ``iomap_iter_next`` as
+described below.
+
+``iomap_iter_next``
+~~~~~~~~~~~~~~~~~~~
+
+Filesystems rarely need a hand-written ``iomap_iter_next_fn`` callback.  The
+``iomap_iter_next`` helper implements the finish-then-produce sequence in
+terms of two smaller callbacks, ``begin`` and ``end``, so most
+``iomap_iter_next_fn`` implementations are simply:
+
+.. code-block:: c
+
+ static int my_iomap_next(const struct iomap_iter *iter,
+                          struct iomap *iomap, struct iomap *srcmap)
+ {
+         return iomap_iter_next(iter, iomap, srcmap,
+                                my_iomap_begin, my_iomap_end);
+ }
+
+This boilerplate is normally generated with the ``DEFINE_IOMAP_ITER_NEXT()``
+and ``DEFINE_IOMAP_ITER_NEXT_END()`` macros rather than written by hand.  Use
+``DEFINE_IOMAP_ITER_NEXT()`` for the common case with no ``end`` callback, and
+``DEFINE_IOMAP_ITER_NEXT_END()`` when an ``end`` callback is needed:
+
+.. code-block:: c
+
+ /* no end() callback */
+ static DEFINE_IOMAP_ITER_NEXT(my_iomap_next, my_iomap_begin);
+
+ /* with an end() callback */
+ static DEFINE_IOMAP_ITER_NEXT_END(my_iomap_next, my_iomap_begin,
+                                   my_iomap_end);
+
+The macro generates a function with external linkage.  Prefix the invocation
+with ``static`` for a file-local callback, or leave it non-``static`` and add
+a matching declaration to a header when the callback is referenced from
+another file.
+
+``end`` may be ``NULL`` when the filesystem has nothing to finish.
+The two callbacks have these prototypes:
+
+.. code-block:: c
+
+ typedef int (*iomap_begin_fn)(struct inode *inode, loff_t pos,
+                               loff_t length, unsigned flags,
+                               struct iomap *iomap, struct iomap *srcmap);
+
+ typedef int (*iomap_end_fn)(struct inode *inode, loff_t pos,
+                             loff_t length, ssize_t written,
+                             unsigned flags, struct iomap *iomap);
+
+``iomap_iter_next`` is an inline helper, so when it is called with fixed
+``begin`` and ``end`` functions the compiler can inline both into the
+filesystem's ``iomap_iter_next_fn`` callback, keeping indirect calls out of the
+iteration hot path.  The two callbacks are described next.
 
-``->iomap_begin``
-~~~~~~~~~~~~~~~~~
+``begin``
+~~~~~~~~~
 
-iomap operations call ``->iomap_begin`` to obtain one file mapping for
-the range of bytes specified by ``pos`` and ``length`` for the file
-``inode``.
+The ``begin`` callback obtains one file mapping for the range of bytes
+specified by ``pos`` and ``length`` for the file ``inode``.
 This mapping should be returned through the ``iomap`` pointer.
 The mapping must cover at least the first byte of the supplied file
 range, but it does not need to cover the entire requested range.
@@ -377,18 +447,18 @@ information via ``srcmap``.
 Only pagecache and fsdax operations support reading from one mapping and
 writing to another.
 
-``->iomap_end``
-~~~~~~~~~~~~~~~
+``end``
+~~~~~~~
 
-After the operation completes, the ``->iomap_end`` function, if present,
-is called to signal that iomap is finished with a mapping.
+After the operation completes, the ``end`` callback, if present, is called to
+signal that iomap is finished with a mapping.
 Typically, implementations will use this function to tear down any
-context that were set up in ``->iomap_begin``.
+context that was set up in ``begin``.
 For example, a write might wish to commit the reservations for the bytes
 that were operated upon and unreserve any space that was not operated
 upon.
 ``written`` might be zero if no bytes were touched.
-``flags`` will contain the same value passed to ``->iomap_begin``.
+``flags`` will contain the same value passed to ``begin``.
 iomap ops for reads are not likely to need to supply this function.
 
 Both functions should return a negative errno code on error, or zero on
@@ -421,8 +491,8 @@ iomap is concerned:
    accessing the folio until writeback is underway.
 
    * The **lower** level primitive is taken by the filesystem in the
-     ``->iomap_begin`` and ``->iomap_end`` functions to coordinate
-     access to the file space mapping information.
+     ``begin`` and ``end`` functions to coordinate access to the file space
+     mapping information.
      The fields of the iomap object should be filled out while holding
      this primitive.
      The upper level synchronization primitive, if any, remains held
diff --git a/Documentation/filesystems/iomap/operations.rst b/Documentation/filesystems/iomap/operations.rst
index da982ca7e413..a5784dee0850 100644
--- a/Documentation/filesystems/iomap/operations.rst
+++ b/Documentation/filesystems/iomap/operations.rst
@@ -17,6 +17,12 @@ Supported File Operations
 Below are a discussion of the high level file operations that iomap
 implements.
 
+Each operation takes an ``iomap_iter_next_fn`` callback that supplies the file
+mappings, as described in the iomap design document.  The per-operation
+``flags`` documented below are passed to that callback; references to
+``begin`` and ``end`` name the two steps a typical callback is built from
+via ``iomap_iter_next``.
+
 Buffered I/O
 ============
 
@@ -90,10 +96,10 @@ iomap calls these functions:
     <https://lore.kernel.org/all/20180619164137.13720-6-hch@lst.de/>`_
     that was set up by ``->get_folio``.
 
-  - ``iomap_valid``: The filesystem may not hold locks between
-    ``->iomap_begin`` and ``->iomap_end`` because pagecache operations
-    can take folio locks, fault on userspace pages, initiate writeback
-    for memory reclamation, or engage in other time-consuming actions.
+  - ``iomap_valid``: The filesystem may not hold locks between ``begin`` and
+    ``end`` because pagecache operations can take folio locks, fault on
+    userspace pages, initiate writeback for memory reclamation, or engage in
+    other time-consuming actions.
     If a file's space mapping data are mutable, it is possible that the
     mapping for a particular pagecache folio can `change in the time it
     takes
@@ -114,12 +120,12 @@ iomap calls these functions:
     If the mapping is not valid, the mapping will be sampled again.
 
     To support making the validity decision, the filesystem's
-    ``->iomap_begin`` function may set ``struct iomap::validity_cookie``
+    ``begin`` function may set ``struct iomap::validity_cookie``
     at the same time that it populates the other iomap fields.
     A simple validation cookie implementation is a sequence counter.
     If the filesystem bumps the sequence counter every time it modifies
     the inode's extent map, it can be placed in the ``struct
-    iomap::validity_cookie`` during ``->iomap_begin``.
+    iomap::validity_cookie`` during ``begin``.
     If the value in the cookie is found to be different to the value
     the filesystem holds when the mapping is passed back to
     ``->iomap_valid``, then the iomap should considered stale and the
@@ -199,7 +205,7 @@ Buffered Readahead and Reads
 The ``iomap_readahead`` function initiates readahead to the pagecache.
 The ``iomap_read_folio`` function reads one folio's worth of data into
 the pagecache.
-The ``flags`` argument to ``->iomap_begin`` will be set to zero.
+The ``flags`` argument to ``begin`` will be set to zero.
 The pagecache takes whatever locks it needs before calling the
 filesystem.
 
@@ -231,7 +237,7 @@ Buffered Writes
 The ``iomap_file_buffered_write`` function writes an ``iocb`` to the
 pagecache.
 ``IOMAP_WRITE`` or ``IOMAP_WRITE`` | ``IOMAP_NOWAIT`` will be passed as
-the ``flags`` argument to ``->iomap_begin``.
+the ``flags`` argument to ``begin``.
 Callers commonly take ``i_rwsem`` in either shared or exclusive mode
 before calling this function.
 
@@ -241,7 +247,7 @@ mmap Write Faults
 The ``iomap_page_mkwrite`` function handles a write fault to a folio in
 the pagecache.
 ``IOMAP_WRITE | IOMAP_FAULT`` will be passed as the ``flags`` argument
-to ``->iomap_begin``.
+to ``begin``.
 Callers commonly take the mmap ``invalidate_lock`` in shared or
 exclusive mode before calling this function.
 
@@ -256,7 +262,7 @@ such `reservations
 <https://lore.kernel.org/linux-xfs/20220817093627.GZ3600936@dread.disaster.area/>`_
 because writeback will not consume the reservation.
 The ``iomap_write_delalloc_release`` can be called from a
-``->iomap_end`` function to find all the clean areas of the folios
+``end`` function to find all the clean areas of the folios
 caching a fresh (``IOMAP_F_NEW``) delalloc mapping.
 It takes the ``invalidate_lock``.
 
@@ -274,7 +280,7 @@ Filesystems can call ``iomap_zero_range`` to perform zeroing of the
 pagecache for non-truncation file operations that are not aligned to
 the fsblock size.
 ``IOMAP_ZERO`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``begin``.
 Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive
 mode before calling this function.
 
@@ -285,7 +291,7 @@ Filesystems can call ``iomap_file_unshare`` to force a file sharing
 storage with another file to preemptively copy the shared data to newly
 allocate storage.
 ``IOMAP_WRITE | IOMAP_UNSHARE`` will be passed as the ``flags`` argument
-to ``->iomap_begin``.
+to ``begin``.
 Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive
 mode before calling this function.
 
@@ -298,7 +304,7 @@ operation.
 ``truncate_setsize`` or ``truncate_pagecache`` will take care of
 everything after the EOF block.
 ``IOMAP_ZERO`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``begin``.
 Callers typically hold ``i_rwsem`` and ``invalidate_lock`` in exclusive
 mode before calling this function.
 
@@ -341,8 +347,8 @@ The fields are as follows:
     though it will `reuse mappings
     <https://lore.kernel.org/all/20231207072710.176093-15-hch@lst.de/>`_
     for runs of contiguous dirty fsblocks within a folio.
-    Do not return ``IOMAP_INLINE`` mappings here; the ``->iomap_end``
-    function must deal with persisting written data.
+    Do not return ``IOMAP_INLINE`` mappings here; the ``end`` function must
+    deal with persisting written data.
     Do not return ``IOMAP_DELALLOC`` mappings here; iomap currently
     requires mapping to allocated space.
     Filesystems can skip a potentially expensive mapping lookup if the
@@ -428,7 +434,7 @@ writes for files.
 .. code-block:: c
 
  ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
-                      const struct iomap_ops *ops,
+                      iomap_iter_next_fn iomap_next,
                       const struct iomap_dio_ops *dops,
                       unsigned int dio_flags, void *private,
                       size_t done_before);
@@ -511,7 +517,7 @@ Return Values
  * ``-ENOTBLK``: Fall back to buffered I/O.
    iomap itself will return this value if it cannot invalidate the page
    cache before issuing the I/O to storage.
-   The ``->iomap_begin`` or ``->iomap_end`` functions may also return
+   The ``begin`` or ``end`` functions may also return
    this value.
 
  * ``-EIOCBQUEUED``: The asynchronous direct I/O request has been
@@ -526,7 +532,7 @@ A direct I/O read initiates a read I/O from the storage device to the
 caller's buffer.
 Dirty parts of the pagecache are flushed to storage before initiating
 the read io.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DIRECT`` with
+The ``flags`` value for ``begin`` will be ``IOMAP_DIRECT`` with
 any combination of the following enhancements:
 
  * ``IOMAP_NOWAIT``, as defined previously.
@@ -542,7 +548,7 @@ caller's buffer.
 Dirty parts of the pagecache are flushed to storage before initiating
 the write io.
 The pagecache is invalidated both before and after the write io.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DIRECT |
+The ``flags`` value for ``begin`` will be ``IOMAP_DIRECT |
 IOMAP_WRITE`` with any combination of the following enhancements:
 
  * ``IOMAP_NOWAIT``, as defined previously.
@@ -644,7 +650,7 @@ fsdax Reads
 
 A fsdax read performs a memcpy from storage device to the caller's
 buffer.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DAX`` with any
+The ``flags`` value for ``begin`` will be ``IOMAP_DAX`` with any
 combination of the following enhancements:
 
  * ``IOMAP_NOWAIT``, as defined previously.
@@ -657,7 +663,7 @@ fsdax Writes
 
 A fsdax write initiates a memcpy to the storage device from the caller's
 buffer.
-The ``flags`` value for ``->iomap_begin`` will be ``IOMAP_DAX |
+The ``flags`` value for ``begin`` will be ``IOMAP_DAX |
 IOMAP_WRITE`` with any combination of the following enhancements:
 
  * ``IOMAP_NOWAIT``, as defined previously.
@@ -680,9 +686,9 @@ fsdax mmap Faults
 The ``dax_iomap_fault`` function handles read and write faults to fsdax
 storage.
 For a read fault, ``IOMAP_DAX | IOMAP_FAULT`` will be passed as the
-``flags`` argument to ``->iomap_begin``.
+``flags`` argument to ``begin``.
 For a write fault, ``IOMAP_DAX | IOMAP_FAULT | IOMAP_WRITE`` will be
-passed as the ``flags`` argument to ``->iomap_begin``.
+passed as the ``flags`` argument to ``begin``.
 
 Callers commonly hold the same locks as they do to call their iomap
 pagecache counterparts.
@@ -692,8 +698,8 @@ fsdax Truncation, fallocate, and Unsharing
 
 For fsdax files, the following functions are provided to replace their
 iomap pagecache I/O counterparts.
-The ``flags`` argument to ``->iomap_begin`` are the same as the
-pagecache counterparts, with ``IOMAP_DAX`` added.
+The ``flags`` argument to ``begin`` are the same as the pagecache counterparts,
+with ``IOMAP_DAX`` added.
 
  * ``dax_file_unshare``
  * ``dax_zero_range``
@@ -719,8 +725,7 @@ SEEK_DATA
 
 The ``iomap_seek_data`` function implements the SEEK_DATA "whence" value
 for llseek.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
 
 For unwritten mappings, the pagecache will be searched.
 Regions of the pagecache with a folio mapped and uptodate fsblocks
@@ -734,8 +739,7 @@ SEEK_HOLE
 
 The ``iomap_seek_hole`` function implements the SEEK_HOLE "whence" value
 for llseek.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
 
 For unwritten mappings, the pagecache will be searched.
 Regions of the pagecache with no folio mapped, or a !uptodate fsblock
@@ -750,8 +754,7 @@ Swap File Activation
 The ``iomap_swapfile_activate`` function finds all the base-page aligned
 regions in a file and sets them up as swap space.
 The file will be ``fsync()``'d before activation.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
 All mappings must be mapped or unwritten; cannot be dirty or shared, and
 cannot span multiple block devices.
 Callers must hold ``i_rwsem`` in exclusive mode; this is already
@@ -767,8 +770,7 @@ FS_IOC_FIEMAP
 
 The ``iomap_fiemap`` function exports file extent mappings to userspace
 in the format specified by the ``FS_IOC_FIEMAP`` ioctl.
-``IOMAP_REPORT`` will be passed as the ``flags`` argument to
-``->iomap_begin``.
+``IOMAP_REPORT`` will be passed as the ``flags`` argument to ``begin``.
 Callers commonly hold ``i_rwsem`` in shared mode before calling this
 function.
 
diff --git a/Documentation/filesystems/iomap/porting.rst b/Documentation/filesystems/iomap/porting.rst
index 3d49a32c0fff..b995f426808a 100644
--- a/Documentation/filesystems/iomap/porting.rst
+++ b/Documentation/filesystems/iomap/porting.rst
@@ -50,8 +50,14 @@ Build the kernel, run fstests with the ``-g all`` option across a wide
 variety of your filesystem's supported configurations to build a
 baseline of which tests pass and which ones fail.
 
-The recommended approach is first to implement ``->iomap_begin`` (and
-``->iomap_end`` if necessary) to allow iomap to obtain a read-only
+Every iomap operation is driven by an ``iomap_iter_next_fn`` callback.
+Filesystems normally do not write one by hand: implement ``begin``
+(and ``end`` if necessary) and generate the callback with the
+``DEFINE_IOMAP_ITER_NEXT()`` macro, or ``DEFINE_IOMAP_ITER_NEXT_END()`` if an
+``end`` callback is needed.
+
+The recommended approach is first to implement ``begin`` (and
+``end`` if necessary) to allow iomap to obtain a read-only
 mapping of a file range.
 In most cases, this is a relatively trivial conversion of the existing
 ``get_block()`` function for read-only mappings.
@@ -62,7 +68,7 @@ If FIEMAP is returning the correct information, it's a good sign that
 other read-only mapping operations will do the right thing.
 
 Next, modify the filesystem's ``get_block(create = false)``
-implementation to use the new ``->iomap_begin`` implementation to map
+implementation to use the new ``begin`` implementation to map
 file space for selected read operations.
 Hide behind a debugging knob the ability to switch on the iomap mapping
 functions for selected call paths.
@@ -82,14 +88,14 @@ I/O path because of bufferheads.
 The buffered read I/O paths doesn't need to be converted yet, though the
 direct I/O read path should be converted in this phase.
 
-At this point, you should look over your ``->iomap_begin`` function.
+At this point, you should look over your ``begin`` function.
 If it switches between large blocks of code based on dispatching of the
 ``flags`` argument, you should consider breaking it up into
 per-operation iomap ops with smaller, more cohesive functions.
 XFS is a good example of this.
 
 The next thing to do is implement ``get_blocks(create == true)``
-functionality in the ``->iomap_begin``/``->iomap_end`` methods.
+functionality in the ``begin``/``end`` methods.
 It is strongly recommended to create separate mapping functions and
 iomap ops for write operations.
 Then convert the direct I/O write path to iomap, and start running fsx
-- 
2.52.0

Re: [PATCH v3 10/20] zonefs: convert iomap ops to ->iomap_next()

[Damien Le Moal]

On 7/21/26 06:01, Joanne Koong wrote:
> Convert zonefs iomap_ops to the new ->iomap_next() callback. Each
> callback is generated with the DEFINE_IOMAP_ITER_NEXT() macro, which
> wraps the iomap_iter_next() helper to finish the previous mapping if
> needed and produce the next one. No functional changes are intended.
> 
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>

Acked-by: Damien Le Moal <dlemoal@kernel.org>

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH v3 08/20] ext4: convert iomap ops to ->iomap_next()

[Baokun Li]

On 2026/7/21 05:01, Joanne Koong wrote:
> Convert ext4 iomap_ops to the new ->iomap_next() callback. Each callback is
> generated with the DEFINE_IOMAP_ITER_NEXT() macro, which wraps the
> iomap_iter_next() helper to finish the previous mapping if needed and
> produce the next one. No functional changes are intended.
>
> Reviewed-by: Jan Kara <jack@suse.cz>
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>

Looks good to me.

Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>

> ---
>  fs/ext4/extents.c | 4 +++-
>  fs/ext4/inode.c   | 8 ++++++--
>  2 files changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
> index 91c97af64b31..15972410d460 100644
> --- a/fs/ext4/extents.c
> +++ b/fs/ext4/extents.c
> @@ -5171,8 +5171,10 @@ static int ext4_iomap_xattr_begin(struct inode *inode, loff_t offset,
>  	return error;
>  }
>  
> +static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_xattr_next, ext4_iomap_xattr_begin);
> +
>  static const struct iomap_ops ext4_iomap_xattr_ops = {
> -	.iomap_begin		= ext4_iomap_xattr_begin,
> +	.iomap_next		= ext4_iomap_xattr_next,
>  };
>  
>  static int ext4_fiemap_check_ranges(struct inode *inode, u64 start, u64 *len)
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index ce99807c5f5b..278a032b300c 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -3850,8 +3850,10 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
>  	return 0;
>  }
>  
> +static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next, ext4_iomap_begin);
> +
>  const struct iomap_ops ext4_iomap_ops = {
> -	.iomap_begin		= ext4_iomap_begin,
> +	.iomap_next		= ext4_iomap_next,
>  };
>  
>  static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
> @@ -3905,8 +3907,10 @@ static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
>  	return 0;
>  }
>  
> +static DEFINE_IOMAP_ITER_NEXT(ext4_iomap_next_report, ext4_iomap_begin_report);
> +
>  const struct iomap_ops ext4_iomap_report_ops = {
> -	.iomap_begin = ext4_iomap_begin_report,
> +	.iomap_next = ext4_iomap_next_report,
>  };
>  
>  /*

Re: [PATCH v3 17/20] exfat: convert iomap ops to ->iomap_next()

[Namjae Jeon]

On Tue, Jul 21, 2026 at 6:02 AM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> Convert exfat iomap_ops to the new ->iomap_next() callback. Each
> callback is generated with the DEFINE_IOMAP_ITER_NEXT() and
> DEFINE_IOMAP_ITER_NEXT_END() macros, which wrap the iomap_iter_next()
> helper to finish the previous mapping if needed and produce the next
> one. No functional changes are intended.
>
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Acked-by: Namjae Jeon <linkinjeon@kernel.org>
Thanks!

Re: [PATCH v3 07/20] ntfs: convert iomap ops to ->iomap_next()

[Namjae Jeon]

On Tue, Jul 21, 2026 at 6:02 AM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> Convert ntfs iomap_ops to the new ->iomap_next() callback. Each callback is
> generated with the DEFINE_IOMAP_ITER_NEXT()/DEFINE_IOMAP_ITER_NEXT_END()
> macros, which wrap the iomap_iter_next() helper to finish the previous
> mapping if needed and produce the next one. No functional changes are
> intended.
>
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Acked-by: Namjae Jeon <linkinjeon@kernel.org>
Thanks!

Re: [PATCH v3 18/20] iomap: remove ->iomap_begin()/->iomap_end() legacy path

[changfengnan]

> From: "Joanne Koong"<joannelkoong@gmail.com>
> Date:  Tue, Jul 21, 2026, 05:03
> Subject:  [PATCH v3 18/20] iomap: remove ->iomap_begin()/->iomap_end() legacy path
> To: "Christian Brauner"<brauner@kernel.org>, <hch@lst.de>, "Darrick J . Wong"<djwong@kernel.org>, <linux-fsdevel@vger.kernel.org>
> Cc: <changfengnan@bytedance.com>, <kbusch@kernel.org>, "Matthew Wilcox"<willy@infradead.org>, "Jan Kara"<jack@suse.cz>, "Jonathan Corbet"<corbet@lwn.net>, "David Sterba"<dsterba@suse.com>, "Dan Williams"<djbw@kernel.org>, "Gao Xiang"<xiang@kernel.org>, "Namjae Jeon"<linkinjeon@kernel.org>, "Theodore Ts'o"<tytso@mit.edu>, "Jaegeuk Kim"<jaegeuk@kernel.org>, "Miklos Szeredi"<miklos@szeredi.hu>, "Andreas Gruenbacher"<agruenba@redhat.com>, "Mikulas Patocka"<mikulas@artax.karlin.mff.cuni.cz>, "Hyunchul Lee"<hyc.lee@gmail.com>, "Konstantin Komarov"<almaz.alexandrovich@paragon-software.com>, "Carlos Maiolino"<cem@kernel.org>, "Damien Le Moal"<dlemoal@kernel.org>
> With all in-tree filesystems implementing ->iomap_next(), drop the
> ->iomap_begin()/->iomap_end() path.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>

Reviewed-by: Fengnan Chang <changfengnan@bytedance.com>

I ran some tests on the simple  path and didn't notice any
performance degradation. 

> ---
>  fs/iomap/direct-io.c  | 47 ++++++++++++++-----------------------------
>  fs/iomap/iter.c       |  6 +-----
>  include/linux/iomap.h |  6 ++----
>  3 files changed, 18 insertions(+), 41 deletions(-)
> 
> diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
> index fcdae3ef6dd0..a47369e6b4e1 100644
> --- a/fs/iomap/direct-io.c
> +++ b/fs/iomap/direct-io.c
> @@ -1001,18 +1001,13 @@ iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
>  }
>  
>  static inline void
> -iomap_dio_simple_finish(struct iomap_iter *iomi, const struct iomap_ops *ops,
> -                size_t len, size_t written)
> +iomap_dio_simple_finish(struct iomap_iter *iomi, iomap_iter_next_fn next,
> +                size_t written)
>  {
> -        if (ops->iomap_next) {
> -                iomi->iter_start_pos = iomi->pos;
> -                iomi->pos += written;
> -                iomi->len -= written;
> -                ops->iomap_next(iomi, &iomi->iomap, &iomi->srcmap);
> -        } else if (ops->iomap_end) {
> -                ops->iomap_end(iomi->inode, iomi->pos, len, written,
> -                               iomi->flags, &iomi->iomap);
> -        }
> +        iomi->iter_start_pos = iomi->pos;
> +        iomi->pos += written;
> +        iomi->len -= written;
> +        next(iomi, &iomi->iomap, &iomi->srcmap);
>  }
>  
>  /*
> @@ -1079,25 +1074,13 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
>  
>          inode_dio_begin(inode);
>  
> -        if (ops->iomap_next) {
> -                ret = ops->iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
> -                if (ret <= 0) {
> -                        inode_dio_end(inode);
> -                        /*
> -                         * the first iteration must yield a mapping (>0) or an
> -                         * error
> -                         */
> -                        if (WARN_ON_ONCE(!ret))
> -                                return -EFAULT;
> -                        return ret;
> -                }
> -        } else {
> -                ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
> -                                       &iomi.iomap, &iomi.srcmap);
> -                if (ret) {
> -                        inode_dio_end(inode);
> -                        return ret;
> -                }
> +        ret = ops->iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
> +        if (ret <= 0) {
> +                inode_dio_end(inode);
> +                /* the first iteration must yield a mapping (>0) or an error */
> +                if (WARN_ON_ONCE(!ret))
> +                        return -EFAULT;
> +                return ret;
>          }
>  
>          if (iomi.iomap.type != IOMAP_MAPPED ||
> @@ -1163,7 +1146,7 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
>           * iomap_next() call on the hot read path.
>           */
>          if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
> -                iomap_dio_simple_finish(&iomi, ops, count, count);
> +                iomap_dio_simple_finish(&iomi, ops->iomap_next, count);
>  
>          if (!wait_for_completion) {
>                  bio->bi_end_io = iomap_dio_simple_end_io;
> @@ -1181,7 +1164,7 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
>          bio_put(bio);
>  out_iomap_end:
>          if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
> -                iomap_dio_simple_finish(&iomi, ops, count, 0);
> +                iomap_dio_simple_finish(&iomi, ops->iomap_next, 0);
>          inode_dio_end(inode);
>          return ret;
>  }
> diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
> index e05574602946..f7c4d5cf56be 100644
> --- a/fs/iomap/iter.c
> +++ b/fs/iomap/iter.c
> @@ -107,11 +107,7 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
>  
>          trace_iomap_iter(iter, ops, _RET_IP_);
>  
> -        if (ops->iomap_next)
> -                ret = ops->iomap_next(iter, &iter->iomap, &iter->srcmap);
> -        else
> -                ret = iomap_iter_next(iter, &iter->iomap, &iter->srcmap,
> -                                ops->iomap_begin, ops->iomap_end);
> +        ret = ops->iomap_next(iter, &iter->iomap, &iter->srcmap);
>  
>          iter->status = 0;
>          if (ret > 0)
> diff --git a/include/linux/iomap.h b/include/linux/iomap.h
> index 8662bd528e9e..66e2f7186299 100644
> --- a/include/linux/iomap.h
> +++ b/include/linux/iomap.h
> @@ -194,7 +194,7 @@ struct iomap_write_ops {
>  };
>  
>  /*
> - * Flags for iomap_begin / iomap_end.  No flag implies a read.
> + * Flags for iomap_next.  No flag implies a read.
>   */
>  #define IOMAP_WRITE                (1 << 0) /* writing, must allocate blocks */
>  #define IOMAP_ZERO                (1 << 1) /* zeroing operation, may skip holes */
> @@ -239,8 +239,6 @@ typedef int (*iomap_iter_next_fn)(const struct iomap_iter *iter,
>                  struct iomap *iomap, struct iomap *srcmap);
>  
>  struct iomap_ops {
> -        iomap_iter_begin_fn iomap_begin;
> -        iomap_iter_end_fn iomap_end;
>          iomap_iter_next_fn iomap_next;
>  };
>  
> @@ -255,7 +253,7 @@ struct iomap_ops {
>   *        incremental iter advance.
>   * @status: Status of the most recent iteration. Zero on success or a negative
>   *        errno on error.
> - * @flags: Zero or more of the iomap_begin flags above.
> + * @flags: Zero or more of the iomap_next flags above.
>   * @iomap: Map describing the I/O iteration
>   * @srcmap: Source map for COW operations
>   */
> -- 
> 2.52.0
> 

Re: [PATCH v3 05/20] btrfs: convert iomap ops to ->iomap_next()

[David Sterba]

On Mon, Jul 20, 2026 at 02:01:47PM -0700, Joanne Koong wrote:
> Convert btrfs iomap_ops to the new ->iomap_next() callback. The callback is
> generated with the DEFINE_IOMAP_ITER_NEXT_END() macro, which wraps the
> iomap_iter_next() helper to finish the previous mapping if needed and
> produce the next one. No functional changes are intended.
> 
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>

Acked-by: David Sterba <dsterba@suse.com>

Re: [PATCH v3 00/20] iomap: convert to in-iter iomap_next() model

[Christoph Hellwig]

The submission seems to be missing most of the relevant mailing lists,
e.g., xfs for iomap itself and the file system-specific lists.

For now I didn't have too much time to look at it, and the little I had
I used to jump on the simple direct I/O thing, as that's the only part
that stood out as something I didn't like.  The new "don't advance the
iter" magic flag just didn't feel right.

What I came up with is the patch below which sits ontop of your entire
series.  I redoes the simple read path very heavily and now requires the
file system to opt in, and provide the begin callback only, i.e. we
require the file system to only call it if it can, which simplifies the
logic a lot, but requires more care from the file systems, and exposing
the old iomap_begin callback.

Fengnan, can you check if this keeps your performance?  It does more
inlining and avoid the direct call, but hopefully doesn't inline too
much like the previous attempt.  If this works, it probably needs
to move before the conversion.

---
diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index adda6342de5b..67b226edb287 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -809,8 +809,8 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
 	if (iocb->ki_flags & IOCB_DIRECT) {
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL,
-				IOMAP_DIO_NO_IOMAP_END, NULL, 0);
+		ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL, 0, NULL,
+				0);
 	} else {
 		ret = generic_file_read_iter(iocb, iter);
 	}
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 755fde1baf03..66d281d00d3e 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -4004,6 +4004,8 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end)
 		io_end->flag &= ~EXT4_IO_END_UNWRITTEN;
 }
 
+inline int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
+		unsigned flags, struct iomap *iomap, struct iomap *srcmap);
 int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
 		struct iomap *srcmap);
 int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index bf0c18cf1017..491854e93f92 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,8 +91,10 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		return generic_file_read_iter(iocb, to);
 	}
 
-	ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL,
-			   IOMAP_DIO_NO_IOMAP_END, NULL, 0);
+
+	ret = iomap_dio_read_simple(iocb, to, ext4_iomap_begin);
+	if (ret == -ENOTBLK)
+		ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
 	inode_unlock_shared(inode);
 
 	file_accessed(iocb->ki_filp);
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index f455708a8c31..973ee7a97f61 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3771,7 +3771,7 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,
 }
 
 
-static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
+inline int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 		unsigned flags, struct iomap *iomap, struct iomap *srcmap)
 {
 	int ret;
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 9c50adc93cfa..33c2c6d3e134 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -894,6 +894,21 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 }
 EXPORT_SYMBOL_GPL(__iomap_dio_rw);
 
+ssize_t
+iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_iter_next_fn next,
+		const struct iomap_dio_ops *dops, unsigned int dio_flags,
+		void *private, size_t done_before)
+{
+	struct iomap_dio *dio;
+
+	dio = __iomap_dio_rw(iocb, iter, next, dops, dio_flags, private,
+			     done_before);
+	if (IS_ERR_OR_NULL(dio))
+		return PTR_ERR_OR_ZERO(dio);
+	return iomap_dio_complete(dio);
+}
+EXPORT_SYMBOL_GPL(iomap_dio_rw);
+
 struct iomap_dio_simple {
 	struct kiocb		*iocb;
 	size_t			size;
@@ -968,159 +983,47 @@ static void iomap_dio_simple_end_io(struct bio *bio)
 	iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
 }
 
-static inline bool
-iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
-			   const struct iomap_dio_ops *dops,
-			   unsigned int dio_flags, size_t done_before)
+ssize_t __iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+		iomap_iter_begin_fn begin, struct iomap_iter *iomi)
 {
-	struct inode *inode = file_inode(iocb->ki_filp);
-	size_t count = iov_iter_count(iter);
-
-	if (dops || done_before)
-		return false;
-	if (iov_iter_rw(iter) != READ)
-		return false;
-	if (!count)
-		return false;
-	/*
-	 * Simple dio is an optimization for small IO. Filter out large IO
-	 * early as it's the most common case to fail for typical direct IO
-	 * workloads.
-	 */
-	if (count > inode->i_sb->s_blocksize)
-		return false;
-	if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
-			 IOMAP_DIO_BOUNCE))
-		return false;
-	if (iocb->ki_pos + count > i_size_read(inode))
-		return false;
-	if (IS_ENCRYPTED(inode))
-		return false;
-
-	return true;
-}
-
-static inline void
-iomap_dio_simple_finish(struct iomap_iter *iomi, iomap_iter_next_fn next,
-		size_t written)
-{
-	iomi->iter_start_pos = iomi->pos;
-	iomi->pos += written;
-	iomi->len -= written;
-	next(iomi, &iomi->iomap, &iomi->srcmap);
-}
-
-/*
- * Fast path for small, block-aligned direct I/Os that map to a single
- * contiguous on-disk extent.
- *
- * iomap_dio_simple_supported() enforces the cheap up-front constraints before
- * entering this path.
- *
- * @dops must be NULL: a non-NULL @dops means the caller wants its
- * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
- * allocated from the filesystem-private @dops->bio_set (whose front_pad sizes a
- * filesystem-private wrapper around the bio).  The fast path instead allocates
- * from the shared iomap_dio_simple_pool, whose front_pad matches struct
- * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
- * back to __iomap_dio_rw() in that case.
- *
- * @done_before must be zero: a non-zero caller-accumulated residual cannot be
- * carried through a single-bio inline completion.
- *
- * @iter must describe a non-empty READ no larger than the inode block size:
- * writes, zero-length I/O, and larger requests need the generic iomap direct
- * I/O path.
- *
- * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
- * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
- * I/O, or bouncing.  The range must also stay within i_size and encrypted
- * inodes must use the generic iomap direct I/O path. IOMAP_DIO_NO_IOMAP_END is
- * a fast-path optimization the caller can set if there is no work that needs to
- * be finished after a mapping.
- *
- * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
- * decides the request does not fit the fast path.  In that case we proceed to
- * the generic __iomap_dio_rw() slow path.  Any other errno is a real result and
- * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
- * caller.
- */
-static ssize_t
-iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
-		 iomap_iter_next_fn next, void *private, unsigned int dio_flags)
-{
-	struct inode *inode = file_inode(iocb->ki_filp);
-	size_t count = iov_iter_count(iter);
-	bool wait_for_completion = is_sync_kiocb(iocb);
-	struct iomap_iter iomi = {
-		.inode		= inode,
-		.pos		= iocb->ki_pos,
-		.len		= count,
-		.flags		= IOMAP_DIRECT,
-		.private	= private,
-	};
 	struct iomap_dio_simple *sr;
 	unsigned int alignment;
 	struct bio *bio;
 	ssize_t ret;
 
-	if (iocb->ki_flags & IOCB_NOWAIT)
-		iomi.flags |= IOMAP_NOWAIT;
-
-	ret = kiocb_write_and_wait(iocb, count);
-	if (ret)
-		return ret;
-
-	inode_dio_begin(inode);
-
-	ret = next(&iomi, &iomi.iomap, &iomi.srcmap);
-	if (ret <= 0) {
-		inode_dio_end(inode);
-		/* the first iteration must yield a mapping (>0) or an error */
-		if (WARN_ON_ONCE(!ret))
-			return -EFAULT;
-		return ret;
-	}
-
-	if (iomi.iomap.type != IOMAP_MAPPED ||
-	    iomi.iomap.offset + iomi.iomap.length < iomi.pos + count ||
-	    (iomi.iomap.flags & IOMAP_F_INTEGRITY)) {
-		ret = -ENOTBLK;
-		goto out_iomap_end;
-	}
+	if (iomi->iomap.type != IOMAP_MAPPED ||
+	    iomi->iomap.offset + iomi->iomap.length < iomi->pos + iomi->len ||
+	    (iomi->iomap.flags & IOMAP_F_INTEGRITY))
+		return -ENOTBLK;
 
-	alignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);
-	if ((iomi.pos | count) & (alignment - 1)) {
-		ret = -EINVAL;
-		goto out_iomap_end;
-	}
+	alignment = iomap_dio_alignment(iomi->inode, iomi->iomap.bdev, 0);
+	if ((iomi->pos | iomi->len) & (alignment - 1))
+		return -EINVAL;
 
-	if (!wait_for_completion && unlikely(!inode->i_sb->s_dio_done_wq)) {
-		ret = sb_init_dio_done_wq(inode->i_sb);
+	if (unlikely(!iomi->inode->i_sb->s_dio_done_wq &&
+			!is_sync_kiocb(iocb))) {
+		ret = sb_init_dio_done_wq(iomi->inode->i_sb);
 		if (ret < 0)
-			goto out_iomap_end;
+			return ret;
 	}
 
-	trace_iomap_dio_rw_begin(iocb, iter, dio_flags, 0);
-
-	if (user_backed_iter(iter))
-		dio_flags |= IOMAP_DIO_USER_BACKED;
+	trace_iomap_dio_rw_begin(iocb, iter, 0, 0);
 
-	bio = bio_alloc_bioset(iomi.iomap.bdev,
+	bio = bio_alloc_bioset(iomi->iomap.bdev,
 			       bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),
 			       REQ_OP_READ, GFP_KERNEL, &iomap_dio_simple_pool);
 	sr = container_of(bio, struct iomap_dio_simple, bio);
 	sr->iocb = iocb;
-	sr->dio_flags = dio_flags;
+	sr->dio_flags = 0;
 
-	bio->bi_iter.bi_sector = iomap_sector(&iomi.iomap, iomi.pos);
+	bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
 	bio->bi_ioprio = iocb->ki_ioprio;
 
 	ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
 	if (unlikely(ret))
 		goto out_bio_put;
 
-	if (bio->bi_iter.bi_size != count) {
+	if (bio->bi_iter.bi_size != iomi->len) {
 		iov_iter_revert(iter, bio->bi_iter.bi_size);
 		ret = -ENOTBLK;
 		goto out_bio_release_pages;
@@ -1128,68 +1031,35 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 
 	sr->size = bio->bi_iter.bi_size;
 
-	if (dio_flags & IOMAP_DIO_USER_BACKED)
+	if (user_backed_iter(iter)) {
 		bio_set_pages_dirty(bio);
+		sr->dio_flags |= IOMAP_DIO_USER_BACKED;
+	}
 
 	if (iocb->ki_flags & IOCB_NOWAIT)
 		bio->bi_opf |= REQ_NOWAIT;
-	if ((iocb->ki_flags & IOCB_HIPRI) && !wait_for_completion) {
-		bio->bi_opf |= REQ_POLLED;
-		WRITE_ONCE(iocb->private, bio);
-	}
 
-	/*
-	 * The finishing iomap_iter() only runs the filesystem's iomap_end()
-	 * work and releases any attached folio batch.  When the caller promised
-	 * neither applies (IOMAP_DIO_NO_IOMAP_END), skip it to avoid an extra
-	 * iomap_next() call on the hot read path.
-	 */
-	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
-		iomap_dio_simple_finish(&iomi, next, count);
-
-	if (!wait_for_completion) {
-		bio->bi_end_io = iomap_dio_simple_end_io;
-		submit_bio(bio);
-		trace_iomap_dio_rw_queued(inode, iocb->ki_pos, count);
-		return -EIOCBQUEUED;
+	if (is_sync_kiocb(iocb)) {
+		submit_bio_wait(bio);
+		return iomap_dio_simple_complete(sr);
 	}
 
-	submit_bio_wait(bio);
-	return iomap_dio_simple_complete(sr);
+	if ((iocb->ki_flags & IOCB_HIPRI)) {
+		bio->bi_opf |= REQ_POLLED;
+		WRITE_ONCE(iocb->private, bio);
+	}
+	bio->bi_end_io = iomap_dio_simple_end_io;
+	submit_bio(bio);
+	trace_iomap_dio_rw_queued(iomi->inode, iocb->ki_pos, iomi->len);
+	return -EIOCBQUEUED;
 
 out_bio_release_pages:
 	bio_release_pages(bio, false);
 out_bio_put:
 	bio_put(bio);
-out_iomap_end:
-	if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
-		iomap_dio_simple_finish(&iomi, next, 0);
-	inode_dio_end(inode);
 	return ret;
 }
-
-ssize_t
-iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_iter_next_fn next,
-		const struct iomap_dio_ops *dops, unsigned int dio_flags,
-		void *private, size_t done_before)
-{
-	struct iomap_dio *dio;
-	ssize_t ret;
-
-	if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
-				       done_before)) {
-		ret = iomap_dio_simple(iocb, iter, next, private, dio_flags);
-		if (ret != -ENOTBLK)
-			return ret;
-	}
-
-	dio = __iomap_dio_rw(iocb, iter, next, dops, dio_flags, private,
-			     done_before);
-	if (IS_ERR_OR_NULL(dio))
-		return PTR_ERR_OR_ZERO(dio);
-	return iomap_dio_complete(dio);
-}
-EXPORT_SYMBOL_GPL(iomap_dio_rw);
+EXPORT_SYMBOL_GPL(__iomap_dio_simple);
 
 static int __init iomap_dio_init(void)
 {
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index 012c073c9b01..d36a77b854aa 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -127,8 +127,7 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
 	{IOMAP_DIO_FORCE_WAIT,		"DIO_FORCE_WAIT" }, \
 	{IOMAP_DIO_OVERWRITE_ONLY,	"DIO_OVERWRITE_ONLY" }, \
 	{IOMAP_DIO_PARTIAL,		"DIO_PARTIAL" }, \
-	{IOMAP_DIO_FSBLOCK_ALIGNED,	"DIO_FSBLOCK_ALIGNED" }, \
-	{IOMAP_DIO_NO_IOMAP_END,	"DIO_NO_IOMAP_END" }
+	{IOMAP_DIO_FSBLOCK_ALIGNED,	"DIO_FSBLOCK_ALIGNED" }
 
 DECLARE_EVENT_CLASS(iomap_class,
 	TP_PROTO(struct inode *inode, struct iomap *iomap),
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index 24c621ac984a..a4f99128b46c 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -458,8 +458,8 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		}
 
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL,
-				IOMAP_DIO_NO_IOMAP_END, NULL, 0);
+		ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL, 0,
+				NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, to);
 	}
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index ea709e4c9607..877f9024d333 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -251,8 +251,6 @@ xfs_file_dio_read(
 	struct iov_iter		*to)
 {
 	struct xfs_inode	*ip = XFS_I(file_inode(iocb->ki_filp));
-	unsigned int		dio_flags = IOMAP_DIO_NO_IOMAP_END;
-	const struct iomap_dio_ops *dio_ops = NULL;
 	ssize_t			ret;
 
 	trace_xfs_file_direct_read(iocb, to);
@@ -266,11 +264,15 @@ xfs_file_dio_read(
 	if (ret)
 		return ret;
 	if (mapping_stable_writes(iocb->ki_filp->f_mapping)) {
-		dio_ops = &xfs_dio_read_bounce_ops;
-		dio_flags |= IOMAP_DIO_BOUNCE;
+		ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next,
+				&xfs_dio_read_bounce_ops, IOMAP_DIO_BOUNCE,
+				NULL, 0);
+	} else {
+		ret = iomap_dio_read_simple(iocb, to, xfs_read_iomap_begin);
+		if (ret == -ENOTBLK)
+			ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, NULL,
+					0, NULL, 0);
 	}
-	ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, dio_ops, dio_flags,
-			NULL, 0);
 	xfs_iunlock(ip, XFS_IOLOCK_SHARED);
 
 	return ret;
diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
index 1314c95f7416..534cdf537224 100644
--- a/fs/xfs/xfs_iomap.c
+++ b/fs/xfs/xfs_iomap.c
@@ -2167,7 +2167,7 @@ xfs_buffered_write_iomap_end(
 DEFINE_IOMAP_ITER_NEXT_END(xfs_buffered_write_iomap_next,
 		xfs_buffered_write_iomap_begin, xfs_buffered_write_iomap_end);
 
-static int
+inline int
 xfs_read_iomap_begin(
 	struct inode		*inode,
 	loff_t			offset,
diff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h
index 01875d20fb66..d271db75d891 100644
--- a/fs/xfs/xfs_iomap.h
+++ b/fs/xfs/xfs_iomap.h
@@ -55,6 +55,9 @@ int xfs_direct_write_iomap_next(const struct iomap_iter *iter,
 		struct iomap *iomap, struct iomap *srcmap);
 int xfs_zoned_direct_write_iomap_next(const struct iomap_iter *iter,
 		struct iomap *iomap, struct iomap *srcmap);
+inline int xfs_read_iomap_begin(struct inode *inode, loff_t offset,
+		loff_t length, unsigned flags, struct iomap *iomap,
+		struct iomap *srcmap);
 int xfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
 		struct iomap *srcmap);
 int xfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 519613f281aa..368be7e87c6a 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -10,6 +10,7 @@
 #include <linux/mm_types.h>
 #include <linux/blkdev.h>
 #include <linux/folio_batch.h>
+#include <linux/pagemap.h>
 
 struct address_space;
 struct fiemap_extent_info;
@@ -668,14 +669,6 @@ struct iomap_dio_ops {
  */
 #define IOMAP_DIO_BOUNCE		(1 << 4)
 
-/*
- * The caller's ->iomap_next() has no ->iomap_end() work to run and this I/O
- * does not use a folio batch, so the iomap_dio_simple() fast path may skip the
- * finishing iomap_iter() call entirely.  Only meaningful on the simple read
- * fast path; ignored by the generic __iomap_dio_rw() slow path.
- */
-#define IOMAP_DIO_NO_IOMAP_END		(1 << 5)
-
 ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 		iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags, void *private, size_t done_before);
@@ -685,6 +678,70 @@ struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 ssize_t iomap_dio_complete(struct iomap_dio *dio);
 void iomap_dio_bio_end_io(struct bio *bio);
 
+/*
+ * Fast path for small, block-aligned direct I/Os that map to a single
+ * contiguous on-disk extent.
+ *
+ * @iter must describe a non-empty READ no larger than the inode block size:
+ * writes, zero-length I/O, and larger requests need the generic iomap direct
+ * I/O path.
+ *
+ * Does not support iomap_dio_ops, dio_flags, done_before or private data.
+ * The range must also stay within i_size and encrypted inodes must use the
+ * generic iomap direct I/O path.
+ *
+ * -ENOTBLK inidicates the generic path must be used by the caller instead.
+ * Any other errno is a real result and is propagated as-is, in particular
+ * -EAGAIN for IOCB_NOWAIT must reach the caller.
+ *
+ * The caller can only provide an iomap begin handler, and the iterator
+ * is never advanced.
+ */
+ssize_t __iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+		iomap_iter_begin_fn begin, struct iomap_iter *iomi);
+static __always_inline ssize_t iomap_dio_read_simple(struct kiocb *iocb,
+		struct iov_iter *iter, iomap_iter_begin_fn begin)
+{
+	struct iomap_iter iomi = {
+		.inode		= file_inode(iocb->ki_filp),
+		.pos		= iocb->ki_pos,
+		.len		= iov_iter_count(iter),
+		.flags		= IOMAP_DIRECT,
+	};
+	ssize_t ret;
+
+	if (!iomi.len)
+		return 0;
+
+	/*
+	 * Simple dio is an optimization for small IO. Filter out large IO
+	 * early as it's the most common case to fail for typical direct IO
+	 * workloads.
+	 */
+	if (iomi.len > iomi.inode->i_sb->s_blocksize)
+		return -ENOTBLK;
+	if (iocb->ki_pos + iomi.len > i_size_read(iomi.inode))
+		return -ENOTBLK;
+	if (IS_ENCRYPTED(iomi.inode))
+		return -ENOTBLK;
+
+	ret = kiocb_write_and_wait(iocb, iomi.len);
+	if (ret)
+		return ret;
+
+	if (iocb->ki_flags & IOCB_NOWAIT)
+		iomi.flags |= IOMAP_NOWAIT;
+
+	inode_dio_begin(iomi.inode);
+	ret = begin(iomi.inode, iomi.pos, iomi.len, iomi.flags, &iomi.iomap,
+			&iomi.srcmap);
+	if (!ret) 
+		ret = __iomap_dio_simple(iocb, iter, begin, &iomi);
+	if (ret <= 0 && ret != -EIOCBQUEUED)
+		inode_dio_end(iomi.inode);
+	return ret;
+}
+
 #ifdef CONFIG_SWAP
 struct file;
 struct swap_info_struct;

Re: [PATCH v3 00/20] iomap: convert to in-iter iomap_next() model

[changfengnan]

> From: "Christoph Hellwig"<hch@lst.de>
> Date:  Tue, Jul 21, 2026, 22:29
> Subject:  Re: [PATCH v3 00/20] iomap: convert to in-iter iomap_next() model
> To: "Joanne Koong"<joannelkoong@gmail.com>
> Cc: "Christian Brauner"<brauner@kernel.org>, <hch@lst.de>, "Darrick J . Wong"<djwong@kernel.org>, <linux-fsdevel@vger.kernel.org>, <changfengnan@bytedance.com>, <kbusch@kernel.org>, "Matthew Wilcox"<willy@infradead.org>, "Jan Kara"<jack@suse.cz>, "Jonathan Corbet"<corbet@lwn.net>, "David Sterba"<dsterba@suse.com>, "Dan Williams"<djbw@kernel.org>, "Gao Xiang"<xiang@kernel.org>, "Namjae Jeon"<linkinjeon@kernel.org>, "Theodore Ts'o"<tytso@mit.edu>, "Jaegeuk Kim"<jaegeuk@kernel.org>, "Miklos Szeredi"<miklos@szeredi.hu>, "Andreas Gruenbacher"<agruenba@redhat.com>, "Mikulas Patocka"<mikulas@artax.karlin.mff.cuni.cz>, "Hyunchul Lee"<hyc.lee@gmail.com>, "Konstantin Komarov"<almaz.alexandrovich@paragon-software.com>, "Carlos Maiolino"<cem@kernel.org>, "Damien Le Moal"<dlemoal@kernel.org>
> The submission seems to be missing most of the relevant mailing lists,
> e.g., xfs for iomap itself and the file system-specific lists.
> 
> For now I didn't have too much time to look at it, and the little I had
> I used to jump on the simple direct I/O thing, as that's the only part
> that stood out as something I didn't like.  The new "don't advance the
> iter" magic flag just didn't feel right.
> 
> What I came up with is the patch below which sits ontop of your entire
> series.  I redoes the simple read path very heavily and now requires the
> file system to opt in, and provide the begin callback only, i.e. we
> require the file system to only call it if it can, which simplifies the
> logic a lot, but requires more care from the file systems, and exposing
> the old iomap_begin callback.
> 
> Fengnan, can you check if this keeps your performance?  It does more
> inlining and avoid the direct call, but hopefully doesn't inline too
> much like the previous attempt.  If this works, it probably needs
> to move before the conversion.

Hi Christoph,
I tested the simple direct I/O path and did not observe any performance
degradation.

> 
> ---
> diff --git a/fs/exfat/file.c b/fs/exfat/file.c
> index adda6342de5b..67b226edb287 100644
> --- a/fs/exfat/file.c
> +++ b/fs/exfat/file.c
> @@ -809,8 +809,8 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
>  
>          if (iocb->ki_flags & IOCB_DIRECT) {
>                  file_accessed(iocb->ki_filp);
> -                ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL,
> -                                IOMAP_DIO_NO_IOMAP_END, NULL, 0);
> +                ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL, 0, NULL,
> +                                0);
>          } else {
>                  ret = generic_file_read_iter(iocb, iter);
>          }
> diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
> index 755fde1baf03..66d281d00d3e 100644
> --- a/fs/ext4/ext4.h
> +++ b/fs/ext4/ext4.h
> @@ -4004,6 +4004,8 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end)
>                  io_end->flag &= ~EXT4_IO_END_UNWRITTEN;
>  }
>  
> +inline int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> +                unsigned flags, struct iomap *iomap, struct iomap *srcmap);
>  int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
>                  struct iomap *srcmap);
>  int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,
> diff --git a/fs/ext4/file.c b/fs/ext4/file.c
> index bf0c18cf1017..491854e93f92 100644
> --- a/fs/ext4/file.c
> +++ b/fs/ext4/file.c
> @@ -91,8 +91,10 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
>                  return generic_file_read_iter(iocb, to);
>          }
>  
> -        ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL,
> -                           IOMAP_DIO_NO_IOMAP_END, NULL, 0);
> +
> +        ret = iomap_dio_read_simple(iocb, to, ext4_iomap_begin);
> +        if (ret == -ENOTBLK)
> +                ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
>          inode_unlock_shared(inode);
>  
>          file_accessed(iocb->ki_filp);
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index f455708a8c31..973ee7a97f61 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -3771,7 +3771,7 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,
>  }
>  
>  
> -static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> +inline int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
>                  unsigned flags, struct iomap *iomap, struct iomap *srcmap)
>  {
>          int ret;
> diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
> index 9c50adc93cfa..33c2c6d3e134 100644
> --- a/fs/iomap/direct-io.c
> +++ b/fs/iomap/direct-io.c
> @@ -894,6 +894,21 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
>  }
>  EXPORT_SYMBOL_GPL(__iomap_dio_rw);
>  
> +ssize_t
> +iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_iter_next_fn next,
> +                const struct iomap_dio_ops *dops, unsigned int dio_flags,
> +                void *private, size_t done_before)
> +{
> +        struct iomap_dio *dio;
> +
> +        dio = __iomap_dio_rw(iocb, iter, next, dops, dio_flags, private,
> +                             done_before);
> +        if (IS_ERR_OR_NULL(dio))
> +                return PTR_ERR_OR_ZERO(dio);
> +        return iomap_dio_complete(dio);
> +}
> +EXPORT_SYMBOL_GPL(iomap_dio_rw);
> +
>  struct iomap_dio_simple {
>          struct kiocb                *iocb;
>          size_t                        size;
> @@ -968,159 +983,47 @@ static void iomap_dio_simple_end_io(struct bio *bio)
>          iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
>  }
>  
> -static inline bool
> -iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
> -                           const struct iomap_dio_ops *dops,
> -                           unsigned int dio_flags, size_t done_before)
> +ssize_t __iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
> +                iomap_iter_begin_fn begin, struct iomap_iter *iomi)
>  {
> -        struct inode *inode = file_inode(iocb->ki_filp);
> -        size_t count = iov_iter_count(iter);
> -
> -        if (dops || done_before)
> -                return false;
> -        if (iov_iter_rw(iter) != READ)
> -                return false;
> -        if (!count)
> -                return false;
> -        /*
> -         * Simple dio is an optimization for small IO. Filter out large IO
> -         * early as it's the most common case to fail for typical direct IO
> -         * workloads.
> -         */
> -        if (count > inode->i_sb->s_blocksize)
> -                return false;
> -        if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
> -                         IOMAP_DIO_BOUNCE))
> -                return false;
> -        if (iocb->ki_pos + count > i_size_read(inode))
> -                return false;
> -        if (IS_ENCRYPTED(inode))
> -                return false;
> -
> -        return true;
> -}
> -
> -static inline void
> -iomap_dio_simple_finish(struct iomap_iter *iomi, iomap_iter_next_fn next,
> -                size_t written)
> -{
> -        iomi->iter_start_pos = iomi->pos;
> -        iomi->pos += written;
> -        iomi->len -= written;
> -        next(iomi, &iomi->iomap, &iomi->srcmap);
> -}
> -
> -/*
> - * Fast path for small, block-aligned direct I/Os that map to a single
> - * contiguous on-disk extent.
> - *
> - * iomap_dio_simple_supported() enforces the cheap up-front constraints before
> - * entering this path.
> - *
> - * @dops must be NULL: a non-NULL @dops means the caller wants its
> - * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
> - * allocated from the filesystem-private @dops->bio_set (whose front_pad sizes a
> - * filesystem-private wrapper around the bio).  The fast path instead allocates
> - * from the shared iomap_dio_simple_pool, whose front_pad matches struct
> - * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
> - * back to __iomap_dio_rw() in that case.
> - *
> - * @done_before must be zero: a non-zero caller-accumulated residual cannot be
> - * carried through a single-bio inline completion.
> - *
> - * @iter must describe a non-empty READ no larger than the inode block size:
> - * writes, zero-length I/O, and larger requests need the generic iomap direct
> - * I/O path.
> - *
> - * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
> - * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
> - * I/O, or bouncing.  The range must also stay within i_size and encrypted
> - * inodes must use the generic iomap direct I/O path. IOMAP_DIO_NO_IOMAP_END is
> - * a fast-path optimization the caller can set if there is no work that needs to
> - * be finished after a mapping.
> - *
> - * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
> - * decides the request does not fit the fast path.  In that case we proceed to
> - * the generic __iomap_dio_rw() slow path.  Any other errno is a real result and
> - * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
> - * caller.
> - */
> -static ssize_t
> -iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
> -                 iomap_iter_next_fn next, void *private, unsigned int dio_flags)
> -{
> -        struct inode *inode = file_inode(iocb->ki_filp);
> -        size_t count = iov_iter_count(iter);
> -        bool wait_for_completion = is_sync_kiocb(iocb);
> -        struct iomap_iter iomi = {
> -                .inode                = inode,
> -                .pos                = iocb->ki_pos,
> -                .len                = count,
> -                .flags                = IOMAP_DIRECT,
> -                .private        = private,
> -        };
>          struct iomap_dio_simple *sr;
>          unsigned int alignment;
>          struct bio *bio;
>          ssize_t ret;
>  
> -        if (iocb->ki_flags & IOCB_NOWAIT)
> -                iomi.flags |= IOMAP_NOWAIT;
> -
> -        ret = kiocb_write_and_wait(iocb, count);
> -        if (ret)
> -                return ret;
> -
> -        inode_dio_begin(inode);
> -
> -        ret = next(&iomi, &iomi.iomap, &iomi.srcmap);
> -        if (ret <= 0) {
> -                inode_dio_end(inode);
> -                /* the first iteration must yield a mapping (>0) or an error */
> -                if (WARN_ON_ONCE(!ret))
> -                        return -EFAULT;
> -                return ret;
> -        }
> -
> -        if (iomi.iomap.type != IOMAP_MAPPED ||
> -            iomi.iomap.offset + iomi.iomap.length < iomi.pos + count ||
> -            (iomi.iomap.flags & IOMAP_F_INTEGRITY)) {
> -                ret = -ENOTBLK;
> -                goto out_iomap_end;
> -        }
> +        if (iomi->iomap.type != IOMAP_MAPPED ||
> +            iomi->iomap.offset + iomi->iomap.length < iomi->pos + iomi->len ||
> +            (iomi->iomap.flags & IOMAP_F_INTEGRITY))
> +                return -ENOTBLK;
>  
> -        alignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);
> -        if ((iomi.pos | count) & (alignment - 1)) {
> -                ret = -EINVAL;
> -                goto out_iomap_end;
> -        }
> +        alignment = iomap_dio_alignment(iomi->inode, iomi->iomap.bdev, 0);
> +        if ((iomi->pos | iomi->len) & (alignment - 1))
> +                return -EINVAL;
>  
> -        if (!wait_for_completion && unlikely(!inode->i_sb->s_dio_done_wq)) {
> -                ret = sb_init_dio_done_wq(inode->i_sb);
> +        if (unlikely(!iomi->inode->i_sb->s_dio_done_wq &&
> +                        !is_sync_kiocb(iocb))) {
> +                ret = sb_init_dio_done_wq(iomi->inode->i_sb);
>                  if (ret < 0)
> -                        goto out_iomap_end;
> +                        return ret;
>          }
>  
> -        trace_iomap_dio_rw_begin(iocb, iter, dio_flags, 0);
> -
> -        if (user_backed_iter(iter))
> -                dio_flags |= IOMAP_DIO_USER_BACKED;
> +        trace_iomap_dio_rw_begin(iocb, iter, 0, 0);
>  
> -        bio = bio_alloc_bioset(iomi.iomap.bdev,
> +        bio = bio_alloc_bioset(iomi->iomap.bdev,
>                                 bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),
>                                 REQ_OP_READ, GFP_KERNEL, &iomap_dio_simple_pool);
>          sr = container_of(bio, struct iomap_dio_simple, bio);
>          sr->iocb = iocb;
> -        sr->dio_flags = dio_flags;
> +        sr->dio_flags = 0;
>  
> -        bio->bi_iter.bi_sector = iomap_sector(&iomi.iomap, iomi.pos);
> +        bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
>          bio->bi_ioprio = iocb->ki_ioprio;
>  
>          ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
>          if (unlikely(ret))
>                  goto out_bio_put;
>  
> -        if (bio->bi_iter.bi_size != count) {
> +        if (bio->bi_iter.bi_size != iomi->len) {
>                  iov_iter_revert(iter, bio->bi_iter.bi_size);
>                  ret = -ENOTBLK;
>                  goto out_bio_release_pages;
> @@ -1128,68 +1031,35 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
>  
>          sr->size = bio->bi_iter.bi_size;
>  
> -        if (dio_flags & IOMAP_DIO_USER_BACKED)
> +        if (user_backed_iter(iter)) {
>                  bio_set_pages_dirty(bio);
> +                sr->dio_flags |= IOMAP_DIO_USER_BACKED;
> +        }
>  
>          if (iocb->ki_flags & IOCB_NOWAIT)
>                  bio->bi_opf |= REQ_NOWAIT;
> -        if ((iocb->ki_flags & IOCB_HIPRI) && !wait_for_completion) {
> -                bio->bi_opf |= REQ_POLLED;
> -                WRITE_ONCE(iocb->private, bio);
> -        }
>  
> -        /*
> -         * The finishing iomap_iter() only runs the filesystem's iomap_end()
> -         * work and releases any attached folio batch.  When the caller promised
> -         * neither applies (IOMAP_DIO_NO_IOMAP_END), skip it to avoid an extra
> -         * iomap_next() call on the hot read path.
> -         */
> -        if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
> -                iomap_dio_simple_finish(&iomi, next, count);
> -
> -        if (!wait_for_completion) {
> -                bio->bi_end_io = iomap_dio_simple_end_io;
> -                submit_bio(bio);
> -                trace_iomap_dio_rw_queued(inode, iocb->ki_pos, count);
> -                return -EIOCBQUEUED;
> +        if (is_sync_kiocb(iocb)) {
> +                submit_bio_wait(bio);
> +                return iomap_dio_simple_complete(sr);
>          }
>  
> -        submit_bio_wait(bio);
> -        return iomap_dio_simple_complete(sr);
> +        if ((iocb->ki_flags & IOCB_HIPRI)) {
> +                bio->bi_opf |= REQ_POLLED;
> +                WRITE_ONCE(iocb->private, bio);
> +        }
> +        bio->bi_end_io = iomap_dio_simple_end_io;
> +        submit_bio(bio);
> +        trace_iomap_dio_rw_queued(iomi->inode, iocb->ki_pos, iomi->len);
> +        return -EIOCBQUEUED;
>  
>  out_bio_release_pages:
>          bio_release_pages(bio, false);
>  out_bio_put:
>          bio_put(bio);
> -out_iomap_end:
> -        if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
> -                iomap_dio_simple_finish(&iomi, next, 0);
> -        inode_dio_end(inode);
>          return ret;
>  }
> -
> -ssize_t
> -iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_iter_next_fn next,
> -                const struct iomap_dio_ops *dops, unsigned int dio_flags,
> -                void *private, size_t done_before)
> -{
> -        struct iomap_dio *dio;
> -        ssize_t ret;
> -
> -        if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
> -                                       done_before)) {
> -                ret = iomap_dio_simple(iocb, iter, next, private, dio_flags);
> -                if (ret != -ENOTBLK)
> -                        return ret;
> -        }
> -
> -        dio = __iomap_dio_rw(iocb, iter, next, dops, dio_flags, private,
> -                             done_before);
> -        if (IS_ERR_OR_NULL(dio))
> -                return PTR_ERR_OR_ZERO(dio);
> -        return iomap_dio_complete(dio);
> -}
> -EXPORT_SYMBOL_GPL(iomap_dio_rw);
> +EXPORT_SYMBOL_GPL(__iomap_dio_simple);
>  
>  static int __init iomap_dio_init(void)
>  {
> diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
> index 012c073c9b01..d36a77b854aa 100644
> --- a/fs/iomap/trace.h
> +++ b/fs/iomap/trace.h
> @@ -127,8 +127,7 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
>          {IOMAP_DIO_FORCE_WAIT,                "DIO_FORCE_WAIT" }, \
>          {IOMAP_DIO_OVERWRITE_ONLY,        "DIO_OVERWRITE_ONLY" }, \
>          {IOMAP_DIO_PARTIAL,                "DIO_PARTIAL" }, \
> -        {IOMAP_DIO_FSBLOCK_ALIGNED,        "DIO_FSBLOCK_ALIGNED" }, \
> -        {IOMAP_DIO_NO_IOMAP_END,        "DIO_NO_IOMAP_END" }
> +        {IOMAP_DIO_FSBLOCK_ALIGNED,        "DIO_FSBLOCK_ALIGNED" }
>  
>  DECLARE_EVENT_CLASS(iomap_class,
>          TP_PROTO(struct inode *inode, struct iomap *iomap),
> diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
> index 24c621ac984a..a4f99128b46c 100644
> --- a/fs/ntfs/file.c
> +++ b/fs/ntfs/file.c
> @@ -458,8 +458,8 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
>                  }
>  
>                  file_accessed(iocb->ki_filp);
> -                ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL,
> -                                IOMAP_DIO_NO_IOMAP_END, NULL, 0);
> +                ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL, 0,
> +                                NULL, 0);
>          } else {
>                  ret = generic_file_read_iter(iocb, to);
>          }
> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> index ea709e4c9607..877f9024d333 100644
> --- a/fs/xfs/xfs_file.c
> +++ b/fs/xfs/xfs_file.c
> @@ -251,8 +251,6 @@ xfs_file_dio_read(
>          struct iov_iter                *to)
>  {
>          struct xfs_inode        *ip = XFS_I(file_inode(iocb->ki_filp));
> -        unsigned int                dio_flags = IOMAP_DIO_NO_IOMAP_END;
> -        const struct iomap_dio_ops *dio_ops = NULL;
>          ssize_t                        ret;
>  
>          trace_xfs_file_direct_read(iocb, to);
> @@ -266,11 +264,15 @@ xfs_file_dio_read(
>          if (ret)
>                  return ret;
>          if (mapping_stable_writes(iocb->ki_filp->f_mapping)) {
> -                dio_ops = &xfs_dio_read_bounce_ops;
> -                dio_flags |= IOMAP_DIO_BOUNCE;
> +                ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next,
> +                                &xfs_dio_read_bounce_ops, IOMAP_DIO_BOUNCE,
> +                                NULL, 0);
> +        } else {
> +                ret = iomap_dio_read_simple(iocb, to, xfs_read_iomap_begin);
> +                if (ret == -ENOTBLK)
> +                        ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, NULL,
> +                                        0, NULL, 0);
>          }
> -        ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, dio_ops, dio_flags,
> -                        NULL, 0);
>          xfs_iunlock(ip, XFS_IOLOCK_SHARED);
>  
>          return ret;
> diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
> index 1314c95f7416..534cdf537224 100644
> --- a/fs/xfs/xfs_iomap.c
> +++ b/fs/xfs/xfs_iomap.c
> @@ -2167,7 +2167,7 @@ xfs_buffered_write_iomap_end(
>  DEFINE_IOMAP_ITER_NEXT_END(xfs_buffered_write_iomap_next,
>                  xfs_buffered_write_iomap_begin, xfs_buffered_write_iomap_end);
>  
> -static int
> +inline int
>  xfs_read_iomap_begin(
>          struct inode                *inode,
>          loff_t                        offset,
> diff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h
> index 01875d20fb66..d271db75d891 100644
> --- a/fs/xfs/xfs_iomap.h
> +++ b/fs/xfs/xfs_iomap.h
> @@ -55,6 +55,9 @@ int xfs_direct_write_iomap_next(const struct iomap_iter *iter,
>                  struct iomap *iomap, struct iomap *srcmap);
>  int xfs_zoned_direct_write_iomap_next(const struct iomap_iter *iter,
>                  struct iomap *iomap, struct iomap *srcmap);
> +inline int xfs_read_iomap_begin(struct inode *inode, loff_t offset,
> +                loff_t length, unsigned flags, struct iomap *iomap,
> +                struct iomap *srcmap);
>  int xfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
>                  struct iomap *srcmap);
>  int xfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
> diff --git a/include/linux/iomap.h b/include/linux/iomap.h
> index 519613f281aa..368be7e87c6a 100644
> --- a/include/linux/iomap.h
> +++ b/include/linux/iomap.h
> @@ -10,6 +10,7 @@
>  #include <linux/mm_types.h>
>  #include <linux/blkdev.h>
>  #include <linux/folio_batch.h>
> +#include <linux/pagemap.h>
>  
>  struct address_space;
>  struct fiemap_extent_info;
> @@ -668,14 +669,6 @@ struct iomap_dio_ops {
>   */
>  #define IOMAP_DIO_BOUNCE                (1 << 4)
>  
> -/*
> - * The caller's ->iomap_next() has no ->iomap_end() work to run and this I/O
> - * does not use a folio batch, so the iomap_dio_simple() fast path may skip the
> - * finishing iomap_iter() call entirely.  Only meaningful on the simple read
> - * fast path; ignored by the generic __iomap_dio_rw() slow path.
> - */
> -#define IOMAP_DIO_NO_IOMAP_END                (1 << 5)
> -
>  ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
>                  iomap_iter_next_fn next, const struct iomap_dio_ops *dops,
>                  unsigned int dio_flags, void *private, size_t done_before);
> @@ -685,6 +678,70 @@ struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
>  ssize_t iomap_dio_complete(struct iomap_dio *dio);
>  void iomap_dio_bio_end_io(struct bio *bio);
>  
> +/*
> + * Fast path for small, block-aligned direct I/Os that map to a single
> + * contiguous on-disk extent.
> + *
> + * @iter must describe a non-empty READ no larger than the inode block size:
> + * writes, zero-length I/O, and larger requests need the generic iomap direct
> + * I/O path.
> + *
> + * Does not support iomap_dio_ops, dio_flags, done_before or private data.
> + * The range must also stay within i_size and encrypted inodes must use the
> + * generic iomap direct I/O path.
> + *
> + * -ENOTBLK inidicates the generic path must be used by the caller instead.
> + * Any other errno is a real result and is propagated as-is, in particular
> + * -EAGAIN for IOCB_NOWAIT must reach the caller.
> + *
> + * The caller can only provide an iomap begin handler, and the iterator
> + * is never advanced.
> + */
> +ssize_t __iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
> +                iomap_iter_begin_fn begin, struct iomap_iter *iomi);
> +static __always_inline ssize_t iomap_dio_read_simple(struct kiocb *iocb,
> +                struct iov_iter *iter, iomap_iter_begin_fn begin)
> +{
> +        struct iomap_iter iomi = {
> +                .inode                = file_inode(iocb->ki_filp),
> +                .pos                = iocb->ki_pos,
> +                .len                = iov_iter_count(iter),
> +                .flags                = IOMAP_DIRECT,
> +        };
> +        ssize_t ret;
> +
> +        if (!iomi.len)
> +                return 0;
> +
> +        /*
> +         * Simple dio is an optimization for small IO. Filter out large IO
> +         * early as it's the most common case to fail for typical direct IO
> +         * workloads.
> +         */
> +        if (iomi.len > iomi.inode->i_sb->s_blocksize)
> +                return -ENOTBLK;
> +        if (iocb->ki_pos + iomi.len > i_size_read(iomi.inode))
> +                return -ENOTBLK;
> +        if (IS_ENCRYPTED(iomi.inode))
> +                return -ENOTBLK;
> +
> +        ret = kiocb_write_and_wait(iocb, iomi.len);
> +        if (ret)
> +                return ret;
> +
> +        if (iocb->ki_flags & IOCB_NOWAIT)
> +                iomi.flags |= IOMAP_NOWAIT;
> +
> +        inode_dio_begin(iomi.inode);
> +        ret = begin(iomi.inode, iomi.pos, iomi.len, iomi.flags, &iomi.iomap,
> +                        &iomi.srcmap);
> +        if (!ret) 
> +                ret = __iomap_dio_simple(iocb, iter, begin, &iomi);
> +        if (ret <= 0 && ret != -EIOCBQUEUED)
> +                inode_dio_end(iomi.inode);
> +        return ret;
> +}
> +
>  #ifdef CONFIG_SWAP
>  struct file;
>  struct swap_info_struct;
> 

Re: [PATCH v3 00/20] iomap: convert to in-iter iomap_next() model

[Christoph Hellwig]

On Wed, Jul 22, 2026 at 03:04:24PM +0800, changfengnan wrote:
> Hi Christoph,
> I tested the simple direct I/O path and did not observe any performance
> degradation.

Thanks.  But no improvement either?

I'll respin this to sit before the iter_next conversion and will
send it out.

Re: [PATCH v3 00/20] iomap: convert to in-iter iomap_next() model

[changfengnan]

> From: "Christoph Hellwig"<hch@lst.de>
> Date:  Wed, Jul 22, 2026, 17:04
> Subject:  Re: [PATCH v3 00/20] iomap: convert to in-iter iomap_next() model
> To: "changfengnan"<changfengnan@bytedance.com>
> Cc: "Christoph Hellwig"<hch@lst.de>, "Joanne Koong"<joannelkoong@gmail.com>, "Christian Brauner"<brauner@kernel.org>, "Darrick J . Wong"<djwong@kernel.org>, <linux-fsdevel@vger.kernel.org>, <kbusch@kernel.org>, "Matthew Wilcox"<willy@infradead.org>, "Jan Kara"<jack@suse.cz>, "Jonathan Corbet"<corbet@lwn.net>, "David Sterba"<dsterba@suse.com>, "Dan Williams"<djbw@kernel.org>, "Gao Xiang"<xiang@kernel.org>, "Namjae Jeon"<linkinjeon@kernel.org>, "Theodore Ts'o"<tytso@mit.edu>, "Jaegeuk Kim"<jaegeuk@kernel.org>, "Miklos Szeredi"<miklos@szeredi.hu>, "Andreas Gruenbacher"<agruenba@redhat.com>, "Mikulas Patocka"<mikulas@artax.karlin.mff.cuni.cz>, "Hyunchul Lee"<hyc.lee@gmail.com>, "Konstantin Komarov"<almaz.alexandrovich@paragon-software.com>, "Carlos Maiolino"<cem@kernel.org>, "Damien Le Moal"<dlemoal@kernel.org>
> On Wed, Jul 22, 2026 at 03:04:24PM +0800, changfengnan wrote:
> > Hi Christoph,
> > I tested the simple direct I/O path and did not observe any performance
> > degradation.
> 
> Thanks.  But no improvement either?

yes, I haven't seen any consistent improvement.

> 
> I'll respin this to sit before the iter_next conversion and will
> send it out.
> 

Re: [PATCH v3 01/20] iomap: split iomap_iter() logic into iomap_iter_next()

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH v3 02/20] iomap: add ->iomap_next()

[Christoph Hellwig]

As mentioned in the coverletter not a fan of the simple dio part
of this, but I'll post a version of my patch for that that slots
in before this.  For the core iomap bits:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH v3 12/20] block: convert iomap ops to ->iomap_next()

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH v3 00/20] iomap: convert to in-iter iomap_next() model

[Joanne Koong]

On Tue, Jul 21, 2026 at 7:28 AM Christoph Hellwig <hch@lst.de> wrote:
>
> The submission seems to be missing most of the relevant mailing lists,
> e.g., xfs for iomap itself and the file system-specific lists.

I'm not sure what the general etiquette is, so I tried to avoid
spamming people's inboxes by removing the filesystem-specific lists
and only cc'ing the maintainers since everyone is cc'ed for the whole
series and not just their individual patch. If the filesystem specific
lists should also be added though, I will make sure to do that going
forward.

>
> For now I didn't have too much time to look at it, and the little I had
> I used to jump on the simple direct I/O thing, as that's the only part
> that stood out as something I didn't like.  The new "don't advance the
> iter" magic flag just didn't feel right.
>
> What I came up with is the patch below which sits ontop of your entire
> series.  I redoes the simple read path very heavily and now requires the
> file system to opt in, and provide the begin callback only, i.e. we
> require the file system to only call it if it can, which simplifies the
> logic a lot, but requires more care from the file systems, and exposing
> the old iomap_begin callback.
>
> Fengnan, can you check if this keeps your performance?  It does more
> inlining and avoid the direct call, but hopefully doesn't inline too
> much like the previous attempt.  If this works, it probably needs
> to move before the conversion.
>

Sounds good. I will fold your series in [1] into v4 after awaiting
replies on that series.

Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/20260722124931.3159664-1-hch@lst.de/