[PATCH v6 0/3] iomap: add simple dio path for small direct I/O

[PATCH v6 0/3] iomap: add simple dio path for small direct I/O

[Fengnan Chang]

When running 4K random read workloads on high-performance Gen5 NVMe
SSDs, the software overhead in the iomap direct I/O path
(__iomap_dio_rw) becomes a significant bottleneck.

Using io_uring with poll mode for a 4K randread test on a raw block
device:
taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1
-n1 -P1 /dev/nvme10n1
Result: ~3.2M IOPS

Running the exact same workload on ext4 and XFS:
taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1
-n1 -P1 /mnt/testfile
Result: ~1.92M IOPS

Profiling the ext4 workload reveals that a significant portion of CPU
time is spent on memory allocation and the iomap state machine
iteration:
  5.33%  [kernel]  [k] __iomap_dio_rw
  3.26%  [kernel]  [k] iomap_iter
  2.37%  [kernel]  [k] iomap_dio_bio_iter
  2.35%  [kernel]  [k] kfree
  1.33%  [kernel]  [k] iomap_dio_complete

Introduce a simple dio path to reduce the overhead of iomap. It is
triggered when the request satisfies all of:
- a READ request whose I/O size is <= inode blocksize (fits in a single
  block, no splits);
- no custom iomap_dio_ops (dops) registered by the filesystem;
- no caller-accumulated residual (done_before == 0);
- none of IOMAP_DIO_FORCE_WAIT / IOMAP_DIO_PARTIAL / IOMAP_DIO_BOUNCE
  set, the range is within i_size, and the inode is not encrypted.

The bio is allocated from a dedicated bioset whose front_pad embeds
struct iomap_dio_simple, so the whole request lives in a single
cacheline-aligned allocation and no separate struct iomap_dio is
needed.  Completion is handled inline from ->bi_end_io for the common
success case, and only punted to the s_dio_done_wq workqueue on error.

After this optimization, the heavy generic functions disappear from the
profile, replaced by a single streamlined execution path:
  4.83%  [kernel]  [k] iomap_dio_simple

With this patch, 4K random read IOPS on ext4 increases from 1.92M to
2.19M in the original single-core io_uring poll-mode workload.

Below are the test results using fio:

  fs    workload       qd    simple=0      simple=1      gain
  ext4  libaio         1     18,740        18,761        +0.11%
  ext4  libaio         64    462,850       480,587       +3.83%
  ext4  libaio         128   459,498       478,824       +4.21%
  ext4  libaio         256   459,938       480,156       +4.40%
  ext4  io_uring       1     18,836        18,880        +0.24%
  ext4  io_uring       64    568,193       600,625       +5.71%
  ext4  io_uring       128   570,998       602,148       +5.46%
  ext4  io_uring       256   572,052       602,536       +5.33%
  ext4  io_uring_poll  1     19,283        19,272        -0.06%
  ext4  io_uring_poll  64    989,735       1,013,342     +2.39%
  ext4  io_uring_poll  128   1,467,336     1,538,444     +4.85%
  ext4  io_uring_poll  256   1,663,498     1,830,842     +10.06%
  xfs   libaio         1     18,764        18,776        +0.06%
  xfs   libaio         64    462,408       480,860       +3.99%
  xfs   libaio         128   461,280       480,819       +4.24%
  xfs   libaio         256   461,626       480,190       +4.02%
  xfs   io_uring       1     18,871        18,903        +0.17%
  xfs   io_uring       64    570,383       597,399       +4.74%
  xfs   io_uring       128   568,290       597,370       +5.12%
  xfs   io_uring       256   570,616       598,775       +4.93%
  xfs   io_uring_poll  1     19,211        19,315        +0.54%
  xfs   io_uring_poll  64    989,726       1,008,455     +1.89%
  xfs   io_uring_poll  128   1,430,426     1,513,064     +5.78%
  xfs   io_uring_poll  256   1,587,339     1,742,220     +9.76%

Changes since v5:
- Collect Reviewed-by tags from Christoph for the two prep patches.
- Drop the iomap_dio_bio_release_pages() helper and open-code the simple
  path page release logic.
- Remove unused kobject.h and sysfs.h includes.
- Clean up iomap_dio_simple_complete() to branch on bio->bi_status and
  pass the final error value to trace_iomap_dio_complete().
- Move the fast path documentation above iomap_dio_simple(), and fold
  the dops and done_before checks into iomap_dio_simple_supported().
- Fix declaration ordering, indentation, and field alignment nits.

Changes since v4:
- Update test data based on v7.2-rc1.
- Split refactoring into prep patches.
- Remove three-state atomic synchronization; use submit_bio_wait for
  sync and direct ki_complete from end_io for async.
- Drop the _read suffix from struct and function names.
- Remove bounce buffer handling as bounce requires dops.
- Remove redundant iomap.offset > pos check.
- Guard s_dio_done_wq allocation with !wait_for_completion.
- Add explicit !count early-return in supported() check.

Changes since v3:
- Fix fserror report and update test data based on v7.1-rc3.

Changes since v2:
- Update test data based on v7.1-rc3.

Fengnan Chang (3):
  iomap: factor out iomap_dio_alignment helper
  iomap: pass error code to should_report_dio_fserror directly
  iomap: add simple dio path for small direct I/O

 fs/iomap/direct-io.c | 293 +++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 286 insertions(+), 7 deletions(-)

-- 
2.39.5 (Apple Git-154)

[PATCH v6 1/3] iomap: factor out iomap_dio_alignment helper

[Fengnan Chang]

Extract the alignment computation from iomap_dio_bio_iter() into a
standalone helper so the upcoming simple direct I/O path can reuse it
without requiring a struct iomap_dio.

No functional change.

Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 fs/iomap/direct-io.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index b485e3b191daf..487c4763f3fde 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -398,6 +398,14 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
 	return ret;
 }
 
+static inline unsigned int iomap_dio_alignment(struct inode *inode,
+		struct block_device *bdev, unsigned int dio_flags)
+{
+	if (dio_flags & IOMAP_DIO_FSBLOCK_ALIGNED)
+		return i_blocksize(inode);
+	return bdev_logical_block_size(bdev);
+}
+
 static int iomap_dio_bio_iter(struct iomap_iter *iter, struct iomap_dio *dio)
 {
 	const struct iomap *iomap = &iter->iomap;
@@ -416,10 +424,7 @@ static int iomap_dio_bio_iter(struct iomap_iter *iter, struct iomap_dio *dio)
 	 * File systems that write out of place and always allocate new blocks
 	 * need each bio to be block aligned as that's the unit of allocation.
 	 */
-	if (dio->flags & IOMAP_DIO_FSBLOCK_ALIGNED)
-		alignment = fs_block_size;
-	else
-		alignment = bdev_logical_block_size(iomap->bdev);
+	alignment = iomap_dio_alignment(inode, iomap->bdev, dio->flags);
 
 	if ((pos | length) & (alignment - 1))
 		return -EINVAL;
-- 
2.39.5 (Apple Git-154)

[PATCH v6 2/3] iomap: pass error code to should_report_dio_fserror directly

[Fengnan Chang]

Change should_report_dio_fserror() to take an error code instead of the
full struct iomap_dio, decoupling it for reuse by the upcoming simple
direct I/O path.

No functional change.

Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 fs/iomap/direct-io.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 487c4763f3fde..1b9abdd831d0b 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -88,9 +88,9 @@ static inline enum fserror_type iomap_dio_err_type(const struct iomap_dio *dio)
 	return FSERR_DIRECTIO_READ;
 }
 
-static inline bool should_report_dio_fserror(const struct iomap_dio *dio)
+static inline bool should_report_dio_fserror(int error)
 {
-	switch (dio->error) {
+	switch (error) {
 	case 0:
 	case -EAGAIN:
 	case -ENOTBLK:
@@ -110,7 +110,7 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio)
 
 	if (dops && dops->end_io)
 		ret = dops->end_io(iocb, dio->size, ret, dio->flags);
-	if (should_report_dio_fserror(dio))
+	if (should_report_dio_fserror(dio->error))
 		fserror_report_io(file_inode(iocb->ki_filp),
 				  iomap_dio_err_type(dio), offset, dio->size,
 				  dio->error, GFP_NOFS);
-- 
2.39.5 (Apple Git-154)

[PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[Fengnan Chang]

When running 4K random read workloads on high-performance Gen5 NVMe
SSDs, the software overhead in the iomap direct I/O path
(__iomap_dio_rw) becomes a significant bottleneck.

Using io_uring with poll mode for a 4K randread test on a raw block
device:
taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1
-n1 -P1 /dev/nvme10n1
Result: ~3.2M IOPS

Running the exact same workload on ext4 and XFS:
taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1
-n1 -P1 /mnt/testfile
Result: ~1.92M IOPS

Profiling the ext4 workload reveals that a significant portion of CPU
time is spent on memory allocation and the iomap state machine
iteration:
  5.33%  [kernel]  [k] __iomap_dio_rw
  3.26%  [kernel]  [k] iomap_iter
  2.37%  [kernel]  [k] iomap_dio_bio_iter
  2.35%  [kernel]  [k] kfree
  1.33%  [kernel]  [k] iomap_dio_complete

Introduce a simple dio path to reduce the overhead of iomap. It is
triggered when the request satisfies all of:
- a READ request whose I/O size is <= inode blocksize (fits in a single
  block, no splits);
- no custom iomap_dio_ops (dops) registered by the filesystem;
- no caller-accumulated residual (done_before == 0);
- none of IOMAP_DIO_FORCE_WAIT / IOMAP_DIO_PARTIAL / IOMAP_DIO_BOUNCE
  set, the range is within i_size, and the inode is not encrypted.

The bio is allocated from a dedicated bioset whose front_pad embeds
struct iomap_dio_simple, so the whole request lives in a single
cacheline-aligned allocation and no separate struct iomap_dio is
needed.  Completion is handled inline from ->bi_end_io for the common
success case, and only punted to the s_dio_done_wq workqueue on error.

After this optimization, the heavy generic functions disappear from the
profile, replaced by a single streamlined execution path:
  4.83%  [kernel]  [k] iomap_dio_simple

With this patch, 4K random read IOPS on ext4 increases from 1.92M to
2.19M in the original single-core io_uring poll-mode workload.

Below are the test results using fio:

  fs    workload       qd    simple=0      simple=1      gain
  ext4  libaio         1     18,740        18,761        +0.11%
  ext4  libaio         64    462,850       480,587       +3.83%
  ext4  libaio         128   459,498       478,824       +4.21%
  ext4  libaio         256   459,938       480,156       +4.40%
  ext4  io_uring       1     18,836        18,880        +0.24%
  ext4  io_uring       64    568,193       600,625       +5.71%
  ext4  io_uring       128   570,998       602,148       +5.46%
  ext4  io_uring       256   572,052       602,536       +5.33%
  ext4  io_uring_poll  1     19,283        19,272        -0.06%
  ext4  io_uring_poll  64    989,735       1,013,342     +2.39%
  ext4  io_uring_poll  128   1,467,336     1,538,444     +4.85%
  ext4  io_uring_poll  256   1,663,498     1,830,842     +10.06%
  xfs   libaio         1     18,764        18,776        +0.06%
  xfs   libaio         64    462,408       480,860       +3.99%
  xfs   libaio         128   461,280       480,819       +4.24%
  xfs   libaio         256   461,626       480,190       +4.02%
  xfs   io_uring       1     18,871        18,903        +0.17%
  xfs   io_uring       64    570,383       597,399       +4.74%
  xfs   io_uring       128   568,290       597,370       +5.12%
  xfs   io_uring       256   570,616       598,775       +4.93%
  xfs   io_uring_poll  1     19,211        19,315        +0.54%
  xfs   io_uring_poll  64    989,726       1,008,455     +1.89%
  xfs   io_uring_poll  128   1,430,426     1,513,064     +5.78%
  xfs   io_uring_poll  256   1,587,339     1,742,220     +9.76%

Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
---
 fs/iomap/direct-io.c | 274 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 274 insertions(+)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 1b9abdd831d0b..ca790239e5eb3 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -10,6 +10,7 @@
 #include <linux/iomap.h>
 #include <linux/task_io_accounting_ops.h>
 #include <linux/fserror.h>
+#include <linux/init.h>
 #include "internal.h"
 #include "trace.h"
 
@@ -893,12 +894,277 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 }
 EXPORT_SYMBOL_GPL(__iomap_dio_rw);
 
+struct iomap_dio_simple {
+	struct kiocb		*iocb;
+	size_t			size;
+	unsigned int		dio_flags;
+	struct work_struct	work;
+	/*
+	 * Align @bio to a cacheline boundary so that, combined with the
+	 * front_pad passed to bioset_init(), the bio sits at the start of
+	 * a cacheline in memory returned by the (HWCACHE-aligned) bio
+	 * slab.  This keeps the hot fields block layer touches on submit
+	 * and completion (bi_iter, bi_status, ...) within a single line.
+	 */
+	struct bio		bio ____cacheline_aligned_in_smp;
+};
+
+static struct bio_set iomap_dio_simple_pool;
+
+static ssize_t iomap_dio_simple_complete(struct iomap_dio_simple *sr)
+{
+	struct bio *bio = &sr->bio;
+	struct kiocb *iocb = sr->iocb;
+	struct inode *inode = file_inode(iocb->ki_filp);
+	ssize_t ret;
+
+	if (unlikely(bio->bi_status)) {
+		ret = blk_status_to_errno(bio->bi_status);
+		if (should_report_dio_fserror(ret))
+			fserror_report_io(inode, FSERR_DIRECTIO_READ,
+					  iocb->ki_pos, sr->size, ret,
+					  GFP_NOFS);
+	} else {
+		ret = sr->size;
+		iocb->ki_pos += ret;
+	}
+
+	if (sr->dio_flags & IOMAP_DIO_USER_BACKED) {
+		bio_check_pages_dirty(bio);
+	} else {
+		bio_release_pages(bio, false);
+		bio_put(bio);
+	}
+	inode_dio_end(inode);
+	trace_iomap_dio_complete(iocb, ret < 0 ? ret : 0, ret);
+	return ret;
+}
+
+static void iomap_dio_simple_complete_work(struct work_struct *work)
+{
+	struct iomap_dio_simple *sr =
+		container_of(work, struct iomap_dio_simple, work);
+	struct kiocb *iocb = sr->iocb;
+
+	WRITE_ONCE(iocb->private, NULL);
+	iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
+}
+
+static void iomap_dio_simple_end_io(struct bio *bio)
+{
+	struct iomap_dio_simple *sr =
+		container_of(bio, struct iomap_dio_simple, bio);
+	struct kiocb *iocb = sr->iocb;
+
+	if (unlikely(sr->bio.bi_status)) {
+		struct inode *inode = file_inode(iocb->ki_filp);
+
+		INIT_WORK(&sr->work, iomap_dio_simple_complete_work);
+		queue_work(inode->i_sb->s_dio_done_wq, &sr->work);
+		return;
+	}
+
+	WRITE_ONCE(iocb->private, NULL);
+	iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
+}
+
+static inline bool
+iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
+			   const struct iomap_dio_ops *dops,
+			   unsigned int dio_flags, size_t done_before)
+{
+	struct inode *inode = file_inode(iocb->ki_filp);
+	size_t count = iov_iter_count(iter);
+
+	if (dops || done_before)
+		return false;
+	if (iov_iter_rw(iter) != READ)
+		return false;
+	if (!count)
+		return false;
+	/*
+	 * Simple dio is an optimization for small IO. Filter out large IO
+	 * early as it's the most common case to fail for typical direct IO
+	 * workloads.
+	 */
+	if (count > inode->i_sb->s_blocksize)
+		return false;
+	if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
+			 IOMAP_DIO_BOUNCE))
+		return false;
+	if (iocb->ki_pos + count > i_size_read(inode))
+		return false;
+	if (IS_ENCRYPTED(inode))
+		return false;
+
+	return true;
+}
+
+/*
+ * Fast path for small, block-aligned direct I/Os that map to a single
+ * contiguous on-disk extent.
+ *
+ * iomap_dio_simple_supported() enforces the cheap up-front constraints before
+ * entering this path.
+ *
+ * @dops must be NULL: a non-NULL @dops means the caller wants its
+ * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
+ * allocated from the filesystem-private @dops->bio_set (whose front_pad sizes a
+ * filesystem-private wrapper around the bio).  The fast path instead allocates
+ * from the shared iomap_dio_simple_pool, whose front_pad matches struct
+ * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
+ * back to __iomap_dio_rw() in that case.
+ *
+ * @done_before must be zero: a non-zero caller-accumulated residual cannot be
+ * carried through a single-bio inline completion.
+ *
+ * @iter must describe a non-empty READ no larger than the inode block size:
+ * writes, zero-length I/O, and larger requests need the generic iomap direct
+ * I/O path.
+ *
+ * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
+ * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
+ * I/O, or bouncing.  The range must also stay within i_size and encrypted
+ * inodes must use the generic iomap direct I/O path.
+ *
+ * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
+ * decides the request does not fit the fast path.  In that case we proceed to
+ * the generic __iomap_dio_rw() slow path.  Any other errno is a real result and
+ * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
+ * caller.
+ */
+static ssize_t
+iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+		 const struct iomap_ops *ops, void *private,
+		 unsigned int dio_flags)
+{
+	struct inode *inode = file_inode(iocb->ki_filp);
+	size_t count = iov_iter_count(iter);
+	bool wait_for_completion = is_sync_kiocb(iocb);
+	struct iomap_iter iomi = {
+		.inode		= inode,
+		.pos		= iocb->ki_pos,
+		.len		= count,
+		.flags		= IOMAP_DIRECT,
+		.private	= private,
+	};
+	struct iomap_dio_simple *sr;
+	unsigned int alignment;
+	struct bio *bio;
+	ssize_t ret;
+
+	if (iocb->ki_flags & IOCB_NOWAIT)
+		iomi.flags |= IOMAP_NOWAIT;
+
+	ret = kiocb_write_and_wait(iocb, count);
+	if (ret)
+		return ret;
+
+	inode_dio_begin(inode);
+
+	ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
+			       &iomi.iomap, &iomi.srcmap);
+	if (ret) {
+		inode_dio_end(inode);
+		return ret;
+	}
+
+	if (iomi.iomap.type != IOMAP_MAPPED ||
+	    iomi.iomap.offset + iomi.iomap.length < iomi.pos + count ||
+	    (iomi.iomap.flags & IOMAP_F_INTEGRITY)) {
+		ret = -ENOTBLK;
+		goto out_iomap_end;
+	}
+
+	alignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);
+	if ((iomi.pos | count) & (alignment - 1)) {
+		ret = -EINVAL;
+		goto out_iomap_end;
+	}
+
+	if (!wait_for_completion && unlikely(!inode->i_sb->s_dio_done_wq)) {
+		ret = sb_init_dio_done_wq(inode->i_sb);
+		if (ret < 0)
+			goto out_iomap_end;
+	}
+
+	trace_iomap_dio_rw_begin(iocb, iter, dio_flags, 0);
+
+	if (user_backed_iter(iter))
+		dio_flags |= IOMAP_DIO_USER_BACKED;
+
+	bio = bio_alloc_bioset(iomi.iomap.bdev,
+			       bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),
+			       REQ_OP_READ, GFP_KERNEL, &iomap_dio_simple_pool);
+	sr = container_of(bio, struct iomap_dio_simple, bio);
+	sr->iocb = iocb;
+	sr->dio_flags = dio_flags;
+
+	bio->bi_iter.bi_sector = iomap_sector(&iomi.iomap, iomi.pos);
+	bio->bi_ioprio = iocb->ki_ioprio;
+
+	ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
+	if (unlikely(ret))
+		goto out_bio_put;
+
+	if (bio->bi_iter.bi_size != count) {
+		iov_iter_revert(iter, bio->bi_iter.bi_size);
+		ret = -ENOTBLK;
+		goto out_bio_release_pages;
+	}
+
+	sr->size = bio->bi_iter.bi_size;
+
+	if (dio_flags & IOMAP_DIO_USER_BACKED)
+		bio_set_pages_dirty(bio);
+
+	if (iocb->ki_flags & IOCB_NOWAIT)
+		bio->bi_opf |= REQ_NOWAIT;
+	if ((iocb->ki_flags & IOCB_HIPRI) && !wait_for_completion) {
+		bio->bi_opf |= REQ_POLLED;
+		WRITE_ONCE(iocb->private, bio);
+	}
+
+	if (ops->iomap_end)
+		ops->iomap_end(inode, iomi.pos, count, count, iomi.flags,
+			       &iomi.iomap);
+
+	if (!wait_for_completion) {
+		bio->bi_end_io = iomap_dio_simple_end_io;
+		submit_bio(bio);
+		trace_iomap_dio_rw_queued(inode, iomi.pos, count);
+		return -EIOCBQUEUED;
+	}
+
+	submit_bio_wait(bio);
+	return iomap_dio_simple_complete(sr);
+
+out_bio_release_pages:
+	bio_release_pages(bio, false);
+out_bio_put:
+	bio_put(bio);
+out_iomap_end:
+	if (ops->iomap_end)
+		ops->iomap_end(inode, iomi.pos, count, 0, iomi.flags,
+			       &iomi.iomap);
+	inode_dio_end(inode);
+	return ret;
+}
+
 ssize_t
 iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags, void *private, size_t done_before)
 {
 	struct iomap_dio *dio;
+	ssize_t ret;
+
+	if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
+				       done_before)) {
+		ret = iomap_dio_simple(iocb, iter, ops, private, dio_flags);
+		if (ret != -ENOTBLK)
+			return ret;
+	}
 
 	dio = __iomap_dio_rw(iocb, iter, ops, dops, dio_flags, private,
 			     done_before);
@@ -907,3 +1173,11 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 	return iomap_dio_complete(dio);
 }
 EXPORT_SYMBOL_GPL(iomap_dio_rw);
+
+static int __init iomap_dio_init(void)
+{
+	return bioset_init(&iomap_dio_simple_pool, 4,
+			   offsetof(struct iomap_dio_simple, bio),
+			   BIOSET_NEED_BVECS | BIOSET_PERCPU_CACHE);
+}
+fs_initcall(iomap_dio_init);
-- 
2.39.5 (Apple Git-154)

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

We'll need to sort out the interaction with the iomap_next work, though.

Joanne, is it ok to get this in first and have Fengnan help your with
the interactions due to the open coded iomap_begin/end calls?

On Wed, Jul 01, 2026 at 11:32:53AM +0800, Fengnan Chang wrote:
> When running 4K random read workloads on high-performance Gen5 NVMe
> SSDs, the software overhead in the iomap direct I/O path
> (__iomap_dio_rw) becomes a significant bottleneck.
> 
> Using io_uring with poll mode for a 4K randread test on a raw block
> device:
> taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1
> -n1 -P1 /dev/nvme10n1
> Result: ~3.2M IOPS
> 
> Running the exact same workload on ext4 and XFS:
> taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1
> -n1 -P1 /mnt/testfile
> Result: ~1.92M IOPS
> 
> Profiling the ext4 workload reveals that a significant portion of CPU
> time is spent on memory allocation and the iomap state machine
> iteration:
>   5.33%  [kernel]  [k] __iomap_dio_rw
>   3.26%  [kernel]  [k] iomap_iter
>   2.37%  [kernel]  [k] iomap_dio_bio_iter
>   2.35%  [kernel]  [k] kfree
>   1.33%  [kernel]  [k] iomap_dio_complete
> 
> Introduce a simple dio path to reduce the overhead of iomap. It is
> triggered when the request satisfies all of:
> - a READ request whose I/O size is <= inode blocksize (fits in a single
>   block, no splits);
> - no custom iomap_dio_ops (dops) registered by the filesystem;
> - no caller-accumulated residual (done_before == 0);
> - none of IOMAP_DIO_FORCE_WAIT / IOMAP_DIO_PARTIAL / IOMAP_DIO_BOUNCE
>   set, the range is within i_size, and the inode is not encrypted.
> 
> The bio is allocated from a dedicated bioset whose front_pad embeds
> struct iomap_dio_simple, so the whole request lives in a single
> cacheline-aligned allocation and no separate struct iomap_dio is
> needed.  Completion is handled inline from ->bi_end_io for the common
> success case, and only punted to the s_dio_done_wq workqueue on error.
> 
> After this optimization, the heavy generic functions disappear from the
> profile, replaced by a single streamlined execution path:
>   4.83%  [kernel]  [k] iomap_dio_simple
> 
> With this patch, 4K random read IOPS on ext4 increases from 1.92M to
> 2.19M in the original single-core io_uring poll-mode workload.
> 
> Below are the test results using fio:
> 
>   fs    workload       qd    simple=0      simple=1      gain
>   ext4  libaio         1     18,740        18,761        +0.11%
>   ext4  libaio         64    462,850       480,587       +3.83%
>   ext4  libaio         128   459,498       478,824       +4.21%
>   ext4  libaio         256   459,938       480,156       +4.40%
>   ext4  io_uring       1     18,836        18,880        +0.24%
>   ext4  io_uring       64    568,193       600,625       +5.71%
>   ext4  io_uring       128   570,998       602,148       +5.46%
>   ext4  io_uring       256   572,052       602,536       +5.33%
>   ext4  io_uring_poll  1     19,283        19,272        -0.06%
>   ext4  io_uring_poll  64    989,735       1,013,342     +2.39%
>   ext4  io_uring_poll  128   1,467,336     1,538,444     +4.85%
>   ext4  io_uring_poll  256   1,663,498     1,830,842     +10.06%
>   xfs   libaio         1     18,764        18,776        +0.06%
>   xfs   libaio         64    462,408       480,860       +3.99%
>   xfs   libaio         128   461,280       480,819       +4.24%
>   xfs   libaio         256   461,626       480,190       +4.02%
>   xfs   io_uring       1     18,871        18,903        +0.17%
>   xfs   io_uring       64    570,383       597,399       +4.74%
>   xfs   io_uring       128   568,290       597,370       +5.12%
>   xfs   io_uring       256   570,616       598,775       +4.93%
>   xfs   io_uring_poll  1     19,211        19,315        +0.54%
>   xfs   io_uring_poll  64    989,726       1,008,455     +1.89%
>   xfs   io_uring_poll  128   1,430,426     1,513,064     +5.78%
>   xfs   io_uring_poll  256   1,587,339     1,742,220     +9.76%
> 
> Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
> ---
>  fs/iomap/direct-io.c | 274 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 274 insertions(+)
> 
> diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
> index 1b9abdd831d0b..ca790239e5eb3 100644
> --- a/fs/iomap/direct-io.c
> +++ b/fs/iomap/direct-io.c
> @@ -10,6 +10,7 @@
>  #include <linux/iomap.h>
>  #include <linux/task_io_accounting_ops.h>
>  #include <linux/fserror.h>
> +#include <linux/init.h>
>  #include "internal.h"
>  #include "trace.h"
>  
> @@ -893,12 +894,277 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
>  }
>  EXPORT_SYMBOL_GPL(__iomap_dio_rw);
>  
> +struct iomap_dio_simple {
> +	struct kiocb		*iocb;
> +	size_t			size;
> +	unsigned int		dio_flags;
> +	struct work_struct	work;
> +	/*
> +	 * Align @bio to a cacheline boundary so that, combined with the
> +	 * front_pad passed to bioset_init(), the bio sits at the start of
> +	 * a cacheline in memory returned by the (HWCACHE-aligned) bio
> +	 * slab.  This keeps the hot fields block layer touches on submit
> +	 * and completion (bi_iter, bi_status, ...) within a single line.
> +	 */
> +	struct bio		bio ____cacheline_aligned_in_smp;
> +};
> +
> +static struct bio_set iomap_dio_simple_pool;
> +
> +static ssize_t iomap_dio_simple_complete(struct iomap_dio_simple *sr)
> +{
> +	struct bio *bio = &sr->bio;
> +	struct kiocb *iocb = sr->iocb;
> +	struct inode *inode = file_inode(iocb->ki_filp);
> +	ssize_t ret;
> +
> +	if (unlikely(bio->bi_status)) {
> +		ret = blk_status_to_errno(bio->bi_status);
> +		if (should_report_dio_fserror(ret))
> +			fserror_report_io(inode, FSERR_DIRECTIO_READ,
> +					  iocb->ki_pos, sr->size, ret,
> +					  GFP_NOFS);
> +	} else {
> +		ret = sr->size;
> +		iocb->ki_pos += ret;
> +	}
> +
> +	if (sr->dio_flags & IOMAP_DIO_USER_BACKED) {
> +		bio_check_pages_dirty(bio);
> +	} else {
> +		bio_release_pages(bio, false);
> +		bio_put(bio);
> +	}
> +	inode_dio_end(inode);
> +	trace_iomap_dio_complete(iocb, ret < 0 ? ret : 0, ret);
> +	return ret;
> +}
> +
> +static void iomap_dio_simple_complete_work(struct work_struct *work)
> +{
> +	struct iomap_dio_simple *sr =
> +		container_of(work, struct iomap_dio_simple, work);
> +	struct kiocb *iocb = sr->iocb;
> +
> +	WRITE_ONCE(iocb->private, NULL);
> +	iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
> +}
> +
> +static void iomap_dio_simple_end_io(struct bio *bio)
> +{
> +	struct iomap_dio_simple *sr =
> +		container_of(bio, struct iomap_dio_simple, bio);
> +	struct kiocb *iocb = sr->iocb;
> +
> +	if (unlikely(sr->bio.bi_status)) {
> +		struct inode *inode = file_inode(iocb->ki_filp);
> +
> +		INIT_WORK(&sr->work, iomap_dio_simple_complete_work);
> +		queue_work(inode->i_sb->s_dio_done_wq, &sr->work);
> +		return;
> +	}
> +
> +	WRITE_ONCE(iocb->private, NULL);
> +	iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
> +}
> +
> +static inline bool
> +iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
> +			   const struct iomap_dio_ops *dops,
> +			   unsigned int dio_flags, size_t done_before)
> +{
> +	struct inode *inode = file_inode(iocb->ki_filp);
> +	size_t count = iov_iter_count(iter);
> +
> +	if (dops || done_before)
> +		return false;
> +	if (iov_iter_rw(iter) != READ)
> +		return false;
> +	if (!count)
> +		return false;
> +	/*
> +	 * Simple dio is an optimization for small IO. Filter out large IO
> +	 * early as it's the most common case to fail for typical direct IO
> +	 * workloads.
> +	 */
> +	if (count > inode->i_sb->s_blocksize)
> +		return false;
> +	if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
> +			 IOMAP_DIO_BOUNCE))
> +		return false;
> +	if (iocb->ki_pos + count > i_size_read(inode))
> +		return false;
> +	if (IS_ENCRYPTED(inode))
> +		return false;
> +
> +	return true;
> +}
> +
> +/*
> + * Fast path for small, block-aligned direct I/Os that map to a single
> + * contiguous on-disk extent.
> + *
> + * iomap_dio_simple_supported() enforces the cheap up-front constraints before
> + * entering this path.
> + *
> + * @dops must be NULL: a non-NULL @dops means the caller wants its
> + * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
> + * allocated from the filesystem-private @dops->bio_set (whose front_pad sizes a
> + * filesystem-private wrapper around the bio).  The fast path instead allocates
> + * from the shared iomap_dio_simple_pool, whose front_pad matches struct
> + * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
> + * back to __iomap_dio_rw() in that case.
> + *
> + * @done_before must be zero: a non-zero caller-accumulated residual cannot be
> + * carried through a single-bio inline completion.
> + *
> + * @iter must describe a non-empty READ no larger than the inode block size:
> + * writes, zero-length I/O, and larger requests need the generic iomap direct
> + * I/O path.
> + *
> + * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
> + * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
> + * I/O, or bouncing.  The range must also stay within i_size and encrypted
> + * inodes must use the generic iomap direct I/O path.
> + *
> + * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
> + * decides the request does not fit the fast path.  In that case we proceed to
> + * the generic __iomap_dio_rw() slow path.  Any other errno is a real result and
> + * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
> + * caller.
> + */
> +static ssize_t
> +iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
> +		 const struct iomap_ops *ops, void *private,
> +		 unsigned int dio_flags)
> +{
> +	struct inode *inode = file_inode(iocb->ki_filp);
> +	size_t count = iov_iter_count(iter);
> +	bool wait_for_completion = is_sync_kiocb(iocb);
> +	struct iomap_iter iomi = {
> +		.inode		= inode,
> +		.pos		= iocb->ki_pos,
> +		.len		= count,
> +		.flags		= IOMAP_DIRECT,
> +		.private	= private,
> +	};
> +	struct iomap_dio_simple *sr;
> +	unsigned int alignment;
> +	struct bio *bio;
> +	ssize_t ret;
> +
> +	if (iocb->ki_flags & IOCB_NOWAIT)
> +		iomi.flags |= IOMAP_NOWAIT;
> +
> +	ret = kiocb_write_and_wait(iocb, count);
> +	if (ret)
> +		return ret;
> +
> +	inode_dio_begin(inode);
> +
> +	ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
> +			       &iomi.iomap, &iomi.srcmap);
> +	if (ret) {
> +		inode_dio_end(inode);
> +		return ret;
> +	}
> +
> +	if (iomi.iomap.type != IOMAP_MAPPED ||
> +	    iomi.iomap.offset + iomi.iomap.length < iomi.pos + count ||
> +	    (iomi.iomap.flags & IOMAP_F_INTEGRITY)) {
> +		ret = -ENOTBLK;
> +		goto out_iomap_end;
> +	}
> +
> +	alignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);
> +	if ((iomi.pos | count) & (alignment - 1)) {
> +		ret = -EINVAL;
> +		goto out_iomap_end;
> +	}
> +
> +	if (!wait_for_completion && unlikely(!inode->i_sb->s_dio_done_wq)) {
> +		ret = sb_init_dio_done_wq(inode->i_sb);
> +		if (ret < 0)
> +			goto out_iomap_end;
> +	}
> +
> +	trace_iomap_dio_rw_begin(iocb, iter, dio_flags, 0);
> +
> +	if (user_backed_iter(iter))
> +		dio_flags |= IOMAP_DIO_USER_BACKED;
> +
> +	bio = bio_alloc_bioset(iomi.iomap.bdev,
> +			       bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),
> +			       REQ_OP_READ, GFP_KERNEL, &iomap_dio_simple_pool);
> +	sr = container_of(bio, struct iomap_dio_simple, bio);
> +	sr->iocb = iocb;
> +	sr->dio_flags = dio_flags;
> +
> +	bio->bi_iter.bi_sector = iomap_sector(&iomi.iomap, iomi.pos);
> +	bio->bi_ioprio = iocb->ki_ioprio;
> +
> +	ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
> +	if (unlikely(ret))
> +		goto out_bio_put;
> +
> +	if (bio->bi_iter.bi_size != count) {
> +		iov_iter_revert(iter, bio->bi_iter.bi_size);
> +		ret = -ENOTBLK;
> +		goto out_bio_release_pages;
> +	}
> +
> +	sr->size = bio->bi_iter.bi_size;
> +
> +	if (dio_flags & IOMAP_DIO_USER_BACKED)
> +		bio_set_pages_dirty(bio);
> +
> +	if (iocb->ki_flags & IOCB_NOWAIT)
> +		bio->bi_opf |= REQ_NOWAIT;
> +	if ((iocb->ki_flags & IOCB_HIPRI) && !wait_for_completion) {
> +		bio->bi_opf |= REQ_POLLED;
> +		WRITE_ONCE(iocb->private, bio);
> +	}
> +
> +	if (ops->iomap_end)
> +		ops->iomap_end(inode, iomi.pos, count, count, iomi.flags,
> +			       &iomi.iomap);
> +
> +	if (!wait_for_completion) {
> +		bio->bi_end_io = iomap_dio_simple_end_io;
> +		submit_bio(bio);
> +		trace_iomap_dio_rw_queued(inode, iomi.pos, count);
> +		return -EIOCBQUEUED;
> +	}
> +
> +	submit_bio_wait(bio);
> +	return iomap_dio_simple_complete(sr);
> +
> +out_bio_release_pages:
> +	bio_release_pages(bio, false);
> +out_bio_put:
> +	bio_put(bio);
> +out_iomap_end:
> +	if (ops->iomap_end)
> +		ops->iomap_end(inode, iomi.pos, count, 0, iomi.flags,
> +			       &iomi.iomap);
> +	inode_dio_end(inode);
> +	return ret;
> +}
> +
>  ssize_t
>  iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
>  		const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
>  		unsigned int dio_flags, void *private, size_t done_before)
>  {
>  	struct iomap_dio *dio;
> +	ssize_t ret;
> +
> +	if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
> +				       done_before)) {
> +		ret = iomap_dio_simple(iocb, iter, ops, private, dio_flags);
> +		if (ret != -ENOTBLK)
> +			return ret;
> +	}
>  
>  	dio = __iomap_dio_rw(iocb, iter, ops, dops, dio_flags, private,
>  			     done_before);
> @@ -907,3 +1173,11 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
>  	return iomap_dio_complete(dio);
>  }
>  EXPORT_SYMBOL_GPL(iomap_dio_rw);
> +
> +static int __init iomap_dio_init(void)
> +{
> +	return bioset_init(&iomap_dio_simple_pool, 4,
> +			   offsetof(struct iomap_dio_simple, bio),
> +			   BIOSET_NEED_BVECS | BIOSET_PERCPU_CACHE);
> +}
> +fs_initcall(iomap_dio_init);
> -- 
> 2.39.5 (Apple Git-154)
---end quoted text---

Re: [PATCH v6 0/3] iomap: add simple dio path for small direct I/O

[Christian Brauner]

On Wed, 01 Jul 2026 11:32:50 +0800, Fengnan Chang wrote:
> iomap: add simple dio path for small direct I/O
> 
> When running 4K random read workloads on high-performance Gen5 NVMe
> SSDs, the software overhead in the iomap direct I/O path
> (__iomap_dio_rw) becomes a significant bottleneck.
> 
> Using io_uring with poll mode for a 4K randread test on a raw block
> device:
> taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1
> -n1 -P1 /dev/nvme10n1
> Result: ~3.2M IOPS
> 
> [...]

Applied to the vfs-7.3.iomap branch of the vfs/vfs.git tree.
Patches in the vfs-7.3.iomap branch should appear in linux-next soon.

Please report any outstanding bugs that were missed during review in a
new review to the original patch series allowing us to drop it.

It's encouraged to provide Acked-bys and Reviewed-bys even though the
patch has now been applied. If possible patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase,
trailer updates or similar. If in doubt, please check the listed branch.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs-7.3.iomap

[1/3] iomap: factor out iomap_dio_alignment helper
      https://git.kernel.org/vfs/vfs/c/acb8815f181f
[2/3] iomap: pass error code to should_report_dio_fserror directly
      https://git.kernel.org/vfs/vfs/c/281e3e52acbf
[3/3] iomap: add simple dio path for small direct I/O
      https://git.kernel.org/vfs/vfs/c/de2f9ebea176

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[Joanne Koong]

On Wed, Jul 1, 2026 at 4:23 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> Looks good:
>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
>
> We'll need to sort out the interaction with the iomap_next work, though.
>
> Joanne, is it ok to get this in first and have Fengnan help your with
> the interactions due to the open coded iomap_begin/end calls?
>

That sounds good, I'll rebase the next version onto these changes.

I think it ends up being pretty simple to get it integrated since
->iomap_next already contains the logic to only begin the mapping at
the start and only finish the mapping when the iter has been fully
consumed, eg

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 3ef5e9adca87..e654ec135d54 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -1035,7 +1035,7 @@ iomap_dio_simple_supported(struct kiocb *iocb,
struct iov_iter *iter,
  */
 static ssize_t
 iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
-                const struct iomap_ops *ops, void *private,
+                iomap_next_fn iomap_next, void *private,
                 unsigned int dio_flags)
 {
        struct inode *inode = file_inode(iocb->ki_filp);
@@ -1062,11 +1062,11 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,

        inode_dio_begin(inode);

-       ret = ops->iomap_begin(inode, iomi.pos, count, iomi.flags,
-                              &iomi.iomap, &iomi.srcmap);
-       if (ret) {
+       ret = iomap_iter(&iomi, iomap_next);
+       if (ret <= 0) {
                inode_dio_end(inode);
-               return ret;
+               /* ret of 0 means no mapping, which should never happen */
+               return ret ? ret : -EFAULT;
        }

        if (iomi.iomap.type != IOMAP_MAPPED ||
@@ -1125,14 +1125,17 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
                WRITE_ONCE(iocb->private, bio);
        }

-       if (ops->iomap_end)
-               ops->iomap_end(inode, iomi.pos, count, count, iomi.flags,
-                              &iomi.iomap);
+       iomap_iter_advance(&iomi, count);
+       /*
+        * ends the mapping if the mapping needs to be ended. no new mappings
+        * are fetched since the entire range of the iter has been consumed
+        */
+       iomap_iter(&iomi, iomap_next);

        if (!wait_for_completion) {
                bio->bi_end_io = iomap_dio_simple_end_io;
                submit_bio(bio);
-               trace_iomap_dio_rw_queued(inode, iomi.pos, count);
+               trace_iomap_dio_rw_queued(inode, iocb->ki_pos, count);
                return -EIOCBQUEUED;
        }

@@ -1144,9 +1147,12 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
 out_bio_put:
        bio_put(bio);
 out_iomap_end:
-       if (ops->iomap_end)
-               ops->iomap_end(inode, iomi.pos, count, 0, iomi.flags,
-                              &iomi.iomap);
+       /*
+        * on failure, iter was never advanced. ->iomap_next() will know that 0
+        * bytes have been processed and the mapping will be ended with no next
+        * mapping fetched
+        */
+       iomap_iter(&iomi, iomap_next);
        inode_dio_end(inode);
        return ret;
 }

Thanks,
Joanne

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[Christoph Hellwig]

On Wed, Jul 01, 2026 at 12:35:24PM -0700, Joanne Koong wrote:
> I think it ends up being pretty simple to get it integrated since
> ->iomap_next already contains the logic to only begin the mapping at
> the start and only finish the mapping when the iter has been fully
> consumed, eg

I wonder if this adds overhead that the simple path removed.

As a first step we could look into inlining iomap_iter_advance as it
is pretty trivial, and iomap_iter once the legacy path is gone.

If it still shows overhead for that (I think we'll want Fengnan in the
loop to benchmark that), we could look into a 'need end' flag in the
iter and allow to skip the call to iomap_iter_advance and iomap_iter
if it set.  Although I'd be happier if we could avoid that..

Talking about iomap_next - I also wonder iwe should move everything
that isn't the call to ->iter_next into iomap_process, which would
remove any calling overhead for ->iter_next itself.

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[Joanne Koong]

On Thu, Jul 2, 2026 at 7:35 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Wed, Jul 01, 2026 at 12:35:24PM -0700, Joanne Koong wrote:
> > I think it ends up being pretty simple to get it integrated since
> > ->iomap_next already contains the logic to only begin the mapping at
> > the start and only finish the mapping when the iter has been fully
> > consumed, eg
>
> I wonder if this adds overhead that the simple path removed.

Ah I see now, the whole point of Fengnan's series is to k eep the
logic minimal...

Fengnan, I think you had mentioned earlier [1] that enabling
iomap_next brought the simple-read gain down from 10% to 7%? Thanks
for running those numbers. Would you be able to share the changes you
made to iomap_dio_rw to support iomap_next for those benchmarks you
ran?

Thanks,
Joanne

[1] https://lore.kernel.org/linux-fsdevel/e85e73bd-5fbe-4d11-89ba-7b42c237dd34@bytedance.com/

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[changfengnan]

[-- Attachment #1: Type: text/plain, Size: 2504 bytes --]


> From: "Joanne Koong"<joannelkoong@gmail.com>
> Date:  Fri, Jul 3, 2026, 11:06
> Subject:  Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O
> To: "Christoph Hellwig"<hch@infradead.org>
> Cc: "Fengnan Chang"<changfengnan@bytedance.com>, <brauner@kernel.org>, <djwong@kernel.org>, <ojaswin@linux.ibm.com>, <dgc@kernel.org>, <linux-xfs@vger.kernel.org>, <linux-fsdevel@vger.kernel.org>, <linux-ext4@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <lidiangang@bytedance.com>, <pankaj.raghav@linux.dev>, "Brian Foster"<bfoster@redhat.com>
> On Thu, Jul 2, 2026 at 7:35 AM Christoph Hellwig <hch@infradead.org> wrote:
> >
> > On Wed, Jul 01, 2026 at 12:35:24PM -0700, Joanne Koong wrote:
> > > I think it ends up being pretty simple to get it integrated since
> > > ->iomap_next already contains the logic to only begin the mapping at
> > > the start and only finish the mapping when the iter has been fully
> > > consumed, eg
> >
> > I wonder if this adds overhead that the simple path removed.
> 
> Ah I see now, the whole point of Fengnan's series is to k eep the
> logic minimal...

Hi Christoph and Joanne:
I ran some tests on the simple read path with iomap next patch, and
the results showed a 2%–3% drop in performance, with IOPS falling
from 2.04–2.05M to 2.0M.
Patch 1 added `iomap_dio_simple_finish_iter`, and now this function
is executed for every I/O operation. However, in the previous code,
when `ops->iomap_end` was null for ext4 and XFS, no action was
taken, which introduced some overhead. 
I implemented some optimizations based on Christoph’s suggestions and
made some modifications to `iomap_process`; 
Now, XFS performance is essentially unchanged, while ext4 still shows a
1.5% drop; I will continue to investigate the cause..   I’ve attached the patch.
Actually, I had expected that switching to `iomap_next` would improve
performance, since it avoids an indirect call.


[1] https://lore.kernel.org/linux-fsdevel/CAJnrk1ZZCBexaYUCe0s2_kfbAZVKrdhSXFCRv=cy1ggcxRhSPw@mail.gmail.com/


> 
> Fengnan, I think you had mentioned earlier [1] that enabling
> iomap_next brought the simple-read gain down from 10% to 7%? Thanks
> for running those numbers. Would you be able to share the changes you
> made to iomap_dio_rw to support iomap_next for those benchmarks you
> ran?
> 
> Thanks,
> Joanne
> 
> [1] https://lore.kernel.org/linux-fsdevel/e85e73bd-5fbe-4d11-89ba-7b42c237dd34@bytedance.com/
> 

[-- Attachment #2: 0002-iomap-inline-iomap_iter_advance-and-hoist-the-zero-l.patch --]
[-- Type: application/octet-stream, Size: 3908 bytes --]

From 26b09b6bb5a2aa3741bebd31bfbd9cb05d2f8ac6 Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan@bytedance.com>
Date: Fri, 3 Jul 2026 16:10:11 +0800
Subject: [PATCH 2/2] iomap: inline iomap_iter_advance and hoist the
 zero-length guard

Make iomap_iter_advance() a static inline in the header, and move the
first-iteration (iomap->length == 0) check from iomap_iter_continue()
into iomap_process() so the out-of-line iomap_iter_continue() call is
skipped until begin() has produced a mapping.

Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
---
 fs/iomap/iter.c       | 13 -------------
 include/linux/iomap.h | 41 +++++++++++++++++++++++++++++------------
 2 files changed, 29 insertions(+), 25 deletions(-)

diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index 984045af310a1..35aef827a31c9 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -15,16 +15,6 @@ static inline void iomap_iter_clean_fbatch(struct iomap_iter *iter)
 	}
 }
 
-/* Advance the current iterator position and decrement the remaining length */
-int iomap_iter_advance(struct iomap_iter *iter, u64 count)
-{
-	if (WARN_ON_ONCE(count > iomap_length(iter)))
-		return -EIO;
-	iter->pos += count;
-	iter->len -= count;
-	return 0;
-}
-
 static inline void iomap_iter_done(struct iomap_iter *iter)
 {
 	WARN_ON_ONCE(iter->iomap.offset > iter->pos);
@@ -89,9 +79,6 @@ int iomap_iter_continue(const struct iomap_iter *iter, struct iomap *iomap,
 	bool stale = iomap->flags & IOMAP_F_STALE;
 	ssize_t advanced = iter->pos - iter->iter_start_pos;
 
-	if (!iomap->length)
-		return 1;
-
 	/*
 	 * Use iter->len to determine whether to continue onto the next mapping.
 	 * Explicitly terminate on error status or if the current iter has not
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index b2e6ad4fa2293..820ee05ef6ae5 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -266,7 +266,6 @@ struct iomap_iter {
 };
 
 int iomap_iter(struct iomap_iter *iter, iomap_next_fn iomap_next);
-int iomap_iter_advance(struct iomap_iter *iter, u64 count);
 
 /**
  * iomap_length_trim - trimmed length of the current iomap iteration
@@ -298,6 +297,16 @@ static inline u64 iomap_length(const struct iomap_iter *iter)
 	return iomap_length_trim(iter, iter->pos, iter->len);
 }
 
+/* Advance the current iterator position and decrement the remaining length */
+static inline int iomap_iter_advance(struct iomap_iter *iter, u64 count)
+{
+	if (WARN_ON_ONCE(count > iomap_length(iter)))
+		return -EIO;
+	iter->pos += count;
+	iter->len -= count;
+	return 0;
+}
+
 /**
  * iomap_iter_advance_full - advance by the full length of current map
  */
@@ -686,20 +695,28 @@ static __always_inline int iomap_process(const struct iomap_iter *iter,
 {
 	int ret = 0;
 
-	if (iomap->length && end) {
-		ssize_t advanced = iter->pos - iter->iter_start_pos;
-		loff_t len;
+	/*
+	 * On the first iteration the mapping is still zeroed, so there is
+	 * nothing to finish and iteration simply continues.  Skip the
+	 * out-of-line iomap_iter_continue() call in that case and go straight
+	 * to @begin; it is only called once @begin has produced a mapping.
+	 */
+	if (iomap->length) {
+		if (end) {
+			ssize_t advanced = iter->pos - iter->iter_start_pos;
+			loff_t len;
 
-		len = iomap_length_trim(iter, iter->iter_start_pos,
-				iter->len + advanced);
+			len = iomap_length_trim(iter, iter->iter_start_pos,
+					iter->len + advanced);
 
-		ret = end(iter->inode, iter->iter_start_pos, len, advanced,
-				iter->flags, iomap);
-	}
+			ret = end(iter->inode, iter->iter_start_pos, len,
+					advanced, iter->flags, iomap);
+		}
 
-	ret = iomap_iter_continue(iter, iomap, srcmap, ret);
-	if (ret <= 0)
-		return ret;
+		ret = iomap_iter_continue(iter, iomap, srcmap, ret);
+		if (ret <= 0)
+			return ret;
+	}
 
 	ret = begin(iter->inode, iter->pos, iter->len, iter->flags, iomap,
 			srcmap);
-- 
2.39.5 (Apple Git-154)


[-- Attachment #3: 0001-iomap-skip-the-finishing-iomap_iter-on-the-simple-DI.patch --]
[-- Type: application/octet-stream, Size: 5860 bytes --]

From 30f674f71f672d6ac83cf7a4763abcf4c65e68ad Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan@bytedance.com>
Date: Fri, 3 Jul 2026 15:40:37 +0800
Subject: [PATCH 1/2] iomap: skip the finishing iomap_iter() on the simple DIO
 read fast path

After the ->iomap_next() conversion, iomap_dio_simple() runs a second,
full iomap_iter() just to finish the range.  For DIO reads on filesystems
with no ->iomap_end() work (ext4, xfs, exfat, ntfs) this is a no-op, but
still costs an extra indirect ->iomap_next() call and the iomap_iter()
frames on every I/O.

Add IOMAP_DIO_NO_END so such callers can promise there is no ->iomap_end()
work and no folio batch, letting iomap_dio_simple() skip the finishing
iomap_iter() on both the success and error-exit paths.  Opt in the ext4,
xfs, exfat and ntfs direct read paths; filesystems with real read-path
->iomap_end() work keep the existing behaviour.

Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
---
 fs/exfat/file.c       |  4 ++--
 fs/ext4/file.c        |  3 ++-
 fs/iomap/direct-io.c  | 17 +++++++++++++----
 fs/iomap/trace.h      |  3 ++-
 fs/ntfs/file.c        |  4 ++--
 fs/xfs/xfs_file.c     |  2 +-
 include/linux/iomap.h |  8 ++++++++
 7 files changed, 30 insertions(+), 11 deletions(-)

diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index c05849d305ae4..451d601023c7d 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -809,8 +809,8 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
 	if (iocb->ki_flags & IOCB_DIRECT) {
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL, 0,
-				NULL, 0);
+		ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL,
+				IOMAP_DIO_NO_END, NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, iter);
 	}
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index dbe073e181a7a..662b4e98079cb 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,8 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		return generic_file_read_iter(iocb, to);
 	}
 
-	ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
+	ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, IOMAP_DIO_NO_END,
+			   NULL, 0);
 	inode_unlock_shared(inode);
 
 	file_accessed(iocb->ki_filp);
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index d08b7d3c52710..5ffb641f76b43 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -1142,9 +1142,17 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 		WRITE_ONCE(iocb->private, bio);
 	}
 
-	ret = iomap_dio_simple_finish_iter(&iomi, iomap_next, count);
-	if (ret)
-		goto out_bio_release_pages_no_end;
+	/*
+	 * The finishing iomap_iter() only runs the filesystem's ->iomap_end()
+	 * work and releases any attached folio batch.  When the caller promised
+	 * neither applies (IOMAP_DIO_NO_END), skip it to avoid an extra indirect
+	 * ->iomap_next() call and the iomap_iter() frames on the hot read path.
+	 */
+	if (!(dio_flags & IOMAP_DIO_NO_END)) {
+		ret = iomap_dio_simple_finish_iter(&iomi, iomap_next, count);
+		if (ret)
+			goto out_bio_release_pages_no_end;
+	}
 
 	if (!wait_for_completion) {
 		bio->bi_end_io = iomap_dio_simple_end_io;
@@ -1161,7 +1169,8 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 out_bio_put:
 	bio_put(bio);
 out_iomap_end:
-	iomap_dio_simple_finish_iter(&iomi, iomap_next, 0);
+	if (!(dio_flags & IOMAP_DIO_NO_END))
+		iomap_dio_simple_finish_iter(&iomi, iomap_next, 0);
 	inode_dio_end(inode);
 	return ret;
 out_bio_release_pages_no_end:
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47d..6cb99eb892ae0 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -127,7 +127,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
 	{IOMAP_DIO_FORCE_WAIT,		"DIO_FORCE_WAIT" }, \
 	{IOMAP_DIO_OVERWRITE_ONLY,	"DIO_OVERWRITE_ONLY" }, \
 	{IOMAP_DIO_PARTIAL,		"DIO_PARTIAL" }, \
-	{IOMAP_DIO_FSBLOCK_ALIGNED,	"DIO_FSBLOCK_ALIGNED" }
+	{IOMAP_DIO_FSBLOCK_ALIGNED,	"DIO_FSBLOCK_ALIGNED" }, \
+	{IOMAP_DIO_NO_END,		"DIO_NO_END" }
 
 DECLARE_EVENT_CLASS(iomap_class,
 	TP_PROTO(struct inode *inode, struct iomap *iomap),
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index a4f99128b46c5..677d794b1f82d 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -458,8 +458,8 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		}
 
 		file_accessed(iocb->ki_filp);
-		ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL, 0,
-				NULL, 0);
+		ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL,
+				IOMAP_DIO_NO_END, NULL, 0);
 	} else {
 		ret = generic_file_read_iter(iocb, to);
 	}
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index a987ffbf3c023..293869eadb4dd 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -251,7 +251,7 @@ xfs_file_dio_read(
 	struct iov_iter		*to)
 {
 	struct xfs_inode	*ip = XFS_I(file_inode(iocb->ki_filp));
-	unsigned int		dio_flags = 0;
+	unsigned int		dio_flags = IOMAP_DIO_NO_END;
 	const struct iomap_dio_ops *dio_ops = NULL;
 	ssize_t			ret;
 
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 3b41f123a92d4..b2e6ad4fa2293 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -605,6 +605,14 @@ struct iomap_dio_ops {
  */
 #define IOMAP_DIO_BOUNCE		(1 << 4)
 
+/*
+ * The caller's ->iomap_next() has no ->iomap_end() work to run and this I/O
+ * does not use a folio batch, so the iomap_dio_simple() fast path may skip the
+ * finishing iomap_iter() call entirely.  Only meaningful on the simple read
+ * fast path; ignored by the generic __iomap_dio_rw() slow path.
+ */
+#define IOMAP_DIO_NO_END		(1 << 5)
+
 ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 		iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
 		unsigned int dio_flags, void *private, size_t done_before);
-- 
2.39.5 (Apple Git-154)

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[changfengnan]

> From: "Joanne Koong"<joannelkoong@gmail.com>
> Date:  Fri, Jul 3, 2026, 11:06
> Subject:  Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O
> To: "Christoph Hellwig"<hch@infradead.org>
> Cc: "Fengnan Chang"<changfengnan@bytedance.com>, <brauner@kernel.org>, <djwong@kernel.org>, <ojaswin@linux.ibm.com>, <dgc@kernel.org>, <linux-xfs@vger.kernel.org>, <linux-fsdevel@vger.kernel.org>, <linux-ext4@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <lidiangang@bytedance.com>, <pankaj.raghav@linux.dev>, "Brian Foster"<bfoster@redhat.com>
> On Thu, Jul 2, 2026 at 7:35 AM Christoph Hellwig <hch@infradead.org> wrote:
> >
> > On Wed, Jul 01, 2026 at 12:35:24PM -0700, Joanne Koong wrote:
> > > I think it ends up being pretty simple to get it integrated since
> > > ->iomap_next already contains the logic to only begin the mapping at
> > > the start and only finish the mapping when the iter has been fully
> > > consumed, eg
> >
> > I wonder if this adds overhead that the simple path removed.
> 
> Ah I see now, the whole point of Fengnan's series is to k eep the
> logic minimal...
> 
> Fengnan, I think you had mentioned earlier [1] that enabling
> iomap_next brought the simple-read gain down from 10% to 7%? Thanks
> for running those numbers. Would you be able to share the changes you
> made to iomap_dio_rw to support iomap_next for those benchmarks you
> ran?

Hi Joanne:
My previous change was to pass `xfs_read_iomap_begin` and
`ext4_iomap_begin` from ext4/xfs directly to `iomap_dio_simple`, since
`iomap_begin` was still present in `struct iomap_ops` at the time.
You can ignore my previous patch; let’s continue the analysis based on
your patch now.

> 
> Thanks,
> Joanne
> 
> [1] https://lore.kernel.org/linux-fsdevel/e85e73bd-5fbe-4d11-89ba-7b42c237dd34@bytedance.com/
> 

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[Christoph Hellwig]

On Fri, Jul 03, 2026 at 05:13:12PM +0800, changfengnan wrote:
> taken, which introduced some overhead. 
> I implemented some optimizations based on Christoph’s suggestions and
> made some modifications to `iomap_process`; 
> Now, XFS performance is essentially unchanged, while ext4 still shows a
> 1.5% drop; I will continue to investigate the cause..   I’ve attached the patch.
> Actually, I had expected that switching to `iomap_next` would improve
> performance, since it avoids an indirect call.

I don't think it will for you - the direct I/O readers don't have
iomap_end methods, so you still have the same number of indirect
calls.  What could help is to inline the iterator as seen in Joannes'
fist demo series.  We could even look into doing that only for simple
dio in a first step, shifting the burden to use the simple method
to the callers (at least for the POC).

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[changfengnan]

> From: "Christoph Hellwig"<hch@infradead.org>
> Date:  Fri, Jul 3, 2026, 20:40
> Subject:  Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O
> To: "changfengnan"<changfengnan@bytedance.com>
> Cc: "Joanne Koong"<joannelkoong@gmail.com>, "Christoph Hellwig"<hch@infradead.org>, "brauner"<brauner@kernel.org>, "djwong"<djwong@kernel.org>, "ojaswin"<ojaswin@linux.ibm.com>, "dgc"<dgc@kernel.org>, "linux-xfs"<linux-xfs@vger.kernel.org>, "linux-fsdevel"<linux-fsdevel@vger.kernel.org>, "linux-ext4"<linux-ext4@vger.kernel.org>, "linux-kernel"<linux-kernel@vger.kernel.org>, <lidiangang@bytedance.com>, "pankaj.raghav"<pankaj.raghav@linux.dev>, "Brian Foster"<bfoster@redhat.com>
> On Fri, Jul 03, 2026 at 05:13:12PM +0800, changfengnan wrote:
> > taken, which introduced some overhead. 
> > I implemented some optimizations based on Christoph’s suggestions and
> > made some modifications to `iomap_process`; 
> > Now, XFS performance is essentially unchanged, while ext4 still shows a
> > 1.5% drop; I will continue to investigate the cause..   I’ve attached the patch.
> > Actually, I had expected that switching to `iomap_next` would improve
> > performance, since it avoids an indirect call.
> 
> I don't think it will for you - the direct I/O readers don't have
> iomap_end methods, so you still have the same number of indirect
> calls.  What could help is to inline the iterator as seen in Joannes'
> fist demo series.  We could even look into doing that only for simple
> dio in a first step, shifting the burden to use the simple method
> to the callers (at least for the POC).

I'm not sure if this is what you had in mind, but I made some
adjustments referencing Joannes' first patch, and using this approach, 
there was no performance degradation.

diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index c05849d305ae4..fd2aeabca892a 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -809,8 +809,8 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 
         if (iocb->ki_flags & IOCB_DIRECT) {
                 file_accessed(iocb->ki_filp);
-                ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL, 0,
-                                NULL, 0);
+                ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL,
+                                   IOMAP_DIO_NO_IOMAP_END, NULL, 0);
         } else {
                 ret = generic_file_read_iter(iocb, iter);
         }
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 755fde1baf036..cf1be36fdb177 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -4004,6 +4004,9 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end)
                 io_end->flag &= ~EXT4_IO_END_UNWRITTEN;
 }
 
+int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
+                     unsigned int flags, struct iomap *iomap,
+                     struct iomap *srcmap);
 int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
                 struct iomap *srcmap);
 int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index dbe073e181a7a..e08edabcdeaa0 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,8 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
                 return generic_file_read_iter(iocb, to);
         }
 
-        ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
+        ret = iomap_dio_rw_begin(iocb, to, ext4_iomap_begin, ext4_iomap_next,
+                                 NULL, IOMAP_DIO_NO_IOMAP_END, NULL, 0);
         inode_unlock_shared(inode);
 
         file_accessed(iocb->ki_filp);
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 4c30dd8dbec7a..479406c84ff61 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3770,9 +3770,9 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,
         return ret;
 }
 
-
-static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
-                unsigned flags, struct iomap *iomap, struct iomap *srcmap)
+int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
+                     unsigned int flags, struct iomap *iomap,
+                     struct iomap *srcmap)
 {
         int ret;
         struct ext4_map_blocks map;
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index d08b7d3c52710..fa17e249b0b03 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -1000,6 +1000,40 @@ iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
         return true;
 }
 
+static inline void
+iomap_dio_simple_iter_done(struct iomap_iter *iomi)
+{
+        WARN_ON_ONCE(iomi->iomap.offset > iomi->pos);
+        WARN_ON_ONCE(iomi->iomap.length == 0);
+        WARN_ON_ONCE(iomi->iomap.offset + iomi->iomap.length <= iomi->pos);
+        WARN_ON_ONCE(iomi->iomap.flags & IOMAP_F_STALE);
+
+        iomi->iter_start_pos = iomi->pos;
+
+        trace_iomap_iter_dstmap(iomi->inode, &iomi->iomap);
+        if (iomi->srcmap.type != IOMAP_HOLE)
+                trace_iomap_iter_srcmap(iomi->inode, &iomi->srcmap);
+}
+
+static inline int
+iomap_dio_simple_begin_iter(struct iomap_iter *iomi,
+                            iomap_begin_fn iomap_begin,
+                            iomap_next_fn iomap_next)
+{
+        int ret;
+
+        trace_iomap_iter(iomi, iomap_next, _RET_IP_);
+
+        ret = iomap_begin(iomi->inode, iomi->pos, iomi->len, iomi->flags,
+                          &iomi->iomap, &iomi->srcmap);
+        iomi->status = 0;
+        if (ret < 0)
+                return ret;
+
+        iomap_dio_simple_iter_done(iomi);
+        return 1;
+}
+
 static int
 iomap_dio_simple_finish_iter(struct iomap_iter *iomi,
                 iomap_next_fn iomap_next, size_t done)
@@ -1053,8 +1087,8 @@ iomap_dio_simple_finish_iter(struct iomap_iter *iomi,
  */
 static ssize_t
 iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
-                 iomap_next_fn iomap_next, void *private,
-                 unsigned int dio_flags)
+                 iomap_next_fn iomap_next, iomap_begin_fn iomap_begin,
+                 void *private, unsigned int dio_flags)
 {
         struct inode *inode = file_inode(iocb->ki_filp);
         size_t count = iov_iter_count(iter);
@@ -1080,7 +1114,11 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 
         inode_dio_begin(inode);
 
-        ret = iomap_iter(&iomi, iomap_next);
+        if (iomap_begin)
+                ret = iomap_dio_simple_begin_iter(&iomi, iomap_begin,
+                                                  iomap_next);
+        else
+                ret = iomap_iter(&iomi, iomap_next);
         if (ret <= 0) {
                 inode_dio_end(inode);
                 return ret ?: -ENOTBLK;
@@ -1142,9 +1180,18 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
                 WRITE_ONCE(iocb->private, bio);
         }
 
-        ret = iomap_dio_simple_finish_iter(&iomi, iomap_next, count);
-        if (ret)
-                goto out_bio_release_pages_no_end;
+        /*
+         * The finishing iomap_iter() only runs the filesystem's ->iomap_end()
+         * work and releases any attached folio batch.  When the caller promised
+         * neither applies (IOMAP_DIO_NO_IOMAP_END), skip it to avoid an extra
+         * indirect ->iomap_next() call and the iomap_iter() frames on the hot
+         * read path.
+         */
+        if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END)) {
+                ret = iomap_dio_simple_finish_iter(&iomi, iomap_next, count);
+                if (ret)
+                        goto out_bio_release_pages_no_end;
+        }
 
         if (!wait_for_completion) {
                 bio->bi_end_io = iomap_dio_simple_end_io;
@@ -1161,7 +1208,8 @@ iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
 out_bio_put:
         bio_put(bio);
 out_iomap_end:
-        iomap_dio_simple_finish_iter(&iomi, iomap_next, 0);
+        if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
+                iomap_dio_simple_finish_iter(&iomi, iomap_next, 0);
         inode_dio_end(inode);
         return ret;
 out_bio_release_pages_no_end:
@@ -1181,8 +1229,8 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 
         if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
                                        done_before)) {
-                ret = iomap_dio_simple(iocb, iter, iomap_next, private,
-                                dio_flags);
+                ret = iomap_dio_simple(iocb, iter, iomap_next, NULL, private,
+                                       dio_flags);
                 if (ret != -ENOTBLK)
                         return ret;
         }
@@ -1195,6 +1243,32 @@ iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
 }
 EXPORT_SYMBOL_GPL(iomap_dio_rw);
 
+ssize_t
+iomap_dio_rw_begin(struct kiocb *iocb, struct iov_iter *iter,
+                   iomap_begin_fn iomap_begin, iomap_next_fn iomap_next,
+                   const struct iomap_dio_ops *dops, unsigned int dio_flags,
+                   void *private, size_t done_before)
+{
+        struct iomap_dio *dio;
+        ssize_t ret;
+
+        if ((dio_flags & IOMAP_DIO_NO_IOMAP_END) &&
+            iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
+                                       done_before)) {
+                ret = iomap_dio_simple(iocb, iter, iomap_next, iomap_begin,
+                                       private, dio_flags);
+                if (ret != -ENOTBLK)
+                        return ret;
+        }
+
+        dio = __iomap_dio_rw(iocb, iter, iomap_next, dops, dio_flags, private,
+                             done_before);
+        if (IS_ERR_OR_NULL(dio))
+                return PTR_ERR_OR_ZERO(dio);
+        return iomap_dio_complete(dio);
+}
+EXPORT_SYMBOL_GPL(iomap_dio_rw_begin);
+
 static int __init iomap_dio_init(void)
 {
         return bioset_init(&iomap_dio_simple_pool, 4,
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index 984045af310a1..7dee4a7944071 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -15,16 +15,6 @@ static inline void iomap_iter_clean_fbatch(struct iomap_iter *iter)
         }
 }
 
-/* Advance the current iterator position and decrement the remaining length */
-int iomap_iter_advance(struct iomap_iter *iter, u64 count)
-{
-        if (WARN_ON_ONCE(count > iomap_length(iter)))
-                return -EIO;
-        iter->pos += count;
-        iter->len -= count;
-        return 0;
-}
-
 static inline void iomap_iter_done(struct iomap_iter *iter)
 {
         WARN_ON_ONCE(iter->iomap.offset > iter->pos);
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47d..e43ca419b13a8 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -127,7 +127,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
         {IOMAP_DIO_FORCE_WAIT,                "DIO_FORCE_WAIT" }, \
         {IOMAP_DIO_OVERWRITE_ONLY,        "DIO_OVERWRITE_ONLY" }, \
         {IOMAP_DIO_PARTIAL,                "DIO_PARTIAL" }, \
-        {IOMAP_DIO_FSBLOCK_ALIGNED,        "DIO_FSBLOCK_ALIGNED" }
+        {IOMAP_DIO_FSBLOCK_ALIGNED,        "DIO_FSBLOCK_ALIGNED" }, \
+        {IOMAP_DIO_NO_IOMAP_END,        "DIO_NO_IOMAP_END" }
 
 DECLARE_EVENT_CLASS(iomap_class,
         TP_PROTO(struct inode *inode, struct iomap *iomap),
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index a4f99128b46c5..babdfd0eff4f9 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -458,8 +458,8 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
                 }
 
                 file_accessed(iocb->ki_filp);
-                ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL, 0,
-                                NULL, 0);
+                ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL,
+                                   IOMAP_DIO_NO_IOMAP_END, NULL, 0);
         } else {
                 ret = generic_file_read_iter(iocb, to);
         }
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index a987ffbf3c023..2d4dbd9fac7b4 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -251,7 +251,7 @@ xfs_file_dio_read(
         struct iov_iter                *to)
 {
         struct xfs_inode        *ip = XFS_I(file_inode(iocb->ki_filp));
-        unsigned int                dio_flags = 0;
+        unsigned int                dio_flags = IOMAP_DIO_NO_IOMAP_END;
         const struct iomap_dio_ops *dio_ops = NULL;
         ssize_t                        ret;
 
@@ -269,8 +269,9 @@ xfs_file_dio_read(
                 dio_ops = &xfs_dio_read_bounce_ops;
                 dio_flags |= IOMAP_DIO_BOUNCE;
         }
-        ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, dio_ops, dio_flags,
-                        NULL, 0);
+        ret = iomap_dio_rw_begin(iocb, to, xfs_read_iomap_begin,
+                                 xfs_read_iomap_next, dio_ops, dio_flags,
+                                 NULL, 0);
         xfs_iunlock(ip, XFS_IOLOCK_SHARED);
 
         return ret;
diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
index 71c4bb024f042..dcc7f1fe2d52b 100644
--- a/fs/xfs/xfs_iomap.c
+++ b/fs/xfs/xfs_iomap.c
@@ -2203,7 +2203,7 @@ xfs_buffered_write_iomap_next(
                         xfs_buffered_write_iomap_end);
 }
 
-static int
+int
 xfs_read_iomap_begin(
         struct inode                *inode,
         loff_t                        offset,
diff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h
index 01875d20fb669..3db533e977e09 100644
--- a/fs/xfs/xfs_iomap.h
+++ b/fs/xfs/xfs_iomap.h
@@ -55,6 +55,9 @@ int xfs_direct_write_iomap_next(const struct iomap_iter *iter,
                 struct iomap *iomap, struct iomap *srcmap);
 int xfs_zoned_direct_write_iomap_next(const struct iomap_iter *iter,
                 struct iomap *iomap, struct iomap *srcmap);
+int xfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
+                         unsigned int flags, struct iomap *iomap,
+                         struct iomap *srcmap);
 int xfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
                 struct iomap *srcmap);
 int xfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 3b41f123a92d4..060051fc79b1f 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -266,7 +266,6 @@ struct iomap_iter {
 };
 
 int iomap_iter(struct iomap_iter *iter, iomap_next_fn iomap_next);
-int iomap_iter_advance(struct iomap_iter *iter, u64 count);
 
 /**
  * iomap_length_trim - trimmed length of the current iomap iteration
@@ -298,6 +297,16 @@ static inline u64 iomap_length(const struct iomap_iter *iter)
         return iomap_length_trim(iter, iter->pos, iter->len);
 }
 
+/* Advance the current iterator position and decrement the remaining length */
+static inline int iomap_iter_advance(struct iomap_iter *iter, u64 count)
+{
+        if (WARN_ON_ONCE(count > iomap_length(iter)))
+                return -EIO;
+        iter->pos += count;
+        iter->len -= count;
+        return 0;
+}
+
 /**
  * iomap_iter_advance_full - advance by the full length of current map
  */
@@ -605,9 +614,23 @@ struct iomap_dio_ops {
  */
 #define IOMAP_DIO_BOUNCE                (1 << 4)
 
+/*
+ * The caller's ->iomap_next() has no ->iomap_end() work to run and this I/O
+ * does not use a folio batch, so the iomap_dio_simple() fast path may skip
+ * the finishing iomap_iter() call entirely.  Only meaningful on the simple
+ * read fast path; ignored by the generic __iomap_dio_rw() slow path.
+ */
+#define IOMAP_DIO_NO_IOMAP_END                BIT(5)
+
 ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
                 iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
                 unsigned int dio_flags, void *private, size_t done_before);
+ssize_t iomap_dio_rw_begin(struct kiocb *iocb, struct iov_iter *iter,
+                           iomap_begin_fn iomap_begin,
+                           iomap_next_fn iomap_next,
+                           const struct iomap_dio_ops *dops,
+                           unsigned int dio_flags, void *private,
+                           size_t done_before);
 struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
                 iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
                 unsigned int dio_flags, void *private, size_t done_before);
> 

Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O

[Joanne Koong]

On Mon, Jul 6, 2026 at 4:10 AM changfengnan <changfengnan@bytedance.com> wrote:
>
> > From: "Christoph Hellwig"<hch@infradead.org>
> > Date:  Fri, Jul 3, 2026, 20:40
> > Subject:  Re: [PATCH v6 3/3] iomap: add simple dio path for small direct I/O
> > To: "changfengnan"<changfengnan@bytedance.com>
> > Cc: "Joanne Koong"<joannelkoong@gmail.com>, "Christoph Hellwig"<hch@infradead.org>, "brauner"<brauner@kernel.org>, "djwong"<djwong@kernel.org>, "ojaswin"<ojaswin@linux.ibm.com>, "dgc"<dgc@kernel.org>, "linux-xfs"<linux-xfs@vger.kernel.org>, "linux-fsdevel"<linux-fsdevel@vger.kernel.org>, "linux-ext4"<linux-ext4@vger.kernel.org>, "linux-kernel"<linux-kernel@vger.kernel.org>, <lidiangang@bytedance.com>, "pankaj.raghav"<pankaj.raghav@linux.dev>, "Brian Foster"<bfoster@redhat.com>
> > On Fri, Jul 03, 2026 at 05:13:12PM +0800, changfengnan wrote:
> > > taken, which introduced some overhead.
> > > I implemented some optimizations based on Christoph’s suggestions and
> > > made some modifications to `iomap_process`;
> > > Now, XFS performance is essentially unchanged, while ext4 still shows a
> > > 1.5% drop; I will continue to investigate the cause..   I’ve attached the patch.
> > > Actually, I had expected that switching to `iomap_next` would improve
> > > performance, since it avoids an indirect call.
> >
> > I don't think it will for you - the direct I/O readers don't have
> > iomap_end methods, so you still have the same number of indirect
> > calls.  What could help is to inline the iterator as seen in Joannes'
> > fist demo series.  We could even look into doing that only for simple
> > dio in a first step, shifting the burden to use the simple method
> > to the callers (at least for the POC).
>
> I'm not sure if this is what you had in mind, but I made some
> adjustments referencing Joannes' first patch, and using this approach,
> there was no performance degradation.

Thanks, Fengnan. This is awesome that it got the degradation down to zero.

Would you be able to test how this change performs on your benchmarks?:

diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index dbe073e181a7..bf0c18cf1017 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,8 @@ static ssize_t ext4_dio_read_iter(struct kiocb
*iocb, struct iov_iter *to)
                return generic_file_read_iter(iocb, to);
        }

-       ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
+       ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL,
+                          IOMAP_DIO_NO_IOMAP_END, NULL, 0);
        inode_unlock_shared(inode);

        file_accessed(iocb->ki_filp);
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index bcd3d4223464..bc3b87e57592 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -968,115 +968,38 @@ static void iomap_dio_simple_end_io(struct bio *bio)
        iocb->ki_complete(iocb, iomap_dio_simple_complete(sr));
 }

-static inline bool
-iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
-                          const struct iomap_dio_ops *dops,
-                          unsigned int dio_flags, size_t done_before)
+static inline void
+iomap_dio_simple_finish(struct iomap_iter *iomi, iomap_next_fn iomap_next,
+               size_t written)
 {
-       struct inode *inode = file_inode(iocb->ki_filp);
-       size_t count = iov_iter_count(iter);
-
-       if (dops || done_before)
-               return false;
-       if (iov_iter_rw(iter) != READ)
-               return false;
-       if (!count)
-               return false;
-       /*
-        * Simple dio is an optimization for small IO. Filter out large IO
-        * early as it's the most common case to fail for typical direct IO
-        * workloads.
-        */
-       if (count > inode->i_sb->s_blocksize)
-               return false;
-       if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
-                        IOMAP_DIO_BOUNCE))
-               return false;
-       if (iocb->ki_pos + count > i_size_read(inode))
-               return false;
-       if (IS_ENCRYPTED(inode))
-               return false;
-
-       return true;
+       iomi->iter_start_pos = iomi->pos;
+       iomi->pos += written;
+       iomi->len -= written;
+       iomap_next(iomi, &iomi->iomap, &iomi->srcmap);
 }

-/*
- * Fast path for small, block-aligned direct I/Os that map to a single
- * contiguous on-disk extent.
- *
- * iomap_dio_simple_supported() enforces the cheap up-front constraints before
- * entering this path.
- *
- * @dops must be NULL: a non-NULL @dops means the caller wants its
- * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
- * allocated from the filesystem-private @dops->bio_set (whose
front_pad sizes a
- * filesystem-private wrapper around the bio).  The fast path instead allocates
- * from the shared iomap_dio_simple_pool, whose front_pad matches struct
- * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
- * back to __iomap_dio_rw() in that case.
- *
- * @done_before must be zero: a non-zero caller-accumulated residual cannot be
- * carried through a single-bio inline completion.
- *
- * @iter must describe a non-empty READ no larger than the inode block size:
- * writes, zero-length I/O, and larger requests need the generic iomap direct
- * I/O path.
- *
- * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
- * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
- * I/O, or bouncing.  The range must also stay within i_size and encrypted
- * inodes must use the generic iomap direct I/O path.
- *
- * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
- * decides the request does not fit the fast path.  In that case we proceed to
- * the generic __iomap_dio_rw() slow path.  Any other errno is a real
result and
- * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
- * caller.
- */
-static ssize_t
-iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
-                iomap_next_fn iomap_next, void *private,
-                unsigned int dio_flags)
+ssize_t
+__iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+               struct iomap_iter *iomi, iomap_next_fn iomap_next,
+               unsigned int dio_flags)
 {
        struct inode *inode = file_inode(iocb->ki_filp);
        size_t count = iov_iter_count(iter);
        bool wait_for_completion = is_sync_kiocb(iocb);
-       struct iomap_iter iomi = {
-               .inode          = inode,
-               .pos            = iocb->ki_pos,
-               .len            = count,
-               .flags          = IOMAP_DIRECT,
-               .private        = private,
-       };
        struct iomap_dio_simple *sr;
        unsigned int alignment;
        struct bio *bio;
        ssize_t ret;

-       if (iocb->ki_flags & IOCB_NOWAIT)
-               iomi.flags |= IOMAP_NOWAIT;
-
-       ret = kiocb_write_and_wait(iocb, count);
-       if (ret)
-               return ret;
-
-       inode_dio_begin(inode);
-
-       ret = iomap_iter(&iomi, iomap_next);
-       if (ret <= 0) {
-               inode_dio_end(inode);
-               return ret ? ret : -EFAULT;
-       }
-
-       if (iomi.iomap.type != IOMAP_MAPPED ||
-           iomi.iomap.offset + iomi.iomap.length < iomi.pos + count ||
-           (iomi.iomap.flags & IOMAP_F_INTEGRITY)) {
+       if (iomi->iomap.type != IOMAP_MAPPED ||
+           iomi->iomap.offset + iomi->iomap.length < iomi->pos + count ||
+           (iomi->iomap.flags & IOMAP_F_INTEGRITY)) {
                ret = -ENOTBLK;
                goto out_iomap_end;
        }

-       alignment = iomap_dio_alignment(inode, iomi.iomap.bdev, dio_flags);
-       if ((iomi.pos | count) & (alignment - 1)) {
+       alignment = iomap_dio_alignment(inode, iomi->iomap.bdev, dio_flags);
+       if ((iomi->pos | count) & (alignment - 1)) {
                ret = -EINVAL;
                goto out_iomap_end;
        }
@@ -1092,14 +1015,14 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
        if (user_backed_iter(iter))
                dio_flags |= IOMAP_DIO_USER_BACKED;

-       bio = bio_alloc_bioset(iomi.iomap.bdev,
+       bio = bio_alloc_bioset(iomi->iomap.bdev,
                               bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS),
                               REQ_OP_READ, GFP_KERNEL, &iomap_dio_simple_pool);
        sr = container_of(bio, struct iomap_dio_simple, bio);
        sr->iocb = iocb;
        sr->dio_flags = dio_flags;

-       bio->bi_iter.bi_sector = iomap_sector(&iomi.iomap, iomi.pos);
+       bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
        bio->bi_ioprio = iocb->ki_ioprio;

        ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
@@ -1124,8 +1047,8 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
                WRITE_ONCE(iocb->private, bio);
        }

-       iomap_iter_advance(&iomi, count);
-       iomap_iter(&iomi, iomap_next);
+       if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
+               iomap_dio_simple_finish(iomi, iomap_next, count);

        if (!wait_for_completion) {
                bio->bi_end_io = iomap_dio_simple_end_io;
@@ -1142,33 +1065,12 @@ iomap_dio_simple(struct kiocb *iocb, struct
iov_iter *iter,
 out_bio_put:
        bio_put(bio);
 out_iomap_end:
-       iomap_iter(&iomi, iomap_next);
+       if (!(dio_flags & IOMAP_DIO_NO_IOMAP_END))
+               iomap_dio_simple_finish(iomi, iomap_next, 0);
        inode_dio_end(inode);
        return ret;
 }
-
-ssize_t
-iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
-               iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
-               unsigned int dio_flags, void *private, size_t done_before)
-{
-       struct iomap_dio *dio;
-       ssize_t ret;
-
-       if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
-                                      done_before)) {
-               ret = iomap_dio_simple(iocb, iter, iomap_next,
private, dio_flags);
-               if (ret != -ENOTBLK)
-                       return ret;
-       }
-
-       dio = __iomap_dio_rw(iocb, iter, iomap_next, dops, dio_flags, private,
-                            done_before);
-       if (IS_ERR_OR_NULL(dio))
-               return PTR_ERR_OR_ZERO(dio);
-       return iomap_dio_complete(dio);
-}
-EXPORT_SYMBOL_GPL(iomap_dio_rw);
+EXPORT_SYMBOL_GPL(__iomap_dio_simple);

 static int __init iomap_dio_init(void)
 {
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index 984045af310a..1128f5b526ef 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -89,9 +89,6 @@ int iomap_iter_continue(const struct iomap_iter
*iter, struct iomap *iomap,
        bool stale = iomap->flags & IOMAP_F_STALE;
        ssize_t advanced = iter->pos - iter->iter_start_pos;

-       if (!iomap->length)
-               return 1;
-
        /*
         * Use iter->len to determine whether to continue onto the next mapping.
         * Explicitly terminate on error status or if the current iter has not
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index e1ea8392cf47..e43ca419b13a 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -127,7 +127,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
        {IOMAP_DIO_FORCE_WAIT,          "DIO_FORCE_WAIT" }, \
        {IOMAP_DIO_OVERWRITE_ONLY,      "DIO_OVERWRITE_ONLY" }, \
        {IOMAP_DIO_PARTIAL,             "DIO_PARTIAL" }, \
-       {IOMAP_DIO_FSBLOCK_ALIGNED,     "DIO_FSBLOCK_ALIGNED" }
+       {IOMAP_DIO_FSBLOCK_ALIGNED,     "DIO_FSBLOCK_ALIGNED" }, \
+       {IOMAP_DIO_NO_IOMAP_END,        "DIO_NO_IOMAP_END" }

 DECLARE_EVENT_CLASS(iomap_class,
        TP_PROTO(struct inode *inode, struct iomap *iomap),
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index a987ffbf3c02..6a551efff66f 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -251,7 +251,7 @@ xfs_file_dio_read(
        struct iov_iter         *to)
 {
        struct xfs_inode        *ip = XFS_I(file_inode(iocb->ki_filp));
-       unsigned int            dio_flags = 0;
+       unsigned int            dio_flags = IOMAP_DIO_NO_IOMAP_END;
        const struct iomap_dio_ops *dio_ops = NULL;
        ssize_t                 ret;

diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 3b41f123a92d..6213a3b1a0b9 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -10,6 +10,7 @@
 #include <linux/mm_types.h>
 #include <linux/blkdev.h>
 #include <linux/folio_batch.h>
+#include <linux/pagemap.h>

 struct address_space;
 struct fiemap_extent_info;
@@ -605,15 +606,136 @@ struct iomap_dio_ops {
  */
 #define IOMAP_DIO_BOUNCE               (1 << 4)

-ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
-               iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
-               unsigned int dio_flags, void *private, size_t done_before);
+/* optimization hint for the simple fast path */
+#define IOMAP_DIO_NO_IOMAP_END         (1 << 5)
+
 struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
                iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
                unsigned int dio_flags, void *private, size_t done_before);
 ssize_t iomap_dio_complete(struct iomap_dio *dio);
 void iomap_dio_bio_end_io(struct bio *bio);

+static inline bool
+iomap_dio_simple_supported(struct kiocb *iocb, struct iov_iter *iter,
+                          const struct iomap_dio_ops *dops,
+                          unsigned int dio_flags, size_t done_before)
+{
+       struct inode *inode = file_inode(iocb->ki_filp);
+       size_t count = iov_iter_count(iter);
+
+       if (dops || done_before)
+               return false;
+       if (iov_iter_rw(iter) != READ)
+               return false;
+       if (!count)
+               return false;
+       if (count > inode->i_sb->s_blocksize)
+               return false;
+       if (dio_flags & (IOMAP_DIO_FORCE_WAIT | IOMAP_DIO_PARTIAL |
+                        IOMAP_DIO_BOUNCE))
+               return false;
+       if (iocb->ki_pos + count > i_size_read(inode))
+               return false;
+       if (IS_ENCRYPTED(inode))
+               return false;
+       return true;
+}
+
+ssize_t __iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+               struct iomap_iter *iomi, iomap_next_fn iomap_next,
+               unsigned int dio_flags);
+
+/*
+ * Fast path for small, block-aligned direct I/Os that map to a single
+ * contiguous on-disk extent.
+ *
+ * iomap_dio_simple_supported() enforces the cheap up-front constraints before
+ * entering this path.
+ *
+ * @dops must be NULL: a non-NULL @dops means the caller wants its
+ * ->end_io / ->submit_io hooks invoked, and in particular wants its bios to be
+ * allocated from the filesystem-private @dops->bio_set (whose
front_pad sizes a
+ * filesystem-private wrapper around the bio).  The fast path instead allocates
+ * from the shared iomap_dio_simple_pool, whose front_pad matches struct
+ * iomap_dio_simple; the two wrappers are not interchangeable, so we must fall
+ * back to __iomap_dio_rw() in that case.
+ *
+ * @done_before must be zero: a non-zero caller-accumulated residual cannot be
+ * carried through a single-bio inline completion.
+ *
+ * @iter must describe a non-empty READ no larger than the inode block size:
+ * writes, zero-length I/O, and larger requests need the generic iomap direct
+ * I/O path.
+ *
+ * @dio_flags must not request IOMAP_DIO_FORCE_WAIT, IOMAP_DIO_PARTIAL, or
+ * IOMAP_DIO_BOUNCE: this path does not support forced waiting, partial direct
+ * I/O, or bouncing.  The range must also stay within i_size and encrypted
+ * inodes must use the generic iomap direct I/O path. IOMAP_DIO_NO_IOMAP_END is
+ * a fast-path optimization the caller can set if there is no work
that needs to
+ * be done after a mapping.
+ *
+ * -ENOTBLK is the private sentinel returned by iomap_dio_simple() when it
+ * decides the request does not fit the fast path.  In that case we proceed to
+ * the generic __iomap_dio_rw() slow path.  Any other errno is a real
result and
+ * is propagated as-is, in particular -EAGAIN for IOCB_NOWAIT must reach the
+ * caller.
+ */
+static __always_inline ssize_t
+iomap_dio_simple(struct kiocb *iocb, struct iov_iter *iter,
+               iomap_next_fn iomap_next, void *private, unsigned int dio_flags)
+{
+       struct inode *inode = file_inode(iocb->ki_filp);
+       size_t count = iov_iter_count(iter);
+       struct iomap_iter iomi = {
+               .inode          = inode,
+               .pos            = iocb->ki_pos,
+               .len            = count,
+               .flags          = IOMAP_DIRECT,
+               .private        = private,
+       };
+       ssize_t ret;
+
+       if (iocb->ki_flags & IOCB_NOWAIT)
+               iomi.flags |= IOMAP_NOWAIT;
+
+       ret = kiocb_write_and_wait(iocb, count);
+       if (ret)
+               return ret;
+
+       inode_dio_begin(inode);
+
+       ret = iomap_next(&iomi, &iomi.iomap, &iomi.srcmap);
+       if (ret <= 0) {
+               inode_dio_end(inode);
+               /* a zero return means no mapping, which should never happen */
+               return ret ? ret : -EFAULT;
+       }
+
+       return __iomap_dio_simple(iocb, iter, &iomi, iomap_next, dio_flags);
+}
+
+static __always_inline ssize_t
+iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomap_next_fn
iomap_next,
+               const struct iomap_dio_ops *dops, unsigned int dio_flags,
+               void *private, size_t done_before)
+{
+       struct iomap_dio *dio;
+       ssize_t ret;
+
+       if (iomap_dio_simple_supported(iocb, iter, dops, dio_flags,
done_before)) {
+               ret = iomap_dio_simple(iocb, iter, iomap_next, private,
+                               dio_flags);
+               if (ret != -ENOTBLK)
+                       return ret;
+       }
+
+       dio = __iomap_dio_rw(iocb, iter, iomap_next, dops, dio_flags, private,
+                            done_before);
+       if (IS_ERR_OR_NULL(dio))
+               return PTR_ERR_OR_ZERO(dio);
+       return iomap_dio_complete(dio);
+}
+
 #ifdef CONFIG_SWAP
 struct file;
 struct swap_info_struct;
@@ -678,21 +800,22 @@ static __always_inline int iomap_process(const
struct iomap_iter *iter,
 {
        int ret = 0;

-       if (iomap->length && end) {
-               ssize_t advanced = iter->pos - iter->iter_start_pos;
-               loff_t len;
+       if (iomap->length) {
+               if (end) {
+                       ssize_t advanced = iter->pos - iter->iter_start_pos;
+                       loff_t len;

-               len = iomap_length_trim(iter, iter->iter_start_pos,
-                               iter->len + advanced);
+                       len = iomap_length_trim(iter, iter->iter_start_pos,
+                                       iter->len + advanced);

-               ret = end(iter->inode, iter->iter_start_pos, len, advanced,
-                               iter->flags, iomap);
+                       ret = end(iter->inode, iter->iter_start_pos,
len, advanced,
+                                       iter->flags, iomap);
+               }
+               ret = iomap_iter_continue(iter, iomap, srcmap, ret);
+               if (ret <= 0)
+                       return ret;
        }

-       ret = iomap_iter_continue(iter, iomap, srcmap, ret);
-       if (ret <= 0)
-               return ret;
-
        ret = begin(iter->inode, iter->pos, iter->len, iter->flags, iomap,
                        srcmap);

I put the changes in a git tree [1] in case that's is more convenient
for you. I think this ends up devirtualizing the iomap_next callback:

before:
$ objdump -dr fs/xfs/xfs.ko | awk '/<xfs_file_dio_read>:/{f=1}
f{print} f&&/^$/{exit}'  | grep -E 'iomap_next|iomap_begin|thunk'
                          a9ac3: R_X86_64_PLT32   __x86_return_thunk-0x4
                          a9b4b: R_X86_64_32S     xfs_read_iomap_next
                          a9b9c: R_X86_64_PLT32   __x86_return_thunk-0x4

after:
$ objdump -dr fs/xfs/xfs_iomap_next_inlined.ko | awk
'/<xfs_file_dio_read>:/{f=1} f{print} f&&/^$/{exit}' | grep -E
'iomap_next|iomap_begin|thunk'
                          aa083: R_X86_64_PLT32   __x86_return_thunk-0x4
                          aa1d4: R_X86_64_PLT32   xfs_read_iomap_next-0x4
                          aa1ec: R_X86_64_32S     xfs_read_iomap_next
                          aa232: R_X86_64_32S     xfs_read_iomap_next


I was unsure how much bloat this adds but I didn't see a noticable
impact when I ran ./scripts/bloat-o-meter:

$ ./scripts/bloat-o-meter fs/xfs/xfs.ko fs/xfs/xfs_iomap_next_inlined.ko
add/remove: 0/0 grow/shrink: 11/1 up/down: 589/-4 (585)
Function                                     old     new   delta
xfs_file_dio_read                            309     685    +376
xfs_file_dio_write_aligned                   355     402     +47
xfs_file_dio_write_unaligned                 545     591     +46
xfs_file_dio_write_atomic                    457     502     +45
xfs_dax_write_iomap_next                     267     291     +24
xfs_atomic_write_cow_iomap_next             1151    1175     +24
xfs_zoned_direct_write_iomap_next            217     224      +7
xfs_direct_write_iomap_next                   77      84      +7
xfs_buffered_write_iomap_next                373     379      +6
xfs_xattr_iomap_next                         360     365      +5
xfs_seek_iomap_next                          665     667      +2
xfs_read_iomap_next                          681     677      -4
Total: Before=1977941, After=1978526, chg +0.03%

where the baseline xfs.ko was built at the commit in [2] and the
xfs_iomap_next_inlined.ko was built at the commit in [3].  If these
changes are acceptable to you, then I'll put this into v3 of the
iomap_next series.

Thanks,
Joanne

[1] https://github.com/joannekoong/linux/commits/iomap_simple_dio_iomap_next/
[2] https://github.com/joannekoong/linux/commit/1e82c8988ee56e9e7202d5ee09c7f7ea7ac7a832
[3] https://github.com/joannekoong/linux/commit/86ecc9de53bb1b8b1287950fbb745a5a8e49f736