lazy bounce buffering for checksummed reads V2

lazy bounce buffering for checksummed reads V2

[Christoph Hellwig]

Hi all,

this series improves performance and resource usage for reads from
devices that require stable pages due to checksumming on XFS.

Currently XFS unconditionally bounce buffers reads on such devices to
prevent user modifications to the buffer from corrupting the data,
leading to checksum failures.

This uses DRAM bandwidth and CPU cycles for copies that are not needed
most of the time, and due to the use of a bio_vec for the bounce
buffer to smaller than wanted and unaligned I/O sizes when using 4k
user pages (i.e. 1MB-4k I/O).

This series addresses this by reading without the bounce buffer first,
and then only allocating a buffer and reading into that again on an
initial checksum failure.  To accommodate for rare (or hypothetical?)
applications that have legitimate needs to frequently modify in-flight
buffers, a sysfs know is provided to revert to the old behavior.

The baseline of this series is the xfs for-next branch that Carlos
just a pull request for.  The patches are split over 3 subsystems,
and I'd love to hear from the maintainers what a good test strategy
is.

A git tree is available to help with the review here:

    git://git.infradead.org/users/hch/misc.git lazy-bounce

Gitweb:

    https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/lazy-bounce

Changes since v1:
 - rebase on 7.3rc-1 and xfs for-next, which has the bio complete in
   task support merged
 - remove the now unused IOMAP_DIO_BOUNCE support for reads
 - fix compilation with integrity disabled
 - make I/O size limitation actually work
 - clear REQ_POLLED when bounce buffering
 - use sysfs string match helpers to allow non -n echo

[PATCH 01/17] block: pass a maxlen argument to bio_iov_iter_get_pages

[Christoph Hellwig]

Writes that require additional per-I/O allocations such as when using
file system level protection information can be limited to a maximum
size.  Allow passing that to bio_iov_iter_get_pages.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 block/bio.c          |  8 ++++++--
 block/blk-map.c      |  2 +-
 block/fops.c         |  3 ++-
 fs/iomap/direct-io.c | 10 +++++-----
 include/linux/bio.h  |  3 ++-
 5 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/block/bio.c b/block/bio.c
index 898b2f5ef8c8..efd9483b8348 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1261,6 +1261,7 @@ static inline bool bio_iov_bvec_aligned(const struct bio *bio,
  * bio_iov_iter_get_pages - add user or kernel pages to a bio
  * @bio: bio to add pages to
  * @iter: iov iterator describing the region to be added
+ * @maxlen: maximum size to consume from @iter
  * @mem_align_mask: the mask the source address and length must be aligned to,
  *	0 for no requirement
  * @len_align_mask: the mask to align the total size to, 0 for any length
@@ -1281,7 +1282,8 @@ static inline bool bio_iov_bvec_aligned(const struct bio *bio,
  * is returned only if 0 pages could be pinned.
  */
 int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
-			   unsigned mem_align_mask, unsigned len_align_mask)
+			   unsigned maxlen, unsigned mem_align_mask,
+			   unsigned len_align_mask)
 {
 	iov_iter_extraction_t flags = 0;
 
@@ -1293,6 +1295,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 		    !bio_iov_bvec_aligned(bio, mem_align_mask))
 			return -EINVAL;
 
+		/* Truncate to the maximum size that the caller can handle */
+		bio->bi_iter.bi_size = min(bio->bi_iter.bi_size, maxlen);
 		iov_iter_advance(iter, bio->bi_iter.bi_size);
 		return 0;
 	}
@@ -1306,7 +1310,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 		ssize_t ret;
 
 		ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec,
-				BIO_MAX_SIZE - bio->bi_iter.bi_size,
+				maxlen - bio->bi_iter.bi_size,
 				&bio->bi_vcnt, bio->bi_max_vecs,
 				mem_align_mask, flags);
 		if (ret <= 0) {
diff --git a/block/blk-map.c b/block/blk-map.c
index 9cb9605d1f62..81cba3af4e9c 100644
--- a/block/blk-map.c
+++ b/block/blk-map.c
@@ -274,7 +274,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,
 	 * No alignment requirements on our part to support arbitrary
 	 * passthrough commands.
 	 */
-	ret = bio_iov_iter_get_pages(bio, iter, 0, 0);
+	ret = bio_iov_iter_get_pages(bio, iter, BIO_MAX_SIZE, 0, 0);
 	if (ret)
 		goto out_put;
 	ret = blk_rq_append_bio(rq, bio);
diff --git a/block/fops.c b/block/fops.c
index 2ce7c6c4714e..a83df69b175a 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -46,7 +46,8 @@ static bool blkdev_dio_invalid(struct block_device *bdev, struct kiocb *iocb,
 static inline int blkdev_iov_iter_get_pages(struct bio *bio,
 		struct iov_iter *iter, struct block_device *bdev)
 {
-	return bio_iov_iter_get_pages(bio, iter, bdev_dma_alignment(bdev),
+	return bio_iov_iter_get_pages(bio, iter, BIO_MAX_SIZE,
+			bdev_dma_alignment(bdev),
 			bdev_logical_block_size(bdev) - 1);
 }
 
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 8b4039d16ce8..bc726d47b7dc 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -359,8 +359,8 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
 				iomap_max_bio_size(&iter->iomap), alignment);
 	else
 		ret = bio_iov_iter_get_pages(bio, dio->submit.iter,
-					     bdev_dma_alignment(bio->bi_bdev),
-					     alignment - 1);
+				BIO_MAX_SIZE, bdev_dma_alignment(bio->bi_bdev),
+				alignment - 1);
 	if (unlikely(ret))
 		goto out_put_bio;
 	ret = bio->bi_iter.bi_size;
@@ -1034,9 +1034,9 @@ ssize_t __iomap_dio_read_simple(struct kiocb *iocb, struct iov_iter *iter,
 	bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
 	bio->bi_ioprio = iocb->ki_ioprio;
 
-	ret = bio_iov_iter_get_pages(bio, iter,
-				bdev_dma_alignment(bio->bi_bdev),
-				alignment - 1);
+	ret = bio_iov_iter_get_pages(bio, iter, BIO_MAX_SIZE,
+			bdev_dma_alignment(bio->bi_bdev),
+			alignment - 1);
 	if (unlikely(ret))
 		goto out_bio_put;
 
diff --git a/include/linux/bio.h b/include/linux/bio.h
index bb3235497e67..0070be355181 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -516,7 +516,8 @@ int bdev_rw_virt(struct block_device *bdev, sector_t sector, void *data,
 		size_t len, enum req_op op);
 
 int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
-		unsigned mem_align_mask, unsigned len_align_mask);
+		unsigned maxlen, unsigned mem_align_mask,
+		unsigned len_align_mask);
 
 bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter);
 void __bio_release_pages(struct bio *bio, bool mark_dirty);
-- 
2.53.0

[PATCH 02/17] block: warn on too larger integrity allocations

[Christoph Hellwig]

Catch cases where upper layer bugs create larger I/Os than the
mempool would return even without dipping into the mempool.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 block/bio-integrity.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/block/bio-integrity.c b/block/bio-integrity.c
index b23e2434d80c..d3df726e0f08 100644
--- a/block/bio-integrity.c
+++ b/block/bio-integrity.c
@@ -72,6 +72,7 @@ void bio_integrity_alloc_buf(struct bio *bio, gfp_t gfp, bool zero_buffer)
 	unsigned int len = bio_integrity_bytes(bi, bio_sectors(bio));
 	void *buf;
 
+	WARN_ON_ONCE(len > BLK_INTEGRITY_MAX_SIZE);
 	buf = kmalloc(len, gfp | __GFP_NOWARN | (zero_buffer ? __GFP_ZERO : 0));
 	if (unlikely(!buf)) {
 		struct page *page;
-- 
2.53.0

[PATCH 03/17] block: split bio_iov_iter_bounce_write

[Christoph Hellwig]

Factor out a bio_alloc_bounce_folios helper that we'll use for a
different take on read-side bounce buffering soon.

For that make it and also bio_free_folios available to callers outside of
bio.c.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 block/bio.c         | 58 ++++++++++++++++++++++++++-------------------
 include/linux/bio.h |  2 ++
 2 files changed, 35 insertions(+), 25 deletions(-)

diff --git a/block/bio.c b/block/bio.c
index efd9483b8348..a87c33fe96ac 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1353,7 +1353,7 @@ static struct folio *folio_alloc_greedy(gfp_t gfp, size_t *size,
 	return folio_alloc(gfp, get_order(*size));
 }
 
-static void bio_free_folios(struct bio *bio)
+void bio_free_folios(struct bio *bio)
 {
 	struct bio_vec *bv;
 	int i;
@@ -1366,11 +1366,8 @@ static void bio_free_folios(struct bio *bio)
 	}
 }
 
-static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
-		size_t maxlen, size_t minsize)
+int bio_alloc_bounce_folios(struct bio *bio, size_t total_len, size_t minsize)
 {
-	size_t total_len = min(maxlen, iov_iter_count(iter));
-
 	if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
 		return -EINVAL;
 	if (WARN_ON_ONCE(bio->bi_iter.bi_size))
@@ -1380,7 +1377,6 @@ static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
 
 	do {
 		size_t this_len = min(total_len, SZ_1M);
-		size_t copied;
 		struct folio *folio;
 
 		if (this_len > minsize * 2)
@@ -1393,32 +1389,44 @@ static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
 		if (!folio)
 			break;
 		bio_add_folio_nofail(bio, folio, this_len, 0);
+		total_len -= this_len;
+	} while (total_len && bio->bi_vcnt < bio->bi_max_vecs);
+
+	if (!bio->bi_iter.bi_size)
+		return -ENOMEM;
+	return 0;
+}
+
+static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
+		size_t maxlen, size_t minsize)
+{
+	size_t total_len = min(maxlen, iov_iter_count(iter));
+	size_t total_copied = 0;
+	struct bio_vec *bv;
+	int i, error;
+
+	error = bio_alloc_bounce_folios(bio, total_len, minsize);
+	if (error)
+		return error;
+
+	bio_for_each_bvec_all(bv, bio, i) {
+		struct folio *folio = page_folio(bv->bv_page);
+		size_t copied;
 
 		if (iter->nofault)
-			copied = copy_folio_from_iter_atomic(folio, 0, this_len,
-							     iter);
+			copied = copy_folio_from_iter_atomic(folio, 0,
+					bv->bv_len, iter);
 		else
-			copied = copy_folio_from_iter(folio, 0, this_len, iter);
-		if (copied < this_len) {
-			/*
-			 * Need to revert the iov iter for all bytes we have
-			 * copied.
-			 *
-			 * However the bio size differs from the real copied
-			 * bytes as @this_len is queued but only advanced
-			 * less than that.
-			 * Need to compensate that for the revert.
-			 */
-			iov_iter_revert(iter, bio->bi_iter.bi_size - this_len +
-					copied);
+			copied = copy_folio_from_iter(folio, 0, bv->bv_len,
+					iter);
+		total_copied += copied;
+		if (copied < bv->bv_len) {
+			iov_iter_revert(iter, total_copied);
 			bio_free_folios(bio);
 			return -EFAULT;
 		}
-		total_len -= this_len;
-	} while (total_len && bio->bi_vcnt < bio->bi_max_vecs);
+	}
 
-	if (!bio->bi_iter.bi_size)
-		return -ENOMEM;
 	return bio_iov_iter_align_down(bio, iter,
 			&bio->bi_io_vec[bio->bi_vcnt - 1], minsize - 1);
 }
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 0070be355181..584b6abf6baf 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -527,6 +527,8 @@ extern void bio_check_pages_dirty(struct bio *bio);
 int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen,
 		size_t minsize);
 void bio_iov_iter_unbounce(struct bio *bio, bool is_error, bool mark_dirty);
+int bio_alloc_bounce_folios(struct bio *bio, size_t total_len, size_t minsize);
+void bio_free_folios(struct bio *bio);
 
 extern void bio_copy_data(struct bio *dst, struct bio *src);
 extern void bio_free_pages(struct bio *bio);
-- 
2.53.0

[PATCH 04/17] block: export fs_bio_integrity_{alloc,free}

[Christoph Hellwig]

We'll move integrity generation and verification into the iomap
submission helpers, which means they will be needed in modular file
system code.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 block/bio-integrity-fs.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/block/bio-integrity-fs.c b/block/bio-integrity-fs.c
index 692403dfa047..4f67b34bf289 100644
--- a/block/bio-integrity-fs.c
+++ b/block/bio-integrity-fs.c
@@ -31,6 +31,7 @@ unsigned int fs_bio_integrity_alloc(struct bio *bio)
 		bio_integrity_setup_default(bio);
 	return action;
 }
+EXPORT_SYMBOL_GPL(fs_bio_integrity_alloc);
 
 void fs_bio_integrity_free(struct bio *bio)
 {
@@ -43,6 +44,7 @@ void fs_bio_integrity_free(struct bio *bio)
 	bio->bi_integrity = NULL;
 	bio->bi_opf &= ~REQ_INTEGRITY;
 }
+EXPORT_SYMBOL_GPL(fs_bio_integrity_free);
 
 void fs_bio_integrity_generate(struct bio *bio)
 {
-- 
2.53.0

[PATCH 05/17] iomap: respect maximum I/O size in iomap_dio_bio_iter_one

[Christoph Hellwig]

Respect the maximum I/O size set for PI-enabled I/O in
iomap_dio_bio_iter_one, otherwise bio_integrity_alloc_buf could
under-allocate the integrity buffer when the initial kmalloc fails.

Currently this should not be triggered as file systems that limit the
size for PI always use bounce buffering, but this will change soon.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 fs/iomap/direct-io.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index bc726d47b7dc..91f5b67718a5 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -336,6 +336,7 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
 		struct iomap_dio *dio, loff_t pos, unsigned int alignment,
 		blk_opf_t op)
 {
+	unsigned int maxsize = iomap_max_bio_size(&iter->iomap);
 	unsigned int nr_vecs;
 	struct bio *bio;
 	ssize_t ret;
@@ -353,14 +354,12 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
 	bio->bi_private = dio;
 	bio->bi_end_io = iomap_dio_bio_end_io;
 
-
 	if (dio->flags & IOMAP_DIO_BOUNCE)
-		ret = bio_iov_iter_bounce(bio, dio->submit.iter,
-				iomap_max_bio_size(&iter->iomap), alignment);
+		ret = bio_iov_iter_bounce(bio, dio->submit.iter, maxsize,
+				alignment);
 	else
-		ret = bio_iov_iter_get_pages(bio, dio->submit.iter,
-				BIO_MAX_SIZE, bdev_dma_alignment(bio->bi_bdev),
-				alignment - 1);
+		ret = bio_iov_iter_get_pages(bio, dio->submit.iter, maxsize,
+				bdev_dma_alignment(bio->bi_bdev), alignment - 1);
 	if (unlikely(ret))
 		goto out_put_bio;
 	ret = bio->bi_iter.bi_size;
-- 
2.53.0

[PATCH 06/17] iomap: add a iomap_ioend_flags helper

[Christoph Hellwig]

Add a helper to initialize the ioend flags to the values that can be
directly derived from the iomap.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/iomap/ioend.c      | 10 +++-------
 include/linux/iomap.h | 12 ++++++++++++
 2 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c
index 7bbbb417f915..2d77c11e3c18 100644
--- a/fs/iomap/ioend.c
+++ b/fs/iomap/ioend.c
@@ -215,7 +215,7 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 {
 	struct iomap_ioend *ioend = wpc->wb_ctx;
 	size_t poff = offset_in_folio(folio, pos);
-	unsigned int ioend_flags = 0;
+	unsigned int ioend_flags = iomap_ioend_flags(&wpc->iomap);
 	unsigned int map_len = min_t(u64, dirty_len,
 		wpc->iomap.offset + wpc->iomap.length - pos);
 	int error;
@@ -225,20 +225,16 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 	WARN_ON_ONCE(!folio->private && map_len < dirty_len);
 
 	switch (wpc->iomap.type) {
+	case IOMAP_HOLE:
+		return map_len;
 	case IOMAP_UNWRITTEN:
-		ioend_flags |= IOMAP_IOEND_UNWRITTEN;
-		break;
 	case IOMAP_MAPPED:
 		break;
-	case IOMAP_HOLE:
-		return map_len;
 	default:
 		WARN_ON_ONCE(1);
 		return -EIO;
 	}
 
-	if (wpc->iomap.flags & IOMAP_F_SHARED)
-		ioend_flags |= IOMAP_IOEND_SHARED;
 	if (pos == wpc->iomap.offset && (wpc->iomap.flags & IOMAP_F_BOUNDARY))
 		ioend_flags |= IOMAP_IOEND_BOUNDARY;
 
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bc7ae6327dbf..8e09b2d4eda0 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -491,6 +491,18 @@ sector_t iomap_bmap(struct address_space *mapping, sector_t bno,
 #define IOMAP_IOEND_NOMERGE_FLAGS \
 	(IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT)
 
+/* ioend flags directly implied by iomap flags */
+static inline u16 iomap_ioend_flags(const struct iomap *iomap)
+{
+	unsigned int flags = 0;
+
+	if (iomap->type == IOMAP_UNWRITTEN)
+		flags |= IOMAP_IOEND_UNWRITTEN;
+	if (iomap->flags & IOMAP_F_SHARED)
+		flags |= IOMAP_IOEND_SHARED;
+	return flags;
+}
+
 /*
  * Structure for writeback I/O completions.
  *
-- 
2.53.0

[PATCH 07/17] iomap: add a IOMAP_IOEND_INTEGRITY flag

[Christoph Hellwig]

This will make it easier to share code for the various submit callback
provided by the file system to iomap.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/iomap/ioend.c      |  2 +-
 fs/xfs/xfs_aops.c     |  5 +++--
 fs/xfs/xfs_file.c     |  5 +++--
 include/linux/iomap.h | 12 +++++++++++-
 4 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c
index 2d77c11e3c18..573fa89c1632 100644
--- a/fs/iomap/ioend.c
+++ b/fs/iomap/ioend.c
@@ -149,7 +149,7 @@ int iomap_ioend_writeback_submit(struct iomap_writepage_ctx *wpc, int error)
 		return error;
 	}
 
-	if (wpc->iomap.flags & IOMAP_F_INTEGRITY)
+	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
 		fs_bio_integrity_generate(&ioend->io_bio);
 	submit_bio(&ioend->io_bio);
 	return 0;
diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 8b6119776fb3..4e862406eee9 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -498,7 +498,7 @@ xfs_zoned_writeback_submit(
 		bio_endio(&ioend->io_bio);
 		return error;
 	}
-	if (wpc->iomap.flags & IOMAP_F_INTEGRITY)
+	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
 		fs_bio_integrity_generate(&ioend->io_bio);
 	xfs_zone_alloc_and_submit(ioend, &XFS_ZWPC(wpc)->open_zone);
 	return 0;
@@ -588,7 +588,8 @@ xfs_bio_submit_read(
 	struct bio			*bio = ctx->read_ctx;
 
 	/* defer read completions to the ioend workqueue */
-	iomap_init_ioend(iter->inode, bio, ctx->read_ctx_file_offset, 0);
+	iomap_init_ioend(iter->inode, bio, ctx->read_ctx_file_offset,
+		iomap_ioend_flags(&iter->iomap));
 	iomap_bio_submit_read_endio(iter, ctx, xfs_end_bio);
 }
 
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 426a67b813a7..4d7a284fcff9 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -241,7 +241,8 @@ xfs_dio_read_bounce_submit_io(
 	struct bio		*bio,
 	loff_t			file_offset)
 {
-	iomap_init_ioend(iter->inode, bio, file_offset, IOMAP_IOEND_DIRECT);
+	iomap_init_ioend(iter->inode, bio, file_offset,
+		iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
 	bio->bi_end_io = xfs_end_bio;
 	submit_bio(bio);
 }
@@ -732,7 +733,7 @@ xfs_dio_zoned_submit_io(
 
 	bio->bi_end_io = xfs_end_bio;
 	ioend = iomap_init_ioend(iter->inode, bio, file_offset,
-			IOMAP_IOEND_DIRECT);
+		iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
 	xfs_zone_alloc_and_submit(ioend, &ac->open_zone);
 }
 
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 8e09b2d4eda0..1cc9a35fd5cc 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -483,13 +483,20 @@ sector_t iomap_bmap(struct address_space *mapping, sector_t bno,
 #define IOMAP_IOEND_BOUNDARY		(1U << 2)
 /* is direct I/O */
 #define IOMAP_IOEND_DIRECT		(1U << 3)
+/* generate integrity (PI) information */
+#ifdef CONFIG_BLK_DEV_INTEGRITY
+#define IOMAP_IOEND_INTEGRITY		(1U << 4)
+#else
+#define IOMAP_IOEND_INTEGRITY		0
+#endif /* CONFIG_BLK_DEV_INTEGRITY */
 
 /*
  * Flags that if set on either ioend prevent the merge of two ioends.
  * (IOMAP_IOEND_BOUNDARY also prevents merges, but only one-way)
  */
 #define IOMAP_IOEND_NOMERGE_FLAGS \
-	(IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT)
+	(IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT | \
+	 IOMAP_IOEND_INTEGRITY)
 
 /* ioend flags directly implied by iomap flags */
 static inline u16 iomap_ioend_flags(const struct iomap *iomap)
@@ -500,6 +507,9 @@ static inline u16 iomap_ioend_flags(const struct iomap *iomap)
 		flags |= IOMAP_IOEND_UNWRITTEN;
 	if (iomap->flags & IOMAP_F_SHARED)
 		flags |= IOMAP_IOEND_SHARED;
+	if (iomap->flags & IOMAP_F_INTEGRITY)
+		flags |= IOMAP_IOEND_INTEGRITY;
+
 	return flags;
 }
 
-- 
2.53.0

[PATCH 08/17] iomap,xfs: move T10 PI handling for direct I/O into ->submit_io

[Christoph Hellwig]

Currently the buffered I/O code defers PI handling into the submit
handlers, while direct I/O does it in common code.  Move the direct I/O
side into the file system callbacks or their generic implementations to
be consistent, and to allow file systems to shared helpers for submission
of buffered and direct bios.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/iomap/direct-io.c | 22 ++++++++++++----------
 fs/xfs/xfs_file.c    | 13 ++++++++++---
 2 files changed, 22 insertions(+), 13 deletions(-)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 91f5b67718a5..4154717a09de 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -76,10 +76,19 @@ static void iomap_dio_submit_bio(const struct iomap_iter *iter,
 
 	if (dio->dops && dio->dops->submit_io) {
 		dio->dops->submit_io(iter, bio, pos);
-	} else {
-		WARN_ON_ONCE(iter->iomap.flags & IOMAP_F_ANON_WRITE);
-		blk_crypto_submit_bio(bio);
+		return;
+	}
+
+	WARN_ON_ONCE(iter->iomap.flags & IOMAP_F_ANON_WRITE);
+
+	if (iter->iomap.flags & IOMAP_F_INTEGRITY) {
+		if (dio->flags & IOMAP_DIO_WRITE)
+			fs_bio_integrity_generate(bio);
+		else
+			fs_bio_integrity_alloc(bio);
 	}
+
+	blk_crypto_submit_bio(bio);
 }
 
 static inline enum fserror_type iomap_dio_err_type(const struct iomap_dio *dio)
@@ -373,13 +382,6 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
 		goto out_bio_release_pages;
 	}
 
-	if (iter->iomap.flags & IOMAP_F_INTEGRITY) {
-		if (dio->flags & IOMAP_DIO_WRITE)
-			fs_bio_integrity_generate(bio);
-		else
-			fs_bio_integrity_alloc(bio);
-	}
-
 	if (dio->flags & IOMAP_DIO_WRITE)
 		task_io_account_write(ret);
 	else if ((dio->flags & IOMAP_DIO_USER_BACKED) &&
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 4d7a284fcff9..acbfd55f56a3 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -37,6 +37,7 @@
 #include <linux/fadvise.h>
 #include <linux/mount.h>
 #include <linux/filelock.h>
+#include <linux/bio-integrity.h>
 
 static const struct vm_operations_struct xfs_file_vm_ops;
 
@@ -241,8 +242,12 @@ xfs_dio_read_bounce_submit_io(
 	struct bio		*bio,
 	loff_t			file_offset)
 {
-	iomap_init_ioend(iter->inode, bio, file_offset,
-		iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
+	struct iomap_ioend	*ioend;
+
+	ioend = iomap_init_ioend(iter->inode, bio, file_offset,
+			iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
+	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
+		fs_bio_integrity_alloc(bio);
 	bio->bi_end_io = xfs_end_bio;
 	submit_bio(bio);
 }
@@ -733,7 +738,9 @@ xfs_dio_zoned_submit_io(
 
 	bio->bi_end_io = xfs_end_bio;
 	ioend = iomap_init_ioend(iter->inode, bio, file_offset,
-		iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
+			iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
+	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
+		fs_bio_integrity_generate(bio);
 	xfs_zone_alloc_and_submit(ioend, &ac->open_zone);
 }
 
-- 
2.53.0

[PATCH 09/17] xfs: move PI generation into xfs_submit_zoned_bio

[Christoph Hellwig]

Keep the code in one place, and do so only after splitting to
keep the allocation size down in case of a split.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 fs/xfs/xfs_aops.c       | 4 +---
 fs/xfs/xfs_file.c       | 2 --
 fs/xfs/xfs_zone_alloc.c | 3 +++
 3 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 4e862406eee9..d2cdb7cffa6b 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -23,7 +23,6 @@
 #include "xfs_ioend.h"
 #include "xfs_zone_alloc.h"
 #include "xfs_rtgroup.h"
-#include <linux/bio-integrity.h>
 
 struct xfs_writepage_ctx {
 	struct iomap_writepage_ctx ctx;
@@ -498,8 +497,7 @@ xfs_zoned_writeback_submit(
 		bio_endio(&ioend->io_bio);
 		return error;
 	}
-	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
-		fs_bio_integrity_generate(&ioend->io_bio);
+
 	xfs_zone_alloc_and_submit(ioend, &XFS_ZWPC(wpc)->open_zone);
 	return 0;
 }
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index acbfd55f56a3..0e029db2e85b 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -739,8 +739,6 @@ xfs_dio_zoned_submit_io(
 	bio->bi_end_io = xfs_end_bio;
 	ioend = iomap_init_ioend(iter->inode, bio, file_offset,
 			iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
-	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
-		fs_bio_integrity_generate(bio);
 	xfs_zone_alloc_and_submit(ioend, &ac->open_zone);
 }
 
diff --git a/fs/xfs/xfs_zone_alloc.c b/fs/xfs/xfs_zone_alloc.c
index bdbb60cc5d5b..e427be806f97 100644
--- a/fs/xfs/xfs_zone_alloc.c
+++ b/fs/xfs/xfs_zone_alloc.c
@@ -26,6 +26,7 @@
 #include "xfs_zones.h"
 #include "xfs_trace.h"
 #include "xfs_mru_cache.h"
+#include <linux/bio-integrity.h>
 
 static void
 xfs_open_zone_free_rcu(
@@ -890,6 +891,8 @@ xfs_submit_zoned_bio(
 		xfs_mark_rtg_boundary(ioend);
 	}
 
+	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
+		fs_bio_integrity_generate(&ioend->io_bio);
 	submit_bio(&ioend->io_bio);
 }
 
-- 
2.53.0

[PATCH 10/17] block,iomap: fix protection information verification with initial bvec offset

[Christoph Hellwig]

When reconstructing a bvec_iter from an ioend for protection information
verification, iomap currently ignores the offset into the initial
bio_vec.

This can't happen for buffered I/O an direct I/O to user addresses, but
is exercised by split on O_DIRECT file descriptors or when using the loop
driver.

Fortunately the only file system PI user (XFS) currently always bounce
buffers, so this can't actually be triggered yet.  But we'll want to make
the bounce buffering conditional soon, for which this needs to be fixed.

Store the initial offset in struct iomap_ioend, and pass a
pre-constructed bvec_iter to fs_bio_integrity_verify.  For the
synchronous read case the fix is even simpler as this path can
simply stash away the original bvec_iter.

Fixes: 0bde8a12b554 ("block: add fs_bio_integrity helpers")
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 block/bio-integrity-fs.c      | 13 +++++--------
 fs/iomap/bio.c                |  4 +++-
 fs/iomap/ioend.c              | 14 ++++++++++----
 include/linux/bio-integrity.h |  3 +--
 include/linux/iomap.h         |  8 ++++++++
 5 files changed, 27 insertions(+), 15 deletions(-)

diff --git a/block/bio-integrity-fs.c b/block/bio-integrity-fs.c
index 4f67b34bf289..c8e91ada8ca6 100644
--- a/block/bio-integrity-fs.c
+++ b/block/bio-integrity-fs.c
@@ -54,14 +54,10 @@ void fs_bio_integrity_generate(struct bio *bio)
 }
 EXPORT_SYMBOL_GPL(fs_bio_integrity_generate);
 
-int fs_bio_integrity_verify(struct bio *bio, sector_t sector, unsigned int size)
+int fs_bio_integrity_verify(struct bio *bio, struct bvec_iter *data_iter)
 {
 	struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
 	struct bio_integrity_payload *bip = bio_integrity(bio);
-	struct bvec_iter data_iter = {
-		.bi_sector	= sector,
-		.bi_size	= size,
-	};
 
 	if (!bip || !(bip->bip_flags & BIP_CHECK_FLAGS))
 		return 0;
@@ -73,9 +69,10 @@ int fs_bio_integrity_verify(struct bio *bio, sector_t sector, unsigned int size)
 	 * bio.  Requires the submitter to remember the sector and the size.
 	 */
 	memset(&bip->bip_iter, 0, sizeof(bip->bip_iter));
-	bip->bip_iter.bi_sector = sector;
-	bip->bip_iter.bi_size = bio_integrity_bytes(bi, size >> SECTOR_SHIFT);
-	return blk_status_to_errno(bio_integrity_verify(bio, &data_iter));
+	bip->bip_iter.bi_sector = data_iter->bi_sector;
+	bip->bip_iter.bi_size =
+		bio_integrity_bytes(bi, data_iter->bi_size >> SECTOR_SHIFT);
+	return blk_status_to_errno(bio_integrity_verify(bio, data_iter));
 }
 
 static int __init fs_bio_integrity_init(void)
diff --git a/fs/iomap/bio.c b/fs/iomap/bio.c
index 48100c614431..d46c2f8ea18c 100644
--- a/fs/iomap/bio.c
+++ b/fs/iomap/bio.c
@@ -169,6 +169,7 @@ int iomap_bio_read_folio_range_sync(const struct iomap_iter *iter,
 {
 	const struct iomap *srcmap = iomap_iter_srcmap(iter);
 	sector_t sector = iomap_sector(srcmap, pos);
+	struct bvec_iter saved_iter;
 	struct bio_vec bvec;
 	struct bio bio;
 	int error;
@@ -178,10 +179,11 @@ int iomap_bio_read_folio_range_sync(const struct iomap_iter *iter,
 	bio_add_folio_nofail(&bio, folio, len, offset_in_folio(folio, pos));
 	if (srcmap->flags & IOMAP_F_INTEGRITY)
 		fs_bio_integrity_alloc(&bio);
+	saved_iter = bio.bi_iter;
 	error = submit_bio_wait(&bio);
 	if (bio_integrity(&bio)) {
 		if (!error)
-			error = fs_bio_integrity_verify(&bio, sector, len);
+			error = fs_bio_integrity_verify(&bio, &saved_iter);
 		fs_bio_integrity_free(&bio);
 	}
 	bio_uninit(&bio);
diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c
index 573fa89c1632..332dbfb2230f 100644
--- a/fs/iomap/ioend.c
+++ b/fs/iomap/ioend.c
@@ -25,6 +25,7 @@ struct iomap_ioend *iomap_init_ioend(struct inode *inode,
 	ioend->io_parent = NULL;
 	INIT_LIST_HEAD(&ioend->io_list);
 	ioend->io_flags = ioend_flags;
+	ioend->io_bvec_offset = bio->bi_iter.bi_offset;
 	ioend->io_inode = inode;
 	ioend->io_offset = file_offset;
 	ioend->io_size = bio->bi_iter.bi_size;
@@ -308,6 +309,13 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 }
 EXPORT_SYMBOL_GPL(iomap_add_to_ioend);
 
+static int iomap_ioend_integrity_verify(struct iomap_ioend *ioend)
+{
+	struct bvec_iter data_iter = BVEC_ITER_IOEND(ioend);
+
+	return fs_bio_integrity_verify(&ioend->io_bio, &data_iter);
+}
+
 static u32 iomap_finish_ioend(struct iomap_ioend *ioend, int error)
 {
 	if (ioend->io_parent) {
@@ -325,10 +333,8 @@ static u32 iomap_finish_ioend(struct iomap_ioend *ioend, int error)
 
 	if (!ioend->io_error &&
 	    bio_integrity(&ioend->io_bio) &&
-	    bio_op(&ioend->io_bio) == REQ_OP_READ) {
-		ioend->io_error = fs_bio_integrity_verify(&ioend->io_bio,
-			ioend->io_sector, ioend->io_size);
-	}
+	    bio_op(&ioend->io_bio) == REQ_OP_READ)
+		ioend->io_error = iomap_ioend_integrity_verify(ioend);
 
 	if (ioend->io_flags & IOMAP_IOEND_DIRECT)
 		return iomap_finish_ioend_direct(ioend);
diff --git a/include/linux/bio-integrity.h b/include/linux/bio-integrity.h
index 0ea2a8bf7efb..a954c97be0b3 100644
--- a/include/linux/bio-integrity.h
+++ b/include/linux/bio-integrity.h
@@ -151,7 +151,6 @@ void bio_integrity_setup_default(struct bio *bio);
 unsigned int fs_bio_integrity_alloc(struct bio *bio);
 void fs_bio_integrity_free(struct bio *bio);
 void fs_bio_integrity_generate(struct bio *bio);
-int fs_bio_integrity_verify(struct bio *bio, sector_t sector,
-		unsigned int size);
+int fs_bio_integrity_verify(struct bio *bio, struct bvec_iter *data_iter);
 
 #endif /* _LINUX_BIO_INTEGRITY_H */
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 1cc9a35fd5cc..bffdc217ad85 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -522,6 +522,7 @@ static inline u16 iomap_ioend_flags(const struct iomap *iomap)
 struct iomap_ioend {
 	struct list_head	io_list;	/* next ioend in chain */
 	u16			io_flags;	/* IOMAP_IOEND_* */
+	u32			io_bvec_offset;	/* offset into first bvec */
 	struct inode		*io_inode;	/* file being written to */
 	size_t			io_size;	/* size of the extent */
 	atomic_t		io_remaining;	/* completetion defer count */
@@ -539,6 +540,13 @@ static inline struct iomap_ioend *iomap_ioend_from_bio(struct bio *bio)
 	return container_of(bio, struct iomap_ioend, io_bio);
 }
 
+#define BVEC_ITER_IOEND(_ioend)				\
+{							\
+	.bi_sector	= (_ioend)->io_sector,		\
+	.bi_size	= (_ioend)->io_size,		\
+	.bi_offset	= (_ioend)->io_bvec_offset,	\
+}
+
 struct iomap_writeback_ops {
 	/*
 	 * Performs writeback on the passed in range
-- 
2.53.0

[PATCH 11/17] iomap: better read bounce buffering support

[Christoph Hellwig]

Add helpers to bounce buffer an upper bio into one or more lower bios
using bounce buffers, and to copy the data back on completion.

Compared to the existing IOMAP_DIO_BOUNCE support for read bios, this
has two advantages:  by removing the special bounce bio_vec it allows
to the full and "round" size of a single bio, i.e., 1MiB when using
4k pages.  This is important for good performance on HDD.  Additionally
it allows to bounce buffer a bio from completion conext, and thus
implement a "lazy" bounce buffering scheme, where the data is only
read into a bounce buffer after an initial checksum validation failure,
thus avoiding the bounce buffering I/O for most I/O.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/iomap/ioend.c      | 90 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/iomap.h |  5 +++
 2 files changed, 95 insertions(+)

diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c
index 332dbfb2230f..ba97eec4ee2c 100644
--- a/fs/iomap/ioend.c
+++ b/fs/iomap/ioend.c
@@ -514,6 +514,96 @@ struct iomap_ioend *iomap_split_ioend(struct iomap_ioend *ioend,
 }
 EXPORT_SYMBOL_GPL(iomap_split_ioend);
 
+void iomap_bounce_read(struct iomap_ioend *orig_ioend, unsigned int minsize,
+		void (*submit_ioend)(struct iomap_ioend *ioend))
+{
+	struct inode *inode = orig_ioend->io_inode;
+	struct bio *orig_bio = &orig_ioend->io_bio;
+	loff_t file_offset = orig_ioend->io_offset;
+	sector_t sector = orig_ioend->io_sector;
+	size_t total_len = round_up(orig_ioend->io_size, minsize);
+
+	WARN_ON_ONCE(!(orig_ioend->io_flags & IOMAP_IOEND_DIRECT));
+
+	/* We can't poll a bio that is not passed on to hardware */
+	orig_bio->bi_opf &= ~REQ_POLLED;
+
+	do {
+		struct iomap_ioend *ioend;
+		struct bio *bio;
+		int error;
+
+		bio = bio_alloc_bioset(orig_bio->bi_bdev,
+				min(total_len / minsize, BIO_MAX_VECS),
+				orig_bio->bi_opf, GFP_KERNEL,
+				&iomap_ioend_split_bioset);
+		error = bio_alloc_bounce_folios(bio, total_len, minsize);
+		if (error) {
+			bio_put(bio);
+			orig_bio->bi_status = errno_to_blk_status(error);
+			break;
+		}
+		bio->bi_ioprio = orig_bio->bi_ioprio;
+		bio->bi_write_hint = orig_bio->bi_write_hint;
+		bio->bi_write_stream = orig_bio->bi_write_stream;
+		bio->bi_iter.bi_sector = sector;
+
+		ioend = iomap_init_ioend(inode, bio, file_offset,
+				orig_ioend->io_flags);
+
+		total_len -= bio->bi_iter.bi_size;
+		file_offset += bio->bi_iter.bi_size;
+		sector += (bio->bi_iter.bi_size >> SECTOR_SHIFT);
+
+		bio->bi_private = orig_bio;
+		bio_inc_remaining(orig_bio);
+		submit_ioend(ioend);
+	} while (total_len > 0);
+
+	bio_endio(&orig_ioend->io_bio);
+}
+EXPORT_SYMBOL_GPL(iomap_bounce_read);
+
+static void iomap_ioend_unbounce(struct iomap_ioend *orig_ioend,
+		struct iomap_ioend *ioend)
+{
+	struct bio *orig_bio = &orig_ioend->io_bio;
+	struct iov_iter to;
+	struct bio_vec *bv;
+	int i;
+
+	iov_iter_bvec(&to, ITER_DEST, orig_bio->bi_io_vec, orig_bio->bi_vcnt,
+			orig_ioend->io_size);
+	to.iov_offset = orig_ioend->io_bvec_offset;
+
+	if (ioend->io_offset != orig_ioend->io_offset) {
+		WARN_ON_ONCE(ioend->io_offset < orig_ioend->io_offset);
+		iov_iter_advance(&to, ioend->io_offset - orig_ioend->io_offset);
+	}
+
+	/* copying to pinned pages should always work */
+	bio_for_each_bvec_all(bv, &ioend->io_bio, i)
+		WARN_ON_ONCE(copy_to_iter(bvec_virt(bv), bv->bv_len, &to) !=
+				bv->bv_len);
+}
+
+void iomap_bounce_read_end_io(struct iomap_ioend *ioend, struct bio *orig_bio,
+		int error)
+{
+	if (error)
+		orig_bio->bi_status = errno_to_blk_status(error);
+	else
+		iomap_ioend_unbounce(iomap_ioend_from_bio(orig_bio), ioend);
+
+	bio_free_folios(&ioend->io_bio);
+	if (bio_integrity(&ioend->io_bio))
+		fs_bio_integrity_free(&ioend->io_bio);
+	bio_put(&ioend->io_bio);
+
+	bio_endio(orig_bio);
+}
+EXPORT_SYMBOL_GPL(iomap_bounce_read_end_io);
+
 static int __init iomap_ioend_init(void)
 {
 	const unsigned int nr_mempool_entries = 4 * (PAGE_SIZE / SECTOR_SIZE);
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bffdc217ad85..fbe051f0b032 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -607,6 +607,11 @@ void iomap_finish_folio_write(struct inode *inode, struct folio *folio,
 int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio);
 int iomap_writepages(struct iomap_writepage_ctx *wpc);
 
+void iomap_bounce_read(struct iomap_ioend *orig_ioend, unsigned int minsize,
+		void (*submit_ioend)(struct iomap_ioend *ioend));
+void iomap_bounce_read_end_io(struct iomap_ioend *ioend, struct bio *orig_bio,
+		int error);
+
 struct iomap_read_folio_ctx {
 	const struct iomap_read_ops *ops;
 	struct folio		*cur_folio;
-- 
2.53.0

[PATCH 12/17] xfs: use BIO_COMPLETE_IN_TASK for bounce buffered read I/Os

[Christoph Hellwig]

Stop using the xfs per-inode work struct for completing read bios, as
unlike writes we don't want to serialize reads on a single inode as
there is no exclusive resource contention for them.

Factor the code for kicking off a read that needs and ioend and the
task context completion into a single helper so that it is split off
the xfs_end_bio machinery, which is not only used for writes.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/xfs/xfs_aops.c  | 10 ++++------
 fs/xfs/xfs_file.c  |  9 +--------
 fs/xfs/xfs_ioend.c | 32 +++++++++++++++++++++++++++-----
 fs/xfs/xfs_ioend.h |  2 ++
 4 files changed, 34 insertions(+), 19 deletions(-)

diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index d2cdb7cffa6b..c30e688cfc9f 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -583,12 +583,10 @@ xfs_bio_submit_read(
 	const struct iomap_iter		*iter,
 	struct iomap_read_folio_ctx	*ctx)
 {
-	struct bio			*bio = ctx->read_ctx;
-
-	/* defer read completions to the ioend workqueue */
-	iomap_init_ioend(iter->inode, bio, ctx->read_ctx_file_offset,
-		iomap_ioend_flags(&iter->iomap));
-	iomap_bio_submit_read_endio(iter, ctx, xfs_end_bio);
+	xfs_ioend_submit_read(iter->inode, ctx->read_ctx,
+			ctx->read_ctx_file_offset,
+			iomap_ioend_flags(&iter->iomap));
+	ctx->read_ctx = NULL;
 }
 
 static const struct iomap_read_ops xfs_iomap_read_ops = {
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 0e029db2e85b..766c4d2055c1 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -37,7 +37,6 @@
 #include <linux/fadvise.h>
 #include <linux/mount.h>
 #include <linux/filelock.h>
-#include <linux/bio-integrity.h>
 
 static const struct vm_operations_struct xfs_file_vm_ops;
 
@@ -242,14 +241,8 @@ xfs_dio_read_bounce_submit_io(
 	struct bio		*bio,
 	loff_t			file_offset)
 {
-	struct iomap_ioend	*ioend;
-
-	ioend = iomap_init_ioend(iter->inode, bio, file_offset,
+	xfs_ioend_submit_read(iter->inode, bio, file_offset,
 			iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
-	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
-		fs_bio_integrity_alloc(bio);
-	bio->bi_end_io = xfs_end_bio;
-	submit_bio(bio);
 }
 
 static const struct iomap_dio_ops xfs_dio_read_bounce_ops = {
diff --git a/fs/xfs/xfs_ioend.c b/fs/xfs/xfs_ioend.c
index 40695d18dac0..37a3ae8066e9 100644
--- a/fs/xfs/xfs_ioend.c
+++ b/fs/xfs/xfs_ioend.c
@@ -16,6 +16,32 @@
 #include "xfs_reflink.h"
 #include "xfs_zone_alloc.h"
 #include "xfs_ioend.h"
+#include <linux/bio-integrity.h>
+
+static void
+xfs_end_io_read(
+	struct bio		*bio)
+{
+	struct iomap_ioend	*ioend = iomap_ioend_from_bio(bio);
+	int			error = blk_status_to_errno(bio->bi_status);
+
+	iomap_finish_ioends(ioend, error);
+}
+
+void
+xfs_ioend_submit_read(
+	struct inode		*inode,
+	struct bio		*bio,
+	loff_t			file_offset,
+	u16			ioend_flags)
+{
+	iomap_init_ioend(inode, bio, file_offset, ioend_flags);
+	if (ioend_flags & IOMAP_IOEND_INTEGRITY)
+		fs_bio_integrity_alloc(bio);
+	bio->bi_end_io = xfs_end_io_read;
+	bio_set_flag(bio, BIO_COMPLETE_IN_TASK);
+	submit_bio(bio);
+}
 
 static void
 xfs_ioend_put_open_zones(
@@ -148,11 +174,7 @@ xfs_end_io(
 			io_list))) {
 		list_del_init(&ioend->io_list);
 		iomap_ioend_try_merge(ioend, &tmp);
-		if (bio_op(&ioend->io_bio) == REQ_OP_READ)
-			iomap_finish_ioends(ioend,
-				blk_status_to_errno(ioend->io_bio.bi_status));
-		else
-			xfs_end_ioend_write(ioend);
+		xfs_end_ioend_write(ioend);
 		cond_resched();
 	}
 }
diff --git a/fs/xfs/xfs_ioend.h b/fs/xfs/xfs_ioend.h
index 525865767fca..7c2a1ea3e6ed 100644
--- a/fs/xfs/xfs_ioend.h
+++ b/fs/xfs/xfs_ioend.h
@@ -12,5 +12,7 @@ static inline bool xfs_ioend_is_append(struct iomap_ioend *ioend)
 }
 
 void xfs_end_bio(struct bio *bio);
+void xfs_ioend_submit_read(struct inode *inode, struct bio *bio,
+		loff_t file_offset, u16 ioend_flags);
 
 #endif /* __XFS_IOEND_H */
-- 
2.53.0

[PATCH 13/17] iomap,xfs: move integrity verification to the file system

[Christoph Hellwig]

Integrity support in file systems already requires file system-specific
completion handling because it must be run in process context.

Move the actual verification to the file system so that it can better
handle errors in file system specific ways, and to support lazy bounce
buffering.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/iomap/ioend.c      | 10 ++++------
 fs/xfs/xfs_ioend.c    |  3 +++
 include/linux/iomap.h |  1 +
 3 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c
index ba97eec4ee2c..bbebecc31670 100644
--- a/fs/iomap/ioend.c
+++ b/fs/iomap/ioend.c
@@ -309,12 +309,15 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 }
 EXPORT_SYMBOL_GPL(iomap_add_to_ioend);
 
-static int iomap_ioend_integrity_verify(struct iomap_ioend *ioend)
+#ifdef CONFIG_BLK_DEV_INTEGRITY
+int iomap_ioend_integrity_verify(struct iomap_ioend *ioend)
 {
 	struct bvec_iter data_iter = BVEC_ITER_IOEND(ioend);
 
 	return fs_bio_integrity_verify(&ioend->io_bio, &data_iter);
 }
+EXPORT_SYMBOL_GPL(iomap_ioend_integrity_verify);
+#endif /* CONFIG_BLK_DEV_INTEGRITY */
 
 static u32 iomap_finish_ioend(struct iomap_ioend *ioend, int error)
 {
@@ -331,11 +334,6 @@ static u32 iomap_finish_ioend(struct iomap_ioend *ioend, int error)
 	if (!atomic_dec_and_test(&ioend->io_remaining))
 		return 0;
 
-	if (!ioend->io_error &&
-	    bio_integrity(&ioend->io_bio) &&
-	    bio_op(&ioend->io_bio) == REQ_OP_READ)
-		ioend->io_error = iomap_ioend_integrity_verify(ioend);
-
 	if (ioend->io_flags & IOMAP_IOEND_DIRECT)
 		return iomap_finish_ioend_direct(ioend);
 	if (bio_op(&ioend->io_bio) == REQ_OP_READ)
diff --git a/fs/xfs/xfs_ioend.c b/fs/xfs/xfs_ioend.c
index 37a3ae8066e9..a095cf217863 100644
--- a/fs/xfs/xfs_ioend.c
+++ b/fs/xfs/xfs_ioend.c
@@ -25,6 +25,9 @@ xfs_end_io_read(
 	struct iomap_ioend	*ioend = iomap_ioend_from_bio(bio);
 	int			error = blk_status_to_errno(bio->bi_status);
 
+	if (!error && (ioend->io_flags & IOMAP_IOEND_INTEGRITY))
+		error = iomap_ioend_integrity_verify(ioend);
+
 	iomap_finish_ioends(ioend, error);
 }
 
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index fbe051f0b032..59718f73c15a 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -595,6 +595,7 @@ void iomap_finish_ioends(struct iomap_ioend *ioend, int error);
 void iomap_ioend_try_merge(struct iomap_ioend *ioend,
 		struct list_head *more_ioends);
 void iomap_sort_ioends(struct list_head *ioend_list);
+int iomap_ioend_integrity_verify(struct iomap_ioend *ioend);
 ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 		loff_t pos, loff_t end_pos, unsigned int dirty_len);
 int iomap_ioend_writeback_submit(struct iomap_writepage_ctx *wpc, int error);
-- 
2.53.0

[PATCH 14/17] xfs: add support for lazy direct read bounce buffering

[Christoph Hellwig]

Currently direct I/O reads always bounce buffer the I/O to deal with
the case where userspace is modifying the buffer in-flight while
reading data into it.

This is a very expensive countermeasure for something no sane application
should do, so try to avoid it by reading without a bounce buffer first,
and retrying the read on a checksum failure.  This avoids the cost of
bounce buffering for sanely behave applications.  For the rare case of
an application regularly modifying in-flight buffers, allow forcing the
always bounce buffer behavior through sysfs.  And now that we have that
knob, allow disabling read-side bounce buffering entirely for those who
live fast and dangerous.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 fs/xfs/xfs_file.c  |   3 +-
 fs/xfs/xfs_ioend.c | 102 +++++++++++++++++++++++++++++++++++++++++++--
 fs/xfs/xfs_mount.h |   8 ++++
 fs/xfs/xfs_super.c |   1 +
 fs/xfs/xfs_sysfs.c |  76 +++++++++++++++++++++++++++++++++
 fs/xfs/xfs_trace.h |   1 +
 6 files changed, 186 insertions(+), 5 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 766c4d2055c1..daa6a854dd5f 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -270,8 +270,7 @@ xfs_file_dio_read(
 		return ret;
 	if (mapping_stable_writes(iocb->ki_filp->f_mapping)) {
 		ret = iomap_dio_rw(iocb, to, &xfs_read_iomap_ops,
-				&xfs_dio_read_bounce_ops, IOMAP_DIO_BOUNCE,
-				NULL, 0);
+				&xfs_dio_read_bounce_ops, 0, NULL, 0);
 	} else {
 		ret = iomap_dio_read_simple(iocb, to, xfs_read_iomap_begin);
 		if (ret == -ENOTBLK)
diff --git a/fs/xfs/xfs_ioend.c b/fs/xfs/xfs_ioend.c
index a095cf217863..c21cbd7b0a6d 100644
--- a/fs/xfs/xfs_ioend.c
+++ b/fs/xfs/xfs_ioend.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Copyright (c) 2016-2025 Christoph Hellwig.
+ * Copyright (c) 2016-2026 Christoph Hellwig.
  * All Rights Reserved.
  */
 #include "xfs_platform.h"
@@ -18,15 +18,100 @@
 #include "xfs_ioend.h"
 #include <linux/bio-integrity.h>
 
+static void
+xfs_end_bio_bounced(
+	struct bio		*bio)
+{
+	iomap_finish_ioends(iomap_ioend_from_bio(bio),
+			blk_status_to_errno(bio->bi_status));
+}
+
+static void
+xfs_dio_bounce_end_io(
+	struct bio		*bio)
+{
+	struct iomap_ioend	*ioend = iomap_ioend_from_bio(bio);
+	int			error = blk_status_to_errno(bio->bi_status);
+	struct bio		*orig_bio = bio->bi_private;
+
+	if ((ioend->io_flags & IOMAP_IOEND_INTEGRITY) && !bio->bi_status)
+		error = iomap_ioend_integrity_verify(ioend);
+	iomap_bounce_read_end_io(ioend, orig_bio, error);
+}
+
+static void
+xfs_bounce_submit_ioend(
+	struct iomap_ioend	*ioend)
+{
+	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
+		fs_bio_integrity_alloc(&ioend->io_bio);
+	ioend->io_bio.bi_end_io = xfs_dio_bounce_end_io;
+	bio_set_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
+	submit_bio(&ioend->io_bio);
+}
+
+static void
+xfs_read_bounce_and_resubmit(
+	struct iomap_ioend	*ioend)
+{
+	struct bio		*bio = &ioend->io_bio;
+	unsigned short		vcnt = bio->bi_vcnt;
+	void			*private = bio->bi_private;
+
+	trace_xfs_bounce_reread(XFS_I(ioend->io_inode), ioend->io_offset,
+			ioend->io_size);
+
+	/*
+	 * Free the bio integrity data for the original bio, as we'll allocate
+	 * ons for each sub-I/O, which could deadlock if we keep the original
+	 * one around.
+	 */
+	if (bio_integrity(bio))
+		fs_bio_integrity_free(bio);
+
+	/*
+	 * Reset the bio to submit the bio to the block layer again.  Switch to
+	 * an end_io handler that simply complets the ioend, as all verification
+	 * is done by the end_I/O handlers for the clone bio(s).
+	 */
+	bio_reset(bio, xfs_inode_buftarg(XFS_I(ioend->io_inode))->bt_bdev,
+			bio->bi_opf);
+	bio->bi_vcnt = vcnt;
+	bio->bi_private = private;
+	bio->bi_end_io = xfs_end_bio_bounced;
+	bio->bi_iter = (struct bvec_iter) {
+		.bi_sector	= ioend->io_sector,
+		.bi_size	= ioend->io_size,
+		.bi_offset	= ioend->io_bvec_offset,
+	};
+	iomap_bounce_read(ioend, bdev_logical_block_size(bio->bi_bdev),
+			xfs_bounce_submit_ioend);
+}
+
 static void
 xfs_end_io_read(
 	struct bio		*bio)
 {
 	struct iomap_ioend	*ioend = iomap_ioend_from_bio(bio);
+	struct xfs_inode	*ip = XFS_I(ioend->io_inode);
+	struct xfs_mount	*mp = ip->i_mount;
 	int			error = blk_status_to_errno(bio->bi_status);
 
-	if (!error && (ioend->io_flags & IOMAP_IOEND_INTEGRITY))
+	if (!error && (ioend->io_flags & IOMAP_IOEND_INTEGRITY)) {
 		error = iomap_ioend_integrity_verify(ioend);
+		if ((ioend->io_flags & IOMAP_IOEND_DIRECT) &&
+		    READ_ONCE(mp->m_read_bounce) == XFS_READ_BOUNCE_LAZY) {
+			/*
+			 * We only really need to retry for guard tag errors,
+			 * but right now we can't distinguish them from other
+			 * (i.e, reftag) errors.
+			 */
+			if (error) {
+				xfs_read_bounce_and_resubmit(ioend);
+				return;
+			}
+		}
+	}
 
 	iomap_finish_ioends(ioend, error);
 }
@@ -38,7 +123,18 @@ xfs_ioend_submit_read(
 	loff_t			file_offset,
 	u16			ioend_flags)
 {
-	iomap_init_ioend(inode, bio, file_offset, ioend_flags);
+	struct xfs_inode	*ip = XFS_I(inode);
+	struct xfs_mount	*mp = ip->i_mount;
+	struct iomap_ioend	*ioend;
+
+	ioend = iomap_init_ioend(inode, bio, file_offset, ioend_flags);
+	if ((ioend_flags & IOMAP_IOEND_DIRECT) &&
+	    READ_ONCE(mp->m_read_bounce) == XFS_READ_BOUNCE_ALWAYS) {
+		iomap_bounce_read(ioend, bdev_logical_block_size(bio->bi_bdev),
+				xfs_bounce_submit_ioend);
+		return;
+	}
+
 	if (ioend_flags & IOMAP_IOEND_INTEGRITY)
 		fs_bio_integrity_alloc(bio);
 	bio->bi_end_io = xfs_end_io_read;
diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h
index 216a38a354e7..894ff2f4ecbd 100644
--- a/fs/xfs/xfs_mount.h
+++ b/fs/xfs/xfs_mount.h
@@ -142,6 +142,12 @@ struct xfs_freecounter {
 	uint64_t		res_saved;
 };
 
+enum xfs_read_bounce {
+	XFS_READ_BOUNCE_NEVER,
+	XFS_READ_BOUNCE_ALWAYS,
+	XFS_READ_BOUNCE_LAZY,
+};
+
 /*
  * The struct xfsmount layout is optimised to separate read-mostly variables
  * from variables that are frequently modified. We put the read-mostly variables
@@ -177,6 +183,7 @@ typedef struct xfs_mount {
 	struct workqueue_struct	*m_sync_workqueue;
 	struct workqueue_struct *m_blockgc_wq;
 	struct workqueue_struct *m_inodegc_wq;
+	enum xfs_read_bounce	m_read_bounce;
 
 	int			m_bsize;	/* fs logical block size */
 	uint8_t			m_blkbit_log;	/* blocklog + NBBY */
@@ -291,6 +298,7 @@ typedef struct xfs_mount {
 	struct xfs_zone_info	*m_zone_info;	/* zone allocator information */
 	struct dentry		*m_debugfs;	/* debugfs parent */
 	struct xfs_kobj		m_kobj;
+	struct xfs_kobj		m_csum_kobj;
 	struct xfs_kobj		m_error_kobj;
 	struct xfs_kobj		m_error_meta_kobj;
 	struct xfs_error_cfg	m_error_cfg[XFS_ERR_CLASS_MAX][XFS_ERR_ERRNO_MAX];
diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
index b24db75eaedc..fce1d2905c94 100644
--- a/fs/xfs/xfs_super.c
+++ b/fs/xfs/xfs_super.c
@@ -2317,6 +2317,7 @@ xfs_init_fs_context(
 	mp->m_logbufs = -1;
 	mp->m_logbsize = -1;
 	mp->m_allocsize_log = 16; /* 64k */
+	mp->m_read_bounce = XFS_READ_BOUNCE_LAZY;
 
 	xfs_hooks_init(&mp->m_dir_update_hooks);
 
diff --git a/fs/xfs/xfs_sysfs.c b/fs/xfs/xfs_sysfs.c
index b62712187324..2e969e8f279f 100644
--- a/fs/xfs/xfs_sysfs.c
+++ b/fs/xfs/xfs_sysfs.c
@@ -392,6 +392,57 @@ const struct kobj_type xfs_stats_ktype = {
 	.default_groups = xfs_stats_groups,
 };
 
+static inline struct xfs_mount *csum_to_mp(struct kobject *kobj)
+{
+	return container_of(to_kobj(kobj), struct xfs_mount, m_csum_kobj);
+}
+
+static const char * const bounce_modes[] = {
+	[XFS_READ_BOUNCE_NEVER]		= "never",
+	[XFS_READ_BOUNCE_ALWAYS]	= "always",
+	[XFS_READ_BOUNCE_LAZY]		= "lazy",
+};
+
+static ssize_t
+read_bounce_show(
+	struct kobject		*kobj,
+	char			*buf)
+{
+	struct xfs_mount	*mp = csum_to_mp(kobj);
+
+	return sysfs_emit(buf, "%s\n",
+			bounce_modes[READ_ONCE(mp->m_read_bounce)]);
+}
+
+static ssize_t
+read_bounce_store(
+	struct kobject		*kobj,
+	const char		*buf,
+	size_t			count)
+{
+	struct xfs_mount	*mp = csum_to_mp(kobj);
+	int			ret;
+
+	ret = sysfs_match_string(bounce_modes, buf);
+	if (ret < 0)
+		return ret;
+	WRITE_ONCE(mp->m_read_bounce, ret);
+	return count;
+}
+XFS_SYSFS_ATTR_RW(read_bounce);
+
+static struct attribute *xfs_csum_attrs[] = {
+	ATTR_LIST(read_bounce),
+	NULL,
+};
+ATTRIBUTE_GROUPS(xfs_csum);
+
+static const struct kobj_type xfs_csum_ktype = {
+	.release = xfs_sysfs_release,
+	.sysfs_ops = &xfs_sysfs_ops,
+	.default_groups = xfs_csum_groups,
+};
+
 /* xlog */
 
 static inline struct xlog *
@@ -797,6 +848,18 @@ xfs_zoned_sysfs_del(struct xfs_mount *mp)
 		xfs_sysfs_del(&mp->m_zoned_kobj);
 }
 
+static bool
+xfs_has_read_bounce(
+	struct xfs_mount	*mp)
+{
+	if (bdev_has_integrity_csum(mp->m_ddev_targp->bt_bdev))
+		return true;
+	if (mp->m_rtdev_targp &&
+	    bdev_has_integrity_csum(mp->m_rtdev_targp->bt_bdev))
+		return true;
+	return false;
+}
+
 int
 xfs_mount_sysfs_init(
 	struct xfs_mount	*mp)
@@ -837,8 +900,18 @@ xfs_mount_sysfs_init(
 	if (error)
 		goto out_remove_error_dir;
 
+	if (xfs_has_read_bounce(mp)) {
+		/* .../xfs/<dev>/csum/ */
+		error = xfs_sysfs_init(&mp->m_csum_kobj, &xfs_csum_ktype,
+				       &mp->m_kobj, "csum");
+		if (error)
+			goto out_remove_error_metadata_dir;
+	}
+
 	return 0;
 
+out_remove_error_metadata_dir:
+	xfs_sysfs_del(&mp->m_error_meta_kobj);
 out_remove_error_dir:
 	xfs_sysfs_del(&mp->m_error_kobj);
 out_remove_stats_dir:
@@ -855,6 +928,9 @@ xfs_mount_sysfs_del(
 	struct xfs_error_cfg	*cfg;
 	int			i, j;
 
+	if (xfs_has_read_bounce(mp))
+		xfs_sysfs_del(&mp->m_csum_kobj);
+
 	for (i = 0; i < XFS_ERR_CLASS_MAX; i++) {
 		for (j = 0; j < XFS_ERR_ERRNO_MAX; j++) {
 			cfg = &mp->m_error_cfg[i][j];
diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h
index f333c938fbd9..2af9a1429ae9 100644
--- a/fs/xfs/xfs_trace.h
+++ b/fs/xfs/xfs_trace.h
@@ -1896,6 +1896,7 @@ DEFINE_SIMPLE_IO_EVENT(xfs_zero_eof);
 DEFINE_SIMPLE_IO_EVENT(xfs_end_io_direct_write);
 DEFINE_SIMPLE_IO_EVENT(xfs_file_splice_read);
 DEFINE_SIMPLE_IO_EVENT(xfs_zoned_map_blocks);
+DEFINE_SIMPLE_IO_EVENT(xfs_bounce_reread);
 
 DECLARE_EVENT_CLASS(xfs_itrunc_class,
 	TP_PROTO(struct xfs_inode *ip, xfs_fsize_t new_size),
-- 
2.53.0

[PATCH 15/17] xfs: add error injection for lazy bounce buffering

[Christoph Hellwig]

Add an error injection know to exercise the lazy bounce buffering
code path, i.e. to inject direct I/O re-read using the bounce buffer.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/xfs/libxfs/xfs_errortag.h | 6 ++++--
 fs/xfs/xfs_ioend.c           | 5 ++++-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_errortag.h b/fs/xfs/libxfs/xfs_errortag.h
index 6de207fed2d8..2dc441da0333 100644
--- a/fs/xfs/libxfs/xfs_errortag.h
+++ b/fs/xfs/libxfs/xfs_errortag.h
@@ -75,7 +75,8 @@
 #define XFS_ERRTAG_METAFILE_RESV_CRITICAL		45
 #define XFS_ERRTAG_FORCE_ZERO_RANGE			46
 #define XFS_ERRTAG_ZONE_RESET				47
-#define XFS_ERRTAG_MAX					48
+#define XFS_ERRTAG_BOUNCE_REREAD			48
+#define XFS_ERRTAG_MAX					49
 
 /*
  * Random factors for above tags, 1 means always, 2 means 1/2 time, etc.
@@ -137,7 +138,8 @@ XFS_ERRTAG(WRITE_DELAY_MS,	write_delay_ms,		3000) \
 XFS_ERRTAG(EXCHMAPS_FINISH_ONE,	exchmaps_finish_one,	1) \
 XFS_ERRTAG(METAFILE_RESV_CRITICAL, metafile_resv_crit,	4) \
 XFS_ERRTAG(FORCE_ZERO_RANGE,	force_zero_range,	4) \
-XFS_ERRTAG(ZONE_RESET,		zone_reset,		1)
+XFS_ERRTAG(ZONE_RESET,		zone_reset,		1)  \
+XFS_ERRTAG(BOUNCE_REREAD,	bounce_reread,		XFS_RANDOM_DEFAULT)
 #endif /* XFS_ERRTAG */
 
 #endif /* __XFS_ERRORTAG_H_ */
diff --git a/fs/xfs/xfs_ioend.c b/fs/xfs/xfs_ioend.c
index c21cbd7b0a6d..17f7fbaef14f 100644
--- a/fs/xfs/xfs_ioend.c
+++ b/fs/xfs/xfs_ioend.c
@@ -16,6 +16,8 @@
 #include "xfs_reflink.h"
 #include "xfs_zone_alloc.h"
 #include "xfs_ioend.h"
+#include "xfs_error.h"
+#include "xfs_errortag.h"
 #include <linux/bio-integrity.h>
 
 static void
@@ -106,7 +108,8 @@ xfs_end_io_read(
 			 * but right now we can't distinguish them from other
 			 * (i.e, reftag) errors.
 			 */
-			if (error) {
+			if (error ||
+			    XFS_TEST_ERROR(mp, XFS_ERRTAG_BOUNCE_REREAD)) {
 				xfs_read_bounce_and_resubmit(ioend);
 				return;
 			}
-- 
2.53.0

[PATCH 16/17] xfs: log a message at mount time when using integrity protection

[Christoph Hellwig]

Log a message in the kernel when using T10 protection information.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/xfs/xfs_buf.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index ee7c2e9c0340..eee491c01d8c 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -5,6 +5,7 @@
  */
 #include "xfs_platform.h"
 #include <linux/backing-dev.h>
+#include <linux/blk-integrity.h>
 #include <linux/dax.h>
 
 #include "xfs_shared.h"
@@ -1694,6 +1695,7 @@ xfs_configure_buftarg(
 	struct xfs_mount	*mp = btp->bt_mount;
 
 	if (btp->bt_bdev) {
+		struct blk_integrity *bi = bdev_get_integrity(btp->bt_bdev);
 		int		error;
 
 		error = bdev_validate_blocksize(btp->bt_bdev, sectorsize);
@@ -1706,6 +1708,15 @@ xfs_configure_buftarg(
 
 		if (bdev_can_atomic_write(btp->bt_bdev))
 			xfs_configure_buftarg_atomic_writes(btp);
+
+		if (!bi)
+			;
+		else if (btp->bt_bdev == btp->bt_mount->m_super->s_bdev)
+			xfs_info(mp, "using %s integrity profile",
+				blk_integrity_profile_name(bi));
+		else
+			xfs_info(mp, "using %s integrity profile for %pg",
+				blk_integrity_profile_name(bi), btp->bt_bdev);
 	}
 
 	btp->bt_meta_sectorsize = sectorsize;
-- 
2.53.0

[PATCH 17/17] block,iomap: remove the old read side bounce buffering support

[Christoph Hellwig]

bio_iov_iter_bounce_read turned to generate suboptimal I/O sizes and
isn't usable for lazy bounce buffering.  Now that is has been replaced
with the iomap side implementation that requires extra bounce bios, it
is unused and can be removed.  Change the interface so that the
previously hidden write-side implementation is directly exposed.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 block/bio.c          | 144 +++++--------------------------------------
 fs/iomap/direct-io.c |   7 +--
 include/linux/bio.h  |   5 +-
 3 files changed, 20 insertions(+), 136 deletions(-)

diff --git a/block/bio.c b/block/bio.c
index a87c33fe96ac..82ac8cda8271 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1397,7 +1397,20 @@ int bio_alloc_bounce_folios(struct bio *bio, size_t total_len, size_t minsize)
 	return 0;
 }
 
-static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
+/**
+ * bio_iov_iter_bounce_write - bounce buffer data from an iter into a bio
+ * @bio:	bio to send
+ * @iter:	iter to read from
+ * @maxlen:	maximum size to bounce
+ * @minsize:	minimum folio allocation size
+ *
+ * Helper for direct I/O write implementations that need to bounce buffer
+ * because they need need to checksum the data or perform other operations that
+ * require consistency.  Allocates folios to back the bounce buffer, and copies
+ * the data into it.  Needs to be paired with bio_free_folios() called on
+ * completion.
+ */
+int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
 		size_t maxlen, size_t minsize)
 {
 	size_t total_len = min(maxlen, iov_iter_count(iter));
@@ -1430,134 +1443,7 @@ static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
 	return bio_iov_iter_align_down(bio, iter,
 			&bio->bi_io_vec[bio->bi_vcnt - 1], minsize - 1);
 }
-
-static int bio_iov_iter_bounce_read(struct bio *bio, struct iov_iter *iter,
-		size_t maxlen, size_t minsize)
-{
-	size_t len = min3(iov_iter_count(iter), maxlen, SZ_1M);
-	struct folio *folio;
-	ssize_t ret;
-
-	folio = folio_alloc_greedy(GFP_KERNEL, &len, minsize);
-	if (!folio)
-		return -ENOMEM;
-
-	do {
-		ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec + 1, len,
-				&bio->bi_vcnt, bio->bi_max_vecs - 1, 0, 0);
-		if (ret <= 0) {
-			if (!bio->bi_vcnt)
-				goto out_folio_put;
-			break;
-		}
-		len -= ret;
-		bio->bi_iter.bi_size += ret;
-	} while (len && bio->bi_vcnt < bio->bi_max_vecs - 1);
-
-	/*
-	 * Set the folio directly here.  The above loop has already calculated
-	 * the correct bi_size, and we use bi_vcnt for the user buffers.  That
-	 * is safe as bi_vcnt is only used by the submitter and not the actual
-	 * I/O path.
-	 */
-	bvec_set_folio(&bio->bi_io_vec[0], folio, bio->bi_iter.bi_size, 0);
-	if (iov_iter_extract_will_pin(iter))
-		bio_set_flag(bio, BIO_PAGE_PINNED);
-
-	/* The first vec stores the bounce buffer, so do not subtract 1 here. */
-	ret = bio_iov_iter_align_down(bio, iter,
-			&bio->bi_io_vec[bio->bi_vcnt], minsize - 1);
-	if (ret)
-		goto out_folio_put;
-
-	/* Update the bounc buffer bv_len to the aligned down size. */
-	bio->bi_io_vec[0].bv_len = bio->bi_iter.bi_size;
-	return 0;
-
-out_folio_put:
-	folio_put(folio);
-	return ret;
-}
-
-/**
- * bio_iov_iter_bounce - bounce buffer data from an iter into a bio
- * @bio:	bio to send
- * @iter:	iter to read from / write into
- * @maxlen:	maximum size to bounce
- * @minsize:	minimum folio allocation size
- *
- * Helper for direct I/O implementations that need to bounce buffer because
- * we need to checksum the data or perform other operations that require
- * consistency.  Allocates folios to back the bounce buffer, and for writes
- * copies the data into it.  Needs to be paired with bio_iov_iter_unbounce()
- * called on completion.
- */
-int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen,
-			size_t minsize)
-{
-	if (op_is_write(bio_op(bio)))
-		return bio_iov_iter_bounce_write(bio, iter, maxlen, minsize);
-	return bio_iov_iter_bounce_read(bio, iter, maxlen, minsize);
-}
-
-static void bvec_unpin(struct bio_vec *bv, bool mark_dirty)
-{
-	struct folio *folio = bvec_folio(bv);
-	size_t nr_pages = (bv->bv_offset + bv->bv_len - 1) / PAGE_SIZE -
-			bv->bv_offset / PAGE_SIZE + 1;
-
-	if (mark_dirty)
-		folio_mark_dirty_lock(folio);
-	unpin_user_folio(folio, nr_pages);
-}
-
-static void bio_iov_iter_unbounce_read(struct bio *bio, bool is_error,
-		bool mark_dirty)
-{
-	unsigned int len = bio->bi_io_vec[0].bv_len;
-
-	if (likely(!is_error)) {
-		void *buf = bvec_virt(&bio->bi_io_vec[0]);
-		struct iov_iter to;
-
-		iov_iter_bvec(&to, ITER_DEST, bio->bi_io_vec + 1, bio->bi_vcnt,
-				len);
-		/* copying to pinned pages should always work */
-		WARN_ON_ONCE(copy_to_iter(buf, len, &to) != len);
-	} else {
-		/* No need to mark folios dirty if never copied to them */
-		mark_dirty = false;
-	}
-
-	if (bio_flagged(bio, BIO_PAGE_PINNED)) {
-		int i;
-
-		for (i = 0; i < bio->bi_vcnt; i++)
-			bvec_unpin(&bio->bi_io_vec[1 + i], mark_dirty);
-	}
-
-	folio_put(bvec_folio(&bio->bi_io_vec[0]));
-}
-
-/**
- * bio_iov_iter_unbounce - finish a bounce buffer operation
- * @bio:	completed bio
- * @is_error:	%true if an I/O error occurred and data should not be copied
- * @mark_dirty:	If %true, folios will be marked dirty.
- *
- * Helper for direct I/O implementations that need to bounce buffer because
- * we need to checksum the data or perform other operations that require
- * consistency.  Called to complete a bio set up by bio_iov_iter_bounce().
- * Copies data back for reads, and marks the original folios dirty if
- * requested and then frees the bounce buffer.
- */
-void bio_iov_iter_unbounce(struct bio *bio, bool is_error, bool mark_dirty)
-{
-	if (op_is_write(bio_op(bio)))
-		bio_free_folios(bio);
-	else
-		bio_iov_iter_unbounce_read(bio, is_error, mark_dirty);
-}
+EXPORT_SYMBOL_GPL(bio_iov_iter_bounce_write);
 
 static void bio_wait_end_io(struct bio *bio)
 {
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 4154717a09de..41fdc90a9094 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -255,8 +255,7 @@ static void __iomap_dio_bio_end_io(struct bio *bio, bool inline_completion)
 		fs_bio_integrity_free(bio);
 
 	if (dio->flags & IOMAP_DIO_BOUNCE) {
-		bio_iov_iter_unbounce(bio, !!dio->error,
-				dio->flags & IOMAP_DIO_USER_BACKED);
+		bio_free_folios(bio);
 		bio_put(bio);
 	} else if (dio->flags & IOMAP_DIO_USER_BACKED) {
 		bio_check_pages_dirty(bio);
@@ -364,7 +363,7 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
 	bio->bi_end_io = iomap_dio_bio_end_io;
 
 	if (dio->flags & IOMAP_DIO_BOUNCE)
-		ret = bio_iov_iter_bounce(bio, dio->submit.iter, maxsize,
+		ret = bio_iov_iter_bounce_write(bio, dio->submit.iter, maxsize,
 				alignment);
 	else
 		ret = bio_iov_iter_get_pages(bio, dio->submit.iter, maxsize,
@@ -398,7 +397,7 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
 
 out_bio_release_pages:
 	if (dio->flags & IOMAP_DIO_BOUNCE)
-		bio_iov_iter_unbounce(bio, true, false);
+		bio_free_folios(bio);
 	else
 		bio_release_pages(bio, false);
 out_put_bio:
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 584b6abf6baf..828ae414e365 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -524,11 +524,10 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty);
 extern void bio_set_pages_dirty(struct bio *bio);
 extern void bio_check_pages_dirty(struct bio *bio);
 
-int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen,
-		size_t minsize);
-void bio_iov_iter_unbounce(struct bio *bio, bool is_error, bool mark_dirty);
 int bio_alloc_bounce_folios(struct bio *bio, size_t total_len, size_t minsize);
 void bio_free_folios(struct bio *bio);
+int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
+		size_t maxlen, size_t minsize);
 
 extern void bio_copy_data(struct bio *dst, struct bio *src);
 extern void bio_free_pages(struct bio *bio);
-- 
2.53.0

Re: [PATCH 14/17] xfs: add support for lazy direct read bounce buffering

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:40:02AM +0300, Christoph Hellwig wrote:
> Currently direct I/O reads always bounce buffer the I/O to deal with
> the case where userspace is modifying the buffer in-flight while
> reading data into it.
> 
> This is a very expensive countermeasure for something no sane application
> should do, so try to avoid it by reading without a bounce buffer first,
> and retrying the read on a checksum failure.  This avoids the cost of
> bounce buffering for sanely behave applications.  For the rare case of
> an application regularly modifying in-flight buffers, allow forcing the
> always bounce buffer behavior through sysfs.  And now that we have that
> knob, allow disabling read-side bounce buffering entirely for those who
> live fast and dangerous.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>

Looks good to me now.  That sysfs_match_string macro is pretty neat.
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> ---
>  fs/xfs/xfs_file.c  |   3 +-
>  fs/xfs/xfs_ioend.c | 102 +++++++++++++++++++++++++++++++++++++++++++--
>  fs/xfs/xfs_mount.h |   8 ++++
>  fs/xfs/xfs_super.c |   1 +
>  fs/xfs/xfs_sysfs.c |  76 +++++++++++++++++++++++++++++++++
>  fs/xfs/xfs_trace.h |   1 +
>  6 files changed, 186 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> index 766c4d2055c1..daa6a854dd5f 100644
> --- a/fs/xfs/xfs_file.c
> +++ b/fs/xfs/xfs_file.c
> @@ -270,8 +270,7 @@ xfs_file_dio_read(
>  		return ret;
>  	if (mapping_stable_writes(iocb->ki_filp->f_mapping)) {
>  		ret = iomap_dio_rw(iocb, to, &xfs_read_iomap_ops,
> -				&xfs_dio_read_bounce_ops, IOMAP_DIO_BOUNCE,
> -				NULL, 0);
> +				&xfs_dio_read_bounce_ops, 0, NULL, 0);
>  	} else {
>  		ret = iomap_dio_read_simple(iocb, to, xfs_read_iomap_begin);
>  		if (ret == -ENOTBLK)
> diff --git a/fs/xfs/xfs_ioend.c b/fs/xfs/xfs_ioend.c
> index a095cf217863..c21cbd7b0a6d 100644
> --- a/fs/xfs/xfs_ioend.c
> +++ b/fs/xfs/xfs_ioend.c
> @@ -1,6 +1,6 @@
>  // SPDX-License-Identifier: GPL-2.0
>  /*
> - * Copyright (c) 2016-2025 Christoph Hellwig.
> + * Copyright (c) 2016-2026 Christoph Hellwig.
>   * All Rights Reserved.
>   */
>  #include "xfs_platform.h"
> @@ -18,15 +18,100 @@
>  #include "xfs_ioend.h"
>  #include <linux/bio-integrity.h>
>  
> +static void
> +xfs_end_bio_bounced(
> +	struct bio		*bio)
> +{
> +	iomap_finish_ioends(iomap_ioend_from_bio(bio),
> +			blk_status_to_errno(bio->bi_status));
> +}
> +
> +static void
> +xfs_dio_bounce_end_io(
> +	struct bio		*bio)
> +{
> +	struct iomap_ioend	*ioend = iomap_ioend_from_bio(bio);
> +	int			error = blk_status_to_errno(bio->bi_status);
> +	struct bio		*orig_bio = bio->bi_private;
> +
> +	if ((ioend->io_flags & IOMAP_IOEND_INTEGRITY) && !bio->bi_status)
> +		error = iomap_ioend_integrity_verify(ioend);
> +	iomap_bounce_read_end_io(ioend, orig_bio, error);
> +}
> +
> +static void
> +xfs_bounce_submit_ioend(
> +	struct iomap_ioend	*ioend)
> +{
> +	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
> +		fs_bio_integrity_alloc(&ioend->io_bio);
> +	ioend->io_bio.bi_end_io = xfs_dio_bounce_end_io;
> +	bio_set_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
> +	submit_bio(&ioend->io_bio);
> +}
> +
> +static void
> +xfs_read_bounce_and_resubmit(
> +	struct iomap_ioend	*ioend)
> +{
> +	struct bio		*bio = &ioend->io_bio;
> +	unsigned short		vcnt = bio->bi_vcnt;
> +	void			*private = bio->bi_private;
> +
> +	trace_xfs_bounce_reread(XFS_I(ioend->io_inode), ioend->io_offset,
> +			ioend->io_size);
> +
> +	/*
> +	 * Free the bio integrity data for the original bio, as we'll allocate
> +	 * ons for each sub-I/O, which could deadlock if we keep the original
> +	 * one around.
> +	 */
> +	if (bio_integrity(bio))
> +		fs_bio_integrity_free(bio);
> +
> +	/*
> +	 * Reset the bio to submit the bio to the block layer again.  Switch to
> +	 * an end_io handler that simply complets the ioend, as all verification
> +	 * is done by the end_I/O handlers for the clone bio(s).
> +	 */
> +	bio_reset(bio, xfs_inode_buftarg(XFS_I(ioend->io_inode))->bt_bdev,
> +			bio->bi_opf);
> +	bio->bi_vcnt = vcnt;
> +	bio->bi_private = private;
> +	bio->bi_end_io = xfs_end_bio_bounced;
> +	bio->bi_iter = (struct bvec_iter) {
> +		.bi_sector	= ioend->io_sector,
> +		.bi_size	= ioend->io_size,
> +		.bi_offset	= ioend->io_bvec_offset,
> +	};
> +	iomap_bounce_read(ioend, bdev_logical_block_size(bio->bi_bdev),
> +			xfs_bounce_submit_ioend);
> +}
> +
>  static void
>  xfs_end_io_read(
>  	struct bio		*bio)
>  {
>  	struct iomap_ioend	*ioend = iomap_ioend_from_bio(bio);
> +	struct xfs_inode	*ip = XFS_I(ioend->io_inode);
> +	struct xfs_mount	*mp = ip->i_mount;
>  	int			error = blk_status_to_errno(bio->bi_status);
>  
> -	if (!error && (ioend->io_flags & IOMAP_IOEND_INTEGRITY))
> +	if (!error && (ioend->io_flags & IOMAP_IOEND_INTEGRITY)) {
>  		error = iomap_ioend_integrity_verify(ioend);
> +		if ((ioend->io_flags & IOMAP_IOEND_DIRECT) &&
> +		    READ_ONCE(mp->m_read_bounce) == XFS_READ_BOUNCE_LAZY) {
> +			/*
> +			 * We only really need to retry for guard tag errors,
> +			 * but right now we can't distinguish them from other
> +			 * (i.e, reftag) errors.
> +			 */
> +			if (error) {
> +				xfs_read_bounce_and_resubmit(ioend);
> +				return;
> +			}
> +		}
> +	}
>  
>  	iomap_finish_ioends(ioend, error);
>  }
> @@ -38,7 +123,18 @@ xfs_ioend_submit_read(
>  	loff_t			file_offset,
>  	u16			ioend_flags)
>  {
> -	iomap_init_ioend(inode, bio, file_offset, ioend_flags);
> +	struct xfs_inode	*ip = XFS_I(inode);
> +	struct xfs_mount	*mp = ip->i_mount;
> +	struct iomap_ioend	*ioend;
> +
> +	ioend = iomap_init_ioend(inode, bio, file_offset, ioend_flags);
> +	if ((ioend_flags & IOMAP_IOEND_DIRECT) &&
> +	    READ_ONCE(mp->m_read_bounce) == XFS_READ_BOUNCE_ALWAYS) {
> +		iomap_bounce_read(ioend, bdev_logical_block_size(bio->bi_bdev),
> +				xfs_bounce_submit_ioend);
> +		return;
> +	}
> +
>  	if (ioend_flags & IOMAP_IOEND_INTEGRITY)
>  		fs_bio_integrity_alloc(bio);
>  	bio->bi_end_io = xfs_end_io_read;
> diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h
> index 216a38a354e7..894ff2f4ecbd 100644
> --- a/fs/xfs/xfs_mount.h
> +++ b/fs/xfs/xfs_mount.h
> @@ -142,6 +142,12 @@ struct xfs_freecounter {
>  	uint64_t		res_saved;
>  };
>  
> +enum xfs_read_bounce {
> +	XFS_READ_BOUNCE_NEVER,
> +	XFS_READ_BOUNCE_ALWAYS,
> +	XFS_READ_BOUNCE_LAZY,
> +};
> +
>  /*
>   * The struct xfsmount layout is optimised to separate read-mostly variables
>   * from variables that are frequently modified. We put the read-mostly variables
> @@ -177,6 +183,7 @@ typedef struct xfs_mount {
>  	struct workqueue_struct	*m_sync_workqueue;
>  	struct workqueue_struct *m_blockgc_wq;
>  	struct workqueue_struct *m_inodegc_wq;
> +	enum xfs_read_bounce	m_read_bounce;
>  
>  	int			m_bsize;	/* fs logical block size */
>  	uint8_t			m_blkbit_log;	/* blocklog + NBBY */
> @@ -291,6 +298,7 @@ typedef struct xfs_mount {
>  	struct xfs_zone_info	*m_zone_info;	/* zone allocator information */
>  	struct dentry		*m_debugfs;	/* debugfs parent */
>  	struct xfs_kobj		m_kobj;
> +	struct xfs_kobj		m_csum_kobj;
>  	struct xfs_kobj		m_error_kobj;
>  	struct xfs_kobj		m_error_meta_kobj;
>  	struct xfs_error_cfg	m_error_cfg[XFS_ERR_CLASS_MAX][XFS_ERR_ERRNO_MAX];
> diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
> index b24db75eaedc..fce1d2905c94 100644
> --- a/fs/xfs/xfs_super.c
> +++ b/fs/xfs/xfs_super.c
> @@ -2317,6 +2317,7 @@ xfs_init_fs_context(
>  	mp->m_logbufs = -1;
>  	mp->m_logbsize = -1;
>  	mp->m_allocsize_log = 16; /* 64k */
> +	mp->m_read_bounce = XFS_READ_BOUNCE_LAZY;
>  
>  	xfs_hooks_init(&mp->m_dir_update_hooks);
>  
> diff --git a/fs/xfs/xfs_sysfs.c b/fs/xfs/xfs_sysfs.c
> index b62712187324..2e969e8f279f 100644
> --- a/fs/xfs/xfs_sysfs.c
> +++ b/fs/xfs/xfs_sysfs.c
> @@ -392,6 +392,57 @@ const struct kobj_type xfs_stats_ktype = {
>  	.default_groups = xfs_stats_groups,
>  };
>  
> +static inline struct xfs_mount *csum_to_mp(struct kobject *kobj)
> +{
> +	return container_of(to_kobj(kobj), struct xfs_mount, m_csum_kobj);
> +}
> +
> +static const char * const bounce_modes[] = {
> +	[XFS_READ_BOUNCE_NEVER]		= "never",
> +	[XFS_READ_BOUNCE_ALWAYS]	= "always",
> +	[XFS_READ_BOUNCE_LAZY]		= "lazy",
> +};
> +
> +static ssize_t
> +read_bounce_show(
> +	struct kobject		*kobj,
> +	char			*buf)
> +{
> +	struct xfs_mount	*mp = csum_to_mp(kobj);
> +
> +	return sysfs_emit(buf, "%s\n",
> +			bounce_modes[READ_ONCE(mp->m_read_bounce)]);
> +}
> +
> +static ssize_t
> +read_bounce_store(
> +	struct kobject		*kobj,
> +	const char		*buf,
> +	size_t			count)
> +{
> +	struct xfs_mount	*mp = csum_to_mp(kobj);
> +	int			ret;
> +
> +	ret = sysfs_match_string(bounce_modes, buf);
> +	if (ret < 0)
> +		return ret;
> +	WRITE_ONCE(mp->m_read_bounce, ret);
> +	return count;
> +}
> +XFS_SYSFS_ATTR_RW(read_bounce);
> +
> +static struct attribute *xfs_csum_attrs[] = {
> +	ATTR_LIST(read_bounce),
> +	NULL,
> +};
> +ATTRIBUTE_GROUPS(xfs_csum);
> +
> +static const struct kobj_type xfs_csum_ktype = {
> +	.release = xfs_sysfs_release,
> +	.sysfs_ops = &xfs_sysfs_ops,
> +	.default_groups = xfs_csum_groups,
> +};
> +
>  /* xlog */
>  
>  static inline struct xlog *
> @@ -797,6 +848,18 @@ xfs_zoned_sysfs_del(struct xfs_mount *mp)
>  		xfs_sysfs_del(&mp->m_zoned_kobj);
>  }
>  
> +static bool
> +xfs_has_read_bounce(
> +	struct xfs_mount	*mp)
> +{
> +	if (bdev_has_integrity_csum(mp->m_ddev_targp->bt_bdev))
> +		return true;
> +	if (mp->m_rtdev_targp &&
> +	    bdev_has_integrity_csum(mp->m_rtdev_targp->bt_bdev))
> +		return true;
> +	return false;
> +}
> +
>  int
>  xfs_mount_sysfs_init(
>  	struct xfs_mount	*mp)
> @@ -837,8 +900,18 @@ xfs_mount_sysfs_init(
>  	if (error)
>  		goto out_remove_error_dir;
>  
> +	if (xfs_has_read_bounce(mp)) {
> +		/* .../xfs/<dev>/csum/ */
> +		error = xfs_sysfs_init(&mp->m_csum_kobj, &xfs_csum_ktype,
> +				       &mp->m_kobj, "csum");
> +		if (error)
> +			goto out_remove_error_metadata_dir;
> +	}
> +
>  	return 0;
>  
> +out_remove_error_metadata_dir:
> +	xfs_sysfs_del(&mp->m_error_meta_kobj);
>  out_remove_error_dir:
>  	xfs_sysfs_del(&mp->m_error_kobj);
>  out_remove_stats_dir:
> @@ -855,6 +928,9 @@ xfs_mount_sysfs_del(
>  	struct xfs_error_cfg	*cfg;
>  	int			i, j;
>  
> +	if (xfs_has_read_bounce(mp))
> +		xfs_sysfs_del(&mp->m_csum_kobj);
> +
>  	for (i = 0; i < XFS_ERR_CLASS_MAX; i++) {
>  		for (j = 0; j < XFS_ERR_ERRNO_MAX; j++) {
>  			cfg = &mp->m_error_cfg[i][j];
> diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h
> index f333c938fbd9..2af9a1429ae9 100644
> --- a/fs/xfs/xfs_trace.h
> +++ b/fs/xfs/xfs_trace.h
> @@ -1896,6 +1896,7 @@ DEFINE_SIMPLE_IO_EVENT(xfs_zero_eof);
>  DEFINE_SIMPLE_IO_EVENT(xfs_end_io_direct_write);
>  DEFINE_SIMPLE_IO_EVENT(xfs_file_splice_read);
>  DEFINE_SIMPLE_IO_EVENT(xfs_zoned_map_blocks);
> +DEFINE_SIMPLE_IO_EVENT(xfs_bounce_reread);
>  
>  DECLARE_EVENT_CLASS(xfs_itrunc_class,
>  	TP_PROTO(struct xfs_inode *ip, xfs_fsize_t new_size),
> -- 
> 2.53.0
> 
> 

Re: [PATCH 10/17] block,iomap: fix protection information verification with initial bvec offset

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:39:58AM +0300, Christoph Hellwig wrote:
> When reconstructing a bvec_iter from an ioend for protection information
> verification, iomap currently ignores the offset into the initial
> bio_vec.
> 
> This can't happen for buffered I/O an direct I/O to user addresses, but

"...buffered I/O or a direct I/O..." ?

> is exercised by split on O_DIRECT file descriptors or when using the loop
> driver.
> 
> Fortunately the only file system PI user (XFS) currently always bounce
> buffers, so this can't actually be triggered yet.  But we'll want to make
> the bounce buffering conditional soon, for which this needs to be fixed.
> 
> Store the initial offset in struct iomap_ioend, and pass a
> pre-constructed bvec_iter to fs_bio_integrity_verify.  For the
> synchronous read case the fix is even simpler as this path can
> simply stash away the original bvec_iter.
> 
> Fixes: 0bde8a12b554 ("block: add fs_bio_integrity helpers")
> Signed-off-by: Christoph Hellwig <hch@lst.de>

With that fixed, this seems reasonable to me.
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> ---
>  block/bio-integrity-fs.c      | 13 +++++--------
>  fs/iomap/bio.c                |  4 +++-
>  fs/iomap/ioend.c              | 14 ++++++++++----
>  include/linux/bio-integrity.h |  3 +--
>  include/linux/iomap.h         |  8 ++++++++
>  5 files changed, 27 insertions(+), 15 deletions(-)
> 
> diff --git a/block/bio-integrity-fs.c b/block/bio-integrity-fs.c
> index 4f67b34bf289..c8e91ada8ca6 100644
> --- a/block/bio-integrity-fs.c
> +++ b/block/bio-integrity-fs.c
> @@ -54,14 +54,10 @@ void fs_bio_integrity_generate(struct bio *bio)
>  }
>  EXPORT_SYMBOL_GPL(fs_bio_integrity_generate);
>  
> -int fs_bio_integrity_verify(struct bio *bio, sector_t sector, unsigned int size)
> +int fs_bio_integrity_verify(struct bio *bio, struct bvec_iter *data_iter)
>  {
>  	struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
>  	struct bio_integrity_payload *bip = bio_integrity(bio);
> -	struct bvec_iter data_iter = {
> -		.bi_sector	= sector,
> -		.bi_size	= size,
> -	};
>  
>  	if (!bip || !(bip->bip_flags & BIP_CHECK_FLAGS))
>  		return 0;
> @@ -73,9 +69,10 @@ int fs_bio_integrity_verify(struct bio *bio, sector_t sector, unsigned int size)
>  	 * bio.  Requires the submitter to remember the sector and the size.
>  	 */
>  	memset(&bip->bip_iter, 0, sizeof(bip->bip_iter));
> -	bip->bip_iter.bi_sector = sector;
> -	bip->bip_iter.bi_size = bio_integrity_bytes(bi, size >> SECTOR_SHIFT);
> -	return blk_status_to_errno(bio_integrity_verify(bio, &data_iter));
> +	bip->bip_iter.bi_sector = data_iter->bi_sector;
> +	bip->bip_iter.bi_size =
> +		bio_integrity_bytes(bi, data_iter->bi_size >> SECTOR_SHIFT);
> +	return blk_status_to_errno(bio_integrity_verify(bio, data_iter));
>  }
>  
>  static int __init fs_bio_integrity_init(void)
> diff --git a/fs/iomap/bio.c b/fs/iomap/bio.c
> index 48100c614431..d46c2f8ea18c 100644
> --- a/fs/iomap/bio.c
> +++ b/fs/iomap/bio.c
> @@ -169,6 +169,7 @@ int iomap_bio_read_folio_range_sync(const struct iomap_iter *iter,
>  {
>  	const struct iomap *srcmap = iomap_iter_srcmap(iter);
>  	sector_t sector = iomap_sector(srcmap, pos);
> +	struct bvec_iter saved_iter;
>  	struct bio_vec bvec;
>  	struct bio bio;
>  	int error;
> @@ -178,10 +179,11 @@ int iomap_bio_read_folio_range_sync(const struct iomap_iter *iter,
>  	bio_add_folio_nofail(&bio, folio, len, offset_in_folio(folio, pos));
>  	if (srcmap->flags & IOMAP_F_INTEGRITY)
>  		fs_bio_integrity_alloc(&bio);
> +	saved_iter = bio.bi_iter;
>  	error = submit_bio_wait(&bio);
>  	if (bio_integrity(&bio)) {
>  		if (!error)
> -			error = fs_bio_integrity_verify(&bio, sector, len);
> +			error = fs_bio_integrity_verify(&bio, &saved_iter);
>  		fs_bio_integrity_free(&bio);
>  	}
>  	bio_uninit(&bio);
> diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c
> index 573fa89c1632..332dbfb2230f 100644
> --- a/fs/iomap/ioend.c
> +++ b/fs/iomap/ioend.c
> @@ -25,6 +25,7 @@ struct iomap_ioend *iomap_init_ioend(struct inode *inode,
>  	ioend->io_parent = NULL;
>  	INIT_LIST_HEAD(&ioend->io_list);
>  	ioend->io_flags = ioend_flags;
> +	ioend->io_bvec_offset = bio->bi_iter.bi_offset;
>  	ioend->io_inode = inode;
>  	ioend->io_offset = file_offset;
>  	ioend->io_size = bio->bi_iter.bi_size;
> @@ -308,6 +309,13 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
>  }
>  EXPORT_SYMBOL_GPL(iomap_add_to_ioend);
>  
> +static int iomap_ioend_integrity_verify(struct iomap_ioend *ioend)
> +{
> +	struct bvec_iter data_iter = BVEC_ITER_IOEND(ioend);
> +
> +	return fs_bio_integrity_verify(&ioend->io_bio, &data_iter);
> +}
> +
>  static u32 iomap_finish_ioend(struct iomap_ioend *ioend, int error)
>  {
>  	if (ioend->io_parent) {
> @@ -325,10 +333,8 @@ static u32 iomap_finish_ioend(struct iomap_ioend *ioend, int error)
>  
>  	if (!ioend->io_error &&
>  	    bio_integrity(&ioend->io_bio) &&
> -	    bio_op(&ioend->io_bio) == REQ_OP_READ) {
> -		ioend->io_error = fs_bio_integrity_verify(&ioend->io_bio,
> -			ioend->io_sector, ioend->io_size);
> -	}
> +	    bio_op(&ioend->io_bio) == REQ_OP_READ)
> +		ioend->io_error = iomap_ioend_integrity_verify(ioend);
>  
>  	if (ioend->io_flags & IOMAP_IOEND_DIRECT)
>  		return iomap_finish_ioend_direct(ioend);
> diff --git a/include/linux/bio-integrity.h b/include/linux/bio-integrity.h
> index 0ea2a8bf7efb..a954c97be0b3 100644
> --- a/include/linux/bio-integrity.h
> +++ b/include/linux/bio-integrity.h
> @@ -151,7 +151,6 @@ void bio_integrity_setup_default(struct bio *bio);
>  unsigned int fs_bio_integrity_alloc(struct bio *bio);
>  void fs_bio_integrity_free(struct bio *bio);
>  void fs_bio_integrity_generate(struct bio *bio);
> -int fs_bio_integrity_verify(struct bio *bio, sector_t sector,
> -		unsigned int size);
> +int fs_bio_integrity_verify(struct bio *bio, struct bvec_iter *data_iter);
>  
>  #endif /* _LINUX_BIO_INTEGRITY_H */
> diff --git a/include/linux/iomap.h b/include/linux/iomap.h
> index 1cc9a35fd5cc..bffdc217ad85 100644
> --- a/include/linux/iomap.h
> +++ b/include/linux/iomap.h
> @@ -522,6 +522,7 @@ static inline u16 iomap_ioend_flags(const struct iomap *iomap)
>  struct iomap_ioend {
>  	struct list_head	io_list;	/* next ioend in chain */
>  	u16			io_flags;	/* IOMAP_IOEND_* */
> +	u32			io_bvec_offset;	/* offset into first bvec */
>  	struct inode		*io_inode;	/* file being written to */
>  	size_t			io_size;	/* size of the extent */
>  	atomic_t		io_remaining;	/* completetion defer count */
> @@ -539,6 +540,13 @@ static inline struct iomap_ioend *iomap_ioend_from_bio(struct bio *bio)
>  	return container_of(bio, struct iomap_ioend, io_bio);
>  }
>  
> +#define BVEC_ITER_IOEND(_ioend)				\
> +{							\
> +	.bi_sector	= (_ioend)->io_sector,		\
> +	.bi_size	= (_ioend)->io_size,		\
> +	.bi_offset	= (_ioend)->io_bvec_offset,	\
> +}
> +
>  struct iomap_writeback_ops {
>  	/*
>  	 * Performs writeback on the passed in range
> -- 
> 2.53.0
> 
> 

Re: [PATCH 09/17] xfs: move PI generation into xfs_submit_zoned_bio

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:39:57AM +0300, Christoph Hellwig wrote:
> Keep the code in one place, and do so only after splitting to
> keep the allocation size down in case of a split.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>

Looks good, just like last time ;)
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> ---
>  fs/xfs/xfs_aops.c       | 4 +---
>  fs/xfs/xfs_file.c       | 2 --
>  fs/xfs/xfs_zone_alloc.c | 3 +++
>  3 files changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
> index 4e862406eee9..d2cdb7cffa6b 100644
> --- a/fs/xfs/xfs_aops.c
> +++ b/fs/xfs/xfs_aops.c
> @@ -23,7 +23,6 @@
>  #include "xfs_ioend.h"
>  #include "xfs_zone_alloc.h"
>  #include "xfs_rtgroup.h"
> -#include <linux/bio-integrity.h>
>  
>  struct xfs_writepage_ctx {
>  	struct iomap_writepage_ctx ctx;
> @@ -498,8 +497,7 @@ xfs_zoned_writeback_submit(
>  		bio_endio(&ioend->io_bio);
>  		return error;
>  	}
> -	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
> -		fs_bio_integrity_generate(&ioend->io_bio);
> +
>  	xfs_zone_alloc_and_submit(ioend, &XFS_ZWPC(wpc)->open_zone);
>  	return 0;
>  }
> diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
> index acbfd55f56a3..0e029db2e85b 100644
> --- a/fs/xfs/xfs_file.c
> +++ b/fs/xfs/xfs_file.c
> @@ -739,8 +739,6 @@ xfs_dio_zoned_submit_io(
>  	bio->bi_end_io = xfs_end_bio;
>  	ioend = iomap_init_ioend(iter->inode, bio, file_offset,
>  			iomap_ioend_flags(&iter->iomap) | IOMAP_IOEND_DIRECT);
> -	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
> -		fs_bio_integrity_generate(bio);
>  	xfs_zone_alloc_and_submit(ioend, &ac->open_zone);
>  }
>  
> diff --git a/fs/xfs/xfs_zone_alloc.c b/fs/xfs/xfs_zone_alloc.c
> index bdbb60cc5d5b..e427be806f97 100644
> --- a/fs/xfs/xfs_zone_alloc.c
> +++ b/fs/xfs/xfs_zone_alloc.c
> @@ -26,6 +26,7 @@
>  #include "xfs_zones.h"
>  #include "xfs_trace.h"
>  #include "xfs_mru_cache.h"
> +#include <linux/bio-integrity.h>
>  
>  static void
>  xfs_open_zone_free_rcu(
> @@ -890,6 +891,8 @@ xfs_submit_zoned_bio(
>  		xfs_mark_rtg_boundary(ioend);
>  	}
>  
> +	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
> +		fs_bio_integrity_generate(&ioend->io_bio);
>  	submit_bio(&ioend->io_bio);
>  }
>  
> -- 
> 2.53.0
> 
> 

Re: [PATCH 05/17] iomap: respect maximum I/O size in iomap_dio_bio_iter_one

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:39:53AM +0300, Christoph Hellwig wrote:
> Respect the maximum I/O size set for PI-enabled I/O in
> iomap_dio_bio_iter_one, otherwise bio_integrity_alloc_buf could
> under-allocate the integrity buffer when the initial kmalloc fails.
> 
> Currently this should not be triggered as file systems that limit the
> size for PI always use bounce buffering, but this will change soon.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>

This makes sense, as the !DIO_BOUNCE path will soon get used for the
"don't bounce, just try it and see if it fails" code path.

Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> ---
>  fs/iomap/direct-io.c | 11 +++++------
>  1 file changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
> index bc726d47b7dc..91f5b67718a5 100644
> --- a/fs/iomap/direct-io.c
> +++ b/fs/iomap/direct-io.c
> @@ -336,6 +336,7 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
>  		struct iomap_dio *dio, loff_t pos, unsigned int alignment,
>  		blk_opf_t op)
>  {
> +	unsigned int maxsize = iomap_max_bio_size(&iter->iomap);
>  	unsigned int nr_vecs;
>  	struct bio *bio;
>  	ssize_t ret;
> @@ -353,14 +354,12 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
>  	bio->bi_private = dio;
>  	bio->bi_end_io = iomap_dio_bio_end_io;
>  
> -
>  	if (dio->flags & IOMAP_DIO_BOUNCE)
> -		ret = bio_iov_iter_bounce(bio, dio->submit.iter,
> -				iomap_max_bio_size(&iter->iomap), alignment);
> +		ret = bio_iov_iter_bounce(bio, dio->submit.iter, maxsize,
> +				alignment);
>  	else
> -		ret = bio_iov_iter_get_pages(bio, dio->submit.iter,
> -				BIO_MAX_SIZE, bdev_dma_alignment(bio->bi_bdev),
> -				alignment - 1);
> +		ret = bio_iov_iter_get_pages(bio, dio->submit.iter, maxsize,
> +				bdev_dma_alignment(bio->bi_bdev), alignment - 1);
>  	if (unlikely(ret))
>  		goto out_put_bio;
>  	ret = bio->bi_iter.bi_size;
> -- 
> 2.53.0
> 
> 

Re: [PATCH 04/17] block: export fs_bio_integrity_{alloc,free}

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:39:52AM +0300, Christoph Hellwig wrote:
> We'll move integrity generation and verification into the iomap
> submission helpers, which means they will be needed in modular file
> system code.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>

Seems fine to me!
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> ---
>  block/bio-integrity-fs.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/block/bio-integrity-fs.c b/block/bio-integrity-fs.c
> index 692403dfa047..4f67b34bf289 100644
> --- a/block/bio-integrity-fs.c
> +++ b/block/bio-integrity-fs.c
> @@ -31,6 +31,7 @@ unsigned int fs_bio_integrity_alloc(struct bio *bio)
>  		bio_integrity_setup_default(bio);
>  	return action;
>  }
> +EXPORT_SYMBOL_GPL(fs_bio_integrity_alloc);
>  
>  void fs_bio_integrity_free(struct bio *bio)
>  {
> @@ -43,6 +44,7 @@ void fs_bio_integrity_free(struct bio *bio)
>  	bio->bi_integrity = NULL;
>  	bio->bi_opf &= ~REQ_INTEGRITY;
>  }
> +EXPORT_SYMBOL_GPL(fs_bio_integrity_free);
>  
>  void fs_bio_integrity_generate(struct bio *bio)
>  {
> -- 
> 2.53.0
> 
> 

Re: [PATCH 03/17] block: split bio_iov_iter_bounce_write

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:39:51AM +0300, Christoph Hellwig wrote:
> Factor out a bio_alloc_bounce_folios helper that we'll use for a
> different take on read-side bounce buffering soon.
> 
> For that make it and also bio_free_folios available to callers outside of
> bio.c.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>

This looks like a simple split of one function into two, right?

If so, then I'm ok with this;
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> ---
>  block/bio.c         | 58 ++++++++++++++++++++++++++-------------------
>  include/linux/bio.h |  2 ++
>  2 files changed, 35 insertions(+), 25 deletions(-)
> 
> diff --git a/block/bio.c b/block/bio.c
> index efd9483b8348..a87c33fe96ac 100644
> --- a/block/bio.c
> +++ b/block/bio.c
> @@ -1353,7 +1353,7 @@ static struct folio *folio_alloc_greedy(gfp_t gfp, size_t *size,
>  	return folio_alloc(gfp, get_order(*size));
>  }
>  
> -static void bio_free_folios(struct bio *bio)
> +void bio_free_folios(struct bio *bio)
>  {
>  	struct bio_vec *bv;
>  	int i;
> @@ -1366,11 +1366,8 @@ static void bio_free_folios(struct bio *bio)
>  	}
>  }
>  
> -static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
> -		size_t maxlen, size_t minsize)
> +int bio_alloc_bounce_folios(struct bio *bio, size_t total_len, size_t minsize)
>  {
> -	size_t total_len = min(maxlen, iov_iter_count(iter));
> -
>  	if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
>  		return -EINVAL;
>  	if (WARN_ON_ONCE(bio->bi_iter.bi_size))
> @@ -1380,7 +1377,6 @@ static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
>  
>  	do {
>  		size_t this_len = min(total_len, SZ_1M);
> -		size_t copied;
>  		struct folio *folio;
>  
>  		if (this_len > minsize * 2)
> @@ -1393,32 +1389,44 @@ static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
>  		if (!folio)
>  			break;
>  		bio_add_folio_nofail(bio, folio, this_len, 0);
> +		total_len -= this_len;
> +	} while (total_len && bio->bi_vcnt < bio->bi_max_vecs);
> +
> +	if (!bio->bi_iter.bi_size)
> +		return -ENOMEM;
> +	return 0;
> +}
> +
> +static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
> +		size_t maxlen, size_t minsize)
> +{
> +	size_t total_len = min(maxlen, iov_iter_count(iter));
> +	size_t total_copied = 0;
> +	struct bio_vec *bv;
> +	int i, error;
> +
> +	error = bio_alloc_bounce_folios(bio, total_len, minsize);
> +	if (error)
> +		return error;
> +
> +	bio_for_each_bvec_all(bv, bio, i) {
> +		struct folio *folio = page_folio(bv->bv_page);
> +		size_t copied;
>  
>  		if (iter->nofault)
> -			copied = copy_folio_from_iter_atomic(folio, 0, this_len,
> -							     iter);
> +			copied = copy_folio_from_iter_atomic(folio, 0,
> +					bv->bv_len, iter);
>  		else
> -			copied = copy_folio_from_iter(folio, 0, this_len, iter);
> -		if (copied < this_len) {
> -			/*
> -			 * Need to revert the iov iter for all bytes we have
> -			 * copied.
> -			 *
> -			 * However the bio size differs from the real copied
> -			 * bytes as @this_len is queued but only advanced
> -			 * less than that.
> -			 * Need to compensate that for the revert.
> -			 */
> -			iov_iter_revert(iter, bio->bi_iter.bi_size - this_len +
> -					copied);
> +			copied = copy_folio_from_iter(folio, 0, bv->bv_len,
> +					iter);
> +		total_copied += copied;
> +		if (copied < bv->bv_len) {
> +			iov_iter_revert(iter, total_copied);
>  			bio_free_folios(bio);
>  			return -EFAULT;
>  		}
> -		total_len -= this_len;
> -	} while (total_len && bio->bi_vcnt < bio->bi_max_vecs);
> +	}
>  
> -	if (!bio->bi_iter.bi_size)
> -		return -ENOMEM;
>  	return bio_iov_iter_align_down(bio, iter,
>  			&bio->bi_io_vec[bio->bi_vcnt - 1], minsize - 1);
>  }
> diff --git a/include/linux/bio.h b/include/linux/bio.h
> index 0070be355181..584b6abf6baf 100644
> --- a/include/linux/bio.h
> +++ b/include/linux/bio.h
> @@ -527,6 +527,8 @@ extern void bio_check_pages_dirty(struct bio *bio);
>  int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen,
>  		size_t minsize);
>  void bio_iov_iter_unbounce(struct bio *bio, bool is_error, bool mark_dirty);
> +int bio_alloc_bounce_folios(struct bio *bio, size_t total_len, size_t minsize);
> +void bio_free_folios(struct bio *bio);
>  
>  extern void bio_copy_data(struct bio *dst, struct bio *src);
>  extern void bio_free_pages(struct bio *bio);
> -- 
> 2.53.0
> 
> 

Re: [PATCH 02/17] block: warn on too larger integrity allocations

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:39:50AM +0300, Christoph Hellwig wrote:
> Catch cases where upper layer bugs create larger I/Os than the
> mempool would return even without dipping into the mempool.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  block/bio-integrity.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/block/bio-integrity.c b/block/bio-integrity.c
> index b23e2434d80c..d3df726e0f08 100644
> --- a/block/bio-integrity.c
> +++ b/block/bio-integrity.c
> @@ -72,6 +72,7 @@ void bio_integrity_alloc_buf(struct bio *bio, gfp_t gfp, bool zero_buffer)
>  	unsigned int len = bio_integrity_bytes(bi, bio_sectors(bio));
>  	void *buf;
>  
> +	WARN_ON_ONCE(len > BLK_INTEGRITY_MAX_SIZE);

Should it fail the bio after complaining, then?

--D

>  	buf = kmalloc(len, gfp | __GFP_NOWARN | (zero_buffer ? __GFP_ZERO : 0));
>  	if (unlikely(!buf)) {
>  		struct page *page;
> -- 
> 2.53.0
> 
> 

Re: [PATCH 01/17] block: pass a maxlen argument to bio_iov_iter_get_pages

[Darrick J. Wong]

On Mon, Aug 31, 2026 at 09:39:49AM +0300, Christoph Hellwig wrote:
> Writes that require additional per-I/O allocations such as when using
> file system level protection information can be limited to a maximum
> size.  Allow passing that to bio_iov_iter_get_pages.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>

I'm slightly wary of what weird bugs might fall out on account of
reducing the size of the bi_iter size, but AFAICT it's a reasonable
thing to do, so

Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> ---
>  block/bio.c          |  8 ++++++--
>  block/blk-map.c      |  2 +-
>  block/fops.c         |  3 ++-
>  fs/iomap/direct-io.c | 10 +++++-----
>  include/linux/bio.h  |  3 ++-
>  5 files changed, 16 insertions(+), 10 deletions(-)
> 
> diff --git a/block/bio.c b/block/bio.c
> index 898b2f5ef8c8..efd9483b8348 100644
> --- a/block/bio.c
> +++ b/block/bio.c
> @@ -1261,6 +1261,7 @@ static inline bool bio_iov_bvec_aligned(const struct bio *bio,
>   * bio_iov_iter_get_pages - add user or kernel pages to a bio
>   * @bio: bio to add pages to
>   * @iter: iov iterator describing the region to be added
> + * @maxlen: maximum size to consume from @iter
>   * @mem_align_mask: the mask the source address and length must be aligned to,
>   *	0 for no requirement
>   * @len_align_mask: the mask to align the total size to, 0 for any length
> @@ -1281,7 +1282,8 @@ static inline bool bio_iov_bvec_aligned(const struct bio *bio,
>   * is returned only if 0 pages could be pinned.
>   */
>  int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
> -			   unsigned mem_align_mask, unsigned len_align_mask)
> +			   unsigned maxlen, unsigned mem_align_mask,
> +			   unsigned len_align_mask)
>  {
>  	iov_iter_extraction_t flags = 0;
>  
> @@ -1293,6 +1295,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
>  		    !bio_iov_bvec_aligned(bio, mem_align_mask))
>  			return -EINVAL;
>  
> +		/* Truncate to the maximum size that the caller can handle */
> +		bio->bi_iter.bi_size = min(bio->bi_iter.bi_size, maxlen);
>  		iov_iter_advance(iter, bio->bi_iter.bi_size);
>  		return 0;
>  	}
> @@ -1306,7 +1310,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
>  		ssize_t ret;
>  
>  		ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec,
> -				BIO_MAX_SIZE - bio->bi_iter.bi_size,
> +				maxlen - bio->bi_iter.bi_size,
>  				&bio->bi_vcnt, bio->bi_max_vecs,
>  				mem_align_mask, flags);
>  		if (ret <= 0) {
> diff --git a/block/blk-map.c b/block/blk-map.c
> index 9cb9605d1f62..81cba3af4e9c 100644
> --- a/block/blk-map.c
> +++ b/block/blk-map.c
> @@ -274,7 +274,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,
>  	 * No alignment requirements on our part to support arbitrary
>  	 * passthrough commands.
>  	 */
> -	ret = bio_iov_iter_get_pages(bio, iter, 0, 0);
> +	ret = bio_iov_iter_get_pages(bio, iter, BIO_MAX_SIZE, 0, 0);
>  	if (ret)
>  		goto out_put;
>  	ret = blk_rq_append_bio(rq, bio);
> diff --git a/block/fops.c b/block/fops.c
> index 2ce7c6c4714e..a83df69b175a 100644
> --- a/block/fops.c
> +++ b/block/fops.c
> @@ -46,7 +46,8 @@ static bool blkdev_dio_invalid(struct block_device *bdev, struct kiocb *iocb,
>  static inline int blkdev_iov_iter_get_pages(struct bio *bio,
>  		struct iov_iter *iter, struct block_device *bdev)
>  {
> -	return bio_iov_iter_get_pages(bio, iter, bdev_dma_alignment(bdev),
> +	return bio_iov_iter_get_pages(bio, iter, BIO_MAX_SIZE,
> +			bdev_dma_alignment(bdev),
>  			bdev_logical_block_size(bdev) - 1);
>  }
>  
> diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
> index 8b4039d16ce8..bc726d47b7dc 100644
> --- a/fs/iomap/direct-io.c
> +++ b/fs/iomap/direct-io.c
> @@ -359,8 +359,8 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
>  				iomap_max_bio_size(&iter->iomap), alignment);
>  	else
>  		ret = bio_iov_iter_get_pages(bio, dio->submit.iter,
> -					     bdev_dma_alignment(bio->bi_bdev),
> -					     alignment - 1);
> +				BIO_MAX_SIZE, bdev_dma_alignment(bio->bi_bdev),
> +				alignment - 1);
>  	if (unlikely(ret))
>  		goto out_put_bio;
>  	ret = bio->bi_iter.bi_size;
> @@ -1034,9 +1034,9 @@ ssize_t __iomap_dio_read_simple(struct kiocb *iocb, struct iov_iter *iter,
>  	bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
>  	bio->bi_ioprio = iocb->ki_ioprio;
>  
> -	ret = bio_iov_iter_get_pages(bio, iter,
> -				bdev_dma_alignment(bio->bi_bdev),
> -				alignment - 1);
> +	ret = bio_iov_iter_get_pages(bio, iter, BIO_MAX_SIZE,
> +			bdev_dma_alignment(bio->bi_bdev),
> +			alignment - 1);
>  	if (unlikely(ret))
>  		goto out_bio_put;
>  
> diff --git a/include/linux/bio.h b/include/linux/bio.h
> index bb3235497e67..0070be355181 100644
> --- a/include/linux/bio.h
> +++ b/include/linux/bio.h
> @@ -516,7 +516,8 @@ int bdev_rw_virt(struct block_device *bdev, sector_t sector, void *data,
>  		size_t len, enum req_op op);
>  
>  int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
> -		unsigned mem_align_mask, unsigned len_align_mask);
> +		unsigned maxlen, unsigned mem_align_mask,
> +		unsigned len_align_mask);
>  
>  bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter);
>  void __bio_release_pages(struct bio *bio, bool mark_dirty);
> -- 
> 2.53.0
> 
> 

Re: [PATCH 02/17] block: warn on too larger integrity allocations

[Christoph Hellwig]

On Mon, Aug 31, 2026 at 11:01:36AM -0700, Darrick J. Wong wrote:
> On Mon, Aug 31, 2026 at 09:39:50AM +0300, Christoph Hellwig wrote:
> > Catch cases where upper layer bugs create larger I/Os than the
> > mempool would return even without dipping into the mempool.
> > 
> > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > ---
> >  block/bio-integrity.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/block/bio-integrity.c b/block/bio-integrity.c
> > index b23e2434d80c..d3df726e0f08 100644
> > --- a/block/bio-integrity.c
> > +++ b/block/bio-integrity.c
> > @@ -72,6 +72,7 @@ void bio_integrity_alloc_buf(struct bio *bio, gfp_t gfp, bool zero_buffer)
> >  	unsigned int len = bio_integrity_bytes(bi, bio_sectors(bio));
> >  	void *buf;
> >  
> > +	WARN_ON_ONCE(len > BLK_INTEGRITY_MAX_SIZE);
> 
> Should it fail the bio after complaining, then?

We can't really fail here as the entire path is build around continuing.
This is really a big warning for developer bugs that must not happen
for properly written callers.

Re: [PATCH 10/17] block,iomap: fix protection information verification with initial bvec offset

[Christoph Hellwig]

On Mon, Aug 31, 2026 at 10:52:22AM -0700, Darrick J. Wong wrote:
> On Mon, Aug 31, 2026 at 09:39:58AM +0300, Christoph Hellwig wrote:
> > When reconstructing a bvec_iter from an ioend for protection information
> > verification, iomap currently ignores the offset into the initial
> > bio_vec.
> > 
> > This can't happen for buffered I/O an direct I/O to user addresses, but
> 
> "...buffered I/O or a direct I/O..." ?

I'd say:

This can't happen for buffered I/O or direct I/O to user addresses

Re: [PATCH 10/17] block,iomap: fix protection information verification with initial bvec offset

[Darrick J. Wong]

On Tue, Sep 01, 2026 at 10:12:32AM +0200, Christoph Hellwig wrote:
> On Mon, Aug 31, 2026 at 10:52:22AM -0700, Darrick J. Wong wrote:
> > On Mon, Aug 31, 2026 at 09:39:58AM +0300, Christoph Hellwig wrote:
> > > When reconstructing a bvec_iter from an ioend for protection information
> > > verification, iomap currently ignores the offset into the initial
> > > bio_vec.
> > > 
> > > This can't happen for buffered I/O an direct I/O to user addresses, but
> > 
> > "...buffered I/O or a direct I/O..." ?
> 
> I'd say:
> 
> This can't happen for buffered I/O or direct I/O to user addresses

Works for me.

--D

Re: [PATCH 09/17] xfs: move PI generation into xfs_submit_zoned_bio

[Anuj Gupta]

>  xfs_open_zone_free_rcu(
> @@ -890,6 +891,8 @@ xfs_submit_zoned_bio(
>  		xfs_mark_rtg_boundary(ioend);
>  	}
>  
> +	if (ioend->io_flags & IOMAP_IOEND_INTEGRITY)
> +		fs_bio_integrity_generate(&ioend->io_bio);
>  	submit_bio(&ioend->io_bio);

It seems to me that IOMAP_IOEND_INTEGRITY might never be set here. The
zoned write iomaps in xfs_zoned_map_blocks() and
xfs_zoned_direct_write_iomap_begin() do not set IOMAP_F_INTEGRITY based
on bdev_has_integrity_csum(). Should both paths set it? I didn't have a
zoned device to confirm this so this is just from a bit of code reading
only.

This looks like a pre-existing bug from 6bbb4d96f797 ("xfs: support T10
protection information") which can be fixed with something like this?

[PATCH] xfs: set IOMAP_F_INTEGRITY for zoned writes on integrity devices

Neither xfs_zoned_map_blocks() nor xfs_zoned_direct_write_iomap_begin()
sets IOMAP_F_INTEGRITY based on bdev_has_integrity_csum(), so PI
generation is silently skipped for zoned writes on integrity-enabled
devices. Add the check in both paths.

Fixes: 6bbb4d96f797 ("xfs: support T10 protection information")
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
---
 fs/xfs/xfs_aops.c  | 2 ++
 fs/xfs/xfs_iomap.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 76918bd15ca8..f813c0150c62 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -459,6 +459,8 @@ xfs_zoned_map_blocks(
 	wpc->iomap.offset = offset;
 	wpc->iomap.length = XFS_FSB_TO_B(mp, count_fsb);
 	wpc->iomap.flags = IOMAP_F_ANON_WRITE;
+	if (bdev_has_integrity_csum(wpc->iomap.bdev))
+		wpc->iomap.flags |= IOMAP_F_INTEGRITY;
 
 	trace_xfs_zoned_map_blocks(ip, offset, wpc->iomap.length);
 	return 0;
diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
index 225c3de88d03..d7d88231d28b 100644
--- a/fs/xfs/xfs_iomap.c
+++ b/fs/xfs/xfs_iomap.c
@@ -1086,6 +1086,8 @@ xfs_zoned_direct_write_iomap_begin(
 	iomap->offset = offset;
 	iomap->length = length;
 	iomap->flags = IOMAP_F_ANON_WRITE;
+	if (bdev_has_integrity_csum(iomap->bdev))
+		iomap->flags |= IOMAP_F_INTEGRITY;
 	return 0;
 }
 
-- 

Re: [PATCH 14/17] xfs: add support for lazy direct read bounce buffering

[Anuj Gupta]

> +	/*
> +	 * Free the bio integrity data for the original bio, as we'll allocate
> +	 * ons for each sub-I/O, which could deadlock if we keep the original

s/ons/ones

> +	 * one around.
> +	 */
> +	if (bio_integrity(bio))
> +		fs_bio_integrity_free(bio);
> +
> +	/*
> +	 * Reset the bio to submit the bio to the block layer again.  Switch to
> +	 * an end_io handler that simply complets the ioend, as all verification

s/complets/completes

 

Re: [PATCH 15/17] xfs: add error injection for lazy bounce buffering

[Anuj Gupta]

On Mon, Aug 31, 2026 at 09:40:03AM +0300, Christoph Hellwig wrote:
> Add an error injection know to exercise the lazy bounce buffering

s/know/knob