[PATCH 00/13] block, ext4: allow encryption with a block size larger than the page size

[PATCH 00/13] block, ext4: allow encryption with a block size larger than the page size

[Baokun Li]

Motivation
==========

ext4 supports a block size larger than the page size (LBS), e.g. a 32k
block size on a machine with 4k pages [1].  Encryption, however, is
still refused on such a filesystem: file contents used to be encrypted
page by page in the fs-layer path, which cannot handle a folio that
covers several data units, so the checks that refuse encryption with
such a block size stayed in place.

Since commit 5acc6f649a68 ("fscrypt: Always use blk-crypto for
contents on block-based filesystems") file contents are always
encrypted through blk-crypto.  Inline encryption hardware can already
en/decrypt a data unit that spans several pages, but the fallback and
the block layer cannot carry such a data unit:

  1. blk-crypto-fallback en/decrypts a data unit with the bio_vecs of
     a single page-clamped segment, so a larger data unit fails with
     BLK_STS_INVAL;
  2. the block layer requires every bio_vec of an encrypted bio to be
     data unit aligned, and splits such a bio without rounding the
     split size down to the data unit size;
  3. iomap cuts the direct I/O bios of an encrypted file only at the
     logical block size of the device, so a bio can end in the middle
     of a data unit.

This series adds the missing pieces on all three layers, removes the
refusals, and enables large folios for encrypted files on the way.

[1]: https://lore.kernel.org/all/20251121090654.631996-1-libaokun@huaweicloud.com

Patch Overview
==============

  * Patches 1-2 (ext4, from Alberto Garcia) [2]: Refuse to set an
    encryption policy when the block size is larger than the page size,
    and reject encrypted inodes of such a filesystem.  They are the
    fix for the stable branches; patch 11 removes the refusals again
    for mainline.  They are carried along unchanged so that the series
    applies cleanly on its own and avoids adaptation conflicts with
    the stable fix.

  * Patch 3 (blk-crypto): Reject the data unit sizes that
    blk-crypto-fallback can't handle when the key is set up, instead of
    failing every I/O with BLK_STS_INVAL.

  * Patches 4-6 (blk-crypto-fallback): En/decrypt one data unit per
    crypto request, so that a data unit can span several bio_vecs, and
    support the data units that are larger than a page, whose bio_vecs
    and scatterlists come from a mempool.

  * Patch 7 (iomap): Cut the direct I/O bios of encrypted files at the
    block size, so that they cover whole data units.

  * Patches 8-10 (block): Round the split size of an encrypted bio
    down to the data unit size, warn when a bio that must be split
    cannot be split into a valid request, and drop the per-bio_vec data
    unit alignment requirement, which a data unit larger than a page
    cannot satisfy.  The contract is documented: an encrypted bio
    starts at a data unit boundary and covers whole data units.

  * Patches 11-13 (ext4): Remove the refusals, route the folios that
    cover a data unit number wraparound to the buffer_head read path,
    and enable large folios for encrypted files.

[2]: https://lore.kernel.org/r/cover.1790343958.git.berto@igalia.com

Benchmark
=========

Hardware: 96-core machine, 3.84T NVMe, 512G memory
Filesystem: ext4, encryption on the fallback path

Buffered sequential 1M writes and reads (1G, fsync) in MiB/s; medians
of 3 runs.

Base tree, 4k-block filesystem:

  job         4k plain    4k enc
  --------    --------    ------
  seqwrite        1966      1134
  seqread         1878       996

With LBS enabled, encrypted 4k-block vs 32k-block filesystem:

  job         4k enc    32k enc    32k plain    4k->32k
  --------    ------    -------    ---------    -------
  seqwrite      1044       1399         2062       +34%
  seqread        931       1793         1791       +93%

Buffered sequential writes (1G, fsync) of the encrypted 4k-block
filesystem across request sizes, base tree vs this series:

  bs      Before    After    gain
  ----    ------    -----    ----
  4k        1118     1162     +4%
  16k       1260     1756    +39%
  64k       1272     1977    +55%
  256k      1308     2081    +59%
  1M        1260     1455    +15%

Testing
=======

  * kvm-xfstests -c ext4/all -g encrypt (which includes ext4/32k) has
    been executed with no new failures.

  * Manually verified read/write, fallocate -z and punch hole on 4k,
    8k, 16k and 32k block sizes, and direct I/O with a properly
    aligned user buffer on 4k, 8k and 32k block sizes.

  * Manually verified fsx and fsstress on a 64k block size, and direct
    I/O to an encrypted file on a 64k block size with a user buffer
    backed by large folios.

Comments and questions are, as always, welcome.

Thanks,
Baokun

Alberto Garcia (2):
  ext4: refuse encryption when block size > page size
  ext4: reject encrypted inodes when block size > page size

Baokun Li (11):
  blk-crypto: reject fallback data unit sizes it can't handle
  blk-crypto: decrypt one data unit per crypto request
  blk-crypto: encrypt one data unit per crypto request
  blk-crypto: support data units larger than a page
  iomap: align the direct I/O bios of encrypted files to the block size
  block: split encrypted bios at data unit boundaries
  block: warn about bios that can't be split
  block: don't require data unit aligned bio_vecs for encrypted bios
  ext4: allow encryption with a block size larger than the page size
  ext4: send folios that cover a wraparound to the buffer_head path
  ext4: allow large folios for encrypted files

 Documentation/block/inline-encryption.rst |  13 +
 block/blk-crypto-fallback.c               | 446 +++++++++++++++++-----
 block/blk-crypto-internal.h               |   4 +-
 block/blk-crypto.c                        |  40 +-
 block/blk-merge.c                         |  48 ++-
 fs/crypto/block.c                         |   3 +
 fs/ext4/readpage.c                        |  11 +
 fs/ext4/super.c                           |  22 +-
 fs/iomap/direct-io.c                      |  10 +
 9 files changed, 462 insertions(+), 135 deletions(-)


base-commit: fd73f4a6659897191fa0d40695fe370925dd3780
-- 
2.43.7

[PATCH 01/13] ext4: refuse encryption when block size > page size

[Baokun Li]

From: Alberto Garcia <berto@igalia.com>

Encryption is not supported when a filesystem's block size is larger
than the page size, and commit 709f0f1f1bf5 ("ext4: add checks for
large folio incompatibilities when BS > PS") added a check to refuse
mounting such a filesystem.

However, it is also possible to enable the "encrypt" feature on a
filesystem that is already mounted (with tune2fs -O encrypt). Once
that happens it won't be possible to mount the filesystem again in
the future, but there is nothing preventing the user from setting an
encryption policy while the filesystem is still mounted.

Fix this by refusing to set an fscrypt context in ext4_set_context()
when the block size is larger than the page size.

Fixes: 709f0f1f1bf5 ("ext4: add checks for large folio incompatibilities when BS > PS")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Reviewed-by: Eric Biggers <ebiggers@kernel.org>
---
 fs/ext4/crypto.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/fs/ext4/crypto.c b/fs/ext4/crypto.c
index 1a0fccb084ef..318b2eaea741 100644
--- a/fs/ext4/crypto.c
+++ b/fs/ext4/crypto.c
@@ -156,6 +156,15 @@ static int ext4_set_context(struct inode *inode, const void *ctx, size_t len,
 	if (ext4_test_inode_flag(inode, EXT4_INODE_DAX))
 		return -EOPNOTSUPP;
 
+	/*
+	 * Encryption is not supported when the block size is larger
+	 * than the page size.  This is rejected at mount time by
+	 * ext4_check_large_folio(), and this check here is for the case
+	 * when the 'encrypt' feature is enabled on a mounted filesystem.
+	 */
+	if (inode->i_sb->s_blocksize > PAGE_SIZE)
+		return -EOPNOTSUPP;
+
 	res = ext4_convert_inline_data(inode);
 	if (res)
 		return res;
-- 
2.43.7

[PATCH 02/13] ext4: reject encrypted inodes when block size > page size

[Baokun Li]

From: Alberto Garcia <berto@igalia.com>

Encryption is not supported when a filesystem's block size is larger
than the page size. ext4_check_large_folio() refuses to mount a
filesystem like that if the "encrypt" feature is already enabled, and
ext4_set_context() refuses to create encrypted inodes if the feature
is enabled after mounting.

However, a corrupted or hand-crafted filesystem can still have
encrypted inodes even if the "encrypt" feature is not set, and those
inodes can be unlocked if they're using v1 policies.

Check this in __ext4_iget() and reject them as corrupted.

Fixes: 709f0f1f1bf5 ("ext4: add checks for large folio incompatibilities when BS > PS")
Suggested-by: Eric Biggers <ebiggers@kernel.org>
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Reviewed-by: Eric Biggers <ebiggers@kernel.org>
---
 fs/ext4/inode.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 26f0f9714f03..31248fef8bce 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -5671,6 +5671,12 @@ struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
 		ret = -EFSCORRUPTED;
 		goto bad_inode;
 	}
+	if (IS_ENCRYPTED(inode) && sb->s_blocksize > PAGE_SIZE) {
+		ext4_error_inode(inode, function, line, 0,
+				 "encrypted inode with block size larger than page size");
+		ret = -EFSCORRUPTED;
+		goto bad_inode;
+	}
 
 	ext4_set_inode_mapping_order(inode);
 
-- 
2.43.7

[PATCH 03/13] blk-crypto: reject fallback data unit sizes it can't handle

[Baokun Li]

blk-crypto-fallback en/decrypts a data unit that fits in one page:
the encryption and the decryption path both require the data unit
to be contained in a single page-clamped bio_vec.

However, the fallback advertises support for all data unit sizes
(modes_supported[] is 0xFFFFFFFF), and blk_crypto_start_using_key()
never checks the fallback's capabilities, so a key with a larger data
unit size is accepted, and fails with BLK_STS_INVAL on the first I/O:

  EXT4-fs warning (device vdc): ext4_end_bio:360:
	I/O error 19 writing to inode 15 starting block 70136)
  EXT4-fs (vdc): failed to convert unwritten extents to written extents
	-- potential data loss!  (inode 15, error -5)

where error 19 is BLK_STS_INVAL.  Such keys exist on filesystems whose
block size is larger than the page size, because fscrypt uses the block
size as its data unit size by default.

Fix this by advertising only the data unit sizes that the fallback
really supports, and by rejecting keys that the fallback can't handle in
blk_crypto_fallback_start_using_key(), so that the failure happens when
the key is set up instead of during I/O, and before any data is written.

Fixes: 488f6682c832 ("block: blk-crypto-fallback for Inline Encryption")
Cc: stable@vger.kernel.org
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 block/blk-crypto-fallback.c | 20 ++++++++++++++++----
 block/blk-crypto-internal.h |  4 ++--
 block/blk-crypto.c          | 14 +++++++-------
 3 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c
index 2a8f40a65158..a75daadba4c8 100644
--- a/block/blk-crypto-fallback.c
+++ b/block/blk-crypto-fallback.c
@@ -83,6 +83,9 @@ static struct workqueue_struct *blk_crypto_wq;
 static mempool_t *blk_crypto_bounce_page_pool;
 static struct bio_set enc_bio_set;
 
+/* Data unit sizes blk-crypto-fallback can en/decrypt. */
+#define BLK_CRYPTO_FALLBACK_DU_MASK	GENMASK(PAGE_SHIFT, SECTOR_SHIFT)
+
 /*
  * This is the key we set when evicting a keyslot. This *should* be the all 0's
  * key, but AES-XTS rejects that key, so we use some random bytes instead.
@@ -561,7 +564,8 @@ static int blk_crypto_fallback_init(void)
 
 	/* All blk-crypto modes have a crypto API fallback. */
 	for (i = 0; i < BLK_ENCRYPTION_MODE_MAX; i++)
-		blk_crypto_fallback_profile->modes_supported[i] = 0xFFFFFFFF;
+		blk_crypto_fallback_profile->modes_supported[i] =
+			BLK_CRYPTO_FALLBACK_DU_MASK;
 	blk_crypto_fallback_profile->modes_supported[BLK_ENCRYPTION_MODE_INVALID] = 0;
 
 	blk_crypto_wq = alloc_workqueue("blk_crypto_wq",
@@ -612,16 +616,24 @@ static int blk_crypto_fallback_init(void)
 }
 
 /*
- * Prepare blk-crypto-fallback for the specified crypto mode.
- * Returns -ENOPKG if the needed crypto API support is missing.
+ * Prepare blk-crypto-fallback for the specified crypto configuration.
+ * Returns -EOPNOTSUPP if the fallback can't handle the data unit size of the
+ * configuration, or -ENOPKG if the needed crypto API support is missing.
  */
-int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num)
+int blk_crypto_fallback_start_using_key(const struct blk_crypto_config *cfg)
 {
+	enum blk_crypto_mode_num mode_num = cfg->crypto_mode;
 	const char *cipher_str = blk_crypto_modes[mode_num].cipher_str;
 	struct blk_crypto_fallback_keyslot *slotp;
 	unsigned int i;
 	int err = 0;
 
+	if (!(BLK_CRYPTO_FALLBACK_DU_MASK & cfg->data_unit_size)) {
+		pr_warn_ratelimited("can't handle data unit size %u\n",
+				    cfg->data_unit_size);
+		return -EOPNOTSUPP;
+	}
+
 	/*
 	 * Fast path
 	 * Ensure that updates to blk_crypto_keyslots[i].tfms[mode_num]
diff --git a/block/blk-crypto-internal.h b/block/blk-crypto-internal.h
index 2c7a0446572a..640527ed05eb 100644
--- a/block/blk-crypto-internal.h
+++ b/block/blk-crypto-internal.h
@@ -219,14 +219,14 @@ bool blk_crypto_fallback_bio_prep(struct bio *bio);
 
 #ifdef CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK
 
-int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num);
+int blk_crypto_fallback_start_using_key(const struct blk_crypto_config *cfg);
 
 int blk_crypto_fallback_evict_key(const struct blk_crypto_key *key);
 
 #else /* CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK */
 
 static inline int
-blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num)
+blk_crypto_fallback_start_using_key(const struct blk_crypto_config *cfg)
 {
 	pr_warn_once("crypto API fallback is disabled\n");
 	return -ENOPKG;
diff --git a/block/blk-crypto.c b/block/blk-crypto.c
index bc3a9f59574b..5c670c3e65c5 100644
--- a/block/blk-crypto.c
+++ b/block/blk-crypto.c
@@ -392,13 +392,13 @@ bool blk_crypto_config_supported_natively(struct block_device *bdev,
  * @key: A key to use on the device
  *
  * Upper layers must call this function to ensure that either the hardware
- * supports the key's crypto settings, or the crypto API fallback has transforms
- * for the needed mode allocated and ready to go. This function may allocate
- * an skcipher, and *should not* be called from the data path, since that might
- * cause a deadlock
+ * supports the key's crypto settings, or that the crypto API fallback is ready
+ * to handle them. This function may allocate an skcipher, and *should not* be
+ * called from the data path, since that might cause a deadlock
  *
- * Return: 0 on success; -EOPNOTSUPP if the key is wrapped but the hardware does
- *	   not support wrapped keys; -ENOPKG if the key is a raw key but the
+ * Return: 0 on success; -EOPNOTSUPP if the key can't be used on @bdev, e.g. a
+ *	   wrapped key without hardware support or a data unit size that the
+ *	   fallback can't handle; -ENOPKG if the key is a raw key but the
  *	   hardware does not support raw keys and blk-crypto-fallback is either
  *	   disabled or the needed algorithm is disabled in the crypto API; or
  *	   another -errno code if something else went wrong.
@@ -412,7 +412,7 @@ int blk_crypto_start_using_key(struct block_device *bdev,
 		pr_warn_ratelimited("%pg: no support for wrapped keys\n", bdev);
 		return -EOPNOTSUPP;
 	}
-	return blk_crypto_fallback_start_using_mode(key->crypto_cfg.crypto_mode);
+	return blk_crypto_fallback_start_using_key(&key->crypto_cfg);
 }
 EXPORT_SYMBOL_GPL(blk_crypto_start_using_key);
 
-- 
2.43.7

[PATCH 04/13] blk-crypto: decrypt one data unit per crypto request

[Baokun Li]

__blk_crypto_fallback_decrypt_bio() decrypts the bio one bio_vec at a
time: it walks the bio_vecs with __bio_for_each_segment(), requires each
bio_vec to be data unit aligned and a multiple of the data unit size, and
decrypts every data unit in it with a scatterlist that covers the whole
bio_vec.

That requires a data unit to be contiguous in memory, and cannot express
a data unit that spans more than a page.  In preparation for supporting
such data units, collect the bio_vecs that make up one data unit in the
new blk_crypto_du_bvecs(), and decrypt the data unit with a single crypto
request whose scatterlist spans those bio_vecs.

Add a struct blk_crypto_du_ctx to describe one data unit, with
blk_crypto_du_ctx_init() to initialize it and blk_crypto_du_bvecs() to
fill in its bio_vecs.  The array that holds them is sized for one
bio_vec and lives on the stack, so a data unit that needs more than
one bio_vec is rejected with BLK_STS_INVAL for now.

Also add a blk_crypto_crypt_du() helper that en/decrypts one data unit.
Only the decryption path uses it in this patch; the encryption path will
be restructured the same way in a follow-up patch, with the bounce page
scatterlist as the destination.

No functional change intended.

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 block/blk-crypto-fallback.c | 162 ++++++++++++++++++++++++++++++------
 1 file changed, 135 insertions(+), 27 deletions(-)

diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c
index a75daadba4c8..e383444686fc 100644
--- a/block/blk-crypto-fallback.c
+++ b/block/blk-crypto-fallback.c
@@ -242,6 +242,116 @@ static void blk_crypto_dun_to_iv(const u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE],
 		iv->dun[i] = cpu_to_le64(dun[i]);
 }
 
+/*
+ * Describes one data unit: the bio_vecs that cover it and the
+ * scatterlists to en/decrypt it with.
+ */
+struct blk_crypto_du_ctx {
+	/* Backing store for bvecs, for the one-bio_vec case. */
+	struct bio_vec		stack_bvec;
+	/* Backing store for src_sg and dst_sg, one entry each. */
+	struct scatterlist	stack_sgs[2];
+
+	/* The data unit size in bytes. */
+	unsigned int		du_size;
+	/* Number of entries available in bvecs. */
+	unsigned int		max_bvecs;
+
+	/* The bio_vecs that cover the data unit. */
+	struct bio_vec		*bvecs;
+	/* The scatterlist for each side of the crypto request. */
+	struct scatterlist	*src_sg;
+	struct scatterlist	*dst_sg;
+
+	/* The crypto request to en/decrypt the data unit with. */
+	struct skcipher_request	*ciph_req;
+	/* The data unit's DUN, advanced on each successful crypt. */
+	u64			curr_dun[BLK_CRYPTO_DUN_ARRAY_SIZE];
+};
+
+static void blk_crypto_du_ctx_init(struct blk_crypto_du_ctx *du_ctx,
+				   struct skcipher_request *ciph_req,
+				   const struct bio_crypt_ctx *bc)
+{
+	du_ctx->ciph_req = ciph_req;
+	du_ctx->du_size = bc->bc_key->crypto_cfg.data_unit_size;
+	memcpy(du_ctx->curr_dun, bc->bc_dun, sizeof(bc->bc_dun));
+
+	du_ctx->max_bvecs = 1;
+	du_ctx->bvecs = &du_ctx->stack_bvec;
+	du_ctx->src_sg = &du_ctx->stack_sgs[0];
+	du_ctx->dst_sg = &du_ctx->stack_sgs[1];
+}
+
+/*
+ * En/decrypt one data unit: run the request over the data unit's
+ * scatterlists with its DUN as the IV, and advance the DUN on success.
+ */
+static int blk_crypto_crypt_du(struct blk_crypto_du_ctx *du_ctx, bool encrypt)
+{
+	union blk_crypto_iv iv;
+	int err;
+
+	blk_crypto_dun_to_iv(du_ctx->curr_dun, &iv);
+	skcipher_request_set_crypt(du_ctx->ciph_req,
+				   du_ctx->src_sg, du_ctx->dst_sg,
+				   du_ctx->du_size, iv.bytes);
+
+	if (encrypt)
+		err = crypto_skcipher_encrypt(du_ctx->ciph_req);
+	else
+		err = crypto_skcipher_decrypt(du_ctx->ciph_req);
+	if (err)
+		return -EIO;
+
+	bio_crypt_dun_increment(du_ctx->curr_dun, 1);
+	return 0;
+}
+
+/**
+ * blk_crypto_du_bvecs() - collect the bio_vecs that make up one data unit
+ * @bio: the bio to collect from
+ * @iter: the position in @bio to collect at.  Not modified.
+ * @du_ctx: the context to collect the data unit into.  Uses
+ *	    @du_ctx->du_size and @du_ctx->max_bvecs, and stores the
+ *	    bio_vecs in @du_ctx->bvecs
+ *
+ * Collect the bio_vecs that cover one data unit starting at @iter.  Each
+ * bio_vec is clamped to its page, so a data unit spans several bio_vecs if it
+ * covers several pages or if a bio_vec boundary splits it.
+ *
+ * Return: the number of bio_vecs collected, or 0 if @bio doesn't have a whole
+ *	   data unit at @iter, the data unit doesn't start at a
+ *	   min(@du_ctx->du_size, PAGE_SIZE) boundary, or the data unit
+ *	   needs more than @du_ctx->max_bvecs bio_vecs.
+ */
+static unsigned int blk_crypto_du_bvecs(struct bio *bio,
+					const struct bvec_iter *iter,
+					struct blk_crypto_du_ctx *du_ctx)
+{
+	const unsigned int align = min(du_ctx->du_size, PAGE_SIZE);
+	struct bvec_iter du_iter = *iter;
+	struct bio_vec bv;
+	unsigned int n = 0;
+
+	if (du_iter.bi_size < du_ctx->du_size)
+		return 0;
+
+	/* Walk a copy of the iterator that ends after one data unit. */
+	du_iter.bi_size = du_ctx->du_size;
+
+	__bio_for_each_segment(bv, bio, du_iter, du_iter) {
+		if (n == du_ctx->max_bvecs)
+			return 0;
+		if (!n && !IS_ALIGNED(bv.bv_offset, align))
+			return 0;
+
+		du_ctx->bvecs[n++] = bv;
+	}
+
+	return n;
+}
+
 static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio,
 		struct crypto_sync_skcipher *tfm)
 {
@@ -378,42 +488,40 @@ static blk_status_t __blk_crypto_fallback_decrypt_bio(struct bio *bio,
 		struct crypto_sync_skcipher *tfm)
 {
 	SYNC_SKCIPHER_REQUEST_ON_STACK(ciph_req, tfm);
-	u64 curr_dun[BLK_CRYPTO_DUN_ARRAY_SIZE];
-	union blk_crypto_iv iv;
-	struct scatterlist sg;
-	struct bio_vec bv;
-	const int data_unit_size = bc->bc_key->crypto_cfg.data_unit_size;
-	unsigned int i;
+	struct blk_crypto_du_ctx du_ctx;
+	blk_status_t status = BLK_STS_OK;
+	unsigned int i, n;
 
 	skcipher_request_set_callback(ciph_req,
 			CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
 			NULL, NULL);
+	blk_crypto_du_ctx_init(&du_ctx, ciph_req, bc);
+	/* Decrypt in place, so both sides use the same scatterlist. */
+	du_ctx.dst_sg = du_ctx.src_sg;
+
+	/* Decrypt each data unit in the bio, in place */
+	while (iter.bi_size) {
+		n = blk_crypto_du_bvecs(bio, &iter, &du_ctx);
+		if (!n) {
+			status = BLK_STS_INVAL;
+			break;
+		}
 
-	memcpy(curr_dun, bc->bc_dun, sizeof(curr_dun));
-	sg_init_table(&sg, 1);
-	skcipher_request_set_crypt(ciph_req, &sg, &sg, data_unit_size,
-				   iv.bytes);
-
-	/* Decrypt each segment in the bio */
-	__bio_for_each_segment(bv, bio, iter, iter) {
-		struct page *page = bv.bv_page;
-
-		if (!IS_ALIGNED(bv.bv_len | bv.bv_offset, data_unit_size))
-			return BLK_STS_INVAL;
-
-		sg_set_page(&sg, page, data_unit_size, bv.bv_offset);
+		sg_init_table(du_ctx.src_sg, n);
+		for (i = 0; i < n; i++)
+			sg_set_page(&du_ctx.src_sg[i], du_ctx.bvecs[i].bv_page,
+				    du_ctx.bvecs[i].bv_len,
+				    du_ctx.bvecs[i].bv_offset);
 
-		/* Decrypt each data unit in the segment */
-		for (i = 0; i < bv.bv_len; i += data_unit_size) {
-			blk_crypto_dun_to_iv(curr_dun, &iv);
-			if (crypto_skcipher_decrypt(ciph_req))
-				return BLK_STS_IOERR;
-			bio_crypt_dun_increment(curr_dun, 1);
-			sg.offset += data_unit_size;
+		if (blk_crypto_crypt_du(&du_ctx, false)) {
+			status = BLK_STS_IOERR;
+			break;
 		}
+
+		bio_advance_iter_single(bio, &iter, du_ctx.du_size);
 	}
 
-	return BLK_STS_OK;
+	return status;
 }
 
 /*
-- 
2.43.7

[PATCH 05/13] blk-crypto: encrypt one data unit per crypto request

[Baokun Li]

__blk_crypto_fallback_encrypt_bio() encrypts the source bio one bio_vec
at a time: it walks the bio with bio_iter_iovec(), which clamps each
bio_vec to a single page, encrypts all the data units of that bio_vec
into its bounce page with one crypto request per data unit, and starts a
new encrypted bio when all of its pages are in use.

That structure cannot express a data unit that spans more than a page.
In preparation for supporting such data units, restructure the loop
around the data unit, the same way as the decryption path: collect the
bio_vecs that make up one data unit in blk_crypto_du_bvecs(), encrypt it
with a single crypto request whose scatterlist covers exactly those
bio_vecs, and advance the source bio by one data unit.

Unlike the decryption path, which decrypts in place using one
scatterlist for both sides, the encryption path writes the ciphertext
to the bounce pages, so it fills the scatterlist for each side of the
crypto request, and calls the same blk_crypto_crypt_du() helper, with
the encryption direction and the bounce page scatterlist as the
destination.

A data unit is now encrypted as soon as it is complete instead of when
all the data units of its bio_vec are, and the first bio_vec of a data
unit is added to the encrypted bio with the whole length of the source
bio_vec it starts in, as the data units that follow it in that bio_vec
share its bounce page.

No functional change intended.

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 block/blk-crypto-fallback.c | 131 +++++++++++++++++++++++-------------
 1 file changed, 85 insertions(+), 46 deletions(-)

diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c
index e383444686fc..12b3ca95db26 100644
--- a/block/blk-crypto-fallback.c
+++ b/block/blk-crypto-fallback.c
@@ -356,81 +356,66 @@ static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio,
 		struct crypto_sync_skcipher *tfm)
 {
 	struct bio_crypt_ctx *bc = src_bio->bi_crypt_context;
-	int data_unit_size = bc->bc_key->crypto_cfg.data_unit_size;
 	SYNC_SKCIPHER_REQUEST_ON_STACK(ciph_req, tfm);
-	u64 curr_dun[BLK_CRYPTO_DUN_ARRAY_SIZE];
-	struct scatterlist src, dst;
-	union blk_crypto_iv iv;
-	unsigned int nr_enc_pages, enc_idx;
+	struct blk_crypto_du_ctx du_ctx;
+	unsigned int nr_enc_pages, enc_idx, nr_new_pages;
 	struct page **enc_pages;
+	struct page *cur_enc_page = NULL;
 	struct bio *enc_bio;
-	unsigned int i;
+	bool reuse_enc_page;
+	unsigned int i, n;
 
 	skcipher_request_set_callback(ciph_req,
 			CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
 			NULL, NULL);
-
-	memcpy(curr_dun, bc->bc_dun, sizeof(curr_dun));
-	sg_init_table(&src, 1);
-	sg_init_table(&dst, 1);
-
-	skcipher_request_set_crypt(ciph_req, &src, &dst, data_unit_size,
-				   iv.bytes);
+	blk_crypto_du_ctx_init(&du_ctx, ciph_req, bc);
 
 	/*
-	 * Encrypt each page in the source bio.  Because the source bio could
-	 * have bio_vecs that span more than a single page, but the encrypted
-	 * bios are limited to a single page per bio_vec, this can generate
-	 * more than a single encrypted bio per source bio.
+	 * Encrypt each data unit in the source bio.  A data unit is described by
+	 * one bio_vec per page.  Because the encrypted bios are limited to a
+	 * single page per bio_vec, this can generate more than a single
+	 * encrypted bio per source bio.
 	 */
 new_bio:
 	nr_enc_pages = min(bio_segments(src_bio), BIO_MAX_VECS);
 	enc_bio = blk_crypto_alloc_enc_bio(src_bio, nr_enc_pages, &enc_pages);
 	enc_idx = 0;
+	reuse_enc_page = false;
 	for (;;) {
 		struct bio_vec src_bv =
 			bio_iter_iovec(src_bio, src_bio->bi_iter);
-		struct page *enc_page = enc_pages[enc_idx];
 
-		if (!IS_ALIGNED(src_bv.bv_len | src_bv.bv_offset,
-				data_unit_size)) {
+		n = blk_crypto_du_bvecs(src_bio, &src_bio->bi_iter, &du_ctx);
+		if (!n) {
 			enc_bio->bi_status = BLK_STS_INVAL;
 			goto out_free_enc_bio;
 		}
 
-		__bio_add_page(enc_bio, enc_page, src_bv.bv_len,
-				src_bv.bv_offset);
-
-		sg_set_page(&src, src_bv.bv_page, data_unit_size,
-			    src_bv.bv_offset);
-		sg_set_page(&dst, enc_page, data_unit_size, src_bv.bv_offset);
-
 		/*
-		 * Increment the index now that the encrypted page is added to
-		 * the bio.  This is important for the error unwind path.
+		 * The first bio_vec of the data unit reuses the bounce page
+		 * that the previous data unit added for the bio_vec it ended
+		 * in, if any; every other bio_vec needs a bounce page of its
+		 * own.
+		 *
+		 * reuse_enc_page implies nr_new_pages == 0: a data unit that
+		 * reuses a bounce page fits in a single bio_vec, as max_bvecs
+		 * is 1 for data units up to a page (see
+		 * blk_crypto_du_ctx_init()).  A submit therefore only happens
+		 * at a segment boundary and the encrypted bio never extends
+		 * past the encrypted data.
 		 */
-		enc_idx++;
+		nr_new_pages = n - reuse_enc_page;
+		WARN_ON_ONCE(reuse_enc_page && nr_new_pages);
 
 		/*
-		 * Encrypt each data unit in this page.
+		 * Is there room for this data unit in the current encrypted bio?
+		 * (There always is in a newly allocated one.)
 		 */
-		for (i = 0; i < src_bv.bv_len; i += data_unit_size) {
-			blk_crypto_dun_to_iv(curr_dun, &iv);
-			if (crypto_skcipher_encrypt(ciph_req)) {
-				enc_bio->bi_status = BLK_STS_IOERR;
+		if (enc_idx + nr_new_pages > nr_enc_pages) {
+			if (WARN_ON_ONCE(!enc_idx)) {
+				enc_bio->bi_status = BLK_STS_INVAL;
 				goto out_free_enc_bio;
 			}
-			bio_crypt_dun_increment(curr_dun, 1);
-			src.offset += data_unit_size;
-			dst.offset += data_unit_size;
-		}
-
-		bio_advance_iter_single(src_bio, &src_bio->bi_iter,
-				src_bv.bv_len);
-		if (!src_bio->bi_iter.bi_size)
-			break;
-
-		if (enc_idx == nr_enc_pages) {
 			/*
 			 * For each additional encrypted bio submitted,
 			 * increment the source bio's remaining count.  Each
@@ -442,6 +427,60 @@ static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio,
 			submit_bio(enc_bio);
 			goto new_bio;
 		}
+
+		/*
+		 * Encrypt the data unit, scattering the ciphertext into the
+		 * bounce pages.  The bounce pages mirror the data layout of the
+		 * source bio, so that the encrypted bio can be submitted for the
+		 * same disk location as the source bio.
+		 */
+		sg_init_table(du_ctx.src_sg, n);
+		sg_init_table(du_ctx.dst_sg, n);
+		for (i = 0; i < n; i++) {
+			struct page *enc_page;
+
+			if (!i && reuse_enc_page) {
+				/* Covered by the previous bounce page. */
+				enc_page = cur_enc_page;
+			} else {
+				/*
+				 * A bio_vec that holds more than one data unit
+				 * is added with its whole length; the data
+				 * units that follow it fill in the same bounce
+				 * page before the encrypted bio is submitted.
+				 */
+				enc_page = enc_pages[enc_idx++];
+				__bio_add_page(enc_bio, enc_page, src_bv.bv_len,
+					       src_bv.bv_offset);
+				cur_enc_page = enc_page;
+			}
+
+			sg_set_page(&du_ctx.src_sg[i], du_ctx.bvecs[i].bv_page,
+				    du_ctx.bvecs[i].bv_len,
+				    du_ctx.bvecs[i].bv_offset);
+			sg_set_page(&du_ctx.dst_sg[i], enc_page,
+				    du_ctx.bvecs[i].bv_len,
+				    du_ctx.bvecs[i].bv_offset);
+		}
+
+		/*
+		 * The next data unit starts in the same bio_vec, and thus
+		 * shares the bounce page added above, unless this data unit
+		 * reaches the end of the bio_vec it is in.  A page can be
+		 * covered by several bio_vecs, so the sharing is per bio_vec,
+		 * not per page.
+		 */
+		reuse_enc_page = n == 1 && src_bv.bv_len > du_ctx.du_size;
+
+		if (blk_crypto_crypt_du(&du_ctx, true)) {
+			enc_bio->bi_status = BLK_STS_IOERR;
+			goto out_free_enc_bio;
+		}
+
+		bio_advance_iter_single(src_bio, &src_bio->bi_iter,
+					du_ctx.du_size);
+		if (!src_bio->bi_iter.bi_size)
+			break;
 	}
 
 	submit_bio(enc_bio);
-- 
2.43.7

[PATCH 06/13] blk-crypto: support data units larger than a page

[Baokun Li]

blk-crypto-fallback requires each data unit to be contained in a single
bio_vec, and fails a bio with a larger data unit with BLK_STS_INVAL.
fscrypt uses the filesystem block size as its data unit size, so
encrypted files are unusable on filesystems with a block size larger
than the page size; the write failure is silent and loses the data.

Since commit 5acc6f649a68 ("fscrypt: Always use blk-crypto for
contents on block-based filesystems") there is no fs-layer path left.
Fix this by describing a data unit by one bio_vec per page: data units
that fit keep using the bio_vec and the scatterlists in the data unit
context, and larger ones get arrays from the new blk_crypto_du_ctx_pool.

The pool is a mempool backed by a dedicated slab cache, as kmalloc()
would round the element size up to the next power of two.  It is set
up when a key with such a data unit size is first used, and one array
is reused for all data units of a bio.

A data unit must start at a data unit boundary (a page boundary for a
larger one), and the accepted data unit sizes grow to the number of
pages that fit in one encrypted bio, rounded down to a power of two.
An encrypted bio can run out of room before the source bio ends; free
its unused bounce pages with the new blk_crypto_free_enc_pages().

One crypto request per data unit is kept, so the ciphertext of data
units that fit in a page is unchanged, and larger data units produce
the same ciphertext as inline encryption hardware.

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 block/blk-crypto-fallback.c | 171 +++++++++++++++++++++++++++++-------
 1 file changed, 140 insertions(+), 31 deletions(-)

diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c
index 12b3ca95db26..b1b71d1938d1 100644
--- a/block/blk-crypto-fallback.c
+++ b/block/blk-crypto-fallback.c
@@ -81,10 +81,40 @@ static struct blk_crypto_fallback_keyslot {
 static struct blk_crypto_profile *blk_crypto_fallback_profile;
 static struct workqueue_struct *blk_crypto_wq;
 static mempool_t *blk_crypto_bounce_page_pool;
+static struct kmem_cache *blk_crypto_du_ctx_cache;
+static mempool_t *blk_crypto_du_ctx_pool;
 static struct bio_set enc_bio_set;
 
-/* Data unit sizes blk-crypto-fallback can en/decrypt. */
-#define BLK_CRYPTO_FALLBACK_DU_MASK	GENMASK(PAGE_SHIFT, SECTOR_SHIFT)
+/*
+ * The maximum number of pages a data unit may span: the largest power of
+ * two that fits in one encrypted bio.  An encrypted bio holds at most
+ * BIO_MAX_VECS bio_vecs, and a data unit can need one more bio_vec than
+ * the pages it spans when a bio_vec boundary splits it.
+ */
+#define BLK_CRYPTO_DU_MAX_PAGES		BIT(ilog2(BIO_MAX_VECS - 1))
+
+/*
+ * Data unit sizes that blk-crypto-fallback can en/decrypt: at least a
+ * sector, and small enough that the bio_vecs of a data unit fit in one
+ * encrypted bio.
+ */
+#define BLK_CRYPTO_FALLBACK_DU_MASK \
+	GENMASK(ilog2(BLK_CRYPTO_DU_MAX_PAGES * PAGE_SIZE), SECTOR_SHIFT)
+
+/*
+ * Number of scratch buffers to keep around: mempool_alloc() only uses them
+ * when the page allocator fails, so this is the number of bios that can
+ * still make progress when memory is short.
+ */
+#define BLK_CRYPTO_DU_CTX_POOL_SIZE	8
+
+/*
+ * Size of one blk_crypto_du_ctx_pool allocation: the bio_vecs of the largest
+ * data unit, and a scatterlist for each side of the crypto request.
+ */
+#define BLK_CRYPTO_DU_CTX_SIZE \
+	((BLK_CRYPTO_DU_MAX_PAGES + 1) * \
+	(sizeof(struct bio_vec) + 2 * sizeof(struct scatterlist)))
 
 /*
  * This is the key we set when evicting a keyslot. This *should* be the all 0's
@@ -144,6 +174,17 @@ static const struct blk_crypto_ll_ops blk_crypto_fallback_ll_ops = {
 	.keyslot_evict          = blk_crypto_fallback_keyslot_evict,
 };
 
+static void blk_crypto_free_enc_pages(struct page **pages, unsigned int nr)
+{
+	unsigned int i;
+
+	if (!nr)
+		return;
+	i = mempool_free_bulk(blk_crypto_bounce_page_pool, (void **)pages, nr);
+	if (i < nr)
+		release_pages(pages + i, nr - i);
+}
+
 static void blk_crypto_fallback_encrypt_endio(struct bio *enc_bio)
 {
 	struct bio *src_bio = enc_bio->bi_private;
@@ -158,10 +199,7 @@ static void blk_crypto_fallback_encrypt_endio(struct bio *enc_bio)
 	bio_for_each_bvec_all(bv, enc_bio, i)
 		pages[i] = bv->bv_page;
 
-	i = mempool_free_bulk(blk_crypto_bounce_page_pool, (void **)pages,
-			enc_bio->bi_vcnt);
-	if (i < enc_bio->bi_vcnt)
-		release_pages(pages + i, enc_bio->bi_vcnt - i);
+	blk_crypto_free_enc_pages(pages, enc_bio->bi_vcnt);
 
 	if (enc_bio->bi_status)
 		cmpxchg(&src_bio->bi_status, 0, enc_bio->bi_status);
@@ -251,6 +289,8 @@ struct blk_crypto_du_ctx {
 	struct bio_vec		stack_bvec;
 	/* Backing store for src_sg and dst_sg, one entry each. */
 	struct scatterlist	stack_sgs[2];
+	/* Backing store for bvecs and the scatterlists, if not on the stack. */
+	void			*pool_buf;
 
 	/* The data unit size in bytes. */
 	unsigned int		du_size;
@@ -269,6 +309,11 @@ struct blk_crypto_du_ctx {
 	u64			curr_dun[BLK_CRYPTO_DUN_ARRAY_SIZE];
 };
 
+/*
+ * Prepare @du_ctx for the data unit of @bc's key.  A data unit that needs
+ * more than one bio_vec gets its arrays from blk_crypto_du_ctx_pool instead
+ * of the ones in @du_ctx; that never fails, as the pool is a mempool.
+ */
 static void blk_crypto_du_ctx_init(struct blk_crypto_du_ctx *du_ctx,
 				   struct skcipher_request *ciph_req,
 				   const struct bio_crypt_ctx *bc)
@@ -277,10 +322,27 @@ static void blk_crypto_du_ctx_init(struct blk_crypto_du_ctx *du_ctx,
 	du_ctx->du_size = bc->bc_key->crypto_cfg.data_unit_size;
 	memcpy(du_ctx->curr_dun, bc->bc_dun, sizeof(bc->bc_dun));
 
-	du_ctx->max_bvecs = 1;
-	du_ctx->bvecs = &du_ctx->stack_bvec;
-	du_ctx->src_sg = &du_ctx->stack_sgs[0];
-	du_ctx->dst_sg = &du_ctx->stack_sgs[1];
+	if (du_ctx->du_size <= PAGE_SIZE) {
+		du_ctx->max_bvecs = 1;
+		du_ctx->bvecs = &du_ctx->stack_bvec;
+		du_ctx->src_sg = &du_ctx->stack_sgs[0];
+		du_ctx->dst_sg = &du_ctx->stack_sgs[1];
+		du_ctx->pool_buf = NULL;
+	} else {
+		/* blk_crypto_fallback_bio_prep() checks the pool is set up. */
+		du_ctx->max_bvecs = PFN_UP(du_ctx->du_size) + 1;
+		du_ctx->pool_buf = mempool_alloc(blk_crypto_du_ctx_pool,
+						 GFP_NOIO);
+		du_ctx->bvecs = du_ctx->pool_buf;
+		du_ctx->src_sg = du_ctx->pool_buf +
+				 du_ctx->max_bvecs * sizeof(*du_ctx->bvecs);
+		du_ctx->dst_sg = du_ctx->src_sg + du_ctx->max_bvecs;
+	}
+}
+
+static void blk_crypto_du_ctx_exit(struct blk_crypto_du_ctx *du_ctx)
+{
+	mempool_free(du_ctx->pool_buf, blk_crypto_du_ctx_pool);
 }
 
 /*
@@ -372,9 +434,9 @@ static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio,
 
 	/*
 	 * Encrypt each data unit in the source bio.  A data unit is described by
-	 * one bio_vec per page.  Because the encrypted bios are limited to a
-	 * single page per bio_vec, this can generate more than a single
-	 * encrypted bio per source bio.
+	 * one bio_vec per page, and may span more than a single page.  Because
+	 * the encrypted bios are limited to a single page per bio_vec, this can
+	 * generate more than a single encrypted bio per source bio.
 	 */
 new_bio:
 	nr_enc_pages = min(bio_segments(src_bio), BIO_MAX_VECS);
@@ -416,6 +478,8 @@ static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio,
 				enc_bio->bi_status = BLK_STS_INVAL;
 				goto out_free_enc_bio;
 			}
+			blk_crypto_free_enc_pages(enc_pages + enc_idx,
+						  nr_enc_pages - enc_idx);
 			/*
 			 * For each additional encrypted bio submitted,
 			 * increment the source bio's remaining count.  Each
@@ -444,14 +508,18 @@ static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio,
 				enc_page = cur_enc_page;
 			} else {
 				/*
-				 * A bio_vec that holds more than one data unit
-				 * is added with its whole length; the data
-				 * units that follow it fill in the same bounce
-				 * page before the encrypted bio is submitted.
+				 * The whole bio_vec is added for the first
+				 * bio_vec of a data unit; the data units that
+				 * follow it fill in the same bounce page
+				 * before the encrypted bio is submitted.
 				 */
+				unsigned int len = i ? du_ctx.bvecs[i].bv_len :
+						      src_bv.bv_len;
+				unsigned int off = i ? du_ctx.bvecs[i].bv_offset :
+						      src_bv.bv_offset;
+
 				enc_page = enc_pages[enc_idx++];
-				__bio_add_page(enc_bio, enc_page, src_bv.bv_len,
-					       src_bv.bv_offset);
+				__bio_add_page(enc_bio, enc_page, len, off);
 				cur_enc_page = enc_page;
 			}
 
@@ -477,24 +545,20 @@ static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio,
 			goto out_free_enc_bio;
 		}
 
-		bio_advance_iter_single(src_bio, &src_bio->bi_iter,
-					du_ctx.du_size);
+		/* A data unit can span several bio_vecs, e.g. for direct I/O. */
+		bio_advance_iter(src_bio, &src_bio->bi_iter, du_ctx.du_size);
 		if (!src_bio->bi_iter.bi_size)
 			break;
 	}
 
+	blk_crypto_du_ctx_exit(&du_ctx);
+	blk_crypto_free_enc_pages(enc_pages + enc_idx, nr_enc_pages - enc_idx);
 	submit_bio(enc_bio);
 	return;
 
 out_free_enc_bio:
-	/*
-	 * Add the remaining pages to the bio so that the normal completion path
-	 * in blk_crypto_fallback_encrypt_endio frees them.  The exact data
-	 * layout does not matter for that, so don't bother iterating the source
-	 * bio.
-	 */
-	for (; enc_idx < nr_enc_pages; enc_idx++)
-		__bio_add_page(enc_bio, enc_pages[enc_idx], PAGE_SIZE, 0);
+	blk_crypto_du_ctx_exit(&du_ctx);
+	blk_crypto_free_enc_pages(enc_pages + enc_idx, nr_enc_pages - enc_idx);
 	bio_endio(enc_bio);
 }
 
@@ -557,9 +621,11 @@ static blk_status_t __blk_crypto_fallback_decrypt_bio(struct bio *bio,
 			break;
 		}
 
-		bio_advance_iter_single(bio, &iter, du_ctx.du_size);
+		/* A data unit can span several bio_vecs, e.g. for direct I/O. */
+		bio_advance_iter(bio, &iter, du_ctx.du_size);
 	}
 
+	blk_crypto_du_ctx_exit(&du_ctx);
 	return status;
 }
 
@@ -650,6 +716,18 @@ bool blk_crypto_fallback_bio_prep(struct bio *bio)
 		bio_endio_status(bio, BLK_STS_NOTSUPP);
 		return false;
 	}
+	if (WARN_ON_ONCE(!(BLK_CRYPTO_FALLBACK_DU_MASK &
+			   bc->bc_key->crypto_cfg.data_unit_size))) {
+		/* The key was set up for a data unit the fallback can't handle. */
+		bio_endio_status(bio, BLK_STS_NOTSUPP);
+		return false;
+	}
+	if (WARN_ON_ONCE(bc->bc_key->crypto_cfg.data_unit_size > PAGE_SIZE &&
+			 !blk_crypto_du_ctx_pool)) {
+		/* The key was set up without the scratch space it needs. */
+		bio_io_error(bio);
+		return false;
+	}
 
 	if (bio_data_dir(bio) == WRITE) {
 		blk_crypto_fallback_encrypt_bio(bio);
@@ -765,7 +843,9 @@ static int blk_crypto_fallback_init(void)
 /*
  * Prepare blk-crypto-fallback for the specified crypto configuration.
  * Returns -EOPNOTSUPP if the fallback can't handle the data unit size of the
- * configuration, or -ENOPKG if the needed crypto API support is missing.
+ * configuration, -ENOPKG if the needed crypto API support is missing, or
+ * -ENOMEM if the scratch space for a data unit larger than a page can't be
+ * allocated.
  */
 int blk_crypto_fallback_start_using_key(const struct blk_crypto_config *cfg)
 {
@@ -781,6 +861,35 @@ int blk_crypto_fallback_start_using_key(const struct blk_crypto_config *cfg)
 		return -EOPNOTSUPP;
 	}
 
+	/*
+	 * A data unit that needs more than one bio_vec, e.g. a data unit that
+	 * is larger than a page, gets its data unit context from this pool.
+	 * Set the pool up before the fast path below, which can't create it.
+	 */
+	if (cfg->data_unit_size > PAGE_SIZE) {
+		mutex_lock(&tfms_init_lock);
+		if (!blk_crypto_du_ctx_pool) {
+			blk_crypto_du_ctx_cache =
+				kmem_cache_create("blk_crypto_du_ctx",
+						  BLK_CRYPTO_DU_CTX_SIZE,
+						  0, 0, NULL);
+			if (!blk_crypto_du_ctx_cache) {
+				mutex_unlock(&tfms_init_lock);
+				return -ENOMEM;
+			}
+			blk_crypto_du_ctx_pool =
+				mempool_create_slab_pool(BLK_CRYPTO_DU_CTX_POOL_SIZE,
+							 blk_crypto_du_ctx_cache);
+			if (!blk_crypto_du_ctx_pool) {
+				kmem_cache_destroy(blk_crypto_du_ctx_cache);
+				blk_crypto_du_ctx_cache = NULL;
+				mutex_unlock(&tfms_init_lock);
+				return -ENOMEM;
+			}
+		}
+		mutex_unlock(&tfms_init_lock);
+	}
+
 	/*
 	 * Fast path
 	 * Ensure that updates to blk_crypto_keyslots[i].tfms[mode_num]
-- 
2.43.7

[PATCH 07/13] iomap: align the direct I/O bios of encrypted files to the block size

[Baokun Li]

A bio that carries an encryption context must cover a whole number of
data units: its data unit number is generated from its first data unit
(see bio_crypt_set_ctx()), and it is only split at data unit
boundaries.

iomap attaches the encryption context of an encrypted file to the bios
that it builds for direct I/O, but it only cuts them at the logical
block size of the device.  A bio ends where the bio_vecs that fit into
it end, not at a fixed number of bytes: the extraction stops when the
bio is out of bio_vecs or when the iterator is exhausted, and a
bio_vec can cover a whole folio.

A bio can end in the middle of a data unit, and a data unit that is
larger than a page makes this reachable without any misaligned input
at all: the extraction then stops in the middle of a folio, and a cut
at a page granularity inside such a data unit is not data unit aligned
even when the position, the length and the buffer are.

The fallback rejects such a bio once it reaches the block layer: one
of its bio_vecs is not data unit aligned, so a direct I/O that should
work fails.

By contrast, the equivalent check on the hardware path lives in the
splitting code, which a small single segment bio skips, so a bio that
starts in the middle of a data unit can reach the hardware without
being rejected.

Fix this by cutting the bios of an encrypted file at the block size,
like the IOMAP_DIO_FSBLOCK_ALIGNED flag does for the filesystems that
pass it.  fscrypt rejects a data unit size larger than the block size,
so a bio that is cut at block size boundaries covers a whole number of
data units.

This also makes the file systems that don't pass the flag follow the
contract, e.g. f2fs, which passes direct I/O that is not aligned to
the logical block size of the device on to iomap expecting it to fail
with -EINVAL, while iomap checks only the file position and the
length, so such a request can reach bio submission.

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 fs/iomap/direct-io.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 8b4039d16ce8..9b6cdf04dced 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -410,6 +410,16 @@ static inline unsigned int iomap_dio_alignment(struct inode *inode,
 {
 	if (dio_flags & IOMAP_DIO_FSBLOCK_ALIGNED)
 		return i_blocksize(inode);
+
+	/*
+	 * A bio of an encrypted file must cover a whole number of data
+	 * units.  The data unit size of an encryption policy never exceeds
+	 * the block size, so cutting the bios at block size boundaries is
+	 * enough.
+	 */
+	if (IS_ENCRYPTED(inode))
+		return i_blocksize(inode);
+
 	return bdev_logical_block_size(bdev);
 }
 
-- 
2.43.7

[PATCH 08/13] block: split encrypted bios at data unit boundaries

[Baokun Li]

Splitting a bio advances the remaining bio's data unit number by the
number of whole data units that were split off:

  bio_split()
   └─ bio_advance()
      └─ bio_crypt_advance()
         └─ bio_crypt_dun_increment(dun, bytes >> data_unit_size_bits)

The shift truncates, so splitting inside a data unit leaves the
remainder with the wrong data unit number and silently corrupts data.

The split size is already rounded down to the logical block size, so
this is only possible when the data unit size is larger than it, e.g.
a fscrypt filesystem with a 4K block size on a device with a 512-byte
logical block size.

Fix this by rounding the split size down to a data unit boundary.  The
split alignment becomes the maximum of the logical block size, the
data unit size, and the zone write granularity, so a zoned write still
meets all three requirements.

Split bios can be up to one data unit smaller than the queue limits
allow, so a bio can be split a little more often.  The number of
segments for the split bio is still counted up to the size before the
rounding, so it can be larger than needed; it is only used for
accounting.

Fixes: a892c8d52c02 ("block: Inline encryption support for blk-mq")
Cc: stable@vger.kernel.org
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 block/blk-merge.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/block/blk-merge.c b/block/blk-merge.c
index 258a726071d1..f92460c283e7 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -308,9 +308,20 @@ static bool bvec_split_segs(const struct queue_limits *lim,
 static unsigned int bio_split_alignment(struct bio *bio,
 		const struct queue_limits *lim)
 {
+	const struct bio_crypt_ctx *bc = bio_crypt_ctx(bio);
+	unsigned int align = lim->logical_block_size;
+
+	/*
+	 * A split advances the remaining bio's data unit number by whole
+	 * data units only (see bio_crypt_advance()), so a split inside a
+	 * data unit would make it use the wrong data unit numbers.
+	 */
+	if (bc)
+		align = max(align, bc->bc_key->crypto_cfg.data_unit_size);
 	if (op_is_write(bio_op(bio)) && lim->zone_write_granularity)
-		return lim->zone_write_granularity;
-	return lim->logical_block_size;
+		align = max(align, lim->zone_write_granularity);
+
+	return align;
 }
 
 static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,
-- 
2.43.7

[PATCH 09/13] block: warn about bios that can't be split

[Baokun Li]

bio_split_io_at() fails a bio that it can't split into a valid request
with -EINVAL: a bio may have too many discontiguous segments for the
queue, may have virtual boundary gaps that leave it without a block
sized split, and now that the split size is rounded down to the data
unit size of an encryption context, it may have a data unit that does
not fit in the room that the queue limits leave for the split.

Nothing else says which queue limit caused the failure, and a limit that
is smaller than the data unit size, e.g. a max_sectors_kb that was
lowered below it, is accepted by the block layer, so warn about it, with
the bytes that fit and the alignment involved.

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 block/blk-merge.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/block/blk-merge.c b/block/blk-merge.c
index f92460c283e7..9bcc9c52eea1 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -349,6 +349,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
 	struct bio_crypt_ctx *bc = bio_crypt_ctx(bio);
 	struct bio_vec bv, bvprv, *bvprvp = NULL;
 	unsigned nsegs = 0, bytes = 0, gaps = 0;
+	unsigned int align, split_bytes;
 	struct bvec_iter iter;
 	unsigned start_align_mask = lim->dma_alignment;
 
@@ -409,14 +410,20 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
 	 * we do not use the full hardware limits.
 	 *
 	 * It is possible to submit a bio that can't be split into a valid io:
-	 * there may either be too many discontiguous vectors for the max
-	 * segments limit, or contain virtual boundary gaps without having a
-	 * valid block sized split. A zero byte result means one of those
-	 * conditions occured.
+	 * it may have too many discontiguous vectors for the max segments
+	 * limit, contain virtual boundary gaps without having a valid block
+	 * sized split, or have a crypto context whose data units are larger
+	 * than the number of bytes that fit. A zero byte result means one of
+	 * those conditions occurred.
 	 */
-	bytes = ALIGN_DOWN(bytes, bio_split_alignment(bio, lim));
-	if (!bytes)
+	align = bio_split_alignment(bio, lim);
+	split_bytes = ALIGN_DOWN(bytes, align);
+	if (!split_bytes) {
+		pr_warn_ratelimited("%pg: split needs %u align, %u bytes fit\n",
+				    bio->bi_bdev, align, bytes);
 		return -EINVAL;
+	}
+	bytes = split_bytes;
 
 	/*
 	 * Bio splitting may cause subtle trouble such as hang when doing sync
-- 
2.43.7

[PATCH 10/13] block: don't require data unit aligned bio_vecs for encrypted bios

[Baokun Li]

bio_split_io_at() rejects an encrypted bio unless every bio_vec is data
unit aligned and a multiple of the data unit size, so that every split
lands on a data unit boundary.  That is stricter than needed:
bio_split_alignment() already rounds the split size down to the data
unit size, which is all a split needs.

The requirement can't be met by all submitters either: a direct I/O
read or write adds the pages of the user buffer as they are extracted,
and fscrypt_zeroout_range() always adds one page per bio_vec, so a data
unit larger than a page is described by bio_vecs that are smaller than
it.

Drop the requirement: the split size rounding keeps the data unit
numbers of the remainder correct.  A bio whose first data unit doesn't
fit in the split still fails with -EINVAL, as the rounded down size is
zero then.

The dropped requirement was only ever a proxy for the invariant that
the submitter has to keep: the bio must start at a data unit boundary
and cover a whole number of data units.  Document that in
bio_crypt_set_ctx() and in the block API documentation.

For safety, warn about a misaligned start in fscrypt_set_bio_crypt_ctx()
and reject bios whose size is not a whole number of data units in
blk_crypto_submit_bio().

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 Documentation/block/inline-encryption.rst | 13 ++++++++++++
 block/blk-crypto.c                        | 26 +++++++++++++++++++++++
 block/blk-merge.c                         | 14 ++++++------
 fs/crypto/block.c                         |  3 +++
 4 files changed, 48 insertions(+), 8 deletions(-)

diff --git a/Documentation/block/inline-encryption.rst b/Documentation/block/inline-encryption.rst
index 0df964507f76..b2d6598ddfb6 100644
--- a/Documentation/block/inline-encryption.rst
+++ b/Documentation/block/inline-encryption.rst
@@ -198,6 +198,19 @@ it to a bio, given the blk_crypto_key and the data unit number that will be used
 for en/decryption.  Users don't need to worry about freeing the bio_crypt_ctx
 later, as that happens automatically when the bio is freed or reset.
 
+Users must keep such a bio aligned to the data unit size: the bio must start at
+a data unit boundary and be a whole number of data units long, since the data
+unit number of a bio describes its first data unit.  ``blk_crypto_submit_bio()``
+rejects a bio whose size is not a whole number of data units, and the block layer
+splits and merges encrypted bios only at data unit boundaries, but a layer that
+splits bios itself, e.g. device-mapper, must keep the alignment itself.
+
+Splitting a bio needs at least one whole data unit of room, so a queue limit
+that leaves less than a data unit for a split, e.g. when ``max_sectors_kb`` is
+lowered below the data unit size used on the device, fails the bio with
+``BLK_STS_INVAL``.  Devices that advertise a data unit size must therefore be
+able to issue requests of at least that size.
+
 To submit a bio that uses inline encryption, users must call
 ``blk_crypto_submit_bio()`` instead of the usual ``submit_bio()``.  This will
 submit the bio to the underlying driver if it supports inline crypto, or else
diff --git a/block/blk-crypto.c b/block/blk-crypto.c
index 5c670c3e65c5..d79c9042ec52 100644
--- a/block/blk-crypto.c
+++ b/block/blk-crypto.c
@@ -98,6 +98,25 @@ static int __init bio_crypt_ctx_init(void)
 }
 subsys_initcall(bio_crypt_ctx_init);
 
+/**
+ * bio_crypt_set_ctx() - attach an encryption context to a bio
+ * @bio: the bio to attach the context to
+ * @key: the key that the bio will be en/decrypted with
+ * @dun: the data unit number of the bio's first data unit
+ * @gfp_mask: memory allocation flags
+ *
+ * The caller must keep the bio aligned to the data unit size of @key: it must
+ * start at a data unit boundary and be a whole number of data units long,
+ * since the data unit number of a bio describes its first data unit.
+ *
+ * blk_crypto_submit_bio() rejects a bio whose size is not a whole number
+ * of data units, and the block layer splits and merges encrypted bios
+ * only at data unit boundaries; a layer that splits bios itself, e.g.
+ * device-mapper, must keep the alignment itself.
+ *
+ * The bio must be submitted with blk_crypto_submit_bio(), after
+ * blk_crypto_start_using_key() was called for @key on the block device.
+ */
 void bio_crypt_set_ctx(struct bio *bio, const struct blk_crypto_key *key,
 		       const u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE], gfp_t gfp_mask)
 {
@@ -260,6 +279,13 @@ bool __blk_crypto_submit_bio(struct bio *bio)
 		return false;
 	}
 
+	/* Error if the bio doesn't cover a whole number of data units. */
+	if (WARN_ON_ONCE(!IS_ALIGNED(bio->bi_iter.bi_size,
+				     bc_key->crypto_cfg.data_unit_size))) {
+		bio_endio_status(bio, BLK_STS_INVAL);
+		return false;
+	}
+
 	/*
 	 * If the device does not natively support the encryption context, try to use
 	 * the fallback if available.
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 9bcc9c52eea1..dd9be4c04041 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -346,20 +346,18 @@ static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,
 int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
 		unsigned *segs, unsigned max_bytes, unsigned len_align_mask)
 {
-	struct bio_crypt_ctx *bc = bio_crypt_ctx(bio);
 	struct bio_vec bv, bvprv, *bvprvp = NULL;
 	unsigned nsegs = 0, bytes = 0, gaps = 0;
 	unsigned int align, split_bytes;
 	struct bvec_iter iter;
-	unsigned start_align_mask = lim->dma_alignment;
-
-	if (bc) {
-		start_align_mask |= (bc->bc_key->crypto_cfg.data_unit_size - 1);
-		len_align_mask |= (bc->bc_key->crypto_cfg.data_unit_size - 1);
-	}
 
 	bio_for_each_bvec(bv, bio, iter) {
-		if (bv.bv_offset & start_align_mask ||
+		/*
+		 * The bio_vecs of a bio with a crypto context don't have to
+		 * be data unit aligned, only the bio as a whole does; see
+		 * bio_crypt_set_ctx() and bio_split_alignment().
+		 */
+		if (bv.bv_offset & lim->dma_alignment ||
 		    bv.bv_len & len_align_mask)
 			return -EINVAL;
 
diff --git a/fs/crypto/block.c b/fs/crypto/block.c
index 5193f8ba3ee0..2040f63eb9a1 100644
--- a/fs/crypto/block.c
+++ b/fs/crypto/block.c
@@ -234,6 +234,9 @@ void fscrypt_set_bio_crypt_ctx(struct bio *bio, const struct inode *inode,
 		return;
 	ci = fscrypt_get_inode_info_raw(inode);
 
+	/* The data unit number of a bio describes its first data unit. */
+	WARN_ON_ONCE(!IS_ALIGNED(pos, 1U << ci->ci_data_unit_bits));
+
 	fscrypt_generate_dun(ci, pos, dun);
 	bio_crypt_set_ctx(bio, ci->ci_enc_key.blk_key, dun, gfp_mask);
 }
-- 
2.43.7

[PATCH 11/13] ext4: allow encryption with a block size larger than the page size

[Baokun Li]

ext4 refuses encryption when the block size is larger than the page
size, both at mount time and on a mounted filesystem, and refuses to
load an existing encrypted inode.  The checks exist because such a
block size needs folios that cover a whole block, while file contents
were encrypted page by page in the fs-layer path, which cannot handle a
folio that covers several data units.

Since commit 5acc6f649a68 ("fscrypt: Always use blk-crypto for contents
on block-based filesystems") file contents are always encrypted through
blk-crypto, and after the adaptations in the preceding patches, both
inline encryption hardware and blk-crypto-fallback support data units
larger than a page, so the checks can be removed.

Encrypted filesystems still keep their folio order at s_min_folio_order;
allowing large folios needs additional work and will be enabled later.

Performance:

Hardware: 96-core machine, 3.84T NVMe, 512G memory
Filesystem: ext4, encryption on the fallback path

Buffered sequential 1M writes and reads (1G, fsync) in MiB/s; medians
of 3 runs.

  job          4k plain      4k enc    32k plain    32k enc    4k->32k
  ---------    --------    --------    ---------    -------    -------
  seqwrite         1981        1044         2062       1399       +34%
  seqread          1139         931         1791       1793       +93%

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 fs/ext4/crypto.c |  9 ---------
 fs/ext4/inode.c  |  6 ------
 fs/ext4/super.c  | 25 +++----------------------
 3 files changed, 3 insertions(+), 37 deletions(-)

diff --git a/fs/ext4/crypto.c b/fs/ext4/crypto.c
index 318b2eaea741..1a0fccb084ef 100644
--- a/fs/ext4/crypto.c
+++ b/fs/ext4/crypto.c
@@ -156,15 +156,6 @@ static int ext4_set_context(struct inode *inode, const void *ctx, size_t len,
 	if (ext4_test_inode_flag(inode, EXT4_INODE_DAX))
 		return -EOPNOTSUPP;
 
-	/*
-	 * Encryption is not supported when the block size is larger
-	 * than the page size.  This is rejected at mount time by
-	 * ext4_check_large_folio(), and this check here is for the case
-	 * when the 'encrypt' feature is enabled on a mounted filesystem.
-	 */
-	if (inode->i_sb->s_blocksize > PAGE_SIZE)
-		return -EOPNOTSUPP;
-
 	res = ext4_convert_inline_data(inode);
 	if (res)
 		return res;
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 31248fef8bce..26f0f9714f03 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -5671,12 +5671,6 @@ struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
 		ret = -EFSCORRUPTED;
 		goto bad_inode;
 	}
-	if (IS_ENCRYPTED(inode) && sb->s_blocksize > PAGE_SIZE) {
-		ext4_error_inode(inode, function, line, 0,
-				 "encrypted inode with block size larger than page size");
-		ret = -EFSCORRUPTED;
-		goto bad_inode;
-	}
 
 	ext4_set_inode_mapping_order(inode);
 
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index bca0dc87d0b7..9cd5b7445a27 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -5116,30 +5116,13 @@ static void ext4_set_max_mapping_order(struct super_block *sb)
 {
 	struct ext4_sb_info *sbi = EXT4_SB(sb);
 
-	if (test_opt(sb, DATA_FLAGS) == EXT4_MOUNT_JOURNAL_DATA)
+	if (test_opt(sb, DATA_FLAGS) == EXT4_MOUNT_JOURNAL_DATA ||
+	    ext4_has_feature_encrypt(sb))
 		sbi->s_max_folio_order = sbi->s_min_folio_order;
 	else
 		sbi->s_max_folio_order = EXT4_MAX_PAGECACHE_ORDER(sb);
 }
 
-static int ext4_check_large_folio(struct super_block *sb)
-{
-	const char *err_str = NULL;
-
-	if (ext4_has_feature_encrypt(sb))
-		err_str = "encrypt";
-
-	if (!err_str) {
-		ext4_set_max_mapping_order(sb);
-	} else if (sb->s_blocksize > PAGE_SIZE) {
-		ext4_msg(sb, KERN_ERR, "bs(%lu) > ps(%lu) unsupported for %s",
-			 sb->s_blocksize, PAGE_SIZE, err_str);
-		return -EINVAL;
-	}
-
-	return 0;
-}
-
 static int ext4_load_super(struct super_block *sb, ext4_fsblk_t *lsb,
 			   int silent)
 {
@@ -5418,9 +5401,7 @@ static int __ext4_fill_super(struct fs_context *fc, struct super_block *sb)
 
 	ext4_apply_options(fc, sb);
 
-	err = ext4_check_large_folio(sb);
-	if (err < 0)
-		goto failed_mount;
+	ext4_set_max_mapping_order(sb);
 
 	err = ext4_encoding_init(sb, es);
 	if (err)
-- 
2.43.7

[PATCH 12/13] ext4: send folios that cover a wraparound to the buffer_head path

[Baokun Li]

ext4_mpage_readpages() puts a whole folio into a bio with a single
crypt context, taken from the position of the folio, and blk-crypto
increments the data unit number for each data unit that follows in
the bio.

By contrast, fscrypt truncates the data unit numbers to 32 bits for
IV_INO_LBLK_32, the only supported IV generation method whose data
unit numbers wrap around within a file.

The blocks after a wraparound inside a folio are then en/decrypted
with IVs of 2^32 + i instead of i.  fscrypt_mergeable_bio() only runs
between folios, so nothing catches this.

This is reachable even without a block size larger than the page
size: when a filesystem is mounted without the encrypt feature,
ext4_set_max_mapping_order() does not cap its folio order, and
enabling the feature on the mounted filesystem does not change that.

The encrypted files of a policy that is set up afterwards then get
large folios, which can cover the data unit number wraparound of the
file.

Fix this by sending folios that cover a wraparound to the buffer_head
based path in ext4_mpage_readpages(), which creates one bio per block:
fscrypt_limit_io_blocks() tells whether the wraparound falls inside
the folio's blocks, and only the folio covering it takes the slower
path.

At most one folio per file is affected: the extra cost is a
buffer_head and a bio per block for that folio, and the block layer
merges those bios back together.

Fixes: 709f0f1f1bf5 ("ext4: add checks for large folio incompatibilities when BS > PS")
Cc: stable@vger.kernel.org # v6.19+
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 fs/ext4/readpage.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c
index c7b6cdb2e124..0c0dfaac6ded 100644
--- a/fs/ext4/readpage.c
+++ b/fs/ext4/readpage.c
@@ -264,6 +264,17 @@ static int ext4_mpage_readpages(struct inode *inode, struct fsverity_info *vi,
 			folio_set_mappedtodisk(folio);
 		}
 
+		/*
+		 * A bio's data unit numbers are derived from its first data
+		 * unit, so a folio that covers a data unit number wraparound
+		 * would be encrypted with the wrong IVs.  Only IV_INO_LBLK_32
+		 * can wrap; route such folios to the buffer_head based path,
+		 * which creates a bio per block.
+		 */
+		if (fscrypt_limit_io_blocks(inode, pos >> blkbits,
+					    first_hole) < first_hole)
+			goto confused;
+
 		/*
 		 * This folio will go to BIO.  Do we need to send this
 		 * BIO off first?
-- 
2.43.7

[PATCH 13/13] ext4: allow large folios for encrypted files

[Baokun Li]

Encrypted filesystems keep their folio order at s_min_folio_order,
because the read path could not handle a folio that covers a data
unit number wraparound.  The read path now sends such folios to the
buffer_head based path, so drop the restriction and give large folios
to all encrypted files, including the ones whose block size is not
larger than the page size.

Writes and direct I/O don't need the same treatment: ext4_writepages()
checks fscrypt_mergeable_bio() for every block, so a write bio never
spans a wraparound, and the bios of direct I/O are cut at the block
size.

Performance:

Hardware: 96-core machine, 3.84T NVMe, 512G memory
Filesystem: ext4 with a 4k block size, encryption on the fallback path

Buffered sequential writes (1G, fsync) in MiB/s; medians of 3 runs.

  bs      Before     After    gain
  ----    ------    ------    ----
  4k        1118      1162     +4%
  16k       1260      1756    +39%
  64k       1272      1977    +55%
  256k      1308      2081    +59%
  1M        1260      1455    +15%

Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
---
 fs/ext4/super.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 9cd5b7445a27..a788613a53a4 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -5116,8 +5116,7 @@ static void ext4_set_max_mapping_order(struct super_block *sb)
 {
 	struct ext4_sb_info *sbi = EXT4_SB(sb);
 
-	if (test_opt(sb, DATA_FLAGS) == EXT4_MOUNT_JOURNAL_DATA ||
-	    ext4_has_feature_encrypt(sb))
+	if (test_opt(sb, DATA_FLAGS) == EXT4_MOUNT_JOURNAL_DATA)
 		sbi->s_max_folio_order = sbi->s_min_folio_order;
 	else
 		sbi->s_max_folio_order = EXT4_MAX_PAGECACHE_ORDER(sb);
-- 
2.43.7

Re: [PATCH 02/13] ext4: reject encrypted inodes when block size > page size

[sashiko-bot]

> From: Alberto Garcia <berto@igalia.com>
> 
> Encryption is not supported when a filesystem's block size is larger
> than the page size. ext4_check_large_folio() refuses to mount a
> filesystem like that if the "encrypt" feature is already enabled, and
> ext4_set_context() refuses to create encrypted inodes if the feature
> is enabled after mounting.
> 
> However, a corrupted or hand-crafted filesystem can still have
> encrypted inodes even if the "encrypt" feature is not set, and those
> inodes can be unlocked if they're using v1 policies.
> 
> Check this in __ext4_iget() and reject them as corrupted.
> 
> Fixes: 709f0f1f1bf5 ("ext4: add checks for large folio incompatibilities when BS > PS")
> Suggested-by: Eric Biggers <ebiggers@kernel.org>
> Cc: stable@vger.kernel.org # v6.19+
> Signed-off-by: Alberto Garcia <berto@igalia.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=2

Re: [PATCH 09/13] block: warn about bios that can't be split

[sashiko-bot]

> bio_split_io_at() fails a bio that it can't split into a valid request
> with -EINVAL: a bio may have too many discontiguous segments for the
> queue, may have virtual boundary gaps that leave it without a block
> sized split, and now that the split size is rounded down to the data
> unit size of an encryption context, it may have a data unit that does
> not fit in the room that the queue limits leave for the split.
> 
> Nothing else says which queue limit caused the failure, and a limit that
> is smaller than the data unit size, e.g. a max_sectors_kb that was
> lowered below it, is accepted by the block layer, so warn about it, with
> the bytes that fit and the alignment involved.
> 
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=9

Re: [PATCH 04/13] blk-crypto: decrypt one data unit per crypto request

[sashiko-bot]

> __blk_crypto_fallback_decrypt_bio() decrypts the bio one bio_vec at a
> time: it walks the bio_vecs with __bio_for_each_segment(), requires each
> bio_vec to be data unit aligned and a multiple of the data unit size, and
> decrypts every data unit in it with a scatterlist that covers the whole
> bio_vec.
> 
> That requires a data unit to be contiguous in memory, and cannot express
> a data unit that spans more than a page.  In preparation for supporting
> such data units, collect the bio_vecs that make up one data unit in the
> new blk_crypto_du_bvecs(), and decrypt the data unit with a single crypto
> request whose scatterlist spans those bio_vecs.
> 
> Add a struct blk_crypto_du_ctx to describe one data unit, with
> blk_crypto_du_ctx_init() to initialize it and blk_crypto_du_bvecs() to
> fill in its bio_vecs.  The array that holds them is sized for one
> bio_vec and lives on the stack, so a data unit that needs more than
> one bio_vec is rejected with BLK_STS_INVAL for now.
> 
> Also add a blk_crypto_crypt_du() helper that en/decrypts one data unit.
> Only the decryption path uses it in this patch; the encryption path will
> be restructured the same way in a follow-up patch, with the bounce page
> scatterlist as the destination.
> 
> No functional change intended.
> 
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=4

Re: [PATCH 03/13] blk-crypto: reject fallback data unit sizes it can't handle

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] block: NULL pointer deref in blk_crypto_fallback_evict_key() on setup failure
--

commit 9e7475fafabf0ae30868c1ceef29977b6846a713
Author: Baokun Li <libaokun@linux.alibaba.com>

    blk-crypto: reject fallback data unit sizes it can't handle

This patch restricts the data unit sizes advertised by blk-crypto-fallback
to only those it actually supports. It also rejects unsupported data unit
sizes early when setting up the key, preventing failures during I/O.

> diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c
> index 2a8f40a65158b..a75daadba4c86 100644
> --- a/block/blk-crypto-fallback.c
> +++ b/block/blk-crypto-fallback.c

[ ... ]

> @@ -612,16 +616,24 @@ static int blk_crypto_fallback_init(void)
>  }
>  
>  /*
> - * Prepare blk-crypto-fallback for the specified crypto mode.
> - * Returns -ENOPKG if the needed crypto API support is missing.
> + * Prepare blk-crypto-fallback for the specified crypto configuration.
> + * Returns -EOPNOTSUPP if the fallback can't handle the data unit size of the
> + * configuration, or -ENOPKG if the needed crypto API support is missing.
>   */
> -int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num)
> +int blk_crypto_fallback_start_using_key(const struct blk_crypto_config *cfg)
>  {
> +	enum blk_crypto_mode_num mode_num = cfg->crypto_mode;
>  	const char *cipher_str = blk_crypto_modes[mode_num].cipher_str;
>  	struct blk_crypto_fallback_keyslot *slotp;
>  	unsigned int i;
>  	int err = 0;
>  
> +	if (!(BLK_CRYPTO_FALLBACK_DU_MASK & cfg->data_unit_size)) {
> +		pr_warn_ratelimited("can't handle data unit size %u\n",
> +				    cfg->data_unit_size);
> +		return -EOPNOTSUPP;
> +	}
> +

[Severity: High]
Does this early return bypass blk_crypto_fallback_init() and leave
blk_crypto_fallback_profile as NULL?

If this is the first fallback usage, returning here means we haven't
initialized the profile. If downstream callers (like dm-inlinecrypt) handle
this -EOPNOTSUPP error by attempting to evict the partially initialized key,
can it lead to a NULL pointer dereference?

When target teardown calls blk_crypto_evict_key(), it will eventually call
blk_crypto_fallback_evict_key(), which passes the uninitialized profile
directly:

block/blk-crypto-fallback.c:blk_crypto_fallback_evict_key() {
	return __blk_crypto_evict_key(blk_crypto_fallback_profile, key);
}

This unvalidated profile pointer is then dereferenced when checking the number
of slots:

block/blk-crypto-profile.c:__blk_crypto_evict_key() {
	...
	if (profile->num_slots == 0) {
	...
}

Could this cause a kernel panic if a user with device mapper access requests
an unsupported data unit size?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=3

Re: [PATCH 08/13] block: split encrypted bios at data unit boundaries

[sashiko-bot]

> Splitting a bio advances the remaining bio's data unit number by the
> number of whole data units that were split off:
> 
>   bio_split()
>    └─ bio_advance()
>       └─ bio_crypt_advance()
>          └─ bio_crypt_dun_increment(dun, bytes >> data_unit_size_bits)
> 
> The shift truncates, so splitting inside a data unit leaves the
> remainder with the wrong data unit number and silently corrupts data.
> 
> The split size is already rounded down to the logical block size, so
> this is only possible when the data unit size is larger than it, e.g.
> a fscrypt filesystem with a 4K block size on a device with a 512-byte
> logical block size.
> 
> Fix this by rounding the split size down to a data unit boundary.  The
> split alignment becomes the maximum of the logical block size, the
> data unit size, and the zone write granularity, so a zoned write still
> meets all three requirements.
> 
> Split bios can be up to one data unit smaller than the queue limits
> allow, so a bio can be split a little more often.  The number of
> segments for the split bio is still counted up to the size before the
> rounding, so it can be larger than needed; it is only used for
> accounting.
> 
> Fixes: a892c8d52c02 ("block: Inline encryption support for blk-mq")
> Cc: stable@vger.kernel.org
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=8

Re: [PATCH 12/13] ext4: send folios that cover a wraparound to the buffer_head path

[sashiko-bot]

> ext4_mpage_readpages() puts a whole folio into a bio with a single
> crypt context, taken from the position of the folio, and blk-crypto
> increments the data unit number for each data unit that follows in
> the bio.
> 
> By contrast, fscrypt truncates the data unit numbers to 32 bits for
> IV_INO_LBLK_32, the only supported IV generation method whose data
> unit numbers wrap around within a file.
> 
> The blocks after a wraparound inside a folio are then en/decrypted
> with IVs of 2^32 + i instead of i.  fscrypt_mergeable_bio() only runs
> between folios, so nothing catches this.
> 
> This is reachable even without a block size larger than the page
> size: when a filesystem is mounted without the encrypt feature,
> [ ... ]
> merges those bios back together.
> 
> Fixes: 709f0f1f1bf5 ("ext4: add checks for large folio incompatibilities when BS > PS")
> Cc: stable@vger.kernel.org # v6.19+
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=12

Re: [PATCH 13/13] ext4: allow large folios for encrypted files

[sashiko-bot]

> Encrypted filesystems keep their folio order at s_min_folio_order,
> because the read path could not handle a folio that covers a data
> unit number wraparound.  The read path now sends such folios to the
> buffer_head based path, so drop the restriction and give large folios
> to all encrypted files, including the ones whose block size is not
> larger than the page size.
> 
> Writes and direct I/O don't need the same treatment: ext4_writepages()
> checks fscrypt_mergeable_bio() for every block, so a write bio never
> spans a wraparound, and the bios of direct I/O are cut at the block
> size.
> 
> Performance:
> 
> Hardware: 96-core machine, 3.84T NVMe, 512G memory
> Filesystem: ext4 with a 4k block size, encryption on the fallback path
> 
> Buffered sequential writes (1G, fsync) in MiB/s; medians of 3 runs.
> 
>   bs      Before     After    gain
>   ----    ------    ------    ----
>   4k        1118      1162     +4%
>   16k       1260      1756    +39%
>   64k       1272      1977    +55%
>   256k      1308      2081    +59%
>   1M        1260      1455    +15%
> 
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=13

Re: [PATCH 10/13] block: don't require data unit aligned bio_vecs for encrypted bios

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] btrfs: infinite loop and double submission in btrfs_submit_bbio() due to unhandled negative error from bio_split_rw_at()
--

commit 7790faffa64672f5a012e4c7a5a36eb8b60cf57a
Author: Baokun Li <libaokun@linux.alibaba.com>

block: don't require data unit aligned bio_vecs for encrypted bios

This patch drops the requirement that every bio_vec in an encrypted bio
must be data unit aligned and a multiple of the data unit size. The
alignment is now handled by bio_split_alignment() rounding the split size
down to the data unit size.

> diff --git a/block/blk-merge.c b/block/blk-merge.c
> index 9bcc9c52eea17..dd9be4c04041a 100644
> --- a/block/blk-merge.c
> +++ b/block/blk-merge.c
> @@ -346,20 +346,18 @@ static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,
>  int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
>  		unsigned *segs, unsigned max_bytes, unsigned len_align_mask)
>  {
> -	struct bio_crypt_ctx *bc = bio_crypt_ctx(bio);
>  	struct bio_vec bv, bvprv, *bvprvp = NULL;
>  	unsigned nsegs = 0, bytes = 0, gaps = 0;
>  	unsigned int align, split_bytes;
>  	struct bvec_iter iter;
> -	unsigned start_align_mask = lim->dma_alignment;
> -
> -	if (bc) {
> -		start_align_mask |= (bc->bc_key->crypto_cfg.data_unit_size - 1);
> -		len_align_mask |= (bc->bc_key->crypto_cfg.data_unit_size - 1);
> -	}
>  
>  	bio_for_each_bvec(bv, bio, iter) {
> -		if (bv.bv_offset & start_align_mask ||
> +		/*
> +		 * The bio_vecs of a bio with a crypto context don't have to
> +		 * be data unit aligned, only the bio as a whole does; see
> +		 * bio_crypt_set_ctx() and bio_split_alignment().
> +		 */
> +		if (bv.bv_offset & lim->dma_alignment ||
>  		    bv.bv_len & len_align_mask)
>  			return -EINVAL;

[Severity: Critical]
Since bio_split_io_at() (and functions wrapping it like bio_split_rw_at())
returns negative error codes such as -EINVAL, can this lead to an infinite
loop in btrfs callers?

In fs/btrfs/bio.c:btrfs_append_map_length(), the return value is not
checked for negative error codes:

	sector_offset = bio_split_rw_at(&bbio->bio, &fs_info->limits,
					&nr_segs, map_length);
	if (sector_offset) {
		/* ... */
		return ALIGN_DOWN(sector_offset << SECTOR_SHIFT,
				  fs_info->sectorsize);
	}

If bio_split_rw_at() returns a negative error code (such as -EINVAL or
-EAGAIN), ALIGN_DOWN() will operate on a negative value and return a massive
u64.

This huge value is then returned as map_length to btrfs_submit_chunk(),
where it causes the bio split to be skipped since map_length < length is
false.

Because btrfs_submit_chunk() returns false, it causes its caller
btrfs_submit_bbio() to enter an infinite loop:

	while (!btrfs_submit_chunk(bbio, mirror_num))
		;

Do the btrfs callers need to be updated to handle these negative return
values, or does this patch change the conditions under which
bio_split_io_at() returns an error for encrypted bios?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=10

Re: [PATCH 07/13] iomap: align the direct I/O bios of encrypted files to the block size

[sashiko-bot]

> A bio that carries an encryption context must cover a whole number of
> data units: its data unit number is generated from its first data unit
> (see bio_crypt_set_ctx()), and it is only split at data unit
> boundaries.
> 
> iomap attaches the encryption context of an encrypted file to the bios
> that it builds for direct I/O, but it only cuts them at the logical
> block size of the device.  A bio ends where the bio_vecs that fit into
> it end, not at a fixed number of bytes: the extraction stops when the
> bio is out of bio_vecs or when the iterator is exhausted, and a
> bio_vec can cover a whole folio.
> 
> A bio can end in the middle of a data unit, and a data unit that is
> larger than a page makes this reachable without any misaligned input
> at all: the extraction then stops in the middle of a folio, and a cut
> [ ... ]
> the logical block size of the device on to iomap expecting it to fail
> with -EINVAL, while iomap checks only the file position and the
> length, so such a request can reach bio submission.
> 
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=7

Re: [PATCH 05/13] blk-crypto: encrypt one data unit per crypto request

[sashiko-bot]

> __blk_crypto_fallback_encrypt_bio() encrypts the source bio one bio_vec
> at a time: it walks the bio with bio_iter_iovec(), which clamps each
> bio_vec to a single page, encrypts all the data units of that bio_vec
> into its bounce page with one crypto request per data unit, and starts a
> new encrypted bio when all of its pages are in use.
> 
> That structure cannot express a data unit that spans more than a page.
> In preparation for supporting such data units, restructure the loop
> around the data unit, the same way as the decryption path: collect the
> bio_vecs that make up one data unit in blk_crypto_du_bvecs(), encrypt it
> with a single crypto request whose scatterlist covers exactly those
> bio_vecs, and advance the source bio by one data unit.
> 
> Unlike the decryption path, which decrypts in place using one
> scatterlist for both sides, the encryption path writes the ciphertext
> to the bounce pages, so it fills the scatterlist for each side of the
> crypto request, and calls the same blk_crypto_crypt_du() helper, with
> the encryption direction and the bounce page scatterlist as the
> destination.
> 
> A data unit is now encrypted as soon as it is complete instead of when
> all the data units of its bio_vec are, and the first bio_vec of a data
> unit is added to the encrypted bio with the whole length of the source
> bio_vec it starts in, as the data units that follow it in that bio_vec
> share its bounce page.
> 
> No functional change intended.
> 
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=5

Re: [PATCH 01/13] ext4: refuse encryption when block size > page size

[sashiko-bot]

> From: Alberto Garcia <berto@igalia.com>
> 
> Encryption is not supported when a filesystem's block size is larger
> than the page size, and commit 709f0f1f1bf5 ("ext4: add checks for
> large folio incompatibilities when BS > PS") added a check to refuse
> mounting such a filesystem.
> 
> However, it is also possible to enable the "encrypt" feature on a
> filesystem that is already mounted (with tune2fs -O encrypt). Once
> that happens it won't be possible to mount the filesystem again in
> the future, but there is nothing preventing the user from setting an
> encryption policy while the filesystem is still mounted.
> 
> Fix this by refusing to set an fscrypt context in ext4_set_context()
> when the block size is larger than the page size.
> 
> Fixes: 709f0f1f1bf5 ("ext4: add checks for large folio incompatibilities when BS > PS")
> Cc: stable@vger.kernel.org # v6.19+
> Signed-off-by: Alberto Garcia <berto@igalia.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=1

Re: [PATCH 11/13] ext4: allow encryption with a block size larger than the page size

[sashiko-bot]

> ext4 refuses encryption when the block size is larger than the page
> size, both at mount time and on a mounted filesystem, and refuses to
> load an existing encrypted inode.  The checks exist because such a
> block size needs folios that cover a whole block, while file contents
> were encrypted page by page in the fs-layer path, which cannot handle a
> folio that covers several data units.
> 
> Since commit 5acc6f649a68 ("fscrypt: Always use blk-crypto for contents
> on block-based filesystems") file contents are always encrypted through
> blk-crypto, and after the adaptations in the preceding patches, both
> inline encryption hardware and blk-crypto-fallback support data units
> larger than a page, so the checks can be removed.
> 
> Encrypted filesystems still keep their folio order at s_min_folio_order;
> allowing large folios needs additional work and will be enabled later.
> 
> Performance:
> 
> Hardware: 96-core machine, 3.84T NVMe, 512G memory
> Filesystem: ext4, encryption on the fallback path
> 
> Buffered sequential 1M writes and reads (1G, fsync) in MiB/s; medians
> of 3 runs.
> 
>   job          4k plain      4k enc    32k plain    32k enc    4k->32k
>   ---------    --------    --------    ---------    -------    -------
>   seqwrite         1981        1044         2062       1399       +34%
>   seqread          1139         931         1791       1793       +93%
> 
> Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261011170958.1759230-1-libaokun@linux.alibaba.com?part=11

Re: [PATCH 03/13] blk-crypto: reject fallback data unit sizes it can't handle

[Baokun Li]

On 2026/10/12 01:20, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [High] block: NULL pointer deref in blk_crypto_fallback_evict_key() on setup failure
[...]
> [Severity: High]
> Does this early return bypass blk_crypto_fallback_init() and leave
> blk_crypto_fallback_profile as NULL?
>
> If this is the first fallback usage, returning here means we haven't
> initialized the profile. If downstream callers (like dm-inlinecrypt) handle
> this -EOPNOTSUPP error by attempting to evict the partially initialized key,
> can it lead to a NULL pointer dereference?
>
> When target teardown calls blk_crypto_evict_key(), it will eventually call
> blk_crypto_fallback_evict_key(), which passes the uninitialized profile
> directly:
>
> block/blk-crypto-fallback.c:blk_crypto_fallback_evict_key() {
> 	return __blk_crypto_evict_key(blk_crypto_fallback_profile, key);
> }
>
> This unvalidated profile pointer is then dereferenced when checking the number
> of slots:
>
> block/blk-crypto-profile.c:__blk_crypto_evict_key() {
> 	...
> 	if (profile->num_slots == 0) {
> 	...
> }
>
> Could this cause a kernel panic if a user with device mapper access requests
> an unsupported data unit size?
>

The early return does leave blk_crypto_fallback_profile uninitialized,
but the NULL dereference that follows is a pre-existing issue, unrelated
to this series: the same eviction path is reachable on the base tree,
and the two flows below trigger a NULL dereference or a use-after-free:

  dm-inlinecrypt ctor            # first use of the fallback
   ├─ blk_crypto_init_key()      # succeeds, ctx->key.size != 0
   ├─ blk_crypto_start_using_key()
   │   ├─ return -EOPNOTSUPP     # uninit profile, will NULL deref
   │   └─ blk_crypto_fallback_start_using_key()
   │      └─ blk_crypto_fallback_init() fails
   │         └─ kfree(profile)   # not NULLed, will UAF
   ├─ goto bad
   └─ inlinecrypt_dtr()          # evicts whenever key.size is set
      └─ blk_crypto_evict_key()
         └─ blk_crypto_config_supported_natively() = false
            └─ blk_crypto_fallback_evict_key()
               └─ __blk_crypto_evict_key(blk_crypto_fallback_profile)
                  └─ profile->num_slots # BOOM!!!

If this series doesn't need a V2, I'll send the fix as a standalone
bugfix.  If a V2 is needed, I'll include it in the series before this
patch.


Thanks,
Baokun

Re: [PATCH 10/13] block: don't require data unit aligned bio_vecs for encrypted bios

[Baokun Li]

Added David, Chris and the btrfs list.


On 2026/10/12 01:24, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Critical] btrfs: infinite loop and double submission in btrfs_submit_bbio() due to unhandled negative error from bio_split_rw_at()
> --
>
> [Severity: Critical]
> Since bio_split_io_at() (and functions wrapping it like bio_split_rw_at())
> returns negative error codes such as -EINVAL, can this lead to an infinite
> loop in btrfs callers?
>
> In fs/btrfs/bio.c:btrfs_append_map_length(), the return value is not
> checked for negative error codes:
>
> 	sector_offset = bio_split_rw_at(&bbio->bio, &fs_info->limits,
> 					&nr_segs, map_length);
> 	if (sector_offset) {
> 		/* ... */
> 		return ALIGN_DOWN(sector_offset << SECTOR_SHIFT,
> 				  fs_info->sectorsize);
> 	}
>
> If bio_split_rw_at() returns a negative error code (such as -EINVAL or
> -EAGAIN), ALIGN_DOWN() will operate on a negative value and return a massive
> u64.
>
> This huge value is then returned as map_length to btrfs_submit_chunk(),
> where it causes the bio split to be skipped since map_length < length is
> false.
>
> Because btrfs_submit_chunk() returns false, it causes its caller
> btrfs_submit_bbio() to enter an infinite loop:
>
> 	while (!btrfs_submit_chunk(bbio, mirror_num))
> 		;
>
> Do the btrfs callers need to be updated to handle these negative return
> values, or does this patch change the conditions under which
> bio_split_io_at() returns an error for encrypted bios?
>

This patch doesn't change when bio_split_io_at() returns an error.
For bios without a crypto context start_align_mask was already
lim->dma_alignment, and the data unit masks were only added inside
the "if (bc)" branch, so the bvec checks are byte-for-byte identical
before and after this patch.

For encrypted bios the patch only drops the data unit alignment
requirement, so it can make -EINVAL less likely, never more.

The missing negative check in btrfs_append_map_length() is thus a
pre-existing btrfs issue.  It came with fec2e705729d ("block: check
for valid bio while splitting"), which added the -EINVAL returns but
didn't update the btrfs caller, and it is only reachable on the
zoned append path.

The fix should check for a negative return and fail the bio, as
bio_submit_split() does.


Regards,
Baokun