* [PATCH v5 0/5] crypto: skcipher - multi-data-unit dispatch as a template
From: Leonid Ravich @ 2026-06-30 8:34 UTC (permalink / raw)
To: linux-crypto, dm-devel
Cc: linux-block, linux-kernel, herbert, davem, ebiggers, snitzer,
mpatocka, axboe
This is v5. It reworks the multi-data-unit support from the in-core
auto-splitter of v4 into a crypto template, dun(...), addressing the v4
review: there is now no added cost on the core skcipher path, no
per-algorithm capability flag, and the per-data-unit split lives in an
algorithm rather than in crypto_skcipher_encrypt/decrypt the shape
Herbert suggested, which removes the "overhead for everyone" Eric
objected to.
v4: https://lore.kernel.org/linux-crypto/20260615105022.8025-1-lravich@amazon.com/
Model
---
A skcipher_request gains a data_unit_size field (patch 1). When set,
the request covers cryptlen / data_unit_size data units sharing one
starting IV; per-unit IVs are derived from the IV as a wide data-unit-
number (DUN) counter the convention blk-crypto already uses for
inline encryption.
dun(...) (patch 2) is a template that wraps an inner skcipher whose IV
is that counter (e.g. dun(xts(aes),le)). Its ->encrypt/->decrypt split
the request into one inner call per data unit, walking the IV +1 each
unit; each inner call is direct, so only the outer dispatch into the API
is indirect. A plain skcipher is unchanged and ignores data_unit_size,
so existing callers pay nothing the field is inert and the core
en/decrypt path is untouched.
The second template parameter selects how the per-unit IV advances. A
neighbour relates by a +1 step in exactly one of two ways, little- or
big-endian, so dun(...,le) / dun(...,be) is a closed parameter space,
not an open-ended set of "IV types". Internally each is one row of a
small struct dun_mode op table (an iv_next walk plus an ivsize
predicate); adding a future convention e.g. a width-bounded counter,
or an affine sector<<shift+k step is one row, with the dispatch loop
unchanged. IV constructions that are not such a counter are simply not
wrapped (the consumer keeps its per-unit path); an IV that is encrypted
(essiv) composes as the inner algorithm, dun(essiv(...),le), since the
encryption already lives in that inner template.
Why a template
--------------
- No core cost for anyone. crypto_skcipher_encrypt/decrypt are stock;
only a dun() tfm reads data_unit_size. (addresses Eric's "adds
checks/overhead for everyone")
- No capability flag. A hardware engine that handles a whole multi-DU
request in one pass registers its own dun(xts(aes),le) at a higher
cra_priority and is picked automatically exactly how
xts-aes-aesni already beats generic xts. No CRYPTO_ALG_* bit, no
core branch choosing native-vs-split. Such a native driver may also
be async (it owns its dispatch); only the generic template is
sync-only.
- The split is in the algorithm. (the direction Herbert described)
- It is the same kind of wrapper crypto/ already has. Like cryptd()
(async dispatch) and pcrypt() (parallel dispatch), dun() wraps an
inner skcipher and changes only how the request is dispatched
here, split across data units performing no cipher transform of
its own.
- It is a reusable primitive, not a dm-crypt feature. Two in-tree
consumers are included: dm-crypt (patch 4) and blk-crypto-fallback
(patch 5), which both open-code the per-DUN loop today; fscrypt's
direct (non-inline) path open-codes the same loop and could follow.
A HW engine is a provider via cra_priority. Consumers and providers
are decoupled through one named algorithm.
What it does and does not buy
-----------------------------
On a software cipher this is not a throughput win: the generic template
still issues one inner encrypt per data unit, so the AES compute is
unchanged. It removes per-request overhead and the consumer's
open-coded per-unit loop, and is byte-for-byte identical to the
per-sector path (Verification). The win is for a one-pass provider; no
software throughput is claimed.
dm-crypt consumer (patch 4)
---------------------------
dm-crypt submits one request per contiguous bio segment with
data_unit_size = cc->sector_size (e.g. the default 512-byte sector with
a 4 KiB bio_vec -> one request of 8 data units), using only its existing
inline single-entry scatterlist no per-bio allocation, no regression.
It allocates dun(<cipher>,<endian>) instead of the bare cipher when the
config can form the DUN counter: a counter IV mode (plain64 -> le,
plain64be -> be; essiv/lmk/tcw etc. are not plain counters and stay
per-sector), single-tfm, non-aead, sector_size 512 or iv_large_sectors.
DM_CRYPT selects CRYPTO_DUN and the template resolves against a sync
inner, so there is no acceptable wrap failure the bare cipher would
survive; an integrity config keeps an inert dun() wrapper but never
batches (one inner call per request == the per-sector path).
blk-crypto-fallback consumer (patch 5)
--------------------------------------
Every blk-crypto inline-encryption mode feeds the DUN as a little-endian
counter, so the fallback wraps its cipher as dun(<cipher>,le)
unconditionally (BLK_INLINE_ENCRYPTION_FALLBACK selects CRYPTO_DUN).
Because the template handles any counter width up to 32 bytes, this
covers all four modes AES-256-XTS, AES-128-CBC-ESSIV, Adiantum
(32-byte IV) and SM4-XTS and the open-coded per-unit loop is removed
from both the encrypt and decrypt paths.
Verification
------------
Regression protocol in the tree, on x86 + arm64 under qemu: build clean
and checkpatch strict clean (the lone warning is the new-file
MAINTAINERS reminder; crypto/ is an F: catch-all); testmgr dun()
cross-check (batched == N x single-DU reference over a fragmented
scatterlist, plus a boundary-seeded IV that forces a carry across a
64-bit limb / byte run) for every accepted ivsize including 32 (Adiantum)
in BOTH dun(...,le) and dun(...,be), so the big-endian counter path is
exercised independently of any consumer; an AF_ALG probe forces the
dun() cross-check to run for each blk-crypto inner cipher
(dun(essiv(cbc(aes),sha256),le), dun(adiantum(xchacha12,aes),le), ...);
dm-crypt plain64/plain64be activate dun() (le/be), essiv / plain fall
back; negative gates (multikey and integrity not batched); plain64 and
plain64be round-trips and a 4096-byte iv_large_sectors round-trip;
low-memory; arm64 functional; an end-to-end blk-crypto-fallback test
(ext4 + fscrypt -o inlinecrypt with no inline HW, driving dun(xts,le)
and verifying a post-cache-drop round-trip); and byte-equivalence:
ciphertext is bit-identical to an unpatched axboe/for-next baseline
(sha256 4913910b...43efc0 le, da0869a9...63004 be).
Changes since v4
----------------
- The in-core auto-splitter and validator are gone; multi-DU dispatch is
the dun(...) template. crypto_skcipher_encrypt/decrypt revert to
stock, so there is no added cost on the core path.
- The CRYPTO_ALG_SKCIPHER_NATIVE_MULTI_DU capability flag is dropped; a
native one-pass driver is selected by cra_priority instead.
- The template is dun(<inner>,<endian>) in the cryptd()/pcrypt() family
of dispatch-only wrappers; the counter endianness (le/be) is its
second parameter, backed by a struct dun_mode op table so a future
counter convention is one table row. It handles any counter width up
to 32 bytes (covering Adiantum) and rejects a data_unit_size 0 /
cryptlen 0 request.
- dm-crypt allocates dun(<cipher>,le|be) when eligible (selecting the IV
mode before tfm allocation); plain64 -> le, plain64be -> be. An
integrity config keeps an inert dun() wrapper but never batches.
DM_CRYPT selects CRYPTO_DUN.
- blk-crypto-fallback is a second consumer (patch 5), demonstrating the
template is a shared primitive, not dm-crypt-only; it wraps every mode
as dun(<cipher>,le) and BLK_INLINE_ENCRYPTION_FALLBACK selects
CRYPTO_DUN.
- testmgr exercises the template via dun(<inner>,le) and dun(<inner>,be),
including ivsize 32 and a carry-boundary IV; an end-to-end fscrypt
-o inlinecrypt test drives the blk-crypto-fallback consumer.
Leonid Ravich (5):
crypto: skcipher - add per-request data_unit_size
crypto: dun - data-unit-number dispatch template
crypto: testmgr - test dun() dispatch
dm crypt: batch a bio segment's sectors via dun()
blk-crypto: fallback - batch a segment's data units via dun()
block/Kconfig | 1 +
block/blk-crypto-fallback.c | 74 ++++----
crypto/Kconfig | 14 ++
crypto/Makefile | 1 +
crypto/dun.c | 359 ++++++++++++++++++++++++++++++++++++
crypto/testmgr.c | 289 +++++++++++++++++++++++++++++
drivers/md/Kconfig | 1 +
drivers/md/dm-crypt.c | 208 ++++++++++++++++-----
include/crypto/skcipher.h | 34 ++++
9 files changed, 899 insertions(+), 82 deletions(-)
create mode 100644 crypto/dun.c
base-commit: a8cafdf8c949f17c92eca0045532e88ac0dac30d
--
2.47.3
^ permalink raw reply
* [PATCH v5 1/5] crypto: skcipher - add per-request data_unit_size
From: Leonid Ravich @ 2026-06-30 8:34 UTC (permalink / raw)
To: linux-crypto, dm-devel
Cc: linux-block, linux-kernel, herbert, davem, ebiggers, snitzer,
mpatocka, axboe
In-Reply-To: <20260630083431.2772-1-lravich@amazon.com>
Add a data_unit_size field to struct skcipher_request. When non-zero,
the request covers cryptlen / data_unit_size data units that share one
starting IV; per-unit IVs are derived from the request IV as a wide
data-unit-number counter (the convention also used by blk-crypto for
inline encryption). cryptlen must be a positive multiple of
data_unit_size.
The field is honoured by an skcipher that understands data units -- an
instance of the dun(...) template (added next), or a driver that handles
a whole multi-DU request natively. A plain skcipher ignores it, so the
field is inert for every existing caller; the core en/decrypt path is
unchanged. skcipher_request_set_tfm() and the on-stack request
initialiser reset it to 0 so a reused request defaults to single-DU.
Signed-off-by: Leonid Ravich <lravich@amazon.com>
---
include/crypto/skcipher.h | 34 ++++++++++++++++++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/include/crypto/skcipher.h b/include/crypto/skcipher.h
index 4efe2ca8c4d1..1121be80cb53 100644
--- a/include/crypto/skcipher.h
+++ b/include/crypto/skcipher.h
@@ -31,6 +31,13 @@ struct scatterlist;
/**
* struct skcipher_request - Symmetric key cipher request
* @cryptlen: Number of bytes to encrypt or decrypt
+ * @data_unit_size: Size in bytes of each data unit, or 0 for a
+ * single-data-unit request (the default). When non-zero,
+ * must be a multiple of the cipher block size, @cryptlen must
+ * be a positive multiple of it, and per-DU IVs are derived from
+ * @iv as a wide counter (the data-unit-number convention); the
+ * counter width and endianness are chosen by the consumer (e.g.
+ * the dun() template's second parameter).
* @iv: Initialisation Vector
* @src: Source SG list
* @dst: Destination SG list
@@ -39,6 +46,7 @@ struct scatterlist;
*/
struct skcipher_request {
unsigned int cryptlen;
+ unsigned int data_unit_size;
u8 *iv;
@@ -225,6 +233,7 @@ struct lskcipher_alg {
struct skcipher_request *name = \
(((struct skcipher_request *)__##name##_desc)->base.tfm = \
crypto_sync_skcipher_tfm((_tfm)), \
+ ((struct skcipher_request *)__##name##_desc)->data_unit_size = 0, \
(void *)__##name##_desc)
/**
@@ -819,6 +828,8 @@ static inline void skcipher_request_set_tfm(struct skcipher_request *req,
struct crypto_skcipher *tfm)
{
req->base.tfm = crypto_skcipher_tfm(tfm);
+ /* Reused requests default to single-data-unit. */
+ req->data_unit_size = 0;
}
static inline void skcipher_request_set_sync_tfm(struct skcipher_request *req,
@@ -937,5 +948,28 @@ static inline void skcipher_request_set_crypt(
req->iv = iv;
}
+/**
+ * skcipher_request_set_data_unit_size() - submit as multiple data units
+ * @req: request handle
+ * @data_unit_size: data-unit size in bytes (a multiple of the cipher block
+ * size), or 0 to disable
+ *
+ * Process @req as @cryptlen / @data_unit_size data units sharing one starting
+ * @iv, with per-DU IVs derived by treating @iv as a wide counter (the data-
+ * unit-number convention). @cryptlen must be a positive multiple of
+ * @data_unit_size. This is honoured only by a tfm that understands data
+ * units -- an instance of the dun(...) template (which splits the request
+ * into one inner call per unit, with the counter endianness given as its
+ * second parameter), or a driver that consumes a whole multi-DU request
+ * natively, which rejects a request violating these constraints with -EINVAL.
+ * A plain skcipher ignores the field.
+ */
+static inline void
+skcipher_request_set_data_unit_size(struct skcipher_request *req,
+ unsigned int data_unit_size)
+{
+ req->data_unit_size = data_unit_size;
+}
+
#endif /* _CRYPTO_SKCIPHER_H */
--
2.47.3
^ permalink raw reply related
* [PATCH v5 2/5] crypto: dun - data-unit-number dispatch template
From: Leonid Ravich @ 2026-06-30 8:34 UTC (permalink / raw)
To: linux-crypto, dm-devel
Cc: linux-block, linux-kernel, herbert, davem, ebiggers, snitzer,
mpatocka, axboe
In-Reply-To: <20260630083431.2772-1-lravich@amazon.com>
Add a dun(...) skcipher template that wraps an inner skcipher whose IV
is a wide data-unit-number counter (e.g. dun(xts(aes),le)). When the
caller sets skcipher_request::data_unit_size, the template splits the
request into cryptlen / data_unit_size sub-requests on the inner cipher,
walking the IV +1 per unit. Each inner ->encrypt/->decrypt is a direct
call, so only the outer dispatch into the crypto API is indirect -- the
per-unit work is not.
The second template parameter selects the counter endianness: dun(...,le)
for a little-endian counter (dm-crypt plain64, blk-crypto inline
encryption) and dun(...,be) for a big-endian one (dm-crypt plain64be).
Those are the only two ways a per-unit IV relates to its neighbour by a
+1 step; IV modes that are not such a counter are simply not wrapped.
Like cryptd() and pcrypt(), dun() wraps an inner skcipher and changes
only how the request is dispatched -- here, split across data units --
performing no cipher transform of its own.
A dun() tfm exists solely for multi-DU dispatch, so a request with
data_unit_size 0 is rejected with -EINVAL; a caller wanting plain
single-DU encryption uses the inner skcipher.
A hardware engine that consumes a whole multi-DU request in one pass
registers its own dun(...) at a higher cra_priority and is selected
automatically by the existing priority mechanism; no per-algorithm
capability flag is needed. The generic template is sync-only (the split
loop treats any non-zero inner return as terminal), so it resolves against
a sync inner cipher (mask | CRYPTO_ALG_ASYNC); async is left to such
native drivers.
The inner IV must be a whole number of 64-bit limbs and no wider than 32
bytes: 16 covers xts(...), 32 covers the widest inline-encryption mode
(Adiantum).
Suggested-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Leonid Ravich <lravich@amazon.com>
---
crypto/Kconfig | 14 ++
crypto/Makefile | 1 +
crypto/dun.c | 359 ++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 374 insertions(+)
create mode 100644 crypto/dun.c
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 103d1f58cb7c..4f90a780c4fc 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -746,6 +746,20 @@ config CRYPTO_XTS
implementation currently can't handle a sectorsize which is not a
multiple of 16 bytes.
+config CRYPTO_DUN
+ tristate "Data-unit-number (DUN) dispatch template"
+ select CRYPTO_SKCIPHER
+ select CRYPTO_MANAGER
+ help
+ dun(...) wraps an skcipher whose IV is a wide data-unit-number
+ counter (e.g. xts(aes)) and lets a caller submit several data units
+ sharing one starting IV in a single request, via
+ skcipher_request::data_unit_size. The counter endianness is the
+ second parameter: dun(xts(aes),le) or dun(xts(aes),be). The template
+ splits the request into one inner call per data unit; a hardware
+ driver may register a higher-priority dun(...) that handles the whole
+ request in one pass. The first user is dm-crypt.
+
endmenu
menu "AEAD (authenticated encryption with associated data) ciphers"
diff --git a/crypto/Makefile b/crypto/Makefile
index 162242593c7c..584d9e8c4347 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -93,6 +93,7 @@ obj-$(CONFIG_CRYPTO_PCBC) += pcbc.o
obj-$(CONFIG_CRYPTO_CTS) += cts.o
obj-$(CONFIG_CRYPTO_LRW) += lrw.o
obj-$(CONFIG_CRYPTO_XTS) += xts.o
+obj-$(CONFIG_CRYPTO_DUN) += dun.o
obj-$(CONFIG_CRYPTO_CTR) += ctr.o
obj-$(CONFIG_CRYPTO_XCTR) += xctr.o
obj-$(CONFIG_CRYPTO_HCTR2) += hctr2.o
diff --git a/crypto/dun.c b/crypto/dun.c
new file mode 100644
index 000000000000..4fcb81a025b9
--- /dev/null
+++ b/crypto/dun.c
@@ -0,0 +1,359 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * dun: data-unit-number dispatch template for skcipher
+ *
+ * Wraps an inner skcipher (e.g. xts(aes)) and, when the caller sets
+ * skcipher_request::data_unit_size, splits the request into cryptlen /
+ * data_unit_size sub-requests, each unit's IV the previous one +1 -- the
+ * data-unit-number (DUN) convention. The second parameter selects the IV
+ * walk (see struct dun_mode): dun(xts(aes),le) or dun(xts(aes),be).
+ *
+ * Like cryptd()/pcrypt(), dun() only changes how a request is dispatched and
+ * performs no transform of its own; a native one-pass multi-DU driver wins by
+ * cra_priority. Callers that never set data_unit_size pay nothing.
+ */
+
+#include <crypto/algapi.h>
+#include <crypto/internal/skcipher.h>
+#include <crypto/scatterwalk.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+
+/* Bounds the on-stack IV buffers: 16 covers xts(...), 32 covers Adiantum. */
+#define DUN_MAX_IVSIZE 32
+
+/*
+ * A dun() mode is the rule for deriving each data unit's IV from the request's
+ * starting IV. @name is the template's second parameter; @iv_next advances the
+ * @ivsize-byte @iv in place to the next data unit. @ivsize_ok rejects IV sizes
+ * the walk can't handle. Add a row to dun_modes[] to support a new convention.
+ */
+struct dun_mode {
+ const char *name;
+ void (*iv_next)(u8 *iv, unsigned int ivsize);
+ bool (*ivsize_ok)(unsigned int ivsize);
+};
+
+struct dun_tfm_ctx {
+ struct crypto_skcipher *child;
+ const struct dun_mode *mode;
+};
+
+struct dun_inst_ctx {
+ struct crypto_skcipher_spawn spawn;
+ const struct dun_mode *mode;
+};
+
+struct dun_request_ctx {
+ /* Must be last; the child request is appended with its own reqsize. */
+ struct skcipher_request subreq;
+};
+
+/* Little-endian counter: increment the IV per __le64 limb, low limb first. */
+static void dun_iv_next_le(u8 *iv, unsigned int ivsize)
+{
+ unsigned int i;
+
+ for (i = 0; i < ivsize; i += sizeof(__le64)) {
+ __le64 limb;
+ u64 v;
+
+ memcpy(&limb, iv + i, sizeof(limb));
+ v = le64_to_cpu(limb) + 1;
+ limb = cpu_to_le64(v);
+ memcpy(iv + i, &limb, sizeof(limb));
+ if (likely(v != 0))
+ break; /* no carry into the next limb */
+ }
+}
+
+/* Big-endian counter: increment the IV byte-wise from the last byte. */
+static void dun_iv_next_be(u8 *iv, unsigned int ivsize)
+{
+ unsigned int i = ivsize;
+
+ while (i--) {
+ if (likely(++iv[i]))
+ break; /* no carry into the next byte */
+ }
+}
+
+/*
+ * le requires this: it walks the IV in __le64 limbs, so the size must be a
+ * whole number of limbs. be increments byte-wise and would accept any size,
+ * but reuses the same check for a uniform value-space.
+ */
+static bool dun_ivsize_whole_limbs(unsigned int ivsize)
+{
+ return IS_ALIGNED(ivsize, sizeof(__le64));
+}
+
+static const struct dun_mode dun_modes[] = {
+ { "le", dun_iv_next_le, dun_ivsize_whole_limbs },
+ { "be", dun_iv_next_be, dun_ivsize_whole_limbs },
+};
+
+static const struct dun_mode *dun_find_mode(const char *name)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(dun_modes); i++)
+ if (!strcmp(name, dun_modes[i].name))
+ return &dun_modes[i];
+ return NULL;
+}
+
+static int dun_setkey(struct crypto_skcipher *parent, const u8 *key,
+ unsigned int keylen)
+{
+ struct dun_tfm_ctx *ctx = crypto_skcipher_ctx(parent);
+ struct crypto_skcipher *child = ctx->child;
+
+ crypto_skcipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
+ crypto_skcipher_set_flags(child, crypto_skcipher_get_flags(parent) &
+ CRYPTO_TFM_REQ_MASK);
+ return crypto_skcipher_setkey(child, key, keylen);
+}
+
+/*
+ * Run one inner ->crypt per data unit, walking the IV as a wide counter.
+ * @req->iv is never modified; the inner cipher only sees the iv_unit copy.
+ */
+static int dun_split(struct skcipher_request *req,
+ int (*crypt)(struct skcipher_request *))
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct dun_tfm_ctx *ctx = crypto_skcipher_ctx(tfm);
+ struct dun_request_ctx *rctx = skcipher_request_ctx(req);
+ struct skcipher_request *subreq = &rctx->subreq;
+ const unsigned int du = req->data_unit_size;
+ const unsigned int total = req->cryptlen;
+ const unsigned int ivsize = crypto_skcipher_ivsize(tfm);
+ const struct dun_mode *mode = ctx->mode;
+ bool inplace = req->src == req->dst;
+ struct scatter_walk src_walk, dst_walk;
+ struct scatterlist src_sg[2], dst_sg[2];
+ u8 iv_ctr[DUN_MAX_IVSIZE];
+ u8 iv_unit[DUN_MAX_IVSIZE];
+ unsigned int off;
+ int err = 0;
+
+ /* iv_ctr is the counter; iv_unit is a per-unit copy an inner may write
+ * back in place (e.g. xts, essiv), so the counter is never mutated.
+ */
+ memcpy(iv_ctr, req->iv, ivsize);
+
+ sg_init_table(src_sg, 2);
+ scatterwalk_start(&src_walk, req->src);
+ if (!inplace) {
+ sg_init_table(dst_sg, 2);
+ scatterwalk_start(&dst_walk, req->dst);
+ }
+
+ skcipher_request_set_tfm(subreq, ctx->child);
+ skcipher_request_set_callback(subreq, skcipher_request_flags(req),
+ NULL, NULL);
+
+ for (off = 0; off < total; off += du) {
+ struct scatterlist *s, *d;
+
+ scatterwalk_get_sglist(&src_walk, src_sg);
+ scatterwalk_skip(&src_walk, du);
+ s = src_sg;
+ if (inplace) {
+ d = src_sg;
+ } else {
+ scatterwalk_get_sglist(&dst_walk, dst_sg);
+ scatterwalk_skip(&dst_walk, du);
+ d = dst_sg;
+ }
+
+ memcpy(iv_unit, iv_ctr, ivsize);
+ skcipher_request_set_crypt(subreq, s, d, du, iv_unit);
+ err = crypt(subreq);
+ if (err)
+ break;
+
+ mode->iv_next(iv_ctr, ivsize);
+ }
+
+ return err;
+}
+
+/*
+ * Validate a multi-DU request: non-zero cryptlen, and a data_unit_size that is
+ * set, a multiple of the block size, and divides cryptlen evenly.
+ */
+static int dun_check(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+
+ if (!req->cryptlen || !req->data_unit_size ||
+ !IS_ALIGNED(req->data_unit_size, crypto_skcipher_blocksize(tfm)) ||
+ (req->cryptlen % req->data_unit_size))
+ return -EINVAL;
+ return 0;
+}
+
+static int dun_encrypt(struct skcipher_request *req)
+{
+ int err = dun_check(req);
+
+ if (err)
+ return err;
+ return dun_split(req, crypto_skcipher_encrypt);
+}
+
+static int dun_decrypt(struct skcipher_request *req)
+{
+ int err = dun_check(req);
+
+ if (err)
+ return err;
+ return dun_split(req, crypto_skcipher_decrypt);
+}
+
+static int dun_init_tfm(struct crypto_skcipher *tfm)
+{
+ struct skcipher_instance *inst = skcipher_alg_instance(tfm);
+ struct dun_inst_ctx *ictx = skcipher_instance_ctx(inst);
+ struct dun_tfm_ctx *ctx = crypto_skcipher_ctx(tfm);
+ struct crypto_skcipher *child;
+
+ child = crypto_spawn_skcipher(&ictx->spawn);
+ if (IS_ERR(child))
+ return PTR_ERR(child);
+
+ ctx->child = child;
+ ctx->mode = ictx->mode;
+ crypto_skcipher_set_reqsize(tfm,
+ sizeof(struct dun_request_ctx) +
+ crypto_skcipher_reqsize(child));
+ return 0;
+}
+
+static void dun_exit_tfm(struct crypto_skcipher *tfm)
+{
+ struct dun_tfm_ctx *ctx = crypto_skcipher_ctx(tfm);
+
+ crypto_free_skcipher(ctx->child);
+}
+
+static void dun_free_instance(struct skcipher_instance *inst)
+{
+ struct dun_inst_ctx *ictx = skcipher_instance_ctx(inst);
+
+ crypto_drop_skcipher(&ictx->spawn);
+ kfree(inst);
+}
+
+static int dun_create(struct crypto_template *tmpl, struct rtattr **tb)
+{
+ struct skcipher_alg_common *alg;
+ struct skcipher_instance *inst;
+ struct dun_inst_ctx *ictx;
+ const struct dun_mode *mode;
+ const char *cipher_name;
+ const char *mode_name;
+ u32 mask;
+ int err;
+
+ err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER, &mask);
+ if (err)
+ return err;
+
+ cipher_name = crypto_attr_alg_name(tb[1]);
+ if (IS_ERR(cipher_name))
+ return PTR_ERR(cipher_name);
+
+ /* Second parameter: the IV-walk mode (see dun_modes[]). */
+ mode_name = crypto_attr_alg_name(tb[2]);
+ if (IS_ERR(mode_name))
+ return PTR_ERR(mode_name);
+ mode = dun_find_mode(mode_name);
+ if (!mode)
+ return -EINVAL;
+
+ inst = kzalloc(sizeof(*inst) + sizeof(*ictx), GFP_KERNEL);
+ if (!inst)
+ return -ENOMEM;
+ ictx = skcipher_instance_ctx(inst);
+ ictx->mode = mode;
+
+ /*
+ * Sync-only: the split loop can't drive an async (-EINPROGRESS) child,
+ * so resolve against a sync inner (mask | CRYPTO_ALG_ASYNC).
+ */
+ err = crypto_grab_skcipher(&ictx->spawn, skcipher_crypto_instance(inst),
+ cipher_name, 0, mask | CRYPTO_ALG_ASYNC);
+ if (err)
+ goto err_free_inst;
+
+ alg = crypto_spawn_skcipher_alg_common(&ictx->spawn);
+
+ /* The mode must accept this IV size, and it must fit our buffers. */
+ err = -EINVAL;
+ if (!alg->ivsize || alg->ivsize > DUN_MAX_IVSIZE ||
+ !mode->ivsize_ok(alg->ivsize))
+ goto err_free_inst;
+
+ err = -ENAMETOOLONG;
+ if (snprintf(inst->alg.base.cra_name, CRYPTO_MAX_ALG_NAME, "dun(%s,%s)",
+ alg->base.cra_name, mode->name) >= CRYPTO_MAX_ALG_NAME)
+ goto err_free_inst;
+ if (snprintf(inst->alg.base.cra_driver_name, CRYPTO_MAX_ALG_NAME,
+ "dun(%s,%s)", alg->base.cra_driver_name,
+ mode->name) >= CRYPTO_MAX_ALG_NAME)
+ goto err_free_inst;
+
+ inst->alg.base.cra_priority = alg->base.cra_priority;
+ inst->alg.base.cra_blocksize = alg->base.cra_blocksize;
+ inst->alg.base.cra_alignmask = alg->base.cra_alignmask;
+ inst->alg.base.cra_ctxsize = sizeof(struct dun_tfm_ctx);
+
+ inst->alg.ivsize = alg->ivsize;
+ inst->alg.chunksize = alg->chunksize;
+ inst->alg.min_keysize = alg->min_keysize;
+ inst->alg.max_keysize = alg->max_keysize;
+
+ inst->alg.init = dun_init_tfm;
+ inst->alg.exit = dun_exit_tfm;
+ inst->alg.setkey = dun_setkey;
+ inst->alg.encrypt = dun_encrypt;
+ inst->alg.decrypt = dun_decrypt;
+
+ inst->free = dun_free_instance;
+
+ err = skcipher_register_instance(tmpl, inst);
+ if (err) {
+err_free_inst:
+ dun_free_instance(inst);
+ }
+ return err;
+}
+
+static struct crypto_template dun_tmpl = {
+ .name = "dun",
+ .create = dun_create,
+ .module = THIS_MODULE,
+};
+
+static int __init dun_module_init(void)
+{
+ return crypto_register_template(&dun_tmpl);
+}
+
+static void __exit dun_module_exit(void)
+{
+ crypto_unregister_template(&dun_tmpl);
+}
+
+module_init(dun_module_init);
+module_exit(dun_module_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Data-unit-number dispatch template for skcipher");
+MODULE_ALIAS_CRYPTO("dun");
--
2.47.3
^ permalink raw reply related
* [PATCH v5 3/5] crypto: testmgr - test dun() dispatch
From: Leonid Ravich @ 2026-06-30 8:34 UTC (permalink / raw)
To: linux-crypto, dm-devel
Cc: linux-block, linux-kernel, herbert, davem, ebiggers, snitzer,
mpatocka, axboe
In-Reply-To: <20260630083431.2772-1-lravich@amazon.com>
For every ivsize-16 skcipher, wrap it in both a dun(<inner>,le) and a
dun(<inner>,be) instance and cross-check each batched output against an
independent N x single-DU reference run directly on the inner tfm (both
keyed with one random key, the reference counter walked in the matching
endianness), over a deliberately fragmented scatterlist whose entries do
not align to the data-unit size. The two must produce byte-identical
ciphertext; the batched ciphertext is then round-tripped and the caller
IV checked unchanged. Testing both endiannesses exercises the be path
independently of any in-tree consumer. Algorithms with no dun wrapper
(ivsize != 16) are skipped; a genuine mismatch returns -EBADMSG.
Signed-off-by: Leonid Ravich <lravich@amazon.com>
---
crypto/testmgr.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 289 insertions(+)
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index 4d86efae65b2..cd9246f432de 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -3211,6 +3211,291 @@ static int test_skcipher(int enc, const struct cipher_test_suite *suite,
return 0;
}
+/* Upper bound on the IVs the dun() template accepts (16: xts; 32: Adiantum). */
+#define TEST_MDU_MAX_IVSIZE 32
+
+/*
+ * Increment an @ivsize-byte IV as a wide counter. Byte-wise with carry --
+ * deliberately independent of crypto/dun.c's per-limb walk, so the two only
+ * agree if the carry is right. LE: byte 0 least significant; BE: last byte.
+ */
+static void test_mdu_iv_inc(u8 *iv, unsigned int ivsize, bool big_endian)
+{
+ int i;
+
+ if (big_endian) {
+ for (i = ivsize - 1; i >= 0; i--)
+ if (++iv[i])
+ break;
+ } else {
+ for (i = 0; i < (int)ivsize; i++)
+ if (++iv[i])
+ break;
+ }
+}
+
+/*
+ * Seed @iv so the low 64-bit limb is all-ones but its least-significant byte:
+ * the 2nd increment wraps the limb and carries into the next. LE limb is
+ * bytes [0,8); BE limb is the last 8 bytes. Bytes outside keep their value.
+ */
+static void test_mdu_iv_boundary(u8 *iv, unsigned int ivsize, bool big_endian)
+{
+ unsigned int i;
+
+ if (big_endian) {
+ for (i = ivsize - 8; i < ivsize; i++)
+ iv[i] = 0xff;
+ iv[ivsize - 1] = 0xfe;
+ } else {
+ for (i = 0; i < 8; i++)
+ iv[i] = 0xff;
+ iv[0] = 0xfe;
+ }
+}
+
+/* Encrypt one du_size block with a plain single-DU request (the reference). */
+static int test_mdu_ref_encrypt(struct crypto_skcipher *tfm, const u8 *in,
+ u8 *out, unsigned int du_size, const u8 *iv,
+ unsigned int ivsize)
+{
+ struct skcipher_request *req;
+ struct scatterlist sg_in;
+ DECLARE_CRYPTO_WAIT(wait);
+ u8 ivbuf[TEST_MDU_MAX_IVSIZE];
+ int err;
+
+ req = skcipher_request_alloc(tfm, GFP_KERNEL);
+ if (!req)
+ return -ENOMEM;
+ memcpy(ivbuf, iv, ivsize);
+ memcpy(out, in, du_size);
+ sg_init_one(&sg_in, out, du_size);
+ skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ crypto_req_done, &wait);
+ skcipher_request_set_crypt(req, &sg_in, &sg_in, du_size, ivbuf);
+ err = crypto_wait_req(crypto_skcipher_encrypt(req), &wait);
+ skcipher_request_free(req);
+ return err;
+}
+
+/*
+ * Build an SG over @buf with du_size-unaligned entries, so the splitter's
+ * per-DU views cross SG entries and exercise the scatter_walk cursor.
+ */
+static void test_mdu_sg_fragment(struct scatterlist *sg, unsigned int nents,
+ u8 *buf, unsigned int total)
+{
+ unsigned int chunk = total / nents;
+ unsigned int off = 0, i;
+
+ sg_init_table(sg, nents);
+ for (i = 0; i < nents; i++) {
+ unsigned int len = (i == nents - 1) ? total - off : chunk;
+
+ sg_set_buf(&sg[i], buf + off, len);
+ off += len;
+ }
+}
+
+#define TEST_MDU_NR_UNITS 4
+#define TEST_MDU_NR_FRAGS 5
+/*
+ * Verify batched dispatch on @mdu (a dun(<inner>,<endian>) tfm) is byte-equal
+ * to an independent N x single-DU reference on @inner with @big_endian-walked
+ * IVs, over a fragmented SG, then round-trips. Both tfms must share a key.
+ * @iv_orig is the ivsize-byte starting IV (the caller varies it to exercise
+ * both a random IV and one seeded to cross a carry boundary).
+ */
+static int test_skcipher_multi_du_one(struct crypto_skcipher *mdu,
+ struct crypto_skcipher *inner,
+ unsigned int du_size, bool big_endian,
+ const u8 *iv_orig)
+{
+ const char *driver = crypto_skcipher_driver_name(mdu);
+ const unsigned int total = du_size * TEST_MDU_NR_UNITS;
+ const unsigned int ivsize = crypto_skcipher_ivsize(mdu);
+ struct skcipher_request *req = NULL;
+ struct scatterlist sg[TEST_MDU_NR_FRAGS];
+ DECLARE_CRYPTO_WAIT(wait);
+ u8 iv_work[TEST_MDU_MAX_IVSIZE], iv_ref[TEST_MDU_MAX_IVSIZE];
+ u8 *plain = NULL, *buf = NULL, *ref = NULL;
+ unsigned int u;
+ int err;
+
+ plain = kmalloc(total, GFP_KERNEL);
+ buf = kmalloc(total, GFP_KERNEL);
+ ref = kmalloc(total, GFP_KERNEL);
+ req = skcipher_request_alloc(mdu, GFP_KERNEL);
+ if (!plain || !buf || !ref || !req) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ get_random_bytes(plain, total);
+
+ /* Reference: per-DU single requests on the inner tfm, counter-walked IVs. */
+ memcpy(iv_ref, iv_orig, ivsize);
+ for (u = 0; u < TEST_MDU_NR_UNITS; u++) {
+ err = test_mdu_ref_encrypt(inner, plain + u * du_size,
+ ref + u * du_size, du_size, iv_ref,
+ ivsize);
+ if (err) {
+ pr_err("alg: skcipher: %s multi-DU ref encrypt failed (du=%u): %d\n",
+ driver, du_size, err);
+ goto out;
+ }
+ test_mdu_iv_inc(iv_ref, ivsize, big_endian);
+ }
+
+ /* Batched: one request over a fragmented SG. */
+ memcpy(buf, plain, total);
+ memcpy(iv_work, iv_orig, ivsize);
+ test_mdu_sg_fragment(sg, TEST_MDU_NR_FRAGS, buf, total);
+ skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ crypto_req_done, &wait);
+ skcipher_request_set_crypt(req, sg, sg, total, iv_work);
+ skcipher_request_set_data_unit_size(req, du_size);
+ err = crypto_wait_req(crypto_skcipher_encrypt(req), &wait);
+ if (err) {
+ pr_err("alg: skcipher: %s multi-DU encrypt failed (du=%u): %d\n",
+ driver, du_size, err);
+ goto out;
+ }
+ if (memcmp(buf, ref, total) != 0) {
+ pr_err("alg: skcipher: %s multi-DU ciphertext differs from single-DU reference (du=%u)\n",
+ driver, du_size);
+ err = -EBADMSG;
+ goto out;
+ }
+ /* req->iv must be unchanged after multi-DU dispatch. */
+ if (memcmp(iv_work, iv_orig, ivsize) != 0) {
+ pr_err("alg: skcipher: %s multi-DU encrypt mutated caller IV (du=%u)\n",
+ driver, du_size);
+ err = -EBADMSG;
+ goto out;
+ }
+
+ /* Round-trip the batched ciphertext back to plaintext. */
+ test_mdu_sg_fragment(sg, TEST_MDU_NR_FRAGS, buf, total);
+ skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ crypto_req_done, &wait);
+ skcipher_request_set_crypt(req, sg, sg, total, iv_work);
+ skcipher_request_set_data_unit_size(req, du_size);
+ err = crypto_wait_req(crypto_skcipher_decrypt(req), &wait);
+ if (err) {
+ pr_err("alg: skcipher: %s multi-DU decrypt failed (du=%u): %d\n",
+ driver, du_size, err);
+ goto out;
+ }
+ if (memcmp(buf, plain, total) != 0) {
+ pr_err("alg: skcipher: %s multi-DU round-trip mismatch (du=%u)\n",
+ driver, du_size);
+ err = -EBADMSG;
+ }
+
+out:
+ skcipher_request_free(req);
+ kfree(ref);
+ kfree(buf);
+ kfree(plain);
+ return err;
+}
+
+/*
+ * Cross-check the dun(<inner>,@endian) wrapper against @tfm over all du sizes.
+ * Returns 0 on success or skip (no wrapper / rejected key); -EBADMSG on a real
+ * mismatch.
+ */
+static int test_skcipher_multi_du_endian(struct crypto_skcipher *tfm,
+ const char *alg_name,
+ const char *endian, bool big_endian,
+ const u8 *keybuf, unsigned int keylen)
+{
+ static const unsigned int du_sizes[] = { 512, 1024, 2048, 4096 };
+ char mdu_name[CRYPTO_MAX_ALG_NAME];
+ struct crypto_skcipher *mdu;
+ unsigned int ivsize;
+ u8 iv[TEST_MDU_MAX_IVSIZE];
+ unsigned int j;
+ int err;
+
+ if (snprintf(mdu_name, sizeof(mdu_name), "dun(%s,%s)", alg_name,
+ endian) >= (int)sizeof(mdu_name))
+ return 0;
+
+ mdu = crypto_alloc_skcipher(mdu_name, 0, 0);
+ if (IS_ERR(mdu)) {
+ /* No dun wrapper (ivsize not a multiple of 8, or too wide): skip. */
+ if (PTR_ERR(mdu) == -ENOENT || PTR_ERR(mdu) == -EINVAL)
+ return 0;
+ return PTR_ERR(mdu);
+ }
+
+ ivsize = crypto_skcipher_ivsize(mdu);
+ if (ivsize > TEST_MDU_MAX_IVSIZE) {
+ err = 0; /* wider than we have buffers for: skip */
+ goto out;
+ }
+
+ err = crypto_skcipher_setkey(mdu, keybuf, keylen);
+ if (err) {
+ err = 0; /* weak/rejected key (e.g. XTS equal halves): skip */
+ goto out;
+ }
+
+ for (j = 0; j < ARRAY_SIZE(du_sizes); j++) {
+ /* A random starting IV. */
+ get_random_bytes(iv, ivsize);
+ err = test_skcipher_multi_du_one(mdu, tfm, du_sizes[j],
+ big_endian, iv);
+ if (err)
+ break;
+ /* And one seeded to carry across a 64-bit limb / byte run. */
+ get_random_bytes(iv, ivsize);
+ test_mdu_iv_boundary(iv, ivsize, big_endian);
+ err = test_skcipher_multi_du_one(mdu, tfm, du_sizes[j],
+ big_endian, iv);
+ if (err)
+ break;
+ cond_resched();
+ }
+out:
+ crypto_free_skcipher(mdu);
+ return err;
+}
+
+/*
+ * Cross-check dun() dispatch against a single-DU reference, in both le and be,
+ * for every ivsize the template accepts (16: xts; 32: Adiantum).
+ */
+static int test_skcipher_multi_du(struct crypto_skcipher *tfm)
+{
+ const char *alg_name = crypto_skcipher_alg(tfm)->base.cra_name;
+ u8 keybuf[128];
+ unsigned int keylen;
+ int err;
+
+ /* Key the inner tfm; each dun() wrapper is keyed identically below. */
+ keylen = crypto_skcipher_min_keysize(tfm);
+ if (keylen > sizeof(keybuf))
+ return 0; /* unusually large key; skip rather than overflow */
+ get_random_bytes(keybuf, keylen);
+ err = crypto_skcipher_setkey(tfm, keybuf, keylen);
+ if (err)
+ return 0; /* weak/rejected key (e.g. XTS equal halves): skip */
+
+ err = test_skcipher_multi_du_endian(tfm, alg_name, "le", false,
+ keybuf, keylen);
+ if (err)
+ return err;
+ return test_skcipher_multi_du_endian(tfm, alg_name, "be", true,
+ keybuf, keylen);
+}
+
static int alg_test_skcipher(const struct alg_test_desc *desc,
const char *driver, u32 type, u32 mask)
{
@@ -3259,6 +3544,10 @@ static int alg_test_skcipher(const struct alg_test_desc *desc,
if (err)
goto out;
+ err = test_skcipher_multi_du(tfm);
+ if (err)
+ goto out;
+
err = test_skcipher_vs_generic_impl(desc->generic_driver, req, tsgls);
out:
free_cipher_test_sglists(tsgls);
--
2.47.3
^ permalink raw reply related
* [PATCH v5 4/5] dm crypt: batch a bio segment's sectors via dun()
From: Leonid Ravich @ 2026-06-30 8:34 UTC (permalink / raw)
To: linux-crypto, dm-devel
Cc: linux-block, linux-kernel, herbert, davem, ebiggers, snitzer,
mpatocka, axboe
In-Reply-To: <20260630083431.2772-1-lravich@amazon.com>
Submit one skcipher request per contiguous bio segment (a single
bio_vec) with data_unit_size = cc->sector_size, instead of one request
per sector. E.g. the default 512-byte sector with a 4 KiB bio_vec
becomes one request of 8 data units; the crypto layer (the dun()
template, or a native driver) walks the per-sector IV as a data-unit
counter. Because a bio_vec is one contiguous segment, the request uses
only the existing inline dmreq->sg_in[0]/sg_out[0] entry -- no per-bio
scatterlist allocation, and no regression on small random I/O.
crypt_alloc_tfms() wraps the skcipher in dun(<cipher>,<endian>) when
crypt_can_batch_dun() holds: an IV mode that is a data-unit counter (its
crypt_iv_operations sets dun_endian to the counter endianness -- "le" for
plain64, "be" for plain64be; non-counter modes such as lmk/tcw/eboiv
leave it NULL and are excluded), single-tfm, non-aead, and sector_size
512 or iv_large_sectors so the per-unit IV step is exactly one. This is
the same kind of name rewrite as essiv(), done in the one alloc helper so
callers are unchanged.
DM_CRYPT selects CRYPTO_DUN and dun() resolves against a sync inner
cipher, so wrapping has no acceptable failure that the bare cipher would
survive -- there is no fallback; any error propagates. (A config whose
only xts provider is async with no generic CRYPTO_XTS would now fail to
activate rather than silently run per-sector; generic xts is selected by
the dependency chain, so this does not arise in practice.)
crypt_convert_block_skcipher() handles both cases in one function: the
length is crypt_skcipher_len() -- a whole contiguous segment when
batching, else a single sector -- and data_unit_size is set
unconditionally (a dun() tfm reads it; a plain skcipher ignores it). It
advances the bio iterators itself (as the aead path already does) and
reports the bytes processed, so crypt_convert() advances cc_sector /
tag_offset uniformly via one helper, no per-case duplication.
Verified byte-equivalent to the per-sector path: plain64 and plain64be
dm-crypt with dun() produce ciphertext bit-identical to an unpatched
kernel over a 256 MB device (xts-aes driving the split).
Signed-off-by: Leonid Ravich <lravich@amazon.com>
---
drivers/md/Kconfig | 1 +
drivers/md/dm-crypt.c | 208 +++++++++++++++++++++++++++++++++---------
2 files changed, 166 insertions(+), 43 deletions(-)
diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
index a3fcdca7e6db..e8e299566374 100644
--- a/drivers/md/Kconfig
+++ b/drivers/md/Kconfig
@@ -299,6 +299,7 @@ config DM_CRYPT
select CRC32
select CRYPTO
select CRYPTO_CBC
+ select CRYPTO_DUN # multi-data-unit batching of contiguous sectors
select CRYPTO_ESSIV
select CRYPTO_LIB_AES
select CRYPTO_LIB_MD5 # needed by lmk IV mode
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 608b617fb817..44938223ad3e 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -115,6 +115,13 @@ struct crypt_iv_operations {
struct dm_crypt_request *dmreq);
void (*post)(struct crypt_config *cc, u8 *iv,
struct dm_crypt_request *dmreq);
+
+ /*
+ * Counter endianness ("le"/"be") for IV modes whose per-sector IV is a
+ * data-unit-number counter (IV(s+i) == IV(s)+i), batchable via
+ * dun(<cipher>,<dun_endian>). NULL for non-counter modes (lmk, tcw, ...).
+ */
+ const char *dun_endian;
};
struct iv_benbi_private {
@@ -151,6 +158,7 @@ enum cipher_flags {
CRYPT_IV_LARGE_SECTORS, /* Calculate IV from sector_size, not 512B sectors */
CRYPT_ENCRYPT_PREPROCESS, /* Must preprocess data for encryption (elephant) */
CRYPT_KEY_MAC_SIZE_SET, /* The integrity_key_size option was used */
+ CRYPT_MULTI_DATA_UNIT, /* Batch a bio segment's sectors per crypto request */
};
/*
@@ -1018,15 +1026,19 @@ static const struct crypt_iv_operations crypt_iv_plain_ops = {
};
static const struct crypt_iv_operations crypt_iv_plain64_ops = {
- .generator = crypt_iv_plain64_gen
+ .generator = crypt_iv_plain64_gen,
+ .dun_endian = "le",
};
static const struct crypt_iv_operations crypt_iv_plain64be_ops = {
- .generator = crypt_iv_plain64be_gen
+ .generator = crypt_iv_plain64be_gen,
+ .dun_endian = "be",
};
static const struct crypt_iv_operations crypt_iv_essiv_ops = {
- .generator = crypt_iv_essiv_gen
+ .generator = crypt_iv_essiv_gen,
+ /* IV input is le64(sector); the salt-encrypt lives in essiv(). */
+ .dun_endian = "le",
};
static const struct crypt_iv_operations crypt_iv_benbi_ops = {
@@ -1349,21 +1361,51 @@ static int crypt_convert_block_aead(struct crypt_config *cc,
return r;
}
+/*
+ * Bytes to process in one skcipher request: a whole contiguous segment when
+ * batching (multi-data-unit), else one sector. 0 means an unusable
+ * (sub-sector / misaligned) segment.
+ */
+static unsigned int crypt_skcipher_len(struct crypt_config *cc,
+ const struct bio_vec *bv_in,
+ const struct bio_vec *bv_out)
+{
+ const unsigned int sector_size = cc->sector_size;
+
+ if (test_bit(CRYPT_MULTI_DATA_UNIT, &cc->cipher_flags))
+ return round_down(min(bv_in->bv_len, bv_out->bv_len),
+ sector_size);
+
+ /* Reject unexpected unaligned bio. */
+ if (unlikely(bv_in->bv_len & (sector_size - 1)))
+ return 0;
+ return sector_size;
+}
+
+/*
+ * Encrypt/decrypt one bio segment (one sector, or a whole segment when
+ * batching) and report the bytes done in *out_processed. The integrity /
+ * preprocess / post handling is inert when batching (crypt_can_batch_dun()
+ * excludes those configs).
+ */
static int crypt_convert_block_skcipher(struct crypt_config *cc,
struct convert_context *ctx,
struct skcipher_request *req,
- unsigned int tag_offset)
+ unsigned int tag_offset,
+ unsigned int *out_processed)
{
struct bio_vec bv_in = bio_iter_iovec(ctx->bio_in, ctx->iter_in);
struct bio_vec bv_out = bio_iter_iovec(ctx->bio_out, ctx->iter_out);
+ const unsigned int sector_size = cc->sector_size;
struct scatterlist *sg_in, *sg_out;
struct dm_crypt_request *dmreq;
u8 *iv, *org_iv, *tag_iv;
__le64 *sector;
+ unsigned int len;
int r = 0;
- /* Reject unexpected unaligned bio. */
- if (unlikely(bv_in.bv_len & (cc->sector_size - 1)))
+ len = crypt_skcipher_len(cc, &bv_in, &bv_out);
+ if (unlikely(!len))
return -EIO;
dmreq = dmreq_of_req(cc, req);
@@ -1386,10 +1428,10 @@ static int crypt_convert_block_skcipher(struct crypt_config *cc,
sg_out = &dmreq->sg_out[0];
sg_init_table(sg_in, 1);
- sg_set_page(sg_in, bv_in.bv_page, cc->sector_size, bv_in.bv_offset);
+ sg_set_page(sg_in, bv_in.bv_page, len, bv_in.bv_offset);
sg_init_table(sg_out, 1);
- sg_set_page(sg_out, bv_out.bv_page, cc->sector_size, bv_out.bv_offset);
+ sg_set_page(sg_out, bv_out.bv_page, len, bv_out.bv_offset);
if (cc->iv_gen_ops) {
/* For READs use IV stored in integrity metadata */
@@ -1410,7 +1452,9 @@ static int crypt_convert_block_skcipher(struct crypt_config *cc,
memcpy(iv, org_iv, cc->iv_size);
}
- skcipher_request_set_crypt(req, sg_in, sg_out, cc->sector_size, iv);
+ skcipher_request_set_crypt(req, sg_in, sg_out, len, iv);
+ /* A dun() tfm reads this; a plain skcipher ignores it (len is one sector). */
+ skcipher_request_set_data_unit_size(req, sector_size);
if (bio_data_dir(ctx->bio_in) == WRITE)
r = crypto_skcipher_encrypt(req);
@@ -1420,9 +1464,10 @@ static int crypt_convert_block_skcipher(struct crypt_config *cc,
if (!r && cc->iv_gen_ops && cc->iv_gen_ops->post)
cc->iv_gen_ops->post(cc, org_iv, dmreq);
- bio_advance_iter(ctx->bio_in, &ctx->iter_in, cc->sector_size);
- bio_advance_iter(ctx->bio_out, &ctx->iter_out, cc->sector_size);
+ bio_advance_iter(ctx->bio_in, &ctx->iter_in, len);
+ bio_advance_iter(ctx->bio_out, &ctx->iter_out, len);
+ *out_processed = len;
return r;
}
@@ -1509,13 +1554,25 @@ static void crypt_free_req(struct crypt_config *cc, void *req, struct bio *base_
crypt_free_req_skcipher(cc, req, base_bio);
}
+/*
+ * Advance the IV-sector and integrity-tag cursors by @processed bytes; the
+ * bio iterators are advanced by the per-block helpers themselves.
+ */
+static void crypt_convert_advance(struct crypt_config *cc,
+ struct convert_context *ctx,
+ unsigned int processed)
+{
+ ctx->cc_sector += processed >> SECTOR_SHIFT;
+ ctx->tag_offset += processed / cc->sector_size;
+}
+
/*
* Encrypt / decrypt data from one bio to another one (can be the same one)
*/
static blk_status_t crypt_convert(struct crypt_config *cc,
struct convert_context *ctx, bool atomic, bool reset_pending)
{
- unsigned int sector_step = cc->sector_size >> SECTOR_SHIFT;
+ unsigned int processed;
int r;
/*
@@ -1536,10 +1593,12 @@ static blk_status_t crypt_convert(struct crypt_config *cc,
atomic_inc(&ctx->cc_pending);
+ processed = cc->sector_size;
if (crypt_integrity_aead(cc))
r = crypt_convert_block_aead(cc, ctx, ctx->r.req_aead, ctx->tag_offset);
else
- r = crypt_convert_block_skcipher(cc, ctx, ctx->r.req, ctx->tag_offset);
+ r = crypt_convert_block_skcipher(cc, ctx, ctx->r.req,
+ ctx->tag_offset, &processed);
switch (r) {
/*
@@ -1559,8 +1618,7 @@ static blk_status_t crypt_convert(struct crypt_config *cc,
* exit and continue processing in a workqueue
*/
ctx->r.req = NULL;
- ctx->tag_offset++;
- ctx->cc_sector += sector_step;
+ crypt_convert_advance(cc, ctx, processed);
return BLK_STS_DEV_RESOURCE;
}
} else {
@@ -1574,16 +1632,14 @@ static blk_status_t crypt_convert(struct crypt_config *cc,
*/
case -EINPROGRESS:
ctx->r.req = NULL;
- ctx->tag_offset++;
- ctx->cc_sector += sector_step;
+ crypt_convert_advance(cc, ctx, processed);
continue;
/*
* The request was already processed (synchronously).
*/
case 0:
atomic_dec(&ctx->cc_pending);
- ctx->cc_sector += sector_step;
- ctx->tag_offset++;
+ crypt_convert_advance(cc, ctx, processed);
if (!atomic)
cond_resched();
continue;
@@ -2345,12 +2401,37 @@ static int crypt_alloc_tfms_aead(struct crypt_config *cc, char *ciphermode)
return 0;
}
+/*
+ * Whether to wrap the cipher in dun() for multi-data-unit batching: a counter
+ * IV mode (dun_endian set: plain64 "le", plain64be "be", essiv "le"), single-
+ * tfm, non-aead, and a per-unit IV step of exactly one (512B sectors or
+ * iv_large_sectors). Integrity is configured
+ * after alloc, so it is re-checked post-alloc in crypt_ctr_cipher(); an
+ * integrity config keeps an inert dun() wrapper but never sets the batch flag.
+ */
+static bool crypt_can_batch_dun(struct crypt_config *cc)
+{
+ return !crypt_integrity_aead(cc) && cc->tfms_count == 1 &&
+ cc->iv_gen_ops && cc->iv_gen_ops->dun_endian &&
+ (cc->sector_size == (1 << SECTOR_SHIFT) ||
+ test_bit(CRYPT_IV_LARGE_SECTORS, &cc->cipher_flags));
+}
+
static int crypt_alloc_tfms(struct crypt_config *cc, char *ciphermode)
{
+ char dun_api[CRYPTO_MAX_ALG_NAME];
+
if (crypt_integrity_aead(cc))
return crypt_alloc_tfms_aead(cc, ciphermode);
- else
- return crypt_alloc_tfms_skcipher(cc, ciphermode);
+
+ /* Wrap in dun() for batching when eligible (like the essiv() rewrite). */
+ if (crypt_can_batch_dun(cc)) {
+ if (snprintf(dun_api, sizeof(dun_api), "dun(%s,%s)", ciphermode,
+ cc->iv_gen_ops->dun_endian) >= (int)sizeof(dun_api))
+ return -ENAMETOOLONG;
+ ciphermode = dun_api;
+ }
+ return crypt_alloc_tfms_skcipher(cc, ciphermode);
}
static unsigned int crypt_subkey_size(struct crypt_config *cc)
@@ -2747,25 +2828,15 @@ static void crypt_dtr(struct dm_target *ti)
dm_audit_log_dtr(DM_MSG_PREFIX, ti, 1);
}
-static int crypt_ctr_ivmode(struct dm_target *ti, const char *ivmode)
+/*
+ * Select cc->iv_gen_ops from the IV mode string -- pure parsing, no tfm
+ * dependency, so it runs before alloc and lets crypt_can_batch_dun() see the
+ * mode. The tfm-dependent IV sizing is finished later by crypt_ctr_ivmode().
+ */
+static int crypt_select_ivmode(struct dm_target *ti, const char *ivmode)
{
struct crypt_config *cc = ti->private;
- if (crypt_integrity_aead(cc))
- cc->iv_size = crypto_aead_ivsize(any_tfm_aead(cc));
- else
- cc->iv_size = crypto_skcipher_ivsize(any_tfm(cc));
-
- if (cc->iv_size)
- /* at least a 64 bit sector number should fit in our buffer */
- cc->iv_size = max(cc->iv_size,
- (unsigned int)(sizeof(u64) / sizeof(u8)));
- else if (ivmode) {
- DMWARN("Selected cipher does not support IVs");
- ivmode = NULL;
- }
-
- /* Choose ivmode, see comments at iv code. */
if (ivmode == NULL)
cc->iv_gen_ops = NULL;
else if (strcmp(ivmode, "plain") == 0)
@@ -2803,12 +2874,8 @@ static int crypt_ctr_ivmode(struct dm_target *ti, const char *ivmode)
}
} else if (strcmp(ivmode, "tcw") == 0) {
cc->iv_gen_ops = &crypt_iv_tcw_ops;
- cc->key_parts += 2; /* IV + whitening */
- cc->key_extra_size = cc->iv_size + TCW_WHITENING_SIZE;
} else if (strcmp(ivmode, "random") == 0) {
cc->iv_gen_ops = &crypt_iv_random_ops;
- /* Need storage space in integrity fields. */
- cc->integrity_iv_size = cc->iv_size;
} else {
ti->error = "Invalid IV mode";
return -EINVAL;
@@ -2817,6 +2884,37 @@ static int crypt_ctr_ivmode(struct dm_target *ti, const char *ivmode)
return 0;
}
+static int crypt_ctr_ivmode(struct dm_target *ti, const char *ivmode)
+{
+ struct crypt_config *cc = ti->private;
+
+ if (crypt_integrity_aead(cc))
+ cc->iv_size = crypto_aead_ivsize(any_tfm_aead(cc));
+ else
+ cc->iv_size = crypto_skcipher_ivsize(any_tfm(cc));
+
+ if (cc->iv_size)
+ /* at least a 64 bit sector number should fit in our buffer */
+ cc->iv_size = max(cc->iv_size,
+ (unsigned int)(sizeof(u64) / sizeof(u8)));
+ else if (ivmode) {
+ DMWARN("Selected cipher does not support IVs");
+ ivmode = NULL;
+ cc->iv_gen_ops = NULL;
+ }
+
+ /* Finish the tfm-dependent IV sizing; modes are already selected. */
+ if (cc->iv_gen_ops == &crypt_iv_tcw_ops) {
+ cc->key_parts += 2; /* IV + whitening */
+ cc->key_extra_size = cc->iv_size + TCW_WHITENING_SIZE;
+ } else if (cc->iv_gen_ops == &crypt_iv_random_ops) {
+ /* Need storage space in integrity fields. */
+ cc->integrity_iv_size = cc->iv_size;
+ }
+
+ return 0;
+}
+
/*
* Workaround to parse HMAC algorithm from AEAD crypto API spec.
* The HMAC is needed to calculate tag size (HMAC digest size).
@@ -2914,7 +3012,12 @@ static int crypt_ctr_cipher_new(struct dm_target *ti, char *cipher_in, char *key
cc->key_parts = cc->tfms_count;
- /* Allocate cipher */
+ /* Select IV mode before alloc so dun() wrapping can be decided. */
+ ret = crypt_select_ivmode(ti, *ivmode);
+ if (ret < 0)
+ return ret;
+
+ /* Allocate cipher (skcipher may be wrapped in dun()). */
ret = crypt_alloc_tfms(cc, cipher_api);
if (ret < 0) {
ti->error = "Error allocating crypto tfm";
@@ -2999,7 +3102,13 @@ static int crypt_ctr_cipher_old(struct dm_target *ti, char *cipher_in, char *key
goto bad_mem;
}
- /* Allocate cipher */
+ /* Select IV mode before alloc so dun() wrapping can be decided. */
+ ret = crypt_select_ivmode(ti, *ivmode);
+ if (ret < 0) {
+ kfree(cipher_api);
+ return ret;
+ }
+
ret = crypt_alloc_tfms(cc, cipher_api);
if (ret < 0) {
ti->error = "Error allocating crypto tfm";
@@ -3063,6 +3172,19 @@ static int crypt_ctr_cipher(struct dm_target *ti, char *cipher_in, char *key)
}
}
+ /*
+ * Enable batching only if the cipher was dun()-wrapped at alloc time and
+ * no integrity was configured (integrity is set up after cipher alloc).
+ */
+ if (!crypt_integrity_aead(cc) && !cc->integrity_tag_size &&
+ !cc->integrity_iv_size &&
+ !strncmp(crypto_skcipher_alg(any_tfm(cc))->base.cra_name,
+ "dun(", 4)) {
+ set_bit(CRYPT_MULTI_DATA_UNIT, &cc->cipher_flags);
+ DMINFO("Using multi-data-unit crypto offload (du=%u)",
+ cc->sector_size);
+ }
+
/* wipe the kernel key payload copy */
if (cc->key_string)
memset(cc->key, 0, cc->key_size * sizeof(u8));
--
2.47.3
^ permalink raw reply related
* Re: [PATCH 3/7] cpufreq: rcpufreq_dt: use vertical import style
From: Miguel Ojeda @ 2026-06-30 8:35 UTC (permalink / raw)
To: Zhongqiu Han
Cc: Guru Das Srinagesh, Miguel Ojeda, rust-for-linux, linux-kernel,
Danilo Krummrich, Abdiel Janulgue, Daniel Almeida, Robin Murphy,
Andreas Hindborg, Boqun Feng, Gary Guo, Björn Roy Baron,
Benno Lossin, Alice Ryhl, Trevor Gross, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Drew Fustini, Guo Ren, Fu Wei,
Michal Wilczynski, Uwe Kleine-König, Rafael J. Wysocki,
Viresh Kumar, Jens Axboe, FUJITA Tomonori, Andrew Lunn,
Heiner Kallweit, Russell King, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, David Airlie, Simona Vetter,
driver-core, linux-riscv, linux-pwm, linux-pm, linux-block,
netdev, nova-gpu, dri-devel
In-Reply-To: <f0ed588a-6cfb-4079-a2fe-7c14f1b42727@oss.qualcomm.com>
On Mon, Jun 29, 2026 at 2:43 PM Zhongqiu Han
<zhongqiu.han@oss.qualcomm.com> wrote:
>
> If the preferred style is to place each imported item on its own line,
> shouldn't imports such as
>
> cpu, cpufreq,
>
> be formatted similarly as well?
Indeed, good eyes.
To do what we want, `rustfmt` needs the `//` at the end of that level
too (in the future, it will be without the `//`), i.e. the patch
probably passes `rustfmtcheck`, but it still needs to split that line
and add the other `//`.
Cheers,
Miguel
^ permalink raw reply
* Re: [PATCH v2 1/6] dt-bindings: net: bluetooth: Document Qualcomm IPQ5018 Bluetooth controller
From: Konrad Dybcio @ 2026-06-30 9:06 UTC (permalink / raw)
To: Krzysztof Kozlowski, George Moussalem, Kathiravan Thirumoorthy
Cc: Jens Axboe, Ulf Hansson, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Johannes Berg, Jeff Johnson, Bartosz Golaszewski,
Marcel Holtmann, Luiz Augusto von Dentz, Balakrishna Godavarthi,
Rocky Liao, Saravana Kannan, Andrew Lunn, Heiner Kallweit,
Russell King, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Bjorn Andersson, Konrad Dybcio,
Mathieu Poirier, Philipp Zabel, linux-block, linux-kernel,
linux-mmc, devicetree, linux-wireless, ath10k, linux-arm-msm,
linux-bluetooth, netdev, linux-remoteproc
In-Reply-To: <20260630-wondrous-lean-stoat-be0b9a@quoll>
On 6/30/26 9:15 AM, Krzysztof Kozlowski wrote:
> On Mon, Jun 29, 2026 at 05:01:44PM +0400, George Moussalem wrote:
>> +unevaluatedProperties: false
>> +
>> +examples:
>> + - |
>> + #include <dt-bindings/clock/qcom,gcc-ipq5018.h>
>> + #include <dt-bindings/interrupt-controller/arm-gic.h>
>> + #include <dt-bindings/reset/qcom,gcc-ipq5018.h>
>> +
>> + bluetooth {
>
> Don't send new versions while discussion is still going. I need to
> repeat my question - what bus does that sit on?
>
> Device nodes represent real devices. Real devices sit on a bus, usually.
> Do you have here a bus?
+Kathiravan would you know what the remoteproc-driven bluetooth is
connected over on IPQ5018?
Konrad
^ permalink raw reply
* Re: [PATCH v2 1/6] dt-bindings: net: bluetooth: Document Qualcomm IPQ5018 Bluetooth controller
From: Krzysztof Kozlowski @ 2026-06-30 9:09 UTC (permalink / raw)
To: Konrad Dybcio, George Moussalem, Kathiravan Thirumoorthy
Cc: Jens Axboe, Ulf Hansson, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Johannes Berg, Jeff Johnson, Bartosz Golaszewski,
Marcel Holtmann, Luiz Augusto von Dentz, Balakrishna Godavarthi,
Rocky Liao, Saravana Kannan, Andrew Lunn, Heiner Kallweit,
Russell King, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, Simon Horman, Bjorn Andersson, Konrad Dybcio,
Mathieu Poirier, Philipp Zabel, linux-block, linux-kernel,
linux-mmc, devicetree, linux-wireless, ath10k, linux-arm-msm,
linux-bluetooth, netdev, linux-remoteproc
In-Reply-To: <7348d7e6-18b3-4064-9fb2-932cce76816e@oss.qualcomm.com>
On 30/06/2026 11:06, Konrad Dybcio wrote:
> On 6/30/26 9:15 AM, Krzysztof Kozlowski wrote:
>> On Mon, Jun 29, 2026 at 05:01:44PM +0400, George Moussalem wrote:
>>> +unevaluatedProperties: false
>>> +
>>> +examples:
>>> + - |
>>> + #include <dt-bindings/clock/qcom,gcc-ipq5018.h>
>>> + #include <dt-bindings/interrupt-controller/arm-gic.h>
>>> + #include <dt-bindings/reset/qcom,gcc-ipq5018.h>
>>> +
>>> + bluetooth {
>>
>> Don't send new versions while discussion is still going. I need to
>> repeat my question - what bus does that sit on?
>>
>> Device nodes represent real devices. Real devices sit on a bus, usually.
>> Do you have here a bus?
>
> +Kathiravan would you know what the remoteproc-driven bluetooth is
> connected over on IPQ5018?
No need, this got resolved. I missed that this was merged with
remoteproc and remoteprocs we represent in top-level as it uses non-bus
way (APCS IPC or whatever its called). We could fix that and maybe this
should be a child of APCS... but maybe not and it feels like out of
scope of this patchset.
Best regards,
Krzysztof
^ permalink raw reply
* Re: [PATCH 1/7] samples: rust_dma: use vertical import style
From: Danilo Krummrich @ 2026-06-30 9:48 UTC (permalink / raw)
To: Guru Das Srinagesh
Cc: Miguel Ojeda, rust-for-linux, linux-kernel, Abdiel Janulgue,
Daniel Almeida, Robin Murphy, Andreas Hindborg, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Alice Ryhl,
Trevor Gross, Tamir Duberstein, Alexandre Courbot,
Onur Özkan, Drew Fustini, Guo Ren, Fu Wei, Michal Wilczynski,
Uwe Kleine-König, Rafael J. Wysocki, Viresh Kumar,
Jens Axboe, FUJITA Tomonori, Andrew Lunn, Heiner Kallweit,
Russell King, David S. Miller, Eric Dumazet, Jakub Kicinski,
Paolo Abeni, David Airlie, Simona Vetter, driver-core,
linux-riscv, linux-pwm, linux-pm, linux-block, netdev, nova-gpu,
dri-devel
In-Reply-To: <20260628-b4-rust-vertical-imports-v1-1-98bc71d4810b@gurudas.dev>
On Mon Jun 29, 2026 at 5:38 AM CEST, Guru Das Srinagesh wrote:
> page, pci,
Can you please also convert this one? Patch 7 also misses at least one case.
Thanks,
Danilo
^ permalink raw reply
* [PATCH blktests] nvme/039: drain udev events before passthru error injection
From: Mateusz Nowicki @ 2026-06-30 10:19 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, Mateusz Nowicki
From: Mateusz Nowicki <Mateusz.Nowicki@posteo.net>
When run repeatedly, nvme/039 fails intermittently with a regular block
read error instead of the expected passthru line, e.g.:
nvme0n1: Read(0x2) @ LBA 0, 8 blocks, Invalid Command Opcode (sct 0x0 / sc 0x1) DNR
inject_write_fault_on_write() writes LBA 0 of the whole-disk device,
whose release emits a change uevent that makes udev run blkid, reading
LBA 0 asynchronously. The following passthru injection arms a one-shot
fault (times=1); if the stray blkid read races in first it consumes the
fault and is logged via nvme_log_error() (no cdw fields), so the expected
passthru line is never emitted and the test fails.
Drain the pending uevent with udevadm settle before the passthru section.
Signed-off-by: Mateusz Nowicki <Mateusz.Nowicki@posteo.net>
---
tests/nvme/039 | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/nvme/039 b/tests/nvme/039
index 7ca48ec..8c70c5c 100755
--- a/tests/nvme/039
+++ b/tests/nvme/039
@@ -206,6 +206,8 @@ test_device() {
inject_invalid_status_on_read "${ns_dev}"
inject_write_fault_on_write "${ns_dev}"
+ udevadm settle
+
if [ -e "$TEST_DEV_SYSFS/passthru_err_log_enabled" ]; then
_nvme_passthru_logging_setup "${ns_dev}" "${ctrl_dev}"
--
2.53.0
^ permalink raw reply related
* [PATCH] drbd: reject oversized DataReply before signed conversion
From: Tianchu Chen @ 2026-06-30 10:59 UTC (permalink / raw)
To: philipp.reisner, lars.ellenberg, christoph.boehmwalder
Cc: drbd-dev, linux-block, linux-kernel, axboe
From: Tianchu Chen <flynnnchen@tencent.com>
Discovered by Atuin - Automated Vulnerability Discovery Engine.
Reject DataReply payload lengths that cannot fit in recv_dless_read()'s
signed size argument so a bogus remote peer cannot wrap the length negative
and turn it into a huge heap OOB-write.
Fixes: b411b3637fa7 ("The DRBD driver")
Cc: stable@vger.kernel.org
Signed-off-by: Tianchu Chen <flynnnchen@tencent.com>
---
drivers/block/drbd/drbd_receiver.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index 58b95bf4b..5bd3df483 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -24,6 +24,7 @@
#include <linux/memcontrol.h>
#include <linux/mm_inline.h>
#include <linux/slab.h>
+#include <linux/limits.h>
#include <uapi/linux/sched/types.h>
#include <linux/sched/signal.h>
#include <linux/pkt_sched.h>
@@ -1947,6 +1948,9 @@ static int receive_DataReply(struct drbd_connection *connection, struct packet_i
if (unlikely(!req))
return -EIO;
+ if (pi->size > INT_MAX)
+ return -EINVAL;
+
err = recv_dless_read(peer_device, req, sector, pi->size);
if (!err)
req_mod(req, DATA_RECEIVED, peer_device);
--
2.51.0
^ permalink raw reply related
* Re: [PATCH blktests] nvme/039: drain udev events before passthru error injection
From: John Garry @ 2026-06-30 11:29 UTC (permalink / raw)
To: Mateusz Nowicki, Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme
In-Reply-To: <20260630101936.2371-1-mateusz.nowicki@posteo.net>
+ linux-nvme
On 30/06/2026 11:19, Mateusz Nowicki wrote:
> From: Mateusz Nowicki <Mateusz.Nowicki@posteo.net>
>
> When run repeatedly, nvme/039 fails intermittently with a regular block
> read error instead of the expected passthru line, e.g.:
>
> nvme0n1: Read(0x2) @ LBA 0, 8 blocks, Invalid Command Opcode (sct 0x0 / sc 0x1) DNR
This is not the error which I was seeing, however this change seems to
make the test pass reliably for the me. I have not debugged why the test
was intermittently failing for me.
Please also note that I did trigger a kernel warn previously for this
test and I posted a proposed fix in:
https://lore.kernel.org/linux-nvme/20260630102717.2671475-1-john.g.garry@oracle.com/T/#u
thanks
>
> inject_write_fault_on_write() writes LBA 0 of the whole-disk device,
> whose release emits a change uevent that makes udev run blkid, reading
> LBA 0 asynchronously. The following passthru injection arms a one-shot
> fault (times=1); if the stray blkid read races in first it consumes the
> fault and is logged via nvme_log_error() (no cdw fields), so the expected
> passthru line is never emitted and the test fails.
>
> Drain the pending uevent with udevadm settle before the passthru section.
>
> Signed-off-by: Mateusz Nowicki <Mateusz.Nowicki@posteo.net>
> ---
> tests/nvme/039 | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tests/nvme/039 b/tests/nvme/039
> index 7ca48ec..8c70c5c 100755
> --- a/tests/nvme/039
> +++ b/tests/nvme/039
> @@ -206,6 +206,8 @@ test_device() {
> inject_invalid_status_on_read "${ns_dev}"
> inject_write_fault_on_write "${ns_dev}"
>
> + udevadm settle
> +
> if [ -e "$TEST_DEV_SYSFS/passthru_err_log_enabled" ]; then
> _nvme_passthru_logging_setup "${ns_dev}" "${ctrl_dev}"
>
^ permalink raw reply
* [PATCH v2 blktests] block/044: basic block error injection sanity test
From: Christoph Hellwig @ 2026-06-30 11:37 UTC (permalink / raw)
To: shinichiro.kawasaki; +Cc: linux-block
Test the basic block layer error injection functionality.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
Changes since v1:
- change license to GPLv2 or later
- incorporate various improvements from Shin'ichiro
tests/block/044 | 62 +++++++++++++++++++++++++++++++++++++++++++++
tests/block/044.out | 9 +++++++
2 files changed, 71 insertions(+)
create mode 100755 tests/block/044
create mode 100644 tests/block/044.out
diff --git a/tests/block/044 b/tests/block/044
new file mode 100755
index 000000000000..097dd79d3bda
--- /dev/null
+++ b/tests/block/044
@@ -0,0 +1,62 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2026 Christoph Hellwig.
+#
+# Basic block error injection test.
+
+. tests/block/rc
+. common/scsi_debug
+
+DESCRIPTION="basic block error injection test"
+QUICK=1
+
+requires()
+{
+ _have_kernel_option BLK_ERROR_INJECTION
+ _have_loadable_scsi_debug
+ _have_program xfs_io
+}
+
+# load and remove scsi_debug once to test the static_key bug in the
+# initial commit
+_test_load_unload()
+{
+ if ! _init_scsi_debug dev_size_mb=500; then
+ return 1
+ fi
+ echo "Testing unload without rules"
+ _exit_scsi_debug
+}
+
+_test_rules()
+{
+ if ! _init_scsi_debug dev_size_mb=500; then
+ return 1
+ fi
+
+ local dev=${SCSI_DEBUG_DEVICES[0]}
+ local debugfs_file="/sys/kernel/debug/block/$dev/error_injection"
+
+ echo "Testing valid rules"
+ echo "add,op=WRITE,status=RESOURCE,start=0,nr_sectors=8" > "$debugfs_file"
+ echo "add,op=READ,status=IOERR,start=16,nr_sectors=8" > "$debugfs_file"
+ xfs_io -d -c 'pwrite -q 0 4096' /dev/"$dev"
+ xfs_io -d -c 'pread -q 0 4096' /dev/"$dev"
+ xfs_io -d -c 'pwrite -q 4096 4096' /dev/"$dev"
+ xfs_io -d -c 'pread -q 8192 8192' /dev/"$dev"
+
+ echo "Testing invalid rules"
+ echo "op=READ,status=IOERR" > "$debugfs_file"
+ echo "add,op=READ,status=EIO,start=32" > "$debugfs_file"
+ _exit_scsi_debug
+}
+
+test()
+{
+ echo "Running ${TEST_NAME}"
+
+ _test_load_unload
+ _test_rules
+
+ echo "Test complete"
+}
diff --git a/tests/block/044.out b/tests/block/044.out
new file mode 100644
index 000000000000..92efcddf7c8e
--- /dev/null
+++ b/tests/block/044.out
@@ -0,0 +1,9 @@
+Running block/044
+Testing unload without rules
+Testing valid rules
+pwrite: Cannot allocate memory
+pread: Input/output error
+Testing invalid rules
+tests/block/044: line 56: echo: write error: Invalid argument
+tests/block/044: line 57: echo: write error: Invalid argument
+Test complete
--
2.53.0
^ permalink raw reply related
* Re: [PATCH v2 5/6] arm64: dts: qcom: ipq5018: add nodes required for Bluetooth support
From: Konrad Dybcio @ 2026-06-30 11:40 UTC (permalink / raw)
To: george.moussalem, Jens Axboe, Ulf Hansson, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Johannes Berg, Jeff Johnson,
Bartosz Golaszewski, Marcel Holtmann, Luiz Augusto von Dentz,
Balakrishna Godavarthi, Rocky Liao, Saravana Kannan, Andrew Lunn,
Heiner Kallweit, Russell King, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Bjorn Andersson,
Konrad Dybcio, Mathieu Poirier, Philipp Zabel
Cc: linux-block, linux-kernel, linux-mmc, devicetree, linux-wireless,
ath10k, linux-arm-msm, linux-bluetooth, netdev, linux-remoteproc
In-Reply-To: <20260629-ipq5018-bluetooth-v2-5-02770f03b6bb@outlook.com>
On 6/29/26 3:01 PM, George Moussalem via B4 Relay wrote:
> From: George Moussalem <george.moussalem@outlook.com>
>
> Add nodes for the reserved memory carveout and Bluetooth.
>
> Signed-off-by: George Moussalem <george.moussalem@outlook.com>
> ---
> arch/arm64/boot/dts/qcom/ipq5018.dtsi | 25 ++++++++++++++++++++++++-
> 1 file changed, 24 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/boot/dts/qcom/ipq5018.dtsi b/arch/arm64/boot/dts/qcom/ipq5018.dtsi
> index 6f8004a22a1f..65a47ba7d3a3 100644
> --- a/arch/arm64/boot/dts/qcom/ipq5018.dtsi
> +++ b/arch/arm64/boot/dts/qcom/ipq5018.dtsi
> @@ -17,6 +17,23 @@ / {
> #address-cells = <2>;
> #size-cells = <2>;
>
> + bluetooth: bluetooth {
> + compatible = "qcom,ipq5018-bt";
> +
> + firmware-name = "qca/bt_fw_patch.mbn";
Is this fw vendor-signed?
Konrad
^ permalink raw reply
* Re: [PATCH v2 5/6] arm64: dts: qcom: ipq5018: add nodes required for Bluetooth support
From: George Moussalem @ 2026-06-30 12:09 UTC (permalink / raw)
To: Konrad Dybcio, Jens Axboe, Ulf Hansson, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Johannes Berg, Jeff Johnson,
Bartosz Golaszewski, Marcel Holtmann, Luiz Augusto von Dentz,
Balakrishna Godavarthi, Rocky Liao, Saravana Kannan, Andrew Lunn,
Heiner Kallweit, Russell King, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Bjorn Andersson,
Konrad Dybcio, Mathieu Poirier, Philipp Zabel
Cc: linux-block, linux-kernel, linux-mmc, devicetree, linux-wireless,
ath10k, linux-arm-msm, linux-bluetooth, netdev, linux-remoteproc
In-Reply-To: <f3c79cb4-02eb-4e4b-b5b4-9732876c075c@oss.qualcomm.com>
On 6/30/26 15:40, Konrad Dybcio wrote:
> On 6/29/26 3:01 PM, George Moussalem via B4 Relay wrote:
>> From: George Moussalem <george.moussalem@outlook.com>
>>
>> Add nodes for the reserved memory carveout and Bluetooth.
>>
>> Signed-off-by: George Moussalem <george.moussalem@outlook.com>
>> ---
>> arch/arm64/boot/dts/qcom/ipq5018.dtsi | 25 ++++++++++++++++++++++++-
>> 1 file changed, 24 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/boot/dts/qcom/ipq5018.dtsi b/arch/arm64/boot/dts/qcom/ipq5018.dtsi
>> index 6f8004a22a1f..65a47ba7d3a3 100644
>> --- a/arch/arm64/boot/dts/qcom/ipq5018.dtsi
>> +++ b/arch/arm64/boot/dts/qcom/ipq5018.dtsi
>> @@ -17,6 +17,23 @@ / {
>> #address-cells = <2>;
>> #size-cells = <2>;
>>
>> + bluetooth: bluetooth {
>> + compatible = "qcom,ipq5018-bt";
>> +
>> + firmware-name = "qca/bt_fw_patch.mbn";
>
> Is this fw vendor-signed?
I've just analyzed the mbn file (and the mdt + b0x files): it only
contains hashes for the mdt and b02 segments, no signature/certs at all.
I've used your pil squasher to create the mbn file. Here are the FW files:
https://github.com/georgemoussalem/openwrt/tree/ipq50xx-bluetooth/package/firmware/qca-bt-firmware/files
Perhaps you can double check?
>
> Konrad
Best regards,
George
^ permalink raw reply
* Re: [PATCH v2 5/6] arm64: dts: qcom: ipq5018: add nodes required for Bluetooth support
From: Konrad Dybcio @ 2026-06-30 12:12 UTC (permalink / raw)
To: George Moussalem, Jens Axboe, Ulf Hansson, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Johannes Berg, Jeff Johnson,
Bartosz Golaszewski, Marcel Holtmann, Luiz Augusto von Dentz,
Balakrishna Godavarthi, Rocky Liao, Saravana Kannan, Andrew Lunn,
Heiner Kallweit, Russell King, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Bjorn Andersson,
Konrad Dybcio, Mathieu Poirier, Philipp Zabel
Cc: linux-block, linux-kernel, linux-mmc, devicetree, linux-wireless,
ath10k, linux-arm-msm, linux-bluetooth, netdev, linux-remoteproc
In-Reply-To: <SN7PR19MB6736F8FEB36D52E867C000E29DF72@SN7PR19MB6736.namprd19.prod.outlook.com>
On 6/30/26 2:09 PM, George Moussalem wrote:
> On 6/30/26 15:40, Konrad Dybcio wrote:
>> On 6/29/26 3:01 PM, George Moussalem via B4 Relay wrote:
>>> From: George Moussalem <george.moussalem@outlook.com>
>>>
>>> Add nodes for the reserved memory carveout and Bluetooth.
>>>
>>> Signed-off-by: George Moussalem <george.moussalem@outlook.com>
>>> ---
>>> arch/arm64/boot/dts/qcom/ipq5018.dtsi | 25 ++++++++++++++++++++++++-
>>> 1 file changed, 24 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/arm64/boot/dts/qcom/ipq5018.dtsi b/arch/arm64/boot/dts/qcom/ipq5018.dtsi
>>> index 6f8004a22a1f..65a47ba7d3a3 100644
>>> --- a/arch/arm64/boot/dts/qcom/ipq5018.dtsi
>>> +++ b/arch/arm64/boot/dts/qcom/ipq5018.dtsi
>>> @@ -17,6 +17,23 @@ / {
>>> #address-cells = <2>;
>>> #size-cells = <2>;
>>>
>>> + bluetooth: bluetooth {
>>> + compatible = "qcom,ipq5018-bt";
>>> +
>>> + firmware-name = "qca/bt_fw_patch.mbn";
>>
>> Is this fw vendor-signed?
>
> I've just analyzed the mbn file (and the mdt + b0x files): it only
> contains hashes for the mdt and b02 segments, no signature/certs at all.
> I've used your pil squasher to create the mbn file. Here are the FW files:
> https://github.com/georgemoussalem/openwrt/tree/ipq50xx-bluetooth/package/firmware/qca-bt-firmware/files
>
> Perhaps you can double check?
Using the not very sophisticated but very quick method of running
strings on it, there's no certificate identifiers indeed
Konrad
^ permalink raw reply
* Re: [PATCH] nbd: don't warn when reclassifying a busy socket lock
From: Hillf Danton @ 2026-06-30 12:24 UTC (permalink / raw)
To: Eric Dumazet
Cc: Deepanshu Kartikey, linux-block, nbd, linux-kernel,
syzbot+6b85d1e39a5b8ed9a954
In-Reply-To: <CANn89iL-RTVME9qcUSVZW6qBEkVh8suw2pcLMCh0WM1fTEz6Jw@mail.gmail.com>
On Tue, 30 Jun 2026 00:19:45 -0700 Eric Dumazet wrote:
> On Sun, Jun 28, 2026 at 10:29 PM Hillf Danton <hdanton@sina.com> wrote:
> > For archive purpose, syzbot report [1] where udp was not invovled defies
> > what is fixed in d532cddb6c60 ("nbd: Reclassify sockets to avoid lockdep
> > circular dependency") -- "Since the UDP socket and the NBD TCP/TLS socket
> > are different, this is a false positive."
> >
> >
> > [1] Subject: [syzbot] [net?] possible deadlock in inet_shutdown (3)
> > https://lore.kernel.org/lkml/69c37e6a.a70a0220.234938.0045.GAE@google.com/
>
> Why don't you send a patch if you think one is needed?
>
Good question. It is a long standing tough issue, frankly more difficult than
I could cure it in one shoot at the cost of a couple weeks, particularly given
the rule of thumb -- delivering half baked croissants, HBC, as few as possible.
^ permalink raw reply
* [PATCH v5] rust: configfs: add procedural macro for declaring configfs attributes
From: Malte Wechter @ 2026-06-30 13:38 UTC (permalink / raw)
To: Andreas Hindborg, Breno Leitao, Miguel Ojeda, Boqun Feng,
Gary Guo, Björn Roy Baron, Benno Lossin, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Jens Axboe, Luis Chamberlain,
Petr Pavlu, Daniel Gomez, Sami Tolvanen, Aaron Tomlin
Cc: linux-kernel, rust-for-linux, linux-block, linux-modules,
Malte Wechter
Implement `configfs_attrs!` as a procedural macro using `syn`, this
improves readability and maintainability. Remove the old macro and
replace all uses with the new macro. Add the new macro implementation
file to MAINTAINERS.
Signed-off-by: Malte Wechter <maltewechter@gmail.com>
---
Changes in v5:
- Assert during parsing that attribute names are unique (sashiko).
- Update safety comment to include that calls to make_group are serialized on the C side (sashiko)
- Add support for raw identifiers as attribute names (sashiko).
- Link to v4: https://lore.kernel.org/r/20260619-configfs-syn-v4-1-207c39504c1e@gmail.com
Changes in v4:
- Update link path to configfs_attr macro in configfs.rs
- Fix doc strings for configfs_attr in macros/lib.rs
- Fix doc strings for parse_ordered_fields in macros/helpers.rs
- Update title prefix to `rust: configfs:`
- Link to v3: https://lore.kernel.org/r/20260612-configfs-syn-v3-1-3292fbc5cc32@gmail.com
Changes in v3:
- Remove 'make_static_ident' function, make names for static variables simpler
- Move 'parse_ordered_fields' macro from module.rs into helpers
- Use 'parse_ordered_fields' macro for parsing instead of doing it ad-hoc
- Link to v2: https://lore.kernel.org/r/20260603-configfs-syn-v2-1-cb58489c2647@gmail.com
Changes in v2:
- Add a try_parse helper function to macros/helpers.rs
- Fix bug where 'child' configuration gets dropped if trailing comma is missing (sashiko)
- Link to v1: https://lore.kernel.org/r/20260520-configfs-syn-v1-1-6c5b80a9cef2@gmail.com
To: Andreas Hindborg <a.hindborg@kernel.org>
To: Boqun Feng <boqun@kernel.org>
To: Jens Axboe <axboe@kernel.dk>
To: Breno Leitao <leitao@debian.org>
To: Miguel Ojeda <ojeda@kernel.org>
To: Gary Guo <gary@garyguo.net>
To: Björn Roy Baron <bjorn3_gh@protonmail.com>
To: Benno Lossin <lossin@kernel.org>
To: Alice Ryhl <aliceryhl@google.com>
To: Trevor Gross <tmgross@umich.edu>
To: Danilo Krummrich <dakr@kernel.org>
To: Luis Chamberlain <mcgrof@kernel.org>
To: Petr Pavlu <petr.pavlu@suse.com>
To: Daniel Gomez <da.gomez@kernel.org>
To: Sami Tolvanen <samitolvanen@google.com>
To: Aaron Tomlin <atomlin@atomlin.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-block@vger.kernel.org
Cc: rust-for-linux@vger.kernel.org
Cc: linux-modules@vger.kernel.org
---
MAINTAINERS | 1 +
drivers/block/rnull/configfs.rs | 2 +-
rust/kernel/configfs.rs | 263 +---------------------------------------
rust/macros/configfs_attrs.rs | 149 +++++++++++++++++++++++
rust/macros/helpers.rs | 139 +++++++++++++++++++++
rust/macros/lib.rs | 87 +++++++++++++
rust/macros/module.rs | 137 ---------------------
samples/rust/rust_configfs.rs | 2 +-
8 files changed, 384 insertions(+), 396 deletions(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 2fb1c75afd16..45f7a1ec93b4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6464,6 +6464,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/a.hindborg/linux.git config
F: fs/configfs/
F: include/linux/configfs.h
F: rust/kernel/configfs.rs
+F: rust/macros/configfs_attrs.rs
F: samples/configfs/
F: samples/rust/rust_configfs.rs
diff --git a/drivers/block/rnull/configfs.rs b/drivers/block/rnull/configfs.rs
index 7c2eb5c0b722..f28ec69d7984 100644
--- a/drivers/block/rnull/configfs.rs
+++ b/drivers/block/rnull/configfs.rs
@@ -4,8 +4,8 @@
use kernel::{
block::mq::gen_disk::{GenDisk, GenDiskBuilder},
configfs::{self, AttributeOperations},
- configfs_attrs,
fmt::{self, Write as _},
+ macros::configfs_attrs,
new_mutex,
page::PAGE_SIZE,
prelude::*,
diff --git a/rust/kernel/configfs.rs b/rust/kernel/configfs.rs
index 2339c6467325..a8995a418136 100644
--- a/rust/kernel/configfs.rs
+++ b/rust/kernel/configfs.rs
@@ -21,7 +21,7 @@
//!
//! ```ignore
//! use kernel::alloc::flags;
-//! use kernel::configfs_attrs;
+//! use macros::configfs_attrs;
//! use kernel::configfs;
//! use kernel::new_mutex;
//! use kernel::page::PAGE_SIZE;
@@ -240,7 +240,7 @@ unsafe fn container_of(group: *const bindings::config_group) -> *const Self {
/// A configfs group.
///
/// To add a subgroup to configfs, pass this type as `ctype` to
-/// [`crate::configfs_attrs`] when creating a group in [`GroupOperations::make_group`].
+/// [`macros::configfs_attrs`] when creating a group in [`GroupOperations::make_group`].
#[pin_data]
pub struct Group<Data> {
#[pin]
@@ -637,7 +637,7 @@ pub const fn new(name: &'static CStr) -> Self {
/// implement `HasGroup<Data>`. The trait must be implemented once for each
/// attribute of the group. The constant type parameter `ID` maps the
/// implementation to a specific `Attribute`. `ID` must be passed when declaring
-/// attributes via the [`kernel::configfs_attrs`] macro, to tie
+/// attributes via the [`macros::configfs_attrs`] macro, to tie
/// `AttributeOperations` implementations to concrete named attributes.
#[vtable]
pub trait AttributeOperations<const ID: u64 = 0> {
@@ -669,13 +669,13 @@ fn store(_data: &Self::Data, _page: &[u8]) -> Result {
/// This type is used to construct a new [`ItemType`]. It represents a list of
/// [`Attribute`] that will appear in the directory representing a [`Group`].
/// Users should not directly instantiate this type, rather they should use the
-/// [`kernel::configfs_attrs`] macro to declare a static set of attributes for a
+/// [`macros::configfs_attrs`] macro to declare a static set of attributes for a
/// group.
///
/// # Note
///
/// Instances of this type are constructed statically at compile by the
-/// [`kernel::configfs_attrs`] macro.
+/// [`macros::configfs_attrs`] macro.
#[repr(transparent)]
pub struct AttributeList<const N: usize, Data>(
/// Null terminated Array of pointers to [`Attribute`]. The type is [`c_void`]
@@ -724,7 +724,7 @@ impl<const N: usize, Data> AttributeList<N, Data> {
/// [`Subsystem`].
///
/// Users should not directly instantiate objects of this type. Rather, they
-/// should use the [`kernel::configfs_attrs`] macro to statically declare the
+/// should use the [`macros::configfs_attrs`] macro to statically declare the
/// shape of a [`Group`] or [`Subsystem`].
#[pin_data]
pub struct ItemType<Container, Data> {
@@ -791,254 +791,3 @@ fn as_ptr(&self) -> *const bindings::config_item_type {
self.item_type.get()
}
}
-
-/// Define a list of configfs attributes statically.
-///
-/// Invoking the macro in the following manner:
-///
-/// ```ignore
-/// let item_type = configfs_attrs! {
-/// container: configfs::Subsystem<Configuration>,
-/// data: Configuration,
-/// child: Child,
-/// attributes: [
-/// message: 0,
-/// bar: 1,
-/// ],
-/// };
-/// ```
-///
-/// Expands the following output:
-///
-/// ```ignore
-/// let item_type = {
-/// static CONFIGURATION_MESSAGE_ATTR: kernel::configfs::Attribute<
-/// 0,
-/// Configuration,
-/// Configuration,
-/// > = unsafe {
-/// kernel::configfs::Attribute::new({
-/// const S: &str = "message\u{0}";
-/// const C: &kernel::str::CStr = match kernel::str::CStr::from_bytes_with_nul(
-/// S.as_bytes()
-/// ) {
-/// Ok(v) => v,
-/// Err(_) => {
-/// core::panicking::panic_fmt(core::const_format_args!(
-/// "string contains interior NUL"
-/// ));
-/// }
-/// };
-/// C
-/// })
-/// };
-///
-/// static CONFIGURATION_BAR_ATTR: kernel::configfs::Attribute<
-/// 1,
-/// Configuration,
-/// Configuration
-/// > = unsafe {
-/// kernel::configfs::Attribute::new({
-/// const S: &str = "bar\u{0}";
-/// const C: &kernel::str::CStr = match kernel::str::CStr::from_bytes_with_nul(
-/// S.as_bytes()
-/// ) {
-/// Ok(v) => v,
-/// Err(_) => {
-/// core::panicking::panic_fmt(core::const_format_args!(
-/// "string contains interior NUL"
-/// ));
-/// }
-/// };
-/// C
-/// })
-/// };
-///
-/// const N: usize = (1usize + (1usize + 0usize)) + 1usize;
-///
-/// static CONFIGURATION_ATTRS: kernel::configfs::AttributeList<N, Configuration> =
-/// unsafe { kernel::configfs::AttributeList::new() };
-///
-/// {
-/// const N: usize = 0usize;
-/// unsafe { CONFIGURATION_ATTRS.add::<N, 0, _>(&CONFIGURATION_MESSAGE_ATTR) };
-/// }
-///
-/// {
-/// const N: usize = (1usize + 0usize);
-/// unsafe { CONFIGURATION_ATTRS.add::<N, 1, _>(&CONFIGURATION_BAR_ATTR) };
-/// }
-///
-/// static CONFIGURATION_TPE:
-/// kernel::configfs::ItemType<configfs::Subsystem<Configuration> ,Configuration>
-/// = kernel::configfs::ItemType::<
-/// configfs::Subsystem<Configuration>,
-/// Configuration
-/// >::new_with_child_ctor::<N,Child>(
-/// &THIS_MODULE,
-/// &CONFIGURATION_ATTRS
-/// );
-///
-/// &CONFIGURATION_TPE
-/// }
-/// ```
-#[macro_export]
-macro_rules! configfs_attrs {
- (
- container: $container:ty,
- data: $data:ty,
- attributes: [
- $($name:ident: $attr:literal),* $(,)?
- ] $(,)?
- ) => {
- $crate::configfs_attrs!(
- count:
- @container($container),
- @data($data),
- @child(),
- @no_child(x),
- @attrs($($name $attr)*),
- @eat($($name $attr,)*),
- @assign(),
- @cnt(0usize),
- )
- };
- (
- container: $container:ty,
- data: $data:ty,
- child: $child:ty,
- attributes: [
- $($name:ident: $attr:literal),* $(,)?
- ] $(,)?
- ) => {
- $crate::configfs_attrs!(
- count:
- @container($container),
- @data($data),
- @child($child),
- @no_child(),
- @attrs($($name $attr)*),
- @eat($($name $attr,)*),
- @assign(),
- @cnt(0usize),
- )
- };
- (count:
- @container($container:ty),
- @data($data:ty),
- @child($($child:ty)?),
- @no_child($($no_child:ident)?),
- @attrs($($aname:ident $aattr:literal)*),
- @eat($name:ident $attr:literal, $($rname:ident $rattr:literal,)*),
- @assign($($assign:block)*),
- @cnt($cnt:expr),
- ) => {
- $crate::configfs_attrs!(
- count:
- @container($container),
- @data($data),
- @child($($child)?),
- @no_child($($no_child)?),
- @attrs($($aname $aattr)*),
- @eat($($rname $rattr,)*),
- @assign($($assign)* {
- const N: usize = $cnt;
- // The following macro text expands to a call to `Attribute::add`.
-
- // SAFETY: By design of this macro, the name of the variable we
- // invoke the `add` method on below, is not visible outside of
- // the macro expansion. The macro does not operate concurrently
- // on this variable, and thus we have exclusive access to the
- // variable.
- unsafe {
- $crate::macros::paste!(
- [< $data:upper _ATTRS >]
- .add::<N, $attr, _>(&[< $data:upper _ $name:upper _ATTR >])
- )
- };
- }),
- @cnt(1usize + $cnt),
- )
- };
- (count:
- @container($container:ty),
- @data($data:ty),
- @child($($child:ty)?),
- @no_child($($no_child:ident)?),
- @attrs($($aname:ident $aattr:literal)*),
- @eat(),
- @assign($($assign:block)*),
- @cnt($cnt:expr),
- ) =>
- {
- $crate::configfs_attrs!(
- final:
- @container($container),
- @data($data),
- @child($($child)?),
- @no_child($($no_child)?),
- @attrs($($aname $aattr)*),
- @assign($($assign)*),
- @cnt($cnt),
- )
- };
- (final:
- @container($container:ty),
- @data($data:ty),
- @child($($child:ty)?),
- @no_child($($no_child:ident)?),
- @attrs($($name:ident $attr:literal)*),
- @assign($($assign:block)*),
- @cnt($cnt:expr),
- ) =>
- {
- $crate::macros::paste!{
- {
- $(
- // SAFETY: We are expanding `configfs_attrs`.
- static [< $data:upper _ $name:upper _ATTR >]:
- $crate::configfs::Attribute<$attr, $data, $data> =
- unsafe {
- $crate::configfs::Attribute::new(
- $crate::c_str!(::core::stringify!($name)),
- )
- };
- )*
-
-
- // We need space for a null terminator.
- const N: usize = $cnt + 1usize;
-
- // SAFETY: We are expanding `configfs_attrs`.
- static [< $data:upper _ATTRS >]:
- $crate::configfs::AttributeList<N, $data> =
- unsafe { $crate::configfs::AttributeList::new() };
-
- $($assign)*
-
- $(
- const [<$no_child:upper>]: bool = true;
-
- static [< $data:upper _TPE >] : $crate::configfs::ItemType<$container, $data> =
- $crate::configfs::ItemType::<$container, $data>::new::<N>(
- &THIS_MODULE, &[<$ data:upper _ATTRS >]
- );
- )?
-
- $(
- static [< $data:upper _TPE >]:
- $crate::configfs::ItemType<$container, $data> =
- $crate::configfs::ItemType::<$container, $data>::
- new_with_child_ctor::<N, $child>(
- &THIS_MODULE, &[<$ data:upper _ATTRS >]
- );
- )?
-
- & [< $data:upper _TPE >]
- }
- }
- };
-
-}
-
-pub use crate::configfs_attrs;
diff --git a/rust/macros/configfs_attrs.rs b/rust/macros/configfs_attrs.rs
new file mode 100644
index 000000000000..f829a42cece8
--- /dev/null
+++ b/rust/macros/configfs_attrs.rs
@@ -0,0 +1,149 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use std::collections::HashSet;
+
+use quote::{
+ format_ident,
+ quote, //
+};
+
+use syn::{
+ bracketed,
+ ext::IdentExt,
+ parse::{
+ Parse,
+ ParseStream, //
+ },
+ punctuated::Punctuated,
+ spanned::Spanned,
+ Error,
+ Ident,
+ LitInt,
+ Token,
+ Type, //
+};
+
+use crate::helpers::parse_ordered_fields;
+
+pub(crate) struct ConfigfsAttrs {
+ container: Type,
+ data: Type,
+ child: Option<Type>,
+ attributes: Vec<(Ident, LitInt)>,
+}
+
+fn parse_attribute_field(stream: ParseStream<'_>) -> syn::Result<(Ident, LitInt)> {
+ let id = stream.parse::<syn::Ident>()?;
+ let _colon = stream.parse::<Token![:]>()?;
+ let v = stream.parse::<LitInt>()?;
+ Ok((id, v))
+}
+
+fn parse_attributes(stream: ParseStream<'_>) -> syn::Result<Vec<(Ident, LitInt)>> {
+ let attr_stream;
+ let _bracket = bracketed!(attr_stream in stream);
+ let attributes = Punctuated::<(Ident, LitInt), Token![,]>::parse_terminated_with(
+ &attr_stream,
+ parse_attribute_field,
+ )?;
+ let mut attr_set = HashSet::new();
+ for attr in attributes.iter() {
+ if !attr_set.insert(attr.0.clone()) {
+ return Err(syn::Error::new(attr.0.span(), "duplicate attribute"));
+ }
+ }
+
+ Ok(attributes.into_iter().collect::<Vec<_>>())
+}
+
+impl Parse for ConfigfsAttrs {
+ fn parse(input: ParseStream<'_>) -> syn::Result<Self> {
+ parse_ordered_fields!(
+ from input;
+ container [required] => (input.parse::<Type>())?,
+ data [required] => (input.parse::<Type>())?,
+ child => (input.parse::<Type>())?,
+ attributes [required] => parse_attributes(input)?,
+ );
+
+ Ok(ConfigfsAttrs {
+ container,
+ data,
+ child,
+ attributes,
+ })
+ }
+}
+
+pub(crate) fn configfs_attrs(cfs_attrs: ConfigfsAttrs) -> proc_macro2::TokenStream {
+ let (container_ty, data_ty) = (&cfs_attrs.container, &cfs_attrs.data);
+
+ let data_tp_ident = Ident::new("DATA_TPE", cfs_attrs.data.span());
+ let data_attr_ident = Ident::new("DATA_ATTR_LIST", cfs_attrs.data.span());
+
+ let n = cfs_attrs.attributes.len() + 1;
+
+ let attr_list = quote! {
+ static #data_attr_ident: kernel::configfs::AttributeList<#n, #data_ty> =
+ // SAFETY: We are expanding `configfs_attrs`.
+ unsafe { kernel::configfs::AttributeList::new() };
+ };
+
+ let mut attrs = Vec::new();
+ for (attr_idx, (name, id)) in cfs_attrs.attributes.iter().enumerate() {
+ let name_with_attr = format_ident!(
+ "{}_ATTR_{}",
+ name.unraw().to_string().to_uppercase(),
+ attr_idx
+ );
+
+ let id: u64 = match id.base10_parse::<u64>() {
+ Ok(v) => v,
+ Err(_) => {
+ return syn::Error::new(id.span(), "Could not parse attribute ID as a u64")
+ .to_compile_error();
+ }
+ };
+
+ attrs.push(quote! {
+ static #name_with_attr: kernel::configfs::Attribute<#id, #data_ty, #data_ty> =
+ // SAFETY: We are expanding `configfs_attrs`.
+ unsafe {
+ kernel::configfs::Attribute::new(kernel::c_str!(::core::stringify!(#name)))
+ };
+
+ // SAFETY: By design of this macro, the name of the variable we
+ // invoke the `add` method on below, is not visible outside of
+ // the macro expansion. The macro does not operate concurrently
+ // on this variable, and thus we have exclusive access to the
+ // variable. Calls to `make_group` are serialized on the C side
+ // by per subsystem mutex.
+ unsafe { #data_attr_ident.add::<#attr_idx, #id, _>(&#name_with_attr) }
+ });
+ }
+
+ let has_child_code = if let Some(child) = cfs_attrs.child {
+ quote! { new_with_child_ctor::<#n, #child>}
+ } else {
+ quote! { new::<#n> }
+ };
+
+ let data_type = quote! {
+ {
+ static #data_tp_ident:
+ kernel::configfs::ItemType<#container_ty, #data_ty> =
+ kernel::configfs::ItemType::<#container_ty, #data_ty>::#has_child_code(
+ &THIS_MODULE, &#data_attr_ident
+ );
+ &#data_tp_ident
+ }
+ };
+
+ quote! {
+ {
+ #attr_list
+ #(#attrs)*
+ #data_type
+ }
+ }
+}
diff --git a/rust/macros/helpers.rs b/rust/macros/helpers.rs
index d18fbf4daa0a..305dcbddf797 100644
--- a/rust/macros/helpers.rs
+++ b/rust/macros/helpers.rs
@@ -58,3 +58,142 @@ pub(crate) fn file() -> String {
pub(crate) fn gather_cfg_attrs(attr: &[Attribute]) -> impl Iterator<Item = &Attribute> + '_ {
attr.iter().filter(|a| a.path().is_ident("cfg"))
}
+
+/// Parse fields that are required to use a specific order.
+///
+/// As fields must follow a specific order, we *could* just parse fields one by one by peeking.
+/// However the error message generated when implementing that way is not very friendly.
+///
+/// So instead we parse fields in an arbitrary order, but only enforce the ordering after parsing,
+/// and if the wrong order is used, the proper order is communicated to the user with error message.
+///
+/// Usage looks like this:
+/// ```ignore
+/// parse_ordered_fields! {
+/// from input;
+///
+/// // This will extract `foo: <field>` into a variable named `foo`.
+/// // The variable will have type `Option<_>`.
+/// foo => <expression that parses the field>,
+///
+/// // If you need the variable name to be different than the key name.
+/// // This extracts `baz: <field>` into a variable named `bar`.
+/// // You might want this if `baz` is a keyword.
+/// baz as bar => <expression that parse the field>,
+///
+/// // You can mark a key as required, and the variable will no longer be `Option`.
+/// // foobar will be of type `Expr` instead of `Option<Expr>`.
+/// foobar [required] => input.parse::<Expr>()?,
+/// }
+/// ```
+macro_rules! parse_ordered_fields {
+ (@gen
+ [$input:expr]
+ [$([$name:ident; $key:ident; $parser:expr])*]
+ [$([$req_name:ident; $req_key:ident])*]
+ ) => {
+ $(let mut $name = None;)*
+
+ const EXPECTED_KEYS: &[&str] = &[$(stringify!($key),)*];
+ const REQUIRED_KEYS: &[&str] = &[$(stringify!($req_key),)*];
+
+ let span = $input.span();
+ let mut seen_keys = Vec::new();
+
+ while !$input.is_empty() {
+ let key = $input.call(Ident::parse_any)?;
+
+ if seen_keys.contains(&key) {
+ Err(Error::new_spanned(
+ &key,
+ format!(r#"duplicated key "{key}". Keys can only be specified once."#),
+ ))?
+ }
+
+ $input.parse::<Token![:]>()?;
+
+ match &*key.to_string() {
+ $(
+ stringify!($key) => $name = Some($parser),
+ )*
+ _ => {
+ Err(Error::new_spanned(
+ &key,
+ format!(r#"unknown key "{key}". Valid keys are: {EXPECTED_KEYS:?}."#),
+ ))?
+ }
+ }
+
+ $input.parse::<Token![,]>()?;
+ seen_keys.push(key);
+ }
+
+ for key in REQUIRED_KEYS {
+ if !seen_keys.iter().any(|e| e == key) {
+ Err(Error::new(span, format!(r#"missing required key "{key}""#)))?
+ }
+ }
+
+ let mut ordered_keys: Vec<&str> = Vec::new();
+ for key in EXPECTED_KEYS {
+ if seen_keys.iter().any(|e| e == key) {
+ ordered_keys.push(key);
+ }
+ }
+
+ if seen_keys != ordered_keys {
+ Err(Error::new(
+ span,
+ format!(r#"keys are not ordered as expected. Order them like: {ordered_keys:?}."#),
+ ))?
+ }
+
+ $(let $req_name = $req_name.expect("required field");)*
+ };
+
+ // Handle required fields.
+ (@gen
+ [$input:expr] [$($tok:tt)*] [$($req:tt)*]
+ $key:ident as $name:ident [required] => $parser:expr,
+ $($rest:tt)*
+ ) => {
+ parse_ordered_fields!(
+ @gen [$input] [$($tok)* [$name; $key; $parser]] [$($req)* [$name; $key]] $($rest)*
+ )
+ };
+ (@gen
+ [$input:expr] [$($tok:tt)*] [$($req:tt)*]
+ $name:ident [required] => $parser:expr,
+ $($rest:tt)*
+ ) => {
+ parse_ordered_fields!(
+ @gen [$input] [$($tok)* [$name; $name; $parser]] [$($req)* [$name; $name]] $($rest)*
+ )
+ };
+
+ // Handle optional fields.
+ (@gen
+ [$input:expr] [$($tok:tt)*] [$($req:tt)*]
+ $key:ident as $name:ident => $parser:expr,
+ $($rest:tt)*
+ ) => {
+ parse_ordered_fields!(
+ @gen [$input] [$($tok)* [$name; $key; $parser]] [$($req)*] $($rest)*
+ )
+ };
+ (@gen
+ [$input:expr] [$($tok:tt)*] [$($req:tt)*]
+ $name:ident => $parser:expr,
+ $($rest:tt)*
+ ) => {
+ parse_ordered_fields!(
+ @gen [$input] [$($tok)* [$name; $name; $parser]] [$($req)*] $($rest)*
+ )
+ };
+
+ (from $input:expr; $($tok:tt)*) => {
+ parse_ordered_fields!(@gen [$input] [] [] $($tok)*)
+ }
+}
+
+pub(crate) use parse_ordered_fields;
diff --git a/rust/macros/lib.rs b/rust/macros/lib.rs
index 2cfd59e0f9e7..ebb41e80ecc7 100644
--- a/rust/macros/lib.rs
+++ b/rust/macros/lib.rs
@@ -15,6 +15,8 @@
#![cfg_attr(not(CONFIG_RUSTC_HAS_SPAN_FILE), feature(proc_macro_span))]
mod concat_idents;
+#[cfg(CONFIG_CONFIGFS_FS)]
+mod configfs_attrs;
mod export;
mod fmt;
mod helpers;
@@ -489,3 +491,88 @@ pub fn kunit_tests(attr: TokenStream, input: TokenStream) -> TokenStream {
.unwrap_or_else(|e| e.into_compile_error())
.into()
}
+
+/// Define a list of configfs attributes statically.
+///
+/// # Examples
+///
+/// ```ignore
+/// let item_type = configfs_attrs! {
+/// container: configfs::Subsystem<Configuration>,
+/// data: Configuration,
+/// child: Child,
+/// attributes: [
+/// message: 0,
+/// bar: 1,
+/// ],
+/// };
+/// ```
+///
+/// Expands the following output:
+///
+/// ```ignore
+/// let item_type = {
+/// static DATA_ATTR_LIST: kernel::configfs::AttributeList<
+/// 3usize,
+/// Configuration,
+/// > = unsafe { kernel::configfs::AttributeList::new() };
+/// static MESSAGE_ATTR_0: kernel::configfs::Attribute<
+/// 0u64,
+/// Configuration,
+/// Configuration,
+/// > = unsafe {
+/// kernel::configfs::Attribute::new({
+/// const S: &str = "message\u{0}";
+/// const C: &kernel::str::CStr = match kernel::str::CStr::from_bytes_with_nul(
+/// S.as_bytes(),
+/// ) {
+/// Ok(v) => v,
+/// Err(_) => {
+/// ::core::panicking::panic_fmt(
+/// format_args!("string contains interior NUL"),
+/// );
+/// }
+/// };
+/// C
+/// })
+/// };
+/// unsafe { DATA_ATTR_LIST.add::<0usize, 0u64, _>(&MESSAGE_ATTR_0) }
+/// static BAR_ATTR_1: kernel::configfs::Attribute<
+/// 1u64,
+/// Configuration,
+/// Configuration,
+/// > = unsafe {
+/// kernel::configfs::Attribute::new({
+/// const S: &str = "bar\u{0}";
+/// const C: &kernel::str::CStr = match kernel::str::CStr::from_bytes_with_nul(
+/// S.as_bytes(),
+/// ) {
+/// Ok(v) => v,
+/// Err(_) => {
+/// ::core::panicking::panic_fmt(
+/// format_args!("string contains interior NUL"),
+/// );
+/// }
+/// };
+/// C
+/// })
+/// };
+/// unsafe { DATA_ATTR_LIST.add::<1usize, 1u64, _>(&BAR_ATTR_1) }
+/// {
+/// static DATA_TPE: kernel::configfs::ItemType<
+/// Subsystem<Configuration>,
+/// Configuration,
+/// > = kernel::configfs::ItemType::<
+/// Subsystem<Configuration>,
+/// Configuration,
+/// >::new_with_child_ctor::<3usize, Child>(&THIS_MODULE, &DATA_ATTR_LIST);
+/// &DATA_TPE
+/// }
+/// };
+/// ```
+#[cfg(CONFIG_CONFIGFS_FS)]
+#[proc_macro]
+pub fn configfs_attrs(input: TokenStream) -> TokenStream {
+ configfs_attrs::configfs_attrs(parse_macro_input!(input as configfs_attrs::ConfigfsAttrs))
+ .into()
+}
diff --git a/rust/macros/module.rs b/rust/macros/module.rs
index 06c18e207508..7ff6ad09b1a2 100644
--- a/rust/macros/module.rs
+++ b/rust/macros/module.rs
@@ -196,143 +196,6 @@ fn param_ops_path(param_type: &str) -> Path {
}
}
-/// Parse fields that are required to use a specific order.
-///
-/// As fields must follow a specific order, we *could* just parse fields one by one by peeking.
-/// However the error message generated when implementing that way is not very friendly.
-///
-/// So instead we parse fields in an arbitrary order, but only enforce the ordering after parsing,
-/// and if the wrong order is used, the proper order is communicated to the user with error message.
-///
-/// Usage looks like this:
-/// ```ignore
-/// parse_ordered_fields! {
-/// from input;
-///
-/// // This will extract "foo: <field>" into a variable named "foo".
-/// // The variable will have type `Option<_>`.
-/// foo => <expression that parses the field>,
-///
-/// // If you need the variable name to be different than the key name.
-/// // This extracts "baz: <field>" into a variable named "bar".
-/// // You might want this if "baz" is a keyword.
-/// baz as bar => <expression that parse the field>,
-///
-/// // You can mark a key as required, and the variable will no longer be `Option`.
-/// // foobar will be of type `Expr` instead of `Option<Expr>`.
-/// foobar [required] => input.parse::<Expr>()?,
-/// }
-/// ```
-macro_rules! parse_ordered_fields {
- (@gen
- [$input:expr]
- [$([$name:ident; $key:ident; $parser:expr])*]
- [$([$req_name:ident; $req_key:ident])*]
- ) => {
- $(let mut $name = None;)*
-
- const EXPECTED_KEYS: &[&str] = &[$(stringify!($key),)*];
- const REQUIRED_KEYS: &[&str] = &[$(stringify!($req_key),)*];
-
- let span = $input.span();
- let mut seen_keys = Vec::new();
-
- while !$input.is_empty() {
- let key = $input.call(Ident::parse_any)?;
-
- if seen_keys.contains(&key) {
- Err(Error::new_spanned(
- &key,
- format!(r#"duplicated key "{key}". Keys can only be specified once."#),
- ))?
- }
-
- $input.parse::<Token![:]>()?;
-
- match &*key.to_string() {
- $(
- stringify!($key) => $name = Some($parser),
- )*
- _ => {
- Err(Error::new_spanned(
- &key,
- format!(r#"unknown key "{key}". Valid keys are: {EXPECTED_KEYS:?}."#),
- ))?
- }
- }
-
- $input.parse::<Token![,]>()?;
- seen_keys.push(key);
- }
-
- for key in REQUIRED_KEYS {
- if !seen_keys.iter().any(|e| e == key) {
- Err(Error::new(span, format!(r#"missing required key "{key}""#)))?
- }
- }
-
- let mut ordered_keys: Vec<&str> = Vec::new();
- for key in EXPECTED_KEYS {
- if seen_keys.iter().any(|e| e == key) {
- ordered_keys.push(key);
- }
- }
-
- if seen_keys != ordered_keys {
- Err(Error::new(
- span,
- format!(r#"keys are not ordered as expected. Order them like: {ordered_keys:?}."#),
- ))?
- }
-
- $(let $req_name = $req_name.expect("required field");)*
- };
-
- // Handle required fields.
- (@gen
- [$input:expr] [$($tok:tt)*] [$($req:tt)*]
- $key:ident as $name:ident [required] => $parser:expr,
- $($rest:tt)*
- ) => {
- parse_ordered_fields!(
- @gen [$input] [$($tok)* [$name; $key; $parser]] [$($req)* [$name; $key]] $($rest)*
- )
- };
- (@gen
- [$input:expr] [$($tok:tt)*] [$($req:tt)*]
- $name:ident [required] => $parser:expr,
- $($rest:tt)*
- ) => {
- parse_ordered_fields!(
- @gen [$input] [$($tok)* [$name; $name; $parser]] [$($req)* [$name; $name]] $($rest)*
- )
- };
-
- // Handle optional fields.
- (@gen
- [$input:expr] [$($tok:tt)*] [$($req:tt)*]
- $key:ident as $name:ident => $parser:expr,
- $($rest:tt)*
- ) => {
- parse_ordered_fields!(
- @gen [$input] [$($tok)* [$name; $key; $parser]] [$($req)*] $($rest)*
- )
- };
- (@gen
- [$input:expr] [$($tok:tt)*] [$($req:tt)*]
- $name:ident => $parser:expr,
- $($rest:tt)*
- ) => {
- parse_ordered_fields!(
- @gen [$input] [$($tok)* [$name; $name; $parser]] [$($req)*] $($rest)*
- )
- };
-
- (from $input:expr; $($tok:tt)*) => {
- parse_ordered_fields!(@gen [$input] [] [] $($tok)*)
- }
-}
-
struct Parameter {
name: Ident,
ptype: Ident,
diff --git a/samples/rust/rust_configfs.rs b/samples/rust/rust_configfs.rs
index a1bd9db6010d..876462f7789d 100644
--- a/samples/rust/rust_configfs.rs
+++ b/samples/rust/rust_configfs.rs
@@ -4,7 +4,7 @@
use kernel::alloc::flags;
use kernel::configfs;
-use kernel::configfs::configfs_attrs;
+use kernel::macros::configfs_attrs;
use kernel::new_mutex;
use kernel::page::PAGE_SIZE;
use kernel::prelude::*;
---
base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731
change-id: 20260417-configfs-syn-191e07130027
Best regards,
--
Malte Wechter <maltewechter@gmail.com>
^ permalink raw reply related
* Re: [PATCH v4 3/3] btrfs: use IOMAP_DIO_BOUNCE flag instead of falling back to buffered IO
From: David Sterba @ 2026-06-30 15:11 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, linux-block, linux-fsdevel, linux-xfs
In-Reply-To: <1a89047dac91b6b12d190c37cd7bb3d8328b2073.1781597506.git.wqu@suse.com>
On Tue, Jun 16, 2026 at 05:42:37PM +0930, Qu Wenruo wrote:
> Previously btrfs forces direct writes to fall back to buffered ones if the
> inode has data checksum or the profile has duplication.
>
> That fallback is to avoid the content being modified that the final
> content may mismatch with the checksum or the other mirrors.
>
> That brings a pretty huge performance cost, which already caused some
> concern at that time.
>
> But later upstream commit c9d114846b38 ("iomap: add a flag to bounce
> buffer direct I/O") introduced a new method by copying the content into
> new pages, and do all the operations based on the newly allocated pages.
>
> So let btrfs to utilize the new flag for direct writes if we require
> stable folios.
>
> There is a quick benchmark, using the following fio setup:
>
> fio --name=randwrite --filename $mnt/foobar --ioengine=libaio --size=4G \
> --rw=randwrite --iodepth=64 --runtime=60 --time_based --direct=1 \
> --bs=$blocksize
>
> Unit is MiB/s.
>
> Blocksize | Zero-copy (*) | Buffered | Bounce
> -----------+---------------+----------+-----------
> 4K | 35.1 | 17.1 | 33.8
> 64K | 522 | 251 | 492
>
> *: This is done by reverting the commit 968f19c5b1b7 ("btrfs: always
> fallback to buffered write if the inode requires checksum")
>
> Although with page bouncing the performance is only around 95% of
> true-zero copy, it's still almost double the performance of buffered
> fallback.
>
> There will be a small change in behavior, since we're using
> IOMAP_DIO_BOUNCE flag to allocate new folios, NOWAIT flag will
> immediately fail.
>
> So for true NOWAIT direct IOs, NODATASUM and RAID0/SINGLE profiles are
> still required.
>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
The block layer patches have been merged and our for-next is now based
on 7.2-rc1 so pleaase add this one too so we can get back the dio
performance. Thanks.
^ permalink raw reply
* Re: [PATCH v6 1/9] block: partitions: of: Skip child nodes without reg property
From: Rob Herring @ 2026-06-30 18:02 UTC (permalink / raw)
To: Loic Poulain
Cc: Ulf Hansson, Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson,
Konrad Dybcio, Jens Axboe, Johannes Berg, Jeff Johnson,
Bartosz Golaszewski, Marcel Holtmann, Luiz Augusto von Dentz,
Balakrishna Godavarthi, Rocky Liao, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Srinivas Kandagatla,
Andrew Lunn, Heiner Kallweit, Russell King, Saravana Kannan,
Christian Marangi, linux-mmc, devicetree, linux-kernel,
linux-arm-msm, linux-block, linux-wireless, ath10k,
linux-bluetooth, netdev, daniel, stable, Bartosz Golaszewski
In-Reply-To: <20260629-block-as-nvmem-v6-1-f02513dcd46d@oss.qualcomm.com>
On Mon, Jun 29, 2026 at 10:55:20AM +0200, Loic Poulain wrote:
> Child nodes of a fixed-partitions node are not necessarily partition
> entries, for example an nvmem-layout node has no reg property. The
> current code passes a NULL reg pointer and uninitialized len to the
> length check, which can result in a kernel panic or silent failure to
> register any partitions.
That does not sound right to me. A fixed-partitions node should only be
defining partitions with address ranges. I would expect a partition node
could be nvmem-layout, but not the whole address range. If you wanted
the latter, then just do:
partitions {
...
};
nvmem-layout {
...
};
Rob
^ permalink raw reply
* Re: [PATCH v6 1/9] block: partitions: of: Skip child nodes without reg property
From: Loic Poulain @ 2026-06-30 19:59 UTC (permalink / raw)
To: Rob Herring
Cc: Ulf Hansson, Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson,
Konrad Dybcio, Jens Axboe, Johannes Berg, Jeff Johnson,
Bartosz Golaszewski, Marcel Holtmann, Luiz Augusto von Dentz,
Balakrishna Godavarthi, Rocky Liao, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Srinivas Kandagatla,
Andrew Lunn, Heiner Kallweit, Russell King, Saravana Kannan,
Christian Marangi, linux-mmc, devicetree, linux-kernel,
linux-arm-msm, linux-block, linux-wireless, ath10k,
linux-bluetooth, netdev, daniel, stable, Bartosz Golaszewski
In-Reply-To: <20260630180219.GA4139943-robh@kernel.org>
Hi Rob,
On Tue, Jun 30, 2026 at 8:02 PM Rob Herring <robh@kernel.org> wrote:
>
> On Mon, Jun 29, 2026 at 10:55:20AM +0200, Loic Poulain wrote:
> > Child nodes of a fixed-partitions node are not necessarily partition
> > entries, for example an nvmem-layout node has no reg property. The
> > current code passes a NULL reg pointer and uninitialized len to the
> > length check, which can result in a kernel panic or silent failure to
> > register any partitions.
>
> That does not sound right to me. A fixed-partitions node should only be
> defining partitions with address ranges. I would expect a partition node
> could be nvmem-layout, but not the whole address range. If you wanted
> the latter, then just do:
>
> partitions {
> ...
> };
>
> nvmem-layout {
> ...
> };
In our case, the nvmem-layout needs to be associated with a specific
eMMC hardware partition, nvmem cells can be a simple sub-range within
the global eMMC, each hardware partition (boot0, boot1, user...)
having its own address spaces.
That said, your point about not abusing fixed-partitions is valid. I
initially dropped the compatible = "fixed-partitions" from the
partitions-boot1 node when it only carries an nvmem-layout and no
actual partition entries, making it a plain named container node. But
it's a bit fragile if we want to support both nvmem-layout and
fixed-partitions.
Regarding your expectation of a partition node being a nvmem-layout,
do you mean that the nvmem-layout should live under a fixed-partitions
node? Something along these lines:
partitions-boot1 {
compatible = "fixed-partitions";
#address-cells = <1>;
#size-cells = <1>;
nvmem@4400 {
reg = <0x4400 0x1000>;
nvmem-layout {
compatible = "fixed-layout";
#address-cells = <1>;
#size-cells = <1>;
wifi_mac_addr: mac-addr@0 {
compatible = "mac-base";
reg = <0x0 0x6>;
#nvmem-cell-cells = <1>;
};
[...]
That makes some sense, this would require extra work for the
emmc/block layer to also associate fwnodes with logical partitions,
not just the whole disk/hw (hw part), Is that the direction you'd like
us to go?
Also, Note that regardless of which approach we settle on, this
specific fix/patch remains necessary to validate the partition node
and prevent NULL-deref.
Regards,
Loic
^ permalink raw reply
* [PATCH] ublk: snapshot batch commands before preparing I/O
From: Yousef Alhouseen @ 2026-06-30 21:18 UTC (permalink / raw)
To: Ming Lei, Jens Axboe
Cc: Caleb Sander Mateos, linux-block, linux-kernel,
syzbot+1a67ee1aa79484801ec6, Yousef Alhouseen
The batch prepare path rereads its userspace element array when rolling
back a partially prepared batch. Userspace can change an already
processed tag before the second read, causing rollback to reject the
replacement tag and leave earlier I/O slots prepared. The
WARN_ON_ONCE() in the rollback path then fires.
Copy the bounded batch into kernel memory before changing any I/O state
and use the same snapshot for preparation and rollback. Commit and fetch
batches retain the existing chunked userspace walk.
Fixes: b256795b3606 ("ublk: handle UBLK_U_IO_PREP_IO_CMDS")
Reported-by: syzbot+1a67ee1aa79484801ec6@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=1a67ee1aa79484801ec6
Signed-off-by: Yousef Alhouseen <alhouseenyousef@gmail.com>
---
drivers/block/ublk_drv.c | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index 4f6d9e652187..c2c11f2a01e7 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -3584,6 +3584,7 @@ ublk_batch_auto_buf_reg(const struct ublk_batch_io *uc,
#define UBLK_CMD_BATCH_TMP_BUF_SZ (48 * 10)
struct ublk_batch_io_iter {
void __user *uaddr;
+ const u8 *kaddr;
unsigned done, total;
unsigned char elem_bytes;
/* copy to this buffer from user space */
@@ -3632,7 +3633,10 @@ static int ublk_walk_cmd_buf(struct ublk_batch_io_iter *iter,
while (iter->done < iter->total) {
unsigned int len = min(sizeof(iter->buf), iter->total - iter->done);
- if (copy_from_user(iter->buf, iter->uaddr + iter->done, len)) {
+ if (iter->kaddr) {
+ memcpy(iter->buf, iter->kaddr + iter->done, len);
+ } else if (copy_from_user(iter->buf, iter->uaddr + iter->done,
+ len)) {
pr_warn("ublk%d: read batch cmd buffer failed\n",
data->ub->dev_info.dev_id);
return -EFAULT;
@@ -3723,14 +3727,21 @@ static int ublk_handle_batch_prep_cmd(const struct ublk_batch_io_data *data)
.total = uc->nr_elem * uc->elem_bytes,
.elem_bytes = uc->elem_bytes,
};
+ void *cmd_buf;
int ret;
+ cmd_buf = vmemdup_user(iter.uaddr, iter.total);
+ if (IS_ERR(cmd_buf))
+ return PTR_ERR(cmd_buf);
+ iter.kaddr = cmd_buf;
+
mutex_lock(&data->ub->mutex);
ret = ublk_walk_cmd_buf(&iter, data, ublk_batch_prep_io);
if (ret && iter.done)
ublk_batch_revert_prep_cmd(&iter, data);
mutex_unlock(&data->ub->mutex);
+ kvfree(cmd_buf);
return ret;
}
--
2.55.0
^ permalink raw reply related
* Re: [PATCH v6 1/9] block: partitions: of: Skip child nodes without reg property
From: Rob Herring @ 2026-06-30 21:45 UTC (permalink / raw)
To: Loic Poulain
Cc: Ulf Hansson, Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson,
Konrad Dybcio, Jens Axboe, Johannes Berg, Jeff Johnson,
Bartosz Golaszewski, Marcel Holtmann, Luiz Augusto von Dentz,
Balakrishna Godavarthi, Rocky Liao, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Srinivas Kandagatla,
Andrew Lunn, Heiner Kallweit, Russell King, Saravana Kannan,
Christian Marangi, linux-mmc, devicetree, linux-kernel,
linux-arm-msm, linux-block, linux-wireless, ath10k,
linux-bluetooth, netdev, daniel, stable, Bartosz Golaszewski
In-Reply-To: <CAFEp6-163adAq8-H_pCzGnq+Fo4jpyKGs6Jv25j3fSpZg3COjQ@mail.gmail.com>
On Tue, Jun 30, 2026 at 2:59 PM Loic Poulain
<loic.poulain@oss.qualcomm.com> wrote:
>
> Hi Rob,
>
> On Tue, Jun 30, 2026 at 8:02 PM Rob Herring <robh@kernel.org> wrote:
> >
> > On Mon, Jun 29, 2026 at 10:55:20AM +0200, Loic Poulain wrote:
> > > Child nodes of a fixed-partitions node are not necessarily partition
> > > entries, for example an nvmem-layout node has no reg property. The
> > > current code passes a NULL reg pointer and uninitialized len to the
> > > length check, which can result in a kernel panic or silent failure to
> > > register any partitions.
> >
> > That does not sound right to me. A fixed-partitions node should only be
> > defining partitions with address ranges. I would expect a partition node
> > could be nvmem-layout, but not the whole address range. If you wanted
> > the latter, then just do:
> >
> > partitions {
> > ...
> > };
> >
> > nvmem-layout {
> > ...
> > };
>
> In our case, the nvmem-layout needs to be associated with a specific
> eMMC hardware partition, nvmem cells can be a simple sub-range within
> the global eMMC, each hardware partition (boot0, boot1, user...)
> having its own address spaces.
>
> That said, your point about not abusing fixed-partitions is valid. I
> initially dropped the compatible = "fixed-partitions" from the
> partitions-boot1 node when it only carries an nvmem-layout and no
> actual partition entries, making it a plain named container node. But
> it's a bit fragile if we want to support both nvmem-layout and
> fixed-partitions.
>
> Regarding your expectation of a partition node being a nvmem-layout,
> do you mean that the nvmem-layout should live under a fixed-partitions
> node? Something along these lines:
>
> partitions-boot1 {
> compatible = "fixed-partitions";
> #address-cells = <1>;
> #size-cells = <1>;
>
> nvmem@4400 {
partition@4400
> reg = <0x4400 0x1000>;
>
> nvmem-layout {
> compatible = "fixed-layout";
> #address-cells = <1>;
> #size-cells = <1>;
>
> wifi_mac_addr: mac-addr@0 {
> compatible = "mac-base";
> reg = <0x0 0x6>;
> #nvmem-cell-cells = <1>;
> };
> [...]
Either this or replacing "fixed-partitions" with "fixed-layout" if you
want to make the whole boot1 partition nvmem-layout looks like the
right way to me.
> That makes some sense, this would require extra work for the
> emmc/block layer to also associate fwnodes with logical partitions,
> not just the whole disk/hw (hw part), Is that the direction you'd like
> us to go?
Yes.
> Also, Note that regardless of which approach we settle on, this
> specific fix/patch remains necessary to validate the partition node
> and prevent NULL-deref.
Fair enough, though the reasoning for it would be different and
perhaps should give a warning.
Rob
^ permalink raw reply
* [PATCH v2 10/18] block: convert iomap ops to ->iomap_next()
From: Joanne Koong @ 2026-07-01 0:09 UTC (permalink / raw)
To: brauner, hch
Cc: djwong, willy, hsiangkao, linux-fsdevel, linux-xfs, Jens Axboe,
open list:BLOCK LAYER, open list
In-Reply-To: <20260701000949.1666714-1-joannelkoong@gmail.com>
Convert block iomap_ops to the new ->iomap_next() callback. This uses the
iomap_process() helper, which finishes the previous mapping if needed
and produces the next one. No functional changes are intended.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
block/fops.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/block/fops.c b/block/fops.c
index 15783a6180de..c2721e2c659b 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -453,8 +453,14 @@ static int blkdev_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
+static int blkdev_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
+{
+ return iomap_process(iter, iomap, srcmap, blkdev_iomap_begin, NULL);
+}
+
static const struct iomap_ops blkdev_iomap_ops = {
- .iomap_begin = blkdev_iomap_begin,
+ .iomap_next = blkdev_iomap_next,
};
#ifdef CONFIG_BUFFER_HEAD
--
2.52.0
^ permalink raw reply related
* [PATCH v2 17/18] iomap: pass iomap_next_fn directly instead of struct iomap_ops
From: Joanne Koong @ 2026-07-01 0:09 UTC (permalink / raw)
To: brauner, hch
Cc: djwong, willy, hsiangkao, linux-fsdevel, linux-xfs, Jens Axboe,
Chris Mason, David Sterba, Alexander Viro, Jan Kara, Dan Williams,
Gao Xiang, Chao Yu, Yue Hu, Jeffle Xu, Sandeep Dhavale, Hongbo Li,
Chunhai Guo, Namjae Jeon, Sungjong Seo, Yuezhang Mo,
Theodore Ts'o, Andreas Dilger, Baokun Li, Ojaswin Mujoo,
Ritesh Harjani (IBM), Zhang Yi, Jaegeuk Kim, Miklos Szeredi,
Andreas Gruenbacher, Mikulas Patocka, Hyunchul Lee,
Konstantin Komarov, Carlos Maiolino, Damien Le Moal, Naohiro Aota,
Johannes Thumshirn, open list:BLOCK LAYER, open list,
open list:BTRFS FILE SYSTEM,
open list:FILESYSTEM DIRECT ACCESS (DAX),
open list:EROFS FILE SYSTEM, open list:EXT2 FILE SYSTEM,
open list:F2FS FILE SYSTEM, open list:FUSE FILESYSTEM [CORE],
open list:GFS2 FILE SYSTEM, open list:NTFS3 FILESYSTEM
In-Reply-To: <20260701000949.1666714-1-joannelkoong@gmail.com>
Now that all filesystems implement ->iomap_next() and the legacy
->iomap_begin()/->iomap_end() fallback is gone, struct iomap_ops only
wraps a single iomap_next function pointer. Drop the struct entirely and
pass the iomap_next_fn directly to iomap_iter() and all the iomap/dax
entry points; filesystems pass their ->iomap_next function instead of an
ops struct.
No functional change intended.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
block/fops.c | 10 +++-----
fs/btrfs/direct-io.c | 8 ++----
fs/dax.c | 47 ++++++++++++++++++------------------
fs/erofs/data.c | 24 ++++++++----------
fs/erofs/internal.h | 3 ++-
fs/erofs/zmap.c | 8 ++----
fs/exfat/file.c | 18 +++++++-------
fs/exfat/inode.c | 6 ++---
fs/exfat/iomap.c | 16 +++---------
fs/exfat/iomap.h | 6 +++--
fs/ext2/ext2.h | 3 ++-
fs/ext2/file.c | 4 +--
fs/ext2/inode.c | 8 ++----
fs/ext4/ext4.h | 6 +++--
fs/ext4/extents.c | 8 ++----
fs/ext4/file.c | 14 +++++------
fs/ext4/inode.c | 20 +++++----------
fs/f2fs/data.c | 9 +++----
fs/f2fs/f2fs.h | 3 ++-
fs/f2fs/file.c | 4 +--
fs/fuse/dax.c | 10 +++-----
fs/fuse/file.c | 10 +++-----
fs/gfs2/aops.c | 6 ++---
fs/gfs2/bmap.c | 10 +++-----
fs/gfs2/bmap.h | 3 ++-
fs/gfs2/file.c | 6 ++---
fs/gfs2/inode.c | 6 ++---
fs/hpfs/file.c | 6 +----
fs/internal.h | 1 -
fs/iomap/buffered-io.c | 38 ++++++++++++++---------------
fs/iomap/direct-io.c | 8 +++---
fs/iomap/fiemap.c | 8 +++---
fs/iomap/iter.c | 8 +++---
fs/iomap/seek.c | 8 +++---
fs/iomap/swapfile.c | 4 +--
fs/ntfs/aops.c | 6 ++---
fs/ntfs/file.c | 24 +++++++++---------
fs/ntfs/inode.c | 2 +-
fs/ntfs/iomap.c | 42 +++++++-------------------------
fs/ntfs/iomap.h | 15 ++++++++----
fs/ntfs3/file.c | 16 ++++++------
fs/ntfs3/inode.c | 12 +++------
fs/ntfs3/ntfs_fs.h | 3 ++-
fs/remap_range.c | 6 ++---
fs/xfs/xfs_aops.c | 8 +++---
fs/xfs/xfs_file.c | 40 +++++++++++++++---------------
fs/xfs/xfs_iomap.c | 55 +++++++++---------------------------------
fs/xfs/xfs_iomap.h | 24 ++++++++++++------
fs/xfs/xfs_iops.c | 4 +--
fs/xfs/xfs_reflink.c | 6 ++---
fs/zonefs/file.c | 22 ++++++-----------
include/linux/dax.h | 18 ++++++--------
include/linux/fs.h | 7 ++++--
include/linux/iomap.h | 46 +++++++++++++++--------------------
54 files changed, 302 insertions(+), 411 deletions(-)
diff --git a/block/fops.c b/block/fops.c
index c2721e2c659b..9ccec477f90d 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -459,10 +459,6 @@ static int blkdev_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
return iomap_process(iter, iomap, srcmap, blkdev_iomap_begin, NULL);
}
-static const struct iomap_ops blkdev_iomap_ops = {
- .iomap_next = blkdev_iomap_next,
-};
-
#ifdef CONFIG_BUFFER_HEAD
static int blkdev_get_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh, int create)
@@ -516,13 +512,13 @@ const struct address_space_operations def_blk_aops = {
#else /* CONFIG_BUFFER_HEAD */
static int blkdev_read_folio(struct file *file, struct folio *folio)
{
- iomap_bio_read_folio(folio, &blkdev_iomap_ops);
+ iomap_bio_read_folio(folio, blkdev_iomap_next);
return 0;
}
static void blkdev_readahead(struct readahead_control *rac)
{
- iomap_bio_readahead(rac, &blkdev_iomap_ops);
+ iomap_bio_readahead(rac, blkdev_iomap_next);
}
static ssize_t blkdev_writeback_range(struct iomap_writepage_ctx *wpc,
@@ -713,7 +709,7 @@ blkdev_direct_write(struct kiocb *iocb, struct iov_iter *from)
static ssize_t blkdev_buffered_write(struct kiocb *iocb, struct iov_iter *from)
{
- return iomap_file_buffered_write(iocb, from, &blkdev_iomap_ops, NULL,
+ return iomap_file_buffered_write(iocb, from, blkdev_iomap_next, NULL,
NULL);
}
diff --git a/fs/btrfs/direct-io.c b/fs/btrfs/direct-io.c
index 46dd72982fba..f1feeb68642d 100644
--- a/fs/btrfs/direct-io.c
+++ b/fs/btrfs/direct-io.c
@@ -805,10 +805,6 @@ static int btrfs_dio_iomap_next(const struct iomap_iter *iter,
btrfs_dio_iomap_end);
}
-static const struct iomap_ops btrfs_dio_iomap_ops = {
- .iomap_next = btrfs_dio_iomap_next,
-};
-
static const struct iomap_dio_ops btrfs_dio_ops = {
.submit_io = btrfs_dio_submit_io,
.bio_set = &btrfs_dio_bioset,
@@ -819,7 +815,7 @@ static ssize_t btrfs_dio_read(struct kiocb *iocb, struct iov_iter *iter,
{
struct btrfs_dio_data data = { 0 };
- return iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
+ return iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, &btrfs_dio_ops,
IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, &data, done_before);
}
@@ -828,7 +824,7 @@ static struct iomap_dio *btrfs_dio_write(struct kiocb *iocb, struct iov_iter *it
{
struct btrfs_dio_data data = { 0 };
- return __iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
+ return __iomap_dio_rw(iocb, iter, btrfs_dio_iomap_next, &btrfs_dio_ops,
IOMAP_DIO_PARTIAL | IOMAP_DIO_FSBLOCK_ALIGNED, &data, done_before);
}
diff --git a/fs/dax.c b/fs/dax.c
index 6d175cd47a99..c0a6b87dc052 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -1492,7 +1492,7 @@ static int dax_unshare_iter(struct iomap_iter *iter)
}
int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct iomap_iter iter = {
.inode = inode,
@@ -1506,7 +1506,7 @@ int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
return 0;
iter.len = min(len, size - pos);
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = dax_unshare_iter(&iter);
return ret;
}
@@ -1584,7 +1584,7 @@ static int dax_zero_iter(struct iomap_iter *iter, bool *did_zero)
}
int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct iomap_iter iter = {
.inode = inode,
@@ -1594,14 +1594,14 @@ int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
};
int ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = dax_zero_iter(&iter, did_zero);
return ret;
}
EXPORT_SYMBOL_GPL(dax_zero_range);
int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
unsigned int blocksize = i_blocksize(inode);
unsigned int off = pos & (blocksize - 1);
@@ -1609,7 +1609,7 @@ int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
/* Block boundary? Nothing to do */
if (!off)
return 0;
- return dax_zero_range(inode, pos, blocksize - off, did_zero, ops);
+ return dax_zero_range(inode, pos, blocksize - off, did_zero, iomap_next);
}
EXPORT_SYMBOL_GPL(dax_truncate_page);
@@ -1734,7 +1734,7 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)
* dax_iomap_rw - Perform I/O to a DAX file
* @iocb: The control block for this I/O
* @iter: The addresses to do I/O from or to
- * @ops: iomap ops passed from the file system
+ * @iomap_next: iomap_next callback passed from the file system
*
* This function performs read and write operations to directly mapped
* persistent memory. The callers needs to take care of read/write exclusion
@@ -1742,7 +1742,7 @@ static int dax_iomap_iter(struct iomap_iter *iomi, struct iov_iter *iter)
*/
ssize_t
dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct iomap_iter iomi = {
.inode = iocb->ki_filp->f_mapping->host,
@@ -1769,7 +1769,7 @@ dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
if (iocb->ki_flags & IOCB_NOWAIT)
iomi.flags |= IOMAP_NOWAIT;
- while ((ret = iomap_iter(&iomi, ops)) > 0)
+ while ((ret = iomap_iter(&iomi, iomap_next)) > 0)
iomi.status = dax_iomap_iter(&iomi, iter);
done = iomi.pos - iocb->ki_pos;
@@ -1897,7 +1897,7 @@ static vm_fault_t dax_fault_iter(struct vm_fault *vmf,
}
static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,
- int *iomap_errp, const struct iomap_ops *ops)
+ int *iomap_errp, iomap_next_fn iomap_next)
{
struct address_space *mapping = vmf->vma->vm_file->f_mapping;
XA_STATE(xas, &mapping->i_pages, vmf->pgoff);
@@ -1942,7 +1942,7 @@ static vm_fault_t dax_iomap_pte_fault(struct vm_fault *vmf, unsigned long *pfnp,
goto unlock_entry;
}
- while ((error = iomap_iter(&iter, ops)) > 0) {
+ while ((error = iomap_iter(&iter, iomap_next)) > 0) {
if (WARN_ON_ONCE(iomap_length(&iter) < PAGE_SIZE)) {
iter.status = -EIO; /* fs corruption? */
continue;
@@ -2007,7 +2007,7 @@ static bool dax_fault_check_fallback(struct vm_fault *vmf, struct xa_state *xas,
}
static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct address_space *mapping = vmf->vma->vm_file->f_mapping;
XA_STATE_ORDER(xas, &mapping->i_pages, vmf->pgoff, PMD_ORDER);
@@ -2064,7 +2064,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
}
iter.pos = (loff_t)xas.xa_index << PAGE_SHIFT;
- while (iomap_iter(&iter, ops) > 0) {
+ while (iomap_iter(&iter, iomap_next) > 0) {
if (iomap_length(&iter) < PMD_SIZE)
continue; /* actually breaks out of the loop */
@@ -2086,7 +2086,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
}
#else
static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
return VM_FAULT_FALLBACK;
}
@@ -2098,7 +2098,7 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
* @order: Order of the page to fault in
* @pfnp: PFN to insert for synchronous faults if fsync is required
* @iomap_errp: Storage for detailed error code in case of error
- * @ops: Iomap ops passed from the file system
+ * @iomap_next: iomap_next callback passed from the file system
*
* When a page fault occurs, filesystems may call this helper in
* their fault handler for DAX files. dax_iomap_fault() assumes the caller
@@ -2107,12 +2107,12 @@ static vm_fault_t dax_iomap_pmd_fault(struct vm_fault *vmf, unsigned long *pfnp,
*/
vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,
unsigned long *pfnp, int *iomap_errp,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
if (order == 0)
- return dax_iomap_pte_fault(vmf, pfnp, iomap_errp, ops);
+ return dax_iomap_pte_fault(vmf, pfnp, iomap_errp, iomap_next);
else if (order == PMD_ORDER)
- return dax_iomap_pmd_fault(vmf, pfnp, ops);
+ return dax_iomap_pmd_fault(vmf, pfnp, iomap_next);
else
return VM_FAULT_FALLBACK;
}
@@ -2240,7 +2240,7 @@ static int dax_range_compare_iter(struct iomap_iter *it_src,
int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
struct inode *dst, loff_t dstoff, loff_t len, bool *same,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct iomap_iter src_iter = {
.inode = src,
@@ -2256,8 +2256,8 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
};
int ret, status;
- while ((ret = iomap_iter(&src_iter, ops)) > 0 &&
- (ret = iomap_iter(&dst_iter, ops)) > 0) {
+ while ((ret = iomap_iter(&src_iter, iomap_next)) > 0 &&
+ (ret = iomap_iter(&dst_iter, iomap_next)) > 0) {
status = dax_range_compare_iter(&src_iter, &dst_iter,
min(src_iter.len, dst_iter.len), same);
if (status < 0)
@@ -2270,9 +2270,10 @@ int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
return __generic_remap_file_range_prep(file_in, pos_in, file_out,
- pos_out, len, remap_flags, ops);
+ pos_out, len, remap_flags,
+ iomap_next);
}
EXPORT_SYMBOL_GPL(dax_remap_file_range_prep);
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 47dba61ec576..f6fe8c7eaf6d 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -387,10 +387,6 @@ static int erofs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
erofs_iomap_end);
}
-static const struct iomap_ops erofs_iomap_ops = {
- .iomap_next = erofs_iomap_next,
-};
-
int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len)
{
@@ -398,9 +394,9 @@ int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
if (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))
return -EOPNOTSUPP;
return iomap_fiemap(inode, fieinfo, start, len,
- &z_erofs_iomap_report_ops);
+ z_erofs_iomap_next_report);
}
- return iomap_fiemap(inode, fieinfo, start, len, &erofs_iomap_ops);
+ return iomap_fiemap(inode, fieinfo, start, len, erofs_iomap_next);
}
/*
@@ -419,7 +415,7 @@ static int erofs_read_folio(struct file *file, struct folio *folio)
};
trace_erofs_read_folio(iter_ctx.realinode, folio, true);
- iomap_read_folio(&erofs_iomap_ops, &read_ctx, &iter_ctx);
+ iomap_read_folio(erofs_iomap_next, &read_ctx, &iter_ctx);
if (need_iput)
iput(iter_ctx.realinode);
return 0;
@@ -438,14 +434,14 @@ static void erofs_readahead(struct readahead_control *rac)
trace_erofs_readahead(iter_ctx.realinode, readahead_index(rac),
readahead_count(rac), true);
- iomap_readahead(&erofs_iomap_ops, &read_ctx, &iter_ctx);
+ iomap_readahead(erofs_iomap_next, &read_ctx, &iter_ctx);
if (need_iput)
iput(iter_ctx.realinode);
}
static sector_t erofs_bmap(struct address_space *mapping, sector_t block)
{
- return iomap_bmap(mapping, block, &erofs_iomap_ops);
+ return iomap_bmap(mapping, block, erofs_iomap_next);
}
static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
@@ -457,14 +453,14 @@ static ssize_t erofs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
return 0;
if (IS_ENABLED(CONFIG_FS_DAX) && IS_DAX(inode))
- return dax_iomap_rw(iocb, to, &erofs_iomap_ops);
+ return dax_iomap_rw(iocb, to, erofs_iomap_next);
if ((iocb->ki_flags & IOCB_DIRECT) && inode->i_sb->s_bdev) {
struct erofs_iomap_iter_ctx iter_ctx = {
.realinode = inode,
};
- return iomap_dio_rw(iocb, to, &erofs_iomap_ops,
+ return iomap_dio_rw(iocb, to, erofs_iomap_next,
NULL, 0, &iter_ctx, 0);
}
return filemap_read(iocb, to, 0);
@@ -484,7 +480,7 @@ const struct address_space_operations erofs_aops = {
static vm_fault_t erofs_dax_huge_fault(struct vm_fault *vmf,
unsigned int order)
{
- return dax_iomap_fault(vmf, order, NULL, NULL, &erofs_iomap_ops);
+ return dax_iomap_fault(vmf, order, NULL, NULL, erofs_iomap_next);
}
static vm_fault_t erofs_dax_fault(struct vm_fault *vmf)
@@ -516,12 +512,12 @@ static int erofs_file_mmap_prepare(struct vm_area_desc *desc)
static loff_t erofs_file_llseek(struct file *file, loff_t offset, int whence)
{
struct inode *inode = file->f_mapping->host;
- const struct iomap_ops *ops = &erofs_iomap_ops;
+ iomap_next_fn ops = erofs_iomap_next;
if (erofs_inode_is_data_compressed(EROFS_I(inode)->datalayout)) {
if (!IS_ENABLED(CONFIG_EROFS_FS_ZIP))
return generic_file_llseek(file, offset, whence);
- ops = &z_erofs_iomap_report_ops;
+ ops = z_erofs_iomap_next_report;
}
if (whence == SEEK_HOLE)
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 580f8d9f14e7..72ccd6f335b8 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -397,7 +397,8 @@ extern const struct file_operations erofs_file_fops;
extern const struct file_operations erofs_dir_fops;
extern const struct file_operations erofs_ishare_fops;
-extern const struct iomap_ops z_erofs_iomap_report_ops;
+int z_erofs_iomap_next_report(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
void *erofs_read_metadata(struct super_block *sb, struct erofs_buf *buf,
erofs_off_t *offset, int *lengthp);
diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c
index dd058413a0b6..59054eecd69e 100644
--- a/fs/erofs/zmap.c
+++ b/fs/erofs/zmap.c
@@ -821,13 +821,9 @@ static int z_erofs_iomap_begin_report(struct inode *inode, loff_t offset,
return 0;
}
-static int z_erofs_iomap_next_report(const struct iomap_iter *iter,
- struct iomap *iomap, struct iomap *srcmap)
+int z_erofs_iomap_next_report(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, z_erofs_iomap_begin_report,
NULL);
}
-
-const struct iomap_ops z_erofs_iomap_report_ops = {
- .iomap_next = z_erofs_iomap_next_report,
-};
diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index 5fc13378d35f..c05849d305ae 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -668,7 +668,7 @@ static int exfat_extend_valid_size(struct inode *inode, loff_t new_valid_size)
ret = iomap_zero_range(inode, old_valid_size,
new_valid_size - old_valid_size, NULL,
- &exfat_write_iomap_ops, NULL, NULL);
+ exfat_write_iomap_next, NULL, NULL);
if (ret) {
truncate_setsize(inode, old_valid_size);
exfat_truncate(inode);
@@ -687,7 +687,7 @@ static ssize_t exfat_fallback_buffered_write(struct kiocb *iocb,
iocb->ki_flags &= ~IOCB_DIRECT;
- written = iomap_file_buffered_write(iocb, from, &exfat_write_iomap_ops,
+ written = iomap_file_buffered_write(iocb, from, exfat_write_iomap_next,
NULL, NULL);
if (written < 0)
return written;
@@ -709,7 +709,7 @@ static ssize_t exfat_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
{
ssize_t ret;
- ret = iomap_dio_rw(iocb, from, &exfat_write_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, exfat_write_iomap_next,
&exfat_write_dio_ops, 0, NULL, 0);
if (ret == -ENOTBLK)
ret = 0;
@@ -773,7 +773,7 @@ static ssize_t exfat_file_write_iter(struct kiocb *iocb, struct iov_iter *iter)
ret = exfat_dio_write_iter(iocb, iter);
else
ret = iomap_file_buffered_write(iocb, iter,
- &exfat_write_iomap_ops, NULL, NULL);
+ exfat_write_iomap_next, NULL, NULL);
if (ret < 0)
goto unlock;
@@ -809,7 +809,7 @@ static ssize_t exfat_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
if (iocb->ki_flags & IOCB_DIRECT) {
file_accessed(iocb->ki_filp);
- ret = iomap_dio_rw(iocb, iter, &exfat_iomap_ops, NULL, 0,
+ ret = iomap_dio_rw(iocb, iter, exfat_iomap_next, NULL, 0,
NULL, 0);
} else {
ret = generic_file_read_iter(iocb, iter);
@@ -850,7 +850,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)
*/
err = iomap_zero_range(inode, ei->zeroed_size,
mmap_valid_size - ei->zeroed_size, NULL,
- &exfat_iomap_ops, NULL, NULL);
+ exfat_iomap_next, NULL, NULL);
if (err < 0) {
inode_unlock(inode);
return vmf_fs_error(err);
@@ -866,7 +866,7 @@ static vm_fault_t exfat_page_mkwrite(struct vm_fault *vmf)
file_update_time(vmf->vma->vm_file);
filemap_invalidate_lock_shared(inode->i_mapping);
- ret = iomap_page_mkwrite(vmf, &exfat_write_iomap_ops, NULL);
+ ret = iomap_page_mkwrite(vmf, exfat_write_iomap_next, NULL);
filemap_invalidate_unlock_shared(inode->i_mapping);
sb_end_pagefault(inode->i_sb);
inode_unlock(inode);
@@ -939,12 +939,12 @@ static loff_t exfat_file_llseek(struct file *file, loff_t offset, int whence)
switch (whence) {
case SEEK_HOLE:
inode_lock_shared(inode);
- offset = iomap_seek_hole(inode, offset, &exfat_iomap_ops);
+ offset = iomap_seek_hole(inode, offset, exfat_iomap_next);
inode_unlock_shared(inode);
break;
case SEEK_DATA:
inode_lock_shared(inode);
- offset = iomap_seek_data(inode, offset, &exfat_iomap_ops);
+ offset = iomap_seek_data(inode, offset, exfat_iomap_next);
inode_unlock_shared(inode);
break;
default:
diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c
index 89826aea5e1e..a6b9aa2ad792 100644
--- a/fs/exfat/inode.c
+++ b/fs/exfat/inode.c
@@ -248,7 +248,7 @@ static int exfat_read_folio(struct file *file, struct folio *folio)
.ops = &exfat_iomap_bio_read_ops,
};
- iomap_read_folio(&exfat_iomap_ops, &ctx, NULL);
+ iomap_read_folio(exfat_iomap_next, &ctx, NULL);
return 0;
}
@@ -269,7 +269,7 @@ static void exfat_readahead(struct readahead_control *rac)
ei->valid_size < pos + readahead_length(rac))
return;
- iomap_readahead(&exfat_iomap_ops, &ctx, NULL);
+ iomap_readahead(exfat_iomap_next, &ctx, NULL);
}
static int exfat_writepages(struct address_space *mapping,
@@ -293,7 +293,7 @@ static sector_t exfat_aop_bmap(struct address_space *mapping, sector_t block)
/* exfat_get_cluster() assumes the requested blocknr isn't truncated. */
down_read(&EXFAT_I(mapping->host)->truncate_lock);
- blocknr = iomap_bmap(mapping, block, &exfat_iomap_ops);
+ blocknr = iomap_bmap(mapping, block, exfat_iomap_next);
up_read(&EXFAT_I(mapping->host)->truncate_lock);
return blocknr;
}
diff --git a/fs/exfat/iomap.c b/fs/exfat/iomap.c
index 8d33690a562d..6120e0758f7b 100644
--- a/fs/exfat/iomap.c
+++ b/fs/exfat/iomap.c
@@ -151,16 +151,12 @@ static int exfat_write_iomap_begin(struct inode *inode, loff_t offset, loff_t le
return __exfat_iomap_begin(inode, offset, length, flags, iomap, true);
}
-static int exfat_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+int exfat_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, exfat_iomap_begin, NULL);
}
-const struct iomap_ops exfat_iomap_ops = {
- .iomap_next = exfat_iomap_next,
-};
-
/*
* exfat_write_iomap_end - Update the state after write
*
@@ -192,17 +188,13 @@ static int exfat_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return written;
}
-static int exfat_write_iomap_next(const struct iomap_iter *iter,
- struct iomap *iomap, struct iomap *srcmap)
+int exfat_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap,
exfat_write_iomap_begin, exfat_write_iomap_end);
}
-const struct iomap_ops exfat_write_iomap_ops = {
- .iomap_next = exfat_write_iomap_next,
-};
-
/*
* exfat_writeback_range - Map folio during writeback
*
@@ -279,5 +271,5 @@ const struct iomap_read_ops exfat_iomap_bio_read_ops = {
int exfat_iomap_swap_activate(struct swap_info_struct *sis,
struct file *file, sector_t *span)
{
- return iomap_swapfile_activate(sis, file, span, &exfat_iomap_ops);
+ return iomap_swapfile_activate(sis, file, span, exfat_iomap_next);
}
diff --git a/fs/exfat/iomap.h b/fs/exfat/iomap.h
index fd8a913f7794..47d7b753735e 100644
--- a/fs/exfat/iomap.h
+++ b/fs/exfat/iomap.h
@@ -7,8 +7,10 @@
#define _LINUX_EXFAT_IOMAP_H
extern const struct iomap_dio_ops exfat_write_dio_ops;
-extern const struct iomap_ops exfat_iomap_ops;
-extern const struct iomap_ops exfat_write_iomap_ops;
+int exfat_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int exfat_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_writeback_ops exfat_writeback_ops;
extern const struct iomap_read_ops exfat_iomap_bio_read_ops;
diff --git a/fs/ext2/ext2.h b/fs/ext2/ext2.h
index 79f7b395258c..59ef8b898940 100644
--- a/fs/ext2/ext2.h
+++ b/fs/ext2/ext2.h
@@ -780,7 +780,8 @@ extern const struct file_operations ext2_file_operations;
/* inode.c */
extern void ext2_set_file_ops(struct inode *inode);
extern const struct address_space_operations ext2_aops;
-extern const struct iomap_ops ext2_iomap_ops;
+int ext2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
/* namei.c */
extern const struct inode_operations ext2_dir_inode_operations;
diff --git a/fs/ext2/file.c b/fs/ext2/file.c
index 8dca9ec4cacd..1fc00ad77517 100644
--- a/fs/ext2/file.c
+++ b/fs/ext2/file.c
@@ -70,7 +70,7 @@ static ssize_t ext2_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
trace_ext2_dio_read_begin(iocb, to, 0);
inode_lock_shared(inode);
- ret = iomap_dio_rw(iocb, to, &ext2_iomap_ops, NULL, 0, NULL, 0);
+ ret = iomap_dio_rw(iocb, to, ext2_iomap_next, NULL, 0, NULL, 0);
inode_unlock_shared(inode);
trace_ext2_dio_read_end(iocb, to, ret);
@@ -134,7 +134,7 @@ static ssize_t ext2_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
(!IS_ALIGNED(iocb->ki_pos | iov_iter_alignment(from), blocksize)))
flags |= IOMAP_DIO_FORCE_WAIT;
- ret = iomap_dio_rw(iocb, from, &ext2_iomap_ops, &ext2_dio_write_ops,
+ ret = iomap_dio_rw(iocb, from, ext2_iomap_next, &ext2_dio_write_ops,
flags, NULL, 0);
/* ENOTBLK is magic return value for fallback to buffered-io */
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 0693059caa35..74d5be85341d 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -860,17 +860,13 @@ ext2_iomap_end(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-static int ext2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+int ext2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ext2_iomap_begin,
ext2_iomap_end);
}
-const struct iomap_ops ext2_iomap_ops = {
- .iomap_next = ext2_iomap_next,
-};
-
int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len)
{
@@ -888,7 +884,7 @@ int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
if (i_size == 0)
i_size = 1;
len = min_t(u64, len, i_size);
- ret = iomap_fiemap(inode, fieinfo, start, len, &ext2_iomap_ops);
+ ret = iomap_fiemap(inode, fieinfo, start, len, ext2_iomap_next);
inode_unlock(inode);
return ret;
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index b37c136ea3ab..755fde1baf03 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -4004,8 +4004,10 @@ static inline void ext4_clear_io_unwritten_flag(ext4_io_end_t *io_end)
io_end->flag &= ~EXT4_IO_END_UNWRITTEN;
}
-extern const struct iomap_ops ext4_iomap_ops;
-extern const struct iomap_ops ext4_iomap_report_ops;
+int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
static inline int ext4_buffer_uptodate(struct buffer_head *bh)
{
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 431298eca7e8..aa3c5c0915c0 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -5177,10 +5177,6 @@ static int ext4_iomap_xattr_next(const struct iomap_iter *iter,
return iomap_process(iter, iomap, srcmap, ext4_iomap_xattr_begin, NULL);
}
-static const struct iomap_ops ext4_iomap_xattr_ops = {
- .iomap_next = ext4_iomap_xattr_next,
-};
-
static int ext4_fiemap_check_ranges(struct inode *inode, u64 start, u64 *len)
{
u64 maxbytes = ext4_get_maxbytes(inode);
@@ -5223,10 +5219,10 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
if (fieinfo->fi_flags & FIEMAP_FLAG_XATTR) {
fieinfo->fi_flags &= ~FIEMAP_FLAG_XATTR;
error = iomap_fiemap(inode, fieinfo, start, len,
- &ext4_iomap_xattr_ops);
+ ext4_iomap_xattr_next);
} else {
error = iomap_fiemap(inode, fieinfo, start, len,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
}
unlock:
inode_unlock_shared(inode);
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index eb1a323962b1..dbe073e181a7 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -91,7 +91,7 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
return generic_file_read_iter(iocb, to);
}
- ret = iomap_dio_rw(iocb, to, &ext4_iomap_ops, NULL, 0, NULL, 0);
+ ret = iomap_dio_rw(iocb, to, ext4_iomap_next, NULL, 0, NULL, 0);
inode_unlock_shared(inode);
file_accessed(iocb->ki_filp);
@@ -119,7 +119,7 @@ static ssize_t ext4_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)
/* Fallback to buffered IO in case we cannot support DAX */
return generic_file_read_iter(iocb, to);
}
- ret = dax_iomap_rw(iocb, to, &ext4_iomap_ops);
+ ret = dax_iomap_rw(iocb, to, ext4_iomap_next);
inode_unlock_shared(inode);
file_accessed(iocb->ki_filp);
@@ -589,7 +589,7 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
goto out;
}
- ret = iomap_dio_rw(iocb, from, &ext4_iomap_ops, &ext4_dio_write_ops,
+ ret = iomap_dio_rw(iocb, from, ext4_iomap_next, &ext4_dio_write_ops,
dio_flags, NULL, 0);
if (ret == -ENOTBLK)
ret = 0;
@@ -688,7 +688,7 @@ ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)
ext4_journal_stop(handle);
}
- ret = dax_iomap_rw(iocb, from, &ext4_iomap_ops);
+ ret = dax_iomap_rw(iocb, from, ext4_iomap_next);
if (extend) {
ret = ext4_handle_inode_extension(inode, offset, ret, count);
@@ -776,7 +776,7 @@ static vm_fault_t ext4_dax_huge_fault(struct vm_fault *vmf, unsigned int order)
} else {
filemap_invalidate_lock_shared(mapping);
}
- result = dax_iomap_fault(vmf, order, &pfn, &error, &ext4_iomap_ops);
+ result = dax_iomap_fault(vmf, order, &pfn, &error, ext4_iomap_next);
if (write) {
ext4_journal_stop(handle);
@@ -955,13 +955,13 @@ loff_t ext4_llseek(struct file *file, loff_t offset, int whence)
case SEEK_HOLE:
inode_lock_shared(inode);
offset = iomap_seek_hole(inode, offset,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
inode_unlock_shared(inode);
break;
case SEEK_DATA:
inode_lock_shared(inode);
offset = iomap_seek_data(inode, offset,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
inode_unlock_shared(inode);
break;
}
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index cf7aa8275651..4c30dd8dbec7 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3391,7 +3391,7 @@ static sector_t ext4_bmap(struct address_space *mapping, sector_t block)
filemap_write_and_wait(mapping);
}
- ret = iomap_bmap(mapping, block, &ext4_iomap_ops);
+ ret = iomap_bmap(mapping, block, ext4_iomap_next);
out:
inode_unlock_shared(inode);
@@ -3850,16 +3850,12 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-static int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+int ext4_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ext4_iomap_begin, NULL);
}
-const struct iomap_ops ext4_iomap_ops = {
- .iomap_next = ext4_iomap_next,
-};
-
static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
struct iomap *iomap, struct iomap *srcmap)
@@ -3911,17 +3907,13 @@ static int ext4_iomap_begin_report(struct inode *inode, loff_t offset,
return 0;
}
-static int ext4_iomap_next_report(const struct iomap_iter *iter,
- struct iomap *iomap, struct iomap *srcmap)
+int ext4_iomap_next_report(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ext4_iomap_begin_report,
NULL);
}
-const struct iomap_ops ext4_iomap_report_ops = {
- .iomap_next = ext4_iomap_next_report,
-};
-
/*
* For data=journal mode, folio should be marked dirty only when it was
* writeably mapped. When that happens, it was already attached to the
@@ -3957,7 +3949,7 @@ static int ext4_iomap_swap_activate(struct swap_info_struct *sis,
struct file *file, sector_t *span)
{
return iomap_swapfile_activate(sis, file, span,
- &ext4_iomap_report_ops);
+ ext4_iomap_next_report);
}
static const struct address_space_operations ext4_aops = {
@@ -4204,7 +4196,7 @@ static int ext4_block_zero_range(struct inode *inode,
if (IS_DAX(inode)) {
return dax_zero_range(inode, from, length, did_zero,
- &ext4_iomap_ops);
+ ext4_iomap_next);
} else if (ext4_should_journal_data(inode)) {
return ext4_block_journalled_zero_range(inode, from, length,
did_zero);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index afc9b2adaa98..9c281336c9b3 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -4171,6 +4171,7 @@ static bool f2fs_dirty_data_folio(struct address_space *mapping,
}
+
static sector_t f2fs_bmap_compress(struct inode *inode, sector_t block)
{
#ifdef CONFIG_F2FS_FS_COMPRESSION
@@ -4653,12 +4654,8 @@ static int f2fs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
return 0;
}
-static int f2fs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
- struct iomap *srcmap)
+int f2fs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, f2fs_iomap_begin, NULL);
}
-
-const struct iomap_ops f2fs_iomap_ops = {
- .iomap_next = f2fs_iomap_next,
-};
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 8f3e632f315c..946a91834aec 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -4216,7 +4216,8 @@ int f2fs_init_post_read_processing(void);
void f2fs_destroy_post_read_processing(void);
int f2fs_init_wq(struct f2fs_sb_info *sbi);
void f2fs_destroy_wq(struct f2fs_sb_info *sbi);
-extern const struct iomap_ops f2fs_iomap_ops;
+int f2fs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
/*
* gc.c
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 4b52c56d71f0..74514b117257 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -4884,7 +4884,7 @@ static ssize_t f2fs_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
* F2FS_DIO_READ counter will be decremented correctly in all cases.
*/
inc_page_count(sbi, F2FS_DIO_READ);
- dio = __iomap_dio_rw(iocb, to, &f2fs_iomap_ops,
+ dio = __iomap_dio_rw(iocb, to, f2fs_iomap_next,
&f2fs_iomap_dio_read_ops, 0, NULL, 0);
if (IS_ERR_OR_NULL(dio)) {
ret = PTR_ERR_OR_ZERO(dio);
@@ -5220,7 +5220,7 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
dio_flags = 0;
if (pos + count > inode->i_size)
dio_flags |= IOMAP_DIO_FORCE_WAIT;
- dio = __iomap_dio_rw(iocb, from, &f2fs_iomap_ops,
+ dio = __iomap_dio_rw(iocb, from, f2fs_iomap_next,
&f2fs_iomap_dio_write_ops, dio_flags, iocb, 0);
if (IS_ERR_OR_NULL(dio)) {
ret = PTR_ERR_OR_ZERO(dio);
diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c
index e8d8c9f5d728..a6e9721552ba 100644
--- a/fs/fuse/dax.c
+++ b/fs/fuse/dax.c
@@ -660,10 +660,6 @@ static int fuse_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
fuse_iomap_end);
}
-static const struct iomap_ops fuse_iomap_ops = {
- .iomap_next = fuse_iomap_next,
-};
-
static void fuse_wait_dax_page(struct inode *inode)
{
filemap_invalidate_unlock(inode->i_mapping);
@@ -691,7 +687,7 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to)
inode_lock_shared(inode);
}
- ret = dax_iomap_rw(iocb, to, &fuse_iomap_ops);
+ ret = dax_iomap_rw(iocb, to, fuse_iomap_next);
inode_unlock_shared(inode);
/* TODO file_accessed(iocb->f_filp) */
@@ -746,7 +742,7 @@ ssize_t fuse_dax_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (file_extending_write(iocb, from))
ret = fuse_dax_direct_write(iocb, from);
else
- ret = dax_iomap_rw(iocb, from, &fuse_iomap_ops);
+ ret = dax_iomap_rw(iocb, from, fuse_iomap_next);
out:
inode_unlock(inode);
@@ -781,7 +777,7 @@ static vm_fault_t __fuse_dax_fault(struct vm_fault *vmf, unsigned int order,
* to populate page cache or access memory we are trying to free.
*/
filemap_invalidate_lock_shared(inode->i_mapping);
- ret = dax_iomap_fault(vmf, order, &pfn, &error, &fuse_iomap_ops);
+ ret = dax_iomap_fault(vmf, order, &pfn, &error, fuse_iomap_next);
if ((ret & VM_FAULT_ERROR) && error == -EAGAIN) {
error = 0;
retry = true;
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 5c0d400629cc..b3e95a28623d 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -896,10 +896,6 @@ static int fuse_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
return iomap_process(iter, iomap, srcmap, fuse_iomap_begin, NULL);
}
-static const struct iomap_ops fuse_iomap_ops = {
- .iomap_next = fuse_iomap_next,
-};
-
struct fuse_fill_read_data {
struct file *file;
@@ -1020,7 +1016,7 @@ static int fuse_read_folio(struct file *file, struct folio *folio)
return -EIO;
}
- iomap_read_folio(&fuse_iomap_ops, &ctx, NULL);
+ iomap_read_folio(fuse_iomap_next, &ctx, NULL);
fuse_invalidate_atime(inode);
return 0;
}
@@ -1121,7 +1117,7 @@ static void fuse_readahead(struct readahead_control *rac)
if (fuse_is_bad(inode))
return;
- iomap_readahead(&fuse_iomap_ops, &ctx, NULL);
+ iomap_readahead(fuse_iomap_next, &ctx, NULL);
}
static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to)
@@ -1553,7 +1549,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from)
* and granular dirty tracking for large folios.
*/
written = iomap_file_buffered_write(iocb, from,
- &fuse_iomap_ops,
+ fuse_iomap_next,
&fuse_iomap_write_ops,
file);
} else {
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index 0a7b8076af3a..66bc19c011cc 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -425,7 +425,7 @@ static int gfs2_read_folio(struct file *file, struct folio *folio)
if (!gfs2_is_jdata(ip) ||
(i_blocksize(inode) == PAGE_SIZE && !folio_buffers(folio))) {
- iomap_bio_read_folio(folio, &gfs2_iomap_ops);
+ iomap_bio_read_folio(folio, gfs2_iomap_next);
} else if (gfs2_is_stuffed(ip)) {
error = stuffed_read_folio(ip, folio);
} else {
@@ -500,7 +500,7 @@ static void gfs2_readahead(struct readahead_control *rac)
else if (gfs2_is_jdata(ip))
mpage_readahead(rac, gfs2_block_map);
else
- iomap_bio_readahead(rac, &gfs2_iomap_ops);
+ iomap_bio_readahead(rac, gfs2_iomap_next);
}
/**
@@ -571,7 +571,7 @@ static sector_t gfs2_bmap(struct address_space *mapping, sector_t lblock)
return 0;
if (!gfs2_is_stuffed(ip))
- dblock = iomap_bmap(mapping, lblock, &gfs2_iomap_ops);
+ dblock = iomap_bmap(mapping, lblock, gfs2_iomap_next);
gfs2_glock_dq_uninit(&i_gh);
diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index 6cb1d4513882..1b96f5622be6 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -1200,17 +1200,13 @@ static int gfs2_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return 0;
}
-static int gfs2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
- struct iomap *srcmap)
+int gfs2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, gfs2_iomap_begin,
gfs2_iomap_end);
}
-const struct iomap_ops gfs2_iomap_ops = {
- .iomap_next = gfs2_iomap_next,
-};
-
/**
* gfs2_block_map - Map one or more blocks of an inode to a disk block
* @inode: The inode
@@ -1324,7 +1320,7 @@ static int gfs2_block_zero_range(struct inode *inode, loff_t from, loff_t length
if (from >= inode->i_size)
return 0;
length = min(length, inode->i_size - from);
- return iomap_zero_range(inode, from, length, NULL, &gfs2_iomap_ops,
+ return iomap_zero_range(inode, from, length, NULL, gfs2_iomap_next,
&gfs2_iomap_write_ops, NULL);
}
diff --git a/fs/gfs2/bmap.h b/fs/gfs2/bmap.h
index e3d6efdfd890..2c2b7ab39259 100644
--- a/fs/gfs2/bmap.h
+++ b/fs/gfs2/bmap.h
@@ -43,7 +43,8 @@ static inline void gfs2_write_calc_reserv(const struct gfs2_inode *ip,
}
}
-extern const struct iomap_ops gfs2_iomap_ops;
+int gfs2_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_write_ops gfs2_iomap_write_ops;
extern const struct iomap_writeback_ops gfs2_writeback_ops;
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index b8c10de113ba..ef5f521a46c0 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -844,7 +844,7 @@ static ssize_t gfs2_file_direct_read(struct kiocb *iocb, struct iov_iter *to,
goto out_uninit;
pagefault_disable();
to->nofault = true;
- ret = iomap_dio_rw(iocb, to, &gfs2_iomap_ops, NULL,
+ ret = iomap_dio_rw(iocb, to, gfs2_iomap_next, NULL,
IOMAP_DIO_PARTIAL, NULL, read);
to->nofault = false;
pagefault_enable();
@@ -910,7 +910,7 @@ static ssize_t gfs2_file_direct_write(struct kiocb *iocb, struct iov_iter *from,
goto out_unlock;
from->nofault = true;
- ret = iomap_dio_rw(iocb, from, &gfs2_iomap_ops, NULL,
+ ret = iomap_dio_rw(iocb, from, gfs2_iomap_next, NULL,
IOMAP_DIO_PARTIAL, NULL, written);
from->nofault = false;
if (ret <= 0) {
@@ -1062,7 +1062,7 @@ static ssize_t gfs2_file_buffered_write(struct kiocb *iocb,
goto out_unlock;
pagefault_disable();
- ret = iomap_file_buffered_write(iocb, from, &gfs2_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, gfs2_iomap_next,
&gfs2_iomap_write_ops, NULL);
pagefault_enable();
if (ret > 0)
diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c
index 8a77794bbd4a..737a3b6c5268 100644
--- a/fs/gfs2/inode.c
+++ b/fs/gfs2/inode.c
@@ -2217,7 +2217,7 @@ static int gfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
goto out;
pagefault_disable();
- ret = iomap_fiemap(inode, fieinfo, start, len, &gfs2_iomap_ops);
+ ret = iomap_fiemap(inode, fieinfo, start, len, gfs2_iomap_next);
pagefault_enable();
gfs2_glock_dq_uninit(&gh);
@@ -2242,7 +2242,7 @@ loff_t gfs2_seek_data(struct file *file, loff_t offset)
inode_lock_shared(inode);
ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, 0, &gh);
if (!ret)
- ret = iomap_seek_data(inode, offset, &gfs2_iomap_ops);
+ ret = iomap_seek_data(inode, offset, gfs2_iomap_next);
gfs2_glock_dq_uninit(&gh);
inode_unlock_shared(inode);
@@ -2261,7 +2261,7 @@ loff_t gfs2_seek_hole(struct file *file, loff_t offset)
inode_lock_shared(inode);
ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, 0, &gh);
if (!ret)
- ret = iomap_seek_hole(inode, offset, &gfs2_iomap_ops);
+ ret = iomap_seek_hole(inode, offset, gfs2_iomap_next);
gfs2_glock_dq_uninit(&gh);
inode_unlock_shared(inode);
diff --git a/fs/hpfs/file.c b/fs/hpfs/file.c
index 1df9f28fb40b..08d5df5fb3cf 100644
--- a/fs/hpfs/file.c
+++ b/fs/hpfs/file.c
@@ -162,10 +162,6 @@ static int hpfs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
return iomap_process(iter, iomap, srcmap, hpfs_iomap_begin, NULL);
}
-static const struct iomap_ops hpfs_iomap_ops = {
- .iomap_next = hpfs_iomap_next,
-};
-
static int hpfs_read_folio(struct file *file, struct folio *folio)
{
return mpage_read_folio(folio, hpfs_get_block);
@@ -242,7 +238,7 @@ static int hpfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
inode_lock(inode);
len = min_t(u64, len, i_size_read(inode));
- ret = iomap_fiemap(inode, fieinfo, start, len, &hpfs_iomap_ops);
+ ret = iomap_fiemap(inode, fieinfo, start, len, hpfs_iomap_next);
inode_unlock(inode);
return ret;
diff --git a/fs/internal.h b/fs/internal.h
index 355d93f92208..19601f8406dc 100644
--- a/fs/internal.h
+++ b/fs/internal.h
@@ -8,7 +8,6 @@
struct super_block;
struct file_system_type;
struct iomap;
-struct iomap_ops;
struct linux_binprm;
struct path;
struct mount;
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 3f0932e46fd6..0aa8abc438c1 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -626,7 +626,7 @@ static int iomap_read_folio_iter(struct iomap_iter *iter,
return 0;
}
-void iomap_read_folio(const struct iomap_ops *ops,
+void iomap_read_folio(iomap_next_fn iomap_next,
struct iomap_read_folio_ctx *ctx, void *private)
{
struct folio *folio = ctx->cur_folio;
@@ -650,7 +650,7 @@ void iomap_read_folio(const struct iomap_ops *ops,
fsverity_readahead(ctx->vi, folio->index,
folio_nr_pages(folio));
- while ((ret = iomap_iter(&iter, ops)) > 0) {
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0) {
iter.status = iomap_read_folio_iter(&iter, ctx,
&bytes_submitted);
iomap_read_submit(&iter, ctx);
@@ -688,22 +688,22 @@ static int iomap_readahead_iter(struct iomap_iter *iter,
/**
* iomap_readahead - Attempt to read pages from a file.
- * @ops: The operations vector for the filesystem.
+ * @iomap_next: The iomap_next callback for the filesystem.
* @ctx: The ctx used for issuing readahead.
* @private: The filesystem-specific information for issuing iomap_iter.
*
* This function is for filesystems to call to implement their readahead
* address_space operation.
*
- * Context: The @ops callbacks may submit I/O (eg to read the addresses of
+ * Context: The @iomap_next callback may submit I/O (eg to read the addresses of
* blocks from disc), and may wait for it. The caller may be trying to
* access a different page, and so sleeping excessively should be avoided.
* It may allocate memory, but should avoid costly allocations. This
* function is called with memalloc_nofs set, so allocations will not cause
* the filesystem to be reentered.
*/
-void iomap_readahead(const struct iomap_ops *ops,
- struct iomap_read_folio_ctx *ctx, void *private)
+void iomap_readahead(iomap_next_fn iomap_next, struct iomap_read_folio_ctx *ctx,
+ void *private)
{
struct readahead_control *rac = ctx->rac;
struct iomap_iter iter = {
@@ -725,7 +725,7 @@ void iomap_readahead(const struct iomap_ops *ops,
fsverity_readahead(ctx->vi, readahead_index(rac),
readahead_count(rac));
- while (iomap_iter(&iter, ops) > 0) {
+ while (iomap_iter(&iter, iomap_next) > 0) {
iter.status = iomap_readahead_iter(&iter, ctx,
&cur_bytes_submitted);
iomap_read_submit(&iter, ctx);
@@ -1268,7 +1268,7 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
ssize_t
iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops, void *private)
{
struct iomap_iter iter = {
@@ -1285,7 +1285,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
if (iocb->ki_flags & IOCB_DONTCACHE)
iter.flags |= IOMAP_DONTCACHE;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = iomap_write_iter(&iter, i, write_ops);
if (unlikely(iter.pos == iocb->ki_pos))
@@ -1297,7 +1297,7 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
EXPORT_SYMBOL_GPL(iomap_file_buffered_write);
int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
- const void *buf, const struct iomap_ops *ops,
+ const void *buf, iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops)
{
int ret;
@@ -1314,7 +1314,7 @@ int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
iov_iter_kvec(&iiter, WRITE, &kvec, 1, length);
- ret = iomap_file_buffered_write(&iocb, &iiter, ops, write_ops, NULL);
+ ret = iomap_file_buffered_write(&iocb, &iiter, iomap_next, write_ops, NULL);
if (ret < 0)
return ret;
return ret == length ? 0 : -EIO;
@@ -1586,7 +1586,7 @@ static int iomap_unshare_iter(struct iomap_iter *iter,
int
iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops)
{
struct iomap_iter iter = {
@@ -1601,7 +1601,7 @@ iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
return 0;
iter.len = min(len, size - pos);
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = iomap_unshare_iter(&iter, write_ops);
return ret;
}
@@ -1710,7 +1710,7 @@ EXPORT_SYMBOL_GPL(iomap_fill_dirty_folios);
int
iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops, void *private)
{
struct folio_batch fbatch;
@@ -1735,7 +1735,7 @@ iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
*/
range_dirty = filemap_range_needs_writeback(mapping, iter.pos,
iter.pos + iter.len - 1);
- while ((ret = iomap_iter(&iter, ops)) > 0) {
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0) {
const struct iomap *srcmap = iomap_iter_srcmap(&iter);
if (!(iter.iomap.flags & IOMAP_F_FOLIO_BATCH) &&
@@ -1761,7 +1761,7 @@ EXPORT_SYMBOL_GPL(iomap_zero_range);
int
iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops, void *private)
{
unsigned int blocksize = i_blocksize(inode);
@@ -1770,7 +1770,7 @@ iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
/* Block boundary? Nothing to do */
if (!off)
return 0;
- return iomap_zero_range(inode, pos, blocksize - off, did_zero, ops,
+ return iomap_zero_range(inode, pos, blocksize - off, did_zero, iomap_next,
write_ops, private);
}
EXPORT_SYMBOL_GPL(iomap_truncate_page);
@@ -1795,7 +1795,7 @@ static int iomap_folio_mkwrite_iter(struct iomap_iter *iter,
return iomap_iter_advance(iter, length);
}
-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_next_fn iomap_next,
void *private)
{
struct iomap_iter iter = {
@@ -1812,7 +1812,7 @@ vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
goto out_unlock;
iter.pos = folio_pos(folio);
iter.len = ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = iomap_folio_mkwrite_iter(&iter, folio);
if (ret < 0)
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index b485e3b191da..e299d186f743 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -676,7 +676,7 @@ static int iomap_dio_iter(struct iomap_iter *iter, struct iomap_dio *dio)
*/
struct iomap_dio *
__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+ iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before)
{
struct inode *inode = file_inode(iocb->ki_filp);
@@ -800,7 +800,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
inode_dio_begin(inode);
blk_start_plug(&plug);
- while ((ret = iomap_iter(&iomi, ops)) > 0) {
+ while ((ret = iomap_iter(&iomi, iomap_next)) > 0) {
iomi.status = iomap_dio_iter(&iomi, dio);
/*
@@ -890,12 +890,12 @@ EXPORT_SYMBOL_GPL(__iomap_dio_rw);
ssize_t
iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+ iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before)
{
struct iomap_dio *dio;
- dio = __iomap_dio_rw(iocb, iter, ops, dops, dio_flags, private,
+ dio = __iomap_dio_rw(iocb, iter, iomap_next, dops, dio_flags, private,
done_before);
if (IS_ERR_OR_NULL(dio))
return PTR_ERR_OR_ZERO(dio);
diff --git a/fs/iomap/fiemap.c b/fs/iomap/fiemap.c
index d11dadff8286..fc488f05d8ce 100644
--- a/fs/iomap/fiemap.c
+++ b/fs/iomap/fiemap.c
@@ -56,7 +56,7 @@ static int iomap_fiemap_iter(struct iomap_iter *iter,
}
int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,
- u64 start, u64 len, const struct iomap_ops *ops)
+ u64 start, u64 len, iomap_next_fn iomap_next)
{
struct iomap_iter iter = {
.inode = inode,
@@ -73,7 +73,7 @@ int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fi,
if (ret)
return ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = iomap_fiemap_iter(&iter, fi, &prev);
if (prev.type != IOMAP_HOLE) {
@@ -92,7 +92,7 @@ EXPORT_SYMBOL_GPL(iomap_fiemap);
/* legacy ->bmap interface. 0 is the error return (!) */
sector_t
iomap_bmap(struct address_space *mapping, sector_t bno,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct iomap_iter iter = {
.inode = mapping->host,
@@ -107,7 +107,7 @@ iomap_bmap(struct address_space *mapping, sector_t bno,
return 0;
bno = 0;
- while ((ret = iomap_iter(&iter, ops)) > 0) {
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0) {
if (iter.iomap.type == IOMAP_MAPPED)
bno = iomap_sector(&iter.iomap, iter.pos) >> blkshift;
/* leave iter.status unset to abort loop */
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index 466c491bdef6..984045af310a 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -42,7 +42,7 @@ static inline void iomap_iter_done(struct iomap_iter *iter)
/**
* iomap_iter - iterate over a ranges in a file
* @iter: iteration structue
- * @ops: iomap ops provided by the file system
+ * @iomap_next: iomap_next callback provided by the file system
*
* Iterate over filesystem-provided space mappings for the provided file range.
*
@@ -54,13 +54,13 @@ static inline void iomap_iter_done(struct iomap_iter *iter)
* of the loop body: leave @iter.status unchanged, or set it to a negative
* errno.
*/
-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
+int iomap_iter(struct iomap_iter *iter, iomap_next_fn iomap_next)
{
int ret;
- trace_iomap_iter(iter, ops, _RET_IP_);
+ trace_iomap_iter(iter, iomap_next, _RET_IP_);
- ret = ops->iomap_next(iter, &iter->iomap, &iter->srcmap);
+ ret = iomap_next(iter, &iter->iomap, &iter->srcmap);
iter->status = 0;
if (ret > 0)
iomap_iter_done(iter);
diff --git a/fs/iomap/seek.c b/fs/iomap/seek.c
index 6cbc587c93da..1bc5053d3fc1 100644
--- a/fs/iomap/seek.c
+++ b/fs/iomap/seek.c
@@ -27,7 +27,7 @@ static int iomap_seek_hole_iter(struct iomap_iter *iter,
}
loff_t
-iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
+iomap_seek_hole(struct inode *inode, loff_t pos, iomap_next_fn iomap_next)
{
loff_t size = i_size_read(inode);
struct iomap_iter iter = {
@@ -42,7 +42,7 @@ iomap_seek_hole(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
return -ENXIO;
iter.len = size - pos;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = iomap_seek_hole_iter(&iter, &pos);
if (ret < 0)
return ret;
@@ -73,7 +73,7 @@ static int iomap_seek_data_iter(struct iomap_iter *iter,
}
loff_t
-iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
+iomap_seek_data(struct inode *inode, loff_t pos, iomap_next_fn iomap_next)
{
loff_t size = i_size_read(inode);
struct iomap_iter iter = {
@@ -88,7 +88,7 @@ iomap_seek_data(struct inode *inode, loff_t pos, const struct iomap_ops *ops)
return -ENXIO;
iter.len = size - pos;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = iomap_seek_data_iter(&iter, &pos);
if (ret < 0)
return ret;
diff --git a/fs/iomap/swapfile.c b/fs/iomap/swapfile.c
index 0db77c449467..b8bb34deddfc 100644
--- a/fs/iomap/swapfile.c
+++ b/fs/iomap/swapfile.c
@@ -139,7 +139,7 @@ static int iomap_swapfile_iter(struct iomap_iter *iter,
*/
int iomap_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file, sector_t *pagespan,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct inode *inode = swap_file->f_mapping->host;
struct iomap_iter iter = {
@@ -163,7 +163,7 @@ int iomap_swapfile_activate(struct swap_info_struct *sis,
if (ret)
return ret;
- while ((ret = iomap_iter(&iter, ops)) > 0)
+ while ((ret = iomap_iter(&iter, iomap_next)) > 0)
iter.status = iomap_swapfile_iter(&iter, &iter.iomap, &isi);
if (ret < 0)
return ret;
diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c
index 1fbf832ad165..43ad597ed491 100644
--- a/fs/ntfs/aops.c
+++ b/fs/ntfs/aops.c
@@ -97,7 +97,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)
return ntfs_read_compressed_block(folio);
}
- iomap_read_folio(&ntfs_read_iomap_ops, &ctx, NULL);
+ iomap_read_folio(ntfs_read_iomap_next, &ctx, NULL);
return 0;
}
@@ -238,7 +238,7 @@ static void ntfs_readahead(struct readahead_control *rac)
*/
if (!NInoNonResident(ni) || NInoCompressed(ni))
return;
- iomap_readahead(&ntfs_read_iomap_ops, &ctx, NULL);
+ iomap_readahead(ntfs_read_iomap_next, &ctx, NULL);
}
static int ntfs_writepages(struct address_space *mapping,
@@ -274,7 +274,7 @@ static int ntfs_swap_activate(struct swap_info_struct *sis,
struct file *swap_file, sector_t *span)
{
return iomap_swapfile_activate(sis, swap_file, span,
- &ntfs_read_iomap_ops);
+ ntfs_read_iomap_next);
}
const struct address_space_operations ntfs_aops = {
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
index 6a7b638e523d..a4f99128b46c 100644
--- a/fs/ntfs/file.c
+++ b/fs/ntfs/file.c
@@ -281,7 +281,7 @@ static int ntfs_setattr_size(struct inode *vi, struct iattr *attr)
round_up(old_size, PAGE_SIZE) - old_size,
attr->ia_size - old_size);
err = iomap_zero_range(vi, old_size, len,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
}
@@ -417,12 +417,12 @@ static loff_t ntfs_file_llseek(struct file *file, loff_t offset, int whence)
switch (whence) {
case SEEK_HOLE:
inode_lock_shared(inode);
- offset = iomap_seek_hole(inode, offset, &ntfs_seek_iomap_ops);
+ offset = iomap_seek_hole(inode, offset, ntfs_seek_iomap_next);
inode_unlock_shared(inode);
break;
case SEEK_DATA:
inode_lock_shared(inode);
- offset = iomap_seek_data(inode, offset, &ntfs_seek_iomap_ops);
+ offset = iomap_seek_data(inode, offset, ntfs_seek_iomap_next);
inode_unlock_shared(inode);
break;
default:
@@ -458,7 +458,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
}
file_accessed(iocb->ki_filp);
- ret = iomap_dio_rw(iocb, to, &ntfs_read_iomap_ops, NULL, 0,
+ ret = iomap_dio_rw(iocb, to, ntfs_read_iomap_next, NULL, 0,
NULL, 0);
} else {
ret = generic_file_read_iter(iocb, to);
@@ -496,7 +496,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
{
ssize_t ret;
- ret = iomap_dio_rw(iocb, from, &ntfs_dio_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, ntfs_dio_iomap_next,
&ntfs_write_dio_ops, 0, NULL, 0);
if (ret == -ENOTBLK)
ret = 0;
@@ -511,7 +511,7 @@ static ssize_t ntfs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
offset = iocb->ki_pos;
iocb->ki_flags &= ~IOCB_DIRECT;
written = iomap_file_buffered_write(iocb, from,
- &ntfs_write_iomap_ops, &ntfs_iomap_folio_ops,
+ ntfs_write_iomap_next, &ntfs_iomap_folio_ops,
NULL);
if (written < 0) {
ret = written;
@@ -594,7 +594,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (NInoNonResident(ni) && iocb->ki_flags & IOCB_DIRECT)
ret = ntfs_dio_write_iter(iocb, from);
else
- ret = iomap_file_buffered_write(iocb, from, &ntfs_write_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, ntfs_write_iomap_next,
&ntfs_iomap_folio_ops, NULL);
out:
if (ret < 0 && ret != -EIOCBQUEUED) {
@@ -623,7 +623,7 @@ static vm_fault_t ntfs_filemap_page_mkwrite(struct vm_fault *vmf)
sb_start_pagefault(inode->i_sb);
file_update_time(vmf->vma->vm_file);
- ret = iomap_page_mkwrite(vmf, &ntfs_page_mkwrite_iomap_ops, NULL);
+ ret = iomap_page_mkwrite(vmf, ntfs_page_mkwrite_iomap_next, NULL);
sb_end_pagefault(inode->i_sb);
return ret;
}
@@ -670,7 +670,7 @@ static int ntfs_file_mmap_prepare(struct vm_area_desc *desc)
static int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len)
{
- return iomap_fiemap(inode, fieinfo, start, len, &ntfs_read_iomap_ops);
+ return iomap_fiemap(inode, fieinfo, start, len, ntfs_read_iomap_next);
}
static const char *ntfs_get_link(struct dentry *dentry, struct inode *inode,
@@ -911,7 +911,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,
ntfs_cluster_to_bytes(vol, start_vcn + 1),
end_offset);
err = iomap_zero_range(vi, offset, to - offset,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err < 0)
goto out;
@@ -927,7 +927,7 @@ static int ntfs_punch_hole(struct ntfs_inode *ni, int mode, loff_t offset,
from = ntfs_cluster_to_bytes(vol, end_vcn - 1);
if (from < ni->initialized_size) {
err = iomap_zero_range(vi, from, end_offset - from,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err < 0)
goto out;
@@ -1131,7 +1131,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t offset, loff_t le
round_up(old_size, PAGE_SIZE) - old_size,
offset - old_size);
err = iomap_zero_range(vi, old_size, len, NULL,
- &ntfs_seek_iomap_ops,
+ ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
}
NInoSetFileNameDirty(ni);
diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c
index c2715521e562..05132d92e87b 100644
--- a/fs/ntfs/inode.c
+++ b/fs/ntfs/inode.c
@@ -2415,7 +2415,7 @@ int ntfs_extend_initialized_size(struct inode *vi, const loff_t offset,
if (!NInoCompressed(ni) && old_init_size < offset) {
err = iomap_zero_range(vi, old_init_size,
offset - old_init_size,
- NULL, &ntfs_seek_iomap_ops,
+ NULL, ntfs_seek_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err)
return err;
diff --git a/fs/ntfs/iomap.c b/fs/ntfs/iomap.c
index 0f9f02e1593e..502f08f01354 100644
--- a/fs/ntfs/iomap.c
+++ b/fs/ntfs/iomap.c
@@ -277,16 +277,12 @@ static int ntfs_read_iomap_begin(struct inode *inode, loff_t offset, loff_t leng
srcmap, true);
}
-static int ntfs_read_iomap_next(const struct iomap_iter *iter,
- struct iomap *iomap, struct iomap *srcmap)
+int ntfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ntfs_read_iomap_begin, NULL);
}
-const struct iomap_ops ntfs_read_iomap_ops = {
- .iomap_next = ntfs_read_iomap_next,
-};
-
/*
* Check that the cached iomap still matches the NTFS runlist before
* iomap_zero_range() is called. if the runlist changes while iomap is
@@ -342,20 +338,12 @@ static int ntfs_zero_read_iomap_next(const struct iomap_iter *iter,
ntfs_zero_read_iomap_end);
}
-static const struct iomap_ops ntfs_zero_read_iomap_ops = {
- .iomap_next = ntfs_zero_read_iomap_next,
-};
-
-static int ntfs_seek_iomap_next(const struct iomap_iter *iter,
+int ntfs_seek_iomap_next(const struct iomap_iter *iter,
struct iomap *iomap, struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ntfs_seek_iomap_begin, NULL);
}
-const struct iomap_ops ntfs_seek_iomap_ops = {
- .iomap_next = ntfs_seek_iomap_next,
-};
-
int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length)
{
if ((offset | length) & (SECTOR_SIZE - 1))
@@ -373,7 +361,7 @@ static int ntfs_zero_range(struct inode *inode, loff_t offset, loff_t length)
return iomap_zero_range(inode,
offset, length,
NULL,
- &ntfs_zero_read_iomap_ops,
+ ntfs_zero_read_iomap_next,
&ntfs_zero_iomap_folio_ops,
NULL);
}
@@ -782,17 +770,13 @@ static int ntfs_write_iomap_end(struct inode *inode, loff_t pos, loff_t length,
return written;
}
-static int ntfs_write_iomap_next(const struct iomap_iter *iter,
- struct iomap *iomap, struct iomap *srcmap)
+int ntfs_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ntfs_write_iomap_begin,
ntfs_write_iomap_end);
}
-const struct iomap_ops ntfs_write_iomap_ops = {
- .iomap_next = ntfs_write_iomap_next,
-};
-
static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
struct iomap *iomap, struct iomap *srcmap)
@@ -801,17 +785,13 @@ static int ntfs_page_mkwrite_iomap_begin(struct inode *inode, loff_t offset,
NTFS_IOMAP_FLAGS_MKWRITE);
}
-static int ntfs_page_mkwrite_iomap_next(const struct iomap_iter *iter,
+int ntfs_page_mkwrite_iomap_next(const struct iomap_iter *iter,
struct iomap *iomap, struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ntfs_page_mkwrite_iomap_begin,
ntfs_write_iomap_end);
}
-const struct iomap_ops ntfs_page_mkwrite_iomap_ops = {
- .iomap_next = ntfs_page_mkwrite_iomap_next,
-};
-
static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
struct iomap *iomap, struct iomap *srcmap)
@@ -820,17 +800,13 @@ static int ntfs_dio_iomap_begin(struct inode *inode, loff_t offset,
NTFS_IOMAP_FLAGS_DIO);
}
-static int ntfs_dio_iomap_next(const struct iomap_iter *iter,
- struct iomap *iomap, struct iomap *srcmap)
+int ntfs_dio_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ntfs_dio_iomap_begin,
ntfs_write_iomap_end);
}
-const struct iomap_ops ntfs_dio_iomap_ops = {
- .iomap_next = ntfs_dio_iomap_next,
-};
-
static ssize_t ntfs_writeback_range(struct iomap_writepage_ctx *wpc,
struct folio *folio, u64 offset, unsigned int len, u64 end_pos)
{
diff --git a/fs/ntfs/iomap.h b/fs/ntfs/iomap.h
index 3abc1d493e91..69443de1fefd 100644
--- a/fs/ntfs/iomap.h
+++ b/fs/ntfs/iomap.h
@@ -12,11 +12,16 @@
#include "volume.h"
#include "inode.h"
-extern const struct iomap_ops ntfs_write_iomap_ops;
-extern const struct iomap_ops ntfs_read_iomap_ops;
-extern const struct iomap_ops ntfs_seek_iomap_ops;
-extern const struct iomap_ops ntfs_page_mkwrite_iomap_ops;
-extern const struct iomap_ops ntfs_dio_iomap_ops;
+int ntfs_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ntfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ntfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int ntfs_page_mkwrite_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int ntfs_dio_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_writeback_ops ntfs_writeback_ops;
extern const struct iomap_write_ops ntfs_iomap_folio_ops;
extern int ntfs_dio_zero_range(struct inode *inode, loff_t offset, loff_t length);
diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c
index d601f088618c..55844b42920a 100644
--- a/fs/ntfs3/file.c
+++ b/fs/ntfs3/file.c
@@ -315,7 +315,7 @@ static int ntfs_extend_initialized_size(struct file *file,
}
err = iomap_zero_range(inode, valid, new_valid - valid, NULL,
- &ntfs_iomap_ops, &ntfs_iomap_folio_ops, NULL);
+ ntfs_iomap_next, &ntfs_iomap_folio_ops, NULL);
if (err) {
ni->i_valid = valid;
ntfs_inode_warn(inode,
@@ -554,7 +554,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)
/* Zero head of punch. */
if (tmp > from) {
err = iomap_zero_range(inode, from, tmp - from, NULL,
- &ntfs_iomap_ops,
+ ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err)
goto out;
@@ -572,7 +572,7 @@ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len)
/* Zero tail of punch. */
if (vbo < end_a && end_a < end) {
err = iomap_zero_range(inode, end_a, end - end_a, NULL,
- &ntfs_iomap_ops,
+ ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err)
goto out;
@@ -872,7 +872,7 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
goto out;
}
- err = iomap_dio_rw(iocb, iter, &ntfs_iomap_ops, NULL, dio_flags,
+ err = iomap_dio_rw(iocb, iter, ntfs_iomap_next, NULL, dio_flags,
NULL, 0);
if (err <= 0)
@@ -1286,7 +1286,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
!ntfs_should_use_dio(iocb, from)) {
iocb->ki_flags &= ~IOCB_DIRECT;
- ret = iomap_file_buffered_write(iocb, from, &ntfs_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
inode_unlock(inode);
@@ -1303,7 +1303,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
goto out;
}
- ret = iomap_dio_rw(iocb, from, &ntfs_iomap_ops, NULL,
+ ret = iomap_dio_rw(iocb, from, ntfs_iomap_next, NULL,
IOMAP_DIO_FORCE_WAIT, NULL, 0);
if (ret == -ENOTBLK) {
@@ -1316,7 +1316,7 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
vbo = iocb->ki_pos;
iocb->ki_flags &= ~IOCB_DIRECT;
- err = iomap_file_buffered_write(iocb, from, &ntfs_iomap_ops,
+ err = iomap_file_buffered_write(iocb, from, ntfs_iomap_next,
&ntfs_iomap_folio_ops, NULL);
if (err < 0) {
ret = err;
@@ -1465,7 +1465,7 @@ int ntfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
inode_lock_shared(inode);
- err = iomap_fiemap(inode, fieinfo, start, len, &ntfs_iomap_ops);
+ err = iomap_fiemap(inode, fieinfo, start, len, ntfs_iomap_next);
inode_unlock_shared(inode);
return err;
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index c5676c51a3a4..8a454ab6ee2a 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -576,7 +576,7 @@ static sector_t ntfs_bmap(struct address_space *mapping, sector_t block)
ni_allocate_da_blocks(ni);
}
- return iomap_bmap(mapping, block, &ntfs_iomap_ops);
+ return iomap_bmap(mapping, block, ntfs_iomap_next);
}
static void ntfs_iomap_read_end_io(struct bio *bio)
@@ -649,7 +649,7 @@ static int ntfs_read_folio(struct file *file, struct folio *folio)
return err;
}
- iomap_read_folio(&ntfs_iomap_ops, &ctx, NULL);
+ iomap_read_folio(ntfs_iomap_next, &ctx, NULL);
return 0;
}
@@ -673,7 +673,7 @@ static void ntfs_readahead(struct readahead_control *rac)
return;
}
- iomap_readahead(&ntfs_iomap_ops, &ctx, NULL);
+ iomap_readahead(ntfs_iomap_next, &ctx, NULL);
}
int ntfs_set_size(struct inode *inode, u64 new_size)
@@ -2101,17 +2101,13 @@ const struct address_space_operations ntfs_aops_cmpr = {
.invalidate_folio = iomap_invalidate_folio,
};
-static int ntfs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+int ntfs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
struct iomap *srcmap)
{
return iomap_process(iter, iomap, srcmap, ntfs_iomap_begin,
ntfs_iomap_end);
}
-const struct iomap_ops ntfs_iomap_ops = {
- .iomap_next = ntfs_iomap_next,
-};
-
const struct iomap_write_ops ntfs_iomap_folio_ops = {
.put_folio = ntfs_iomap_put_folio,
};
diff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h
index d98d7e474476..e00dae3ce700 100644
--- a/fs/ntfs3/ntfs_fs.h
+++ b/fs/ntfs3/ntfs_fs.h
@@ -785,7 +785,8 @@ int ntfs_create_inode(struct mnt_idmap *idmap, struct inode *dir,
int ntfs_link_inode(struct inode *inode, struct dentry *dentry);
int ntfs_unlink_inode(struct inode *dir, const struct dentry *dentry);
void ntfs_evict_inode(struct inode *inode);
-extern const struct iomap_ops ntfs_iomap_ops;
+int ntfs_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
extern const struct iomap_write_ops ntfs_iomap_folio_ops;
extern const struct inode_operations ntfs_link_inode_operations;
extern const struct address_space_operations ntfs_aops;
diff --git a/fs/remap_range.c b/fs/remap_range.c
index 26afbbbfb10c..3d0a355dc90e 100644
--- a/fs/remap_range.c
+++ b/fs/remap_range.c
@@ -277,7 +277,7 @@ int
__generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *dax_read_ops)
+ iomap_next_fn dax_read_next)
{
struct inode *inode_in = file_inode(file_in);
struct inode *inode_out = file_inode(file_out);
@@ -340,10 +340,10 @@ __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
if (!IS_DAX(inode_in))
ret = vfs_dedupe_file_range_compare(file_in, pos_in,
file_out, pos_out, *len, &is_same);
- else if (dax_read_ops)
+ else if (dax_read_next)
ret = dax_dedupe_file_range_compare(inode_in, pos_in,
inode_out, pos_out, *len, &is_same,
- dax_read_ops);
+ dax_read_next);
else
return -EINVAL;
if (ret)
diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 2a0c54256e93..91480cb6a4d8 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -752,7 +752,7 @@ xfs_vm_bmap(
*/
if (xfs_is_cow_inode(ip) || XFS_IS_REALTIME_INODE(ip))
return 0;
- return iomap_bmap(mapping, block, &xfs_read_iomap_ops);
+ return iomap_bmap(mapping, block, xfs_read_iomap_next);
}
static void
@@ -793,7 +793,7 @@ xfs_vm_read_folio(
struct iomap_read_folio_ctx ctx = { .cur_folio = folio };
ctx.ops = xfs_get_iomap_read_ops(folio->mapping);
- iomap_read_folio(&xfs_read_iomap_ops, &ctx, NULL);
+ iomap_read_folio(xfs_read_iomap_next, &ctx, NULL);
return 0;
}
@@ -804,7 +804,7 @@ xfs_vm_readahead(
struct iomap_read_folio_ctx ctx = { .rac = rac };
ctx.ops = xfs_get_iomap_read_ops(rac->mapping),
- iomap_readahead(&xfs_read_iomap_ops, &ctx, NULL);
+ iomap_readahead(xfs_read_iomap_next, &ctx, NULL);
}
static int
@@ -850,7 +850,7 @@ xfs_vm_swap_activate(
sis->bdev = xfs_inode_buftarg(ip)->bt_bdev;
return iomap_swapfile_activate(sis, swap_file, span,
- &xfs_read_iomap_ops);
+ xfs_read_iomap_next);
}
const struct address_space_operations xfs_address_space_operations = {
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 7f8bef1a9954..a987ffbf3c02 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -269,7 +269,7 @@ xfs_file_dio_read(
dio_ops = &xfs_dio_read_bounce_ops;
dio_flags |= IOMAP_DIO_BOUNCE;
}
- ret = iomap_dio_rw(iocb, to, &xfs_read_iomap_ops, dio_ops, dio_flags,
+ ret = iomap_dio_rw(iocb, to, xfs_read_iomap_next, dio_ops, dio_flags,
NULL, 0);
xfs_iunlock(ip, XFS_IOLOCK_SHARED);
@@ -292,7 +292,7 @@ xfs_file_dax_read(
ret = xfs_ilock_iocb(iocb, XFS_IOLOCK_SHARED);
if (ret)
return ret;
- ret = dax_iomap_rw(iocb, to, &xfs_read_iomap_ops);
+ ret = dax_iomap_rw(iocb, to, xfs_read_iomap_next);
xfs_iunlock(ip, XFS_IOLOCK_SHARED);
file_accessed(iocb->ki_filp);
@@ -742,7 +742,7 @@ xfs_file_dio_write_aligned(
struct xfs_inode *ip,
struct kiocb *iocb,
struct iov_iter *from,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_dio_ops *dops,
struct xfs_zone_alloc_ctx *ac)
{
@@ -777,7 +777,7 @@ xfs_file_dio_write_aligned(
if (mapping_stable_writes(iocb->ki_filp->f_mapping))
dio_flags |= IOMAP_DIO_BOUNCE;
trace_xfs_file_direct_write(iocb, from);
- ret = iomap_dio_rw(iocb, from, ops, dops, dio_flags, ac, 0);
+ ret = iomap_dio_rw(iocb, from, iomap_next, dops, dio_flags, ac, 0);
out_unlock:
xfs_iunlock(ip, iolock);
return ret;
@@ -799,7 +799,7 @@ xfs_file_dio_write_zoned(
if (ret < 0)
return ret;
ret = xfs_file_dio_write_aligned(ip, iocb, from,
- &xfs_zoned_direct_write_iomap_ops,
+ xfs_zoned_direct_write_iomap_next,
&xfs_dio_zoned_write_ops, &ac);
xfs_zoned_space_unreserve(ip->i_mount, &ac);
return ret;
@@ -824,16 +824,16 @@ xfs_file_dio_write_atomic(
unsigned int iolock = XFS_IOLOCK_SHARED;
ssize_t ret, ocount = iov_iter_count(from);
unsigned int dio_flags = 0;
- const struct iomap_ops *dops;
+ iomap_next_fn dops;
/*
* HW offload should be faster, so try that first if it is already
* known that the write length is not too large.
*/
if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
- dops = &xfs_atomic_write_cow_iomap_ops;
+ dops = xfs_atomic_write_cow_iomap_next;
else
- dops = &xfs_direct_write_iomap_ops;
+ dops = xfs_direct_write_iomap_next;
retry:
ret = xfs_ilock_iocb_for_write(iocb, &iolock);
@@ -862,9 +862,9 @@ xfs_file_dio_write_atomic(
* possible. The REQ_ATOMIC-based method is typically not possible if
* the write spans multiple extents or the disk blocks are misaligned.
*/
- if (ret == -ENOPROTOOPT && dops == &xfs_direct_write_iomap_ops) {
+ if (ret == -ENOPROTOOPT && dops == xfs_direct_write_iomap_next) {
xfs_iunlock(ip, iolock);
- dops = &xfs_atomic_write_cow_iomap_ops;
+ dops = xfs_atomic_write_cow_iomap_next;
goto retry;
}
@@ -947,7 +947,7 @@ xfs_file_dio_write_unaligned(
flags |= IOMAP_DIO_BOUNCE;
trace_xfs_file_direct_write(iocb, from);
- ret = iomap_dio_rw(iocb, from, &xfs_direct_write_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, xfs_direct_write_iomap_next,
&xfs_dio_write_ops, flags, NULL, 0);
/*
@@ -987,7 +987,7 @@ xfs_file_dio_write(
if (iocb->ki_flags & IOCB_ATOMIC)
return xfs_file_dio_write_atomic(ip, iocb, from);
return xfs_file_dio_write_aligned(ip, iocb, from,
- &xfs_direct_write_iomap_ops, &xfs_dio_write_ops, NULL);
+ xfs_direct_write_iomap_next, &xfs_dio_write_ops, NULL);
}
static noinline ssize_t
@@ -1011,7 +1011,7 @@ xfs_file_dax_write(
pos = iocb->ki_pos;
trace_xfs_file_dax_write(iocb, from);
- ret = dax_iomap_rw(iocb, from, &xfs_dax_write_iomap_ops);
+ ret = dax_iomap_rw(iocb, from, xfs_dax_write_iomap_next);
if (ret > 0 && iocb->ki_pos > i_size_read(inode)) {
i_size_write(inode, iocb->ki_pos);
error = xfs_setfilesize(ip, pos, ret);
@@ -1054,7 +1054,7 @@ xfs_file_buffered_write(
trace_xfs_file_buffered_write(iocb, from);
ret = iomap_file_buffered_write(iocb, from,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
NULL);
/*
@@ -1135,7 +1135,7 @@ xfs_file_buffered_write_zoned(
retry:
trace_xfs_file_buffered_write(iocb, from);
ret = iomap_file_buffered_write(iocb, from,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
&ac);
if (ret == -ENOSPC && !cleared_space) {
/*
@@ -1856,10 +1856,10 @@ xfs_file_llseek(
default:
return generic_file_llseek(file, offset, whence);
case SEEK_HOLE:
- offset = iomap_seek_hole(inode, offset, &xfs_seek_iomap_ops);
+ offset = iomap_seek_hole(inode, offset, xfs_seek_iomap_next);
break;
case SEEK_DATA:
- offset = iomap_seek_data(inode, offset, &xfs_seek_iomap_ops);
+ offset = iomap_seek_data(inode, offset, xfs_seek_iomap_next);
break;
}
@@ -1883,8 +1883,8 @@ xfs_dax_fault_locked(
}
ret = dax_iomap_fault(vmf, order, &pfn, NULL,
(write_fault && !vmf->cow_page) ?
- &xfs_dax_write_iomap_ops :
- &xfs_read_iomap_ops);
+ xfs_dax_write_iomap_next :
+ xfs_read_iomap_next);
if (ret & VM_FAULT_NEEDDSYNC)
ret = dax_finish_sync_fault(vmf, order, pfn);
return ret;
@@ -1948,7 +1948,7 @@ __xfs_write_fault(
if (IS_DAX(inode))
ret = xfs_dax_fault_locked(vmf, order, true);
else
- ret = iomap_page_mkwrite(vmf, &xfs_buffered_write_iomap_ops,
+ ret = iomap_page_mkwrite(vmf, xfs_buffered_write_iomap_next,
ac);
xfs_iunlock(ip, lock_mode);
diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c
index 4fa1a5c985db..71c4bb024f04 100644
--- a/fs/xfs/xfs_iomap.c
+++ b/fs/xfs/xfs_iomap.c
@@ -1037,7 +1037,7 @@ xfs_direct_write_iomap_begin(
return error;
}
-static int
+int
xfs_direct_write_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -1047,10 +1047,6 @@ xfs_direct_write_iomap_next(
NULL);
}
-const struct iomap_ops xfs_direct_write_iomap_ops = {
- .iomap_next = xfs_direct_write_iomap_next,
-};
-
#ifdef CONFIG_XFS_RT
/*
* This is really simple. The space has already been reserved before taking the
@@ -1099,7 +1095,7 @@ xfs_zoned_direct_write_iomap_begin(
return 0;
}
-static int
+int
xfs_zoned_direct_write_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -1109,9 +1105,6 @@ xfs_zoned_direct_write_iomap_next(
xfs_zoned_direct_write_iomap_begin, NULL);
}
-const struct iomap_ops xfs_zoned_direct_write_iomap_ops = {
- .iomap_next = xfs_zoned_direct_write_iomap_next,
-};
#endif /* CONFIG_XFS_RT */
#ifdef DEBUG
@@ -1294,7 +1287,7 @@ xfs_atomic_write_cow_iomap_begin(
return error;
}
-static int
+int
xfs_atomic_write_cow_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -1304,10 +1297,6 @@ xfs_atomic_write_cow_iomap_next(
xfs_atomic_write_cow_iomap_begin, NULL);
}
-const struct iomap_ops xfs_atomic_write_cow_iomap_ops = {
- .iomap_next = xfs_atomic_write_cow_iomap_next,
-};
-
static int
xfs_dax_write_iomap_end(
struct inode *inode,
@@ -1328,7 +1317,7 @@ xfs_dax_write_iomap_end(
return xfs_reflink_end_cow(ip, pos, written);
}
-static int
+int
xfs_dax_write_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -1338,10 +1327,6 @@ xfs_dax_write_iomap_next(
xfs_dax_write_iomap_end);
}
-const struct iomap_ops xfs_dax_write_iomap_ops = {
- .iomap_next = xfs_dax_write_iomap_next,
-};
-
/*
* Convert a hole to a delayed allocation.
*/
@@ -2207,7 +2192,7 @@ xfs_buffered_write_iomap_end(
return 0;
}
-static int
+int
xfs_buffered_write_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -2218,10 +2203,6 @@ xfs_buffered_write_iomap_next(
xfs_buffered_write_iomap_end);
}
-const struct iomap_ops xfs_buffered_write_iomap_ops = {
- .iomap_next = xfs_buffered_write_iomap_next,
-};
-
static int
xfs_read_iomap_begin(
struct inode *inode,
@@ -2263,7 +2244,7 @@ xfs_read_iomap_begin(
shared ? IOMAP_F_SHARED : 0, seq);
}
-static int
+int
xfs_read_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -2272,10 +2253,6 @@ xfs_read_iomap_next(
return iomap_process(iter, iomap, srcmap, xfs_read_iomap_begin, NULL);
}
-const struct iomap_ops xfs_read_iomap_ops = {
- .iomap_next = xfs_read_iomap_next,
-};
-
static int
xfs_seek_iomap_begin(
struct inode *inode,
@@ -2360,7 +2337,7 @@ xfs_seek_iomap_begin(
return error;
}
-static int
+int
xfs_seek_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -2369,10 +2346,6 @@ xfs_seek_iomap_next(
return iomap_process(iter, iomap, srcmap, xfs_seek_iomap_begin, NULL);
}
-const struct iomap_ops xfs_seek_iomap_ops = {
- .iomap_next = xfs_seek_iomap_next,
-};
-
static int
xfs_xattr_iomap_begin(
struct inode *inode,
@@ -2416,7 +2389,7 @@ xfs_xattr_iomap_begin(
return xfs_bmbt_to_iomap(ip, iomap, &imap, flags, IOMAP_F_XATTR, seq);
}
-static int
+int
xfs_xattr_iomap_next(
const struct iomap_iter *iter,
struct iomap *iomap,
@@ -2425,10 +2398,6 @@ xfs_xattr_iomap_next(
return iomap_process(iter, iomap, srcmap, xfs_xattr_iomap_begin, NULL);
}
-const struct iomap_ops xfs_xattr_iomap_ops = {
- .iomap_next = xfs_xattr_iomap_next,
-};
-
int
xfs_zero_range(
struct xfs_inode *ip,
@@ -2443,9 +2412,9 @@ xfs_zero_range(
if (IS_DAX(inode))
return dax_zero_range(inode, pos, len, did_zero,
- &xfs_dax_write_iomap_ops);
+ xfs_dax_write_iomap_next);
return iomap_zero_range(inode, pos, len, did_zero,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
ac);
}
@@ -2460,8 +2429,8 @@ xfs_truncate_page(
if (IS_DAX(inode))
return dax_truncate_page(inode, pos, did_zero,
- &xfs_dax_write_iomap_ops);
+ xfs_dax_write_iomap_next);
return iomap_truncate_page(inode, pos, did_zero,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
+ xfs_buffered_write_iomap_next, &xfs_iomap_write_ops,
ac);
}
diff --git a/fs/xfs/xfs_iomap.h b/fs/xfs/xfs_iomap.h
index ebcce7d49446..01875d20fb66 100644
--- a/fs/xfs/xfs_iomap.h
+++ b/fs/xfs/xfs_iomap.h
@@ -49,14 +49,22 @@ xfs_aligned_fsb_count(
return count_fsb;
}
-extern const struct iomap_ops xfs_buffered_write_iomap_ops;
-extern const struct iomap_ops xfs_direct_write_iomap_ops;
-extern const struct iomap_ops xfs_zoned_direct_write_iomap_ops;
-extern const struct iomap_ops xfs_read_iomap_ops;
-extern const struct iomap_ops xfs_seek_iomap_ops;
-extern const struct iomap_ops xfs_xattr_iomap_ops;
-extern const struct iomap_ops xfs_dax_write_iomap_ops;
-extern const struct iomap_ops xfs_atomic_write_cow_iomap_ops;
+int xfs_buffered_write_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int xfs_direct_write_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int xfs_zoned_direct_write_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
+int xfs_read_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_seek_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_xattr_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_dax_write_iomap_next(const struct iomap_iter *iter, struct iomap *iomap,
+ struct iomap *srcmap);
+int xfs_atomic_write_cow_iomap_next(const struct iomap_iter *iter,
+ struct iomap *iomap, struct iomap *srcmap);
extern const struct iomap_write_ops xfs_iomap_write_ops;
#endif /* __XFS_IOMAP_H__*/
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 6339f4956ecb..5c3d9a365f93 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -1239,10 +1239,10 @@ xfs_vn_fiemap(
if (fieinfo->fi_flags & FIEMAP_FLAG_XATTR) {
fieinfo->fi_flags &= ~FIEMAP_FLAG_XATTR;
error = iomap_fiemap(inode, fieinfo, start, length,
- &xfs_xattr_iomap_ops);
+ xfs_xattr_iomap_next);
} else {
error = iomap_fiemap(inode, fieinfo, start, length,
- &xfs_read_iomap_ops);
+ xfs_read_iomap_next);
}
xfs_iunlock(XFS_I(inode), XFS_IOLOCK_SHARED);
diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c
index a5c188b78138..2b9792626bab 100644
--- a/fs/xfs/xfs_reflink.c
+++ b/fs/xfs/xfs_reflink.c
@@ -1683,7 +1683,7 @@ xfs_reflink_remap_prep(
pos_out, len, remap_flags);
else
ret = dax_remap_file_range_prep(file_in, pos_in, file_out,
- pos_out, len, remap_flags, &xfs_read_iomap_ops);
+ pos_out, len, remap_flags, xfs_read_iomap_next);
if (ret || *len == 0)
goto out_unlock;
@@ -1878,10 +1878,10 @@ xfs_reflink_unshare(
if (IS_DAX(inode))
error = dax_file_unshare(inode, offset, len,
- &xfs_dax_write_iomap_ops);
+ xfs_dax_write_iomap_next);
else
error = iomap_file_unshare(inode, offset, len,
- &xfs_buffered_write_iomap_ops,
+ xfs_buffered_write_iomap_next,
&xfs_iomap_write_ops);
if (error)
goto out;
diff --git a/fs/zonefs/file.c b/fs/zonefs/file.c
index a29a8756d660..3ef1a655dbfe 100644
--- a/fs/zonefs/file.c
+++ b/fs/zonefs/file.c
@@ -64,10 +64,6 @@ static int zonefs_read_iomap_next(const struct iomap_iter *iter,
NULL);
}
-static const struct iomap_ops zonefs_read_iomap_ops = {
- .iomap_next = zonefs_read_iomap_next,
-};
-
static int zonefs_write_iomap_begin(struct inode *inode, loff_t offset,
loff_t length, unsigned int flags,
struct iomap *iomap, struct iomap *srcmap)
@@ -120,19 +116,15 @@ static int zonefs_write_iomap_next(const struct iomap_iter *iter,
NULL);
}
-static const struct iomap_ops zonefs_write_iomap_ops = {
- .iomap_next = zonefs_write_iomap_next,
-};
-
static int zonefs_read_folio(struct file *unused, struct folio *folio)
{
- iomap_bio_read_folio(folio, &zonefs_read_iomap_ops);
+ iomap_bio_read_folio(folio, zonefs_read_iomap_next);
return 0;
}
static void zonefs_readahead(struct readahead_control *rac)
{
- iomap_bio_readahead(rac, &zonefs_read_iomap_ops);
+ iomap_bio_readahead(rac, zonefs_read_iomap_next);
}
/*
@@ -193,7 +185,7 @@ static int zonefs_swap_activate(struct swap_info_struct *sis,
}
return iomap_swapfile_activate(sis, swap_file, span,
- &zonefs_read_iomap_ops);
+ zonefs_read_iomap_next);
}
const struct address_space_operations zonefs_file_aops = {
@@ -323,7 +315,7 @@ static vm_fault_t zonefs_filemap_page_mkwrite(struct vm_fault *vmf)
/* Serialize against truncates */
filemap_invalidate_lock_shared(inode->i_mapping);
- ret = iomap_page_mkwrite(vmf, &zonefs_write_iomap_ops, NULL);
+ ret = iomap_page_mkwrite(vmf, zonefs_write_iomap_next, NULL);
filemap_invalidate_unlock_shared(inode->i_mapping);
sb_end_pagefault(inode->i_sb);
@@ -539,7 +531,7 @@ static ssize_t zonefs_file_dio_write(struct kiocb *iocb, struct iov_iter *from)
* page invalidation. Overwrite that error code with EBUSY so that
* the user can make sense of the error.
*/
- ret = iomap_dio_rw(iocb, from, &zonefs_write_iomap_ops,
+ ret = iomap_dio_rw(iocb, from, zonefs_write_iomap_next,
&zonefs_write_dio_ops, 0, NULL, 0);
if (ret == -ENOTBLK)
ret = -EBUSY;
@@ -589,7 +581,7 @@ static ssize_t zonefs_file_buffered_write(struct kiocb *iocb,
if (ret <= 0)
goto inode_unlock;
- ret = iomap_file_buffered_write(iocb, from, &zonefs_write_iomap_ops,
+ ret = iomap_file_buffered_write(iocb, from, zonefs_write_iomap_next,
NULL, NULL);
if (ret == -EIO)
zonefs_io_error(inode, true);
@@ -684,7 +676,7 @@ static ssize_t zonefs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
goto inode_unlock;
}
file_accessed(iocb->ki_filp);
- ret = iomap_dio_rw(iocb, to, &zonefs_read_iomap_ops,
+ ret = iomap_dio_rw(iocb, to, zonefs_read_iomap_next,
&zonefs_read_dio_ops, 0, NULL, 0);
} else {
ret = generic_file_read_iter(iocb, to);
diff --git a/include/linux/dax.h b/include/linux/dax.h
index fe6c3ded1b50..a5a88f5186bf 100644
--- a/include/linux/dax.h
+++ b/include/linux/dax.h
@@ -3,6 +3,7 @@
#define _LINUX_DAX_H
#include <linux/fs.h>
+#include <linux/iomap.h>
#include <linux/mm.h>
#include <linux/radix-tree.h>
@@ -10,9 +11,6 @@ typedef unsigned long dax_entry_t;
struct dax_device;
struct gendisk;
-struct iomap_ops;
-struct iomap_iter;
-struct iomap;
enum dax_access_mode {
DAX_ACCESS,
@@ -213,11 +211,11 @@ static inline void dax_unlock_mapping_entry(struct address_space *mapping,
#endif
int dax_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
int dax_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
int dax_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
static inline bool dax_page_is_idle(struct page *page)
{
@@ -266,10 +264,10 @@ int dax_holder_notify_failure(struct dax_device *dax_dev, u64 off, u64 len,
void dax_flush(struct dax_device *dax_dev, void *addr, size_t size);
ssize_t dax_iomap_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
vm_fault_t dax_iomap_fault(struct vm_fault *vmf, unsigned int order,
unsigned long *pfnp, int *errp,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf,
unsigned int order, unsigned long pfn);
int dax_delete_mapping_entry(struct address_space *mapping, pgoff_t index);
@@ -288,11 +286,11 @@ void dax_break_layout_final(struct inode *inode);
int dax_dedupe_file_range_compare(struct inode *src, loff_t srcoff,
struct inode *dest, loff_t destoff,
loff_t len, bool *is_same,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
int dax_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
static inline bool dax_mapping(struct address_space *mapping)
{
return mapping->host && IS_DAX(mapping->host);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index d10897b3a1e3..2eb063438a3b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -70,7 +70,8 @@ struct fsnotify_mark_connector;
struct fs_context;
struct fs_parameter_spec;
struct file_kattr;
-struct iomap_ops;
+struct iomap_iter;
+struct iomap;
struct delegated_inode;
extern void __init inode_init(void);
@@ -2079,7 +2080,9 @@ int remap_verify_area(struct file *file, loff_t pos, loff_t len, bool write);
int __generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *len, unsigned int remap_flags,
- const struct iomap_ops *dax_read_ops);
+ int (*dax_read_next)(const struct iomap_iter *iter,
+ struct iomap *iomap,
+ struct iomap *srcmap));
int generic_remap_file_range_prep(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t *count, unsigned int remap_flags);
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 52d6f585b941..3b41f123a92d 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -237,12 +237,6 @@ typedef int (*iomap_end_fn)(struct inode *inode, loff_t pos, loff_t length,
typedef int (*iomap_next_fn)(const struct iomap_iter *iter, struct iomap *iomap,
struct iomap *srcmap);
-struct iomap_ops {
- iomap_begin_fn iomap_begin;
- iomap_end_fn iomap_end;
- iomap_next_fn iomap_next;
-};
-
/**
* struct iomap_iter - Iterate through a range of a file
* @inode: Set at the start of the iteration and should not change.
@@ -271,7 +265,7 @@ struct iomap_iter {
void *private;
};
-int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops);
+int iomap_iter(struct iomap_iter *iter, iomap_next_fn iomap_next);
int iomap_iter_advance(struct iomap_iter *iter, u64 count);
/**
@@ -365,14 +359,14 @@ static inline bool iomap_want_unshare_iter(const struct iomap_iter *iter)
}
ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *from,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops, void *private);
int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
- const void *buf, const struct iomap_ops *ops,
+ const void *buf, iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops);
-void iomap_read_folio(const struct iomap_ops *ops,
+void iomap_read_folio(iomap_next_fn iomap_next,
struct iomap_read_folio_ctx *ctx, void *private);
-void iomap_readahead(const struct iomap_ops *ops,
+void iomap_readahead(iomap_next_fn iomap_next,
struct iomap_read_folio_ctx *ctx, void *private);
bool iomap_is_partially_uptodate(struct folio *, size_t from, size_t count);
struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len);
@@ -380,17 +374,17 @@ bool iomap_release_folio(struct folio *folio, gfp_t gfp_flags);
void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len);
bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio);
int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops);
unsigned int iomap_fill_dirty_folios(struct iomap_iter *iter, loff_t *start,
loff_t end, unsigned int *iomap_flags);
int iomap_zero_range(struct inode *inode, loff_t pos, loff_t len,
- bool *did_zero, const struct iomap_ops *ops,
+ bool *did_zero, iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops, void *private);
int iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero,
- const struct iomap_ops *ops,
+ iomap_next_fn iomap_next,
const struct iomap_write_ops *write_ops, void *private);
-vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops,
+vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, iomap_next_fn iomap_next,
void *private);
typedef void (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length,
struct iomap *iomap);
@@ -399,13 +393,13 @@ void iomap_write_delalloc_release(struct inode *inode, loff_t start_byte,
iomap_punch_t punch);
int iomap_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
- u64 start, u64 len, const struct iomap_ops *ops);
+ u64 start, u64 len, iomap_next_fn iomap_next);
loff_t iomap_seek_hole(struct inode *inode, loff_t offset,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
loff_t iomap_seek_data(struct inode *inode, loff_t offset,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
sector_t iomap_bmap(struct address_space *mapping, sector_t bno,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
/*
* Flags for iomap_ioend->io_flags.
@@ -612,10 +606,10 @@ struct iomap_dio_ops {
#define IOMAP_DIO_BOUNCE (1 << 4)
ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+ iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before);
struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter,
- const struct iomap_ops *ops, const struct iomap_dio_ops *dops,
+ iomap_next_fn iomap_next, const struct iomap_dio_ops *dops,
unsigned int dio_flags, void *private, size_t done_before);
ssize_t iomap_dio_complete(struct iomap_dio *dio);
void iomap_dio_bio_end_io(struct bio *bio);
@@ -626,7 +620,7 @@ struct swap_info_struct;
int iomap_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file, sector_t *pagespan,
- const struct iomap_ops *ops);
+ iomap_next_fn iomap_next);
#else
# define iomap_swapfile_activate(sis, swapfile, pagespan, ops) (-EIO)
#endif /* CONFIG_SWAP */
@@ -640,25 +634,25 @@ int iomap_bio_read_folio_range(const struct iomap_iter *iter,
extern const struct iomap_read_ops iomap_bio_read_ops;
static inline void iomap_bio_read_folio(struct folio *folio,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct iomap_read_folio_ctx ctx = {
.ops = &iomap_bio_read_ops,
.cur_folio = folio,
};
- iomap_read_folio(ops, &ctx, NULL);
+ iomap_read_folio(iomap_next, &ctx, NULL);
}
static inline void iomap_bio_readahead(struct readahead_control *rac,
- const struct iomap_ops *ops)
+ iomap_next_fn iomap_next)
{
struct iomap_read_folio_ctx ctx = {
.ops = &iomap_bio_read_ops,
.rac = rac,
};
- iomap_readahead(ops, &ctx, NULL);
+ iomap_readahead(iomap_next, &ctx, NULL);
}
#endif /* CONFIG_BLOCK */
--
2.52.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox