* Re: [PATCH v2] lib/raid/xor: x86: Add AVX-512 optimized xor_gen()
From: David Laight @ 2026-06-15 22:57 UTC (permalink / raw)
To: Eric Biggers
Cc: Andrew Morton, linux-kernel, Christoph Hellwig, linux-crypto, x86,
linux-raid
In-Reply-To: <20260615184435.GA17731@quark>
On Mon, 15 Jun 2026 11:44:35 -0700
Eric Biggers <ebiggers@kernel.org> wrote:
> On Sun, Jun 14, 2026 at 11:16:28AM +0100, David Laight wrote:
> > On Sat, 13 Jun 2026 18:03:57 -0700
> > Eric Biggers <ebiggers@kernel.org> wrote:
...
> > Some 'not very important' comments:
> >
> > I did wonder whether moving the loop into the asm() would help.
> > gcc has a nasty habit of pessimising loops when you try to be clever.
> > It is certainly safer for tight loops like these.
>
> I originally tried leaving the loops to the compiler, but gcc unrolled
> the 1x ones by 2x, despite it having no visibility into the asm block.
> That broke the intent with the indexed addressing, since to achieve the
> unrolling it generated code that incremented the pointers.
I did suspect that might happen.
> So I just ended up moving the loop to the asm, which reliably gives us
> the code we want.
Yep...
...
> > The code should be limited by the memory reads, so the 3-argument xor and
> > the interleave of the unroll may make no difference.
>
> The unroll by 2x in the 2 and 3-buffer cases helped a little bit on
> Sapphire Rapids. I don't know exactly why, but it makes sense that
> those cases are where the loop overhead is most likely to matter.
Each iteration does 2 (or 3) reads and a write.
The cpu can do two reads and a write every clock.
However Intel cpu can only execute a branch every other clock,
so the shortest loop is two clocks.
That means you need need to unroll once to keep the memory logic busy.
The zen5 seems to be able to execute 1-clock loops, so wouldn't need
the unroll.
> > Some cpu do have constraints on the cache alignment in order to do two
> > reads per clock, but I've forgotten them and they got better before AVX-512.
> > If that were affecting this code (on the tested cpu) then I'd expect the
> > interleaved unroll would improve the _4 and -5 functions.
> > So it probably doesn't affect this code.
>
> The buffers are always 64-byte aligned here, as documented.
It is all more complex that that.
Whether you can do two reads/clock depends on whether the reads manage to
avoid needing the same buffers (etc) in the cache logic.
For instance it might not work if the addresses differ by the size of the
cache (one of Agner's books might have the answer).
(It was pretty hard to get two reads/clock on Sandy Bridge.)
Then there are some really strange effects.
On zen5 (at least on the one I've got) 'rep movsb' is very slow (setup and copy)
if (IIRC) (%di - %si) mod 4k is between 1 and 127.
The only other alignment that makes much difference is 64byte aligning %di (which
doubles throughput).
-- David
^ permalink raw reply
* Re: [PATCH v4 0/3] crypto: skcipher - per-request multi-data-unit batching
From: Eric Biggers @ 2026-06-15 22:53 UTC (permalink / raw)
To: Leonid Ravich
Cc: Herbert Xu, Alasdair Kergon, Ard Biesheuvel, Jens Axboe,
Horia Geanta, Gilad Ben-Yossef, linux-crypto, dm-devel,
linux-block
In-Reply-To: <20260615111459.9452-1-lravich@amazon.com>
On Mon, Jun 15, 2026 at 11:14:56AM +0000, Leonid Ravich wrote:
> The series adds a per-request "data unit size" to the skcipher API
> so a caller can submit several data units (typically 512..4096-byte
> sectors) sharing one starting IV in a single request. Algorithms
> derive each data unit's IV from the caller-supplied IV by treating
> it as a 128-bit little-endian counter and adding the data-unit
> index, matching the layout produced by dm-crypt's plain64 IV mode
> and by typical inline-encryption hardware.
>
> This mirrors the data_unit_size concept already exposed by
> struct blk_crypto_config for inline encryption.
>
> The first user is dm-crypt, which today issues one skcipher request
> per sector and so pays a per-sector cost in request allocation,
> callback dispatch, completion handling, and scatterlist setup.
>
> Proof-of-concept performance numbers from the RFC reply [1]: +19%
> throughput / -40% CPU on a single-core arm64 system with a hardware
> XTS-AES-256 accelerator running fio 4 KiB sequential writes through
> dm-crypt, when an out-of-tree arm64 xts driver advertises
> CRYPTO_ALG_SKCIPHER_NATIVE_MULTI_DU. This series itself does not
> include arch enablement; the fast path is opt-in per driver, the
> slow path is universal via the auto-splitter.
>
> The native fast path amortises both per-sector dispatch and per-sector
> crypto setup across a bio - the measured win above, on an engine that
> offloads the AES compute. The auto-splitter is for correctness and
> reach: any consumer can set data_unit_size and get correct output with
> the per-request allocation/callback/completion cost removed, but it
> still issues one alg->encrypt per data unit, so on a software cipher it
> saves only dispatch overhead (no throughput figure claimed - that is
> hardware- and workload-dependent). What it guarantees unconditionally
> is byte-identical output (Verification below) at O(entries + units),
> walking the scatterlists with a pair of struct scatter_walk cursors
> rather than rescanning from the head per unit.
So in other words, this series slows down dm-crypt and crypto_skcipher
for everyone to optimize for an out-of-tree driver. And there's also no
benchmark showing that your driver is even worth it over just using the
CPU.
- Eric
^ permalink raw reply
* [PATCH 7/7] crypto: caam - Remove crypto_rng interface
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers
In-Reply-To: <20260615224131.69370-1-ebiggers@kernel.org>
Since the crypto_rng interface for hardware PRNGs is unused and is
redundant with hwrng and the actual Linux RNG, it's being phased out.
Most drivers for it were already removed. Go ahead and remove the CAAM
support which is one of the only remaining ones.
Note that the CAAM support for hwrng remains in place. That is the
interface that actually matters.
Note that this code also had several issues, including dlen > 65535
causing corruption of the CAAM descriptor.
Cc: Gaurav Jain <gaurav.jain@nxp.com>
Cc: Horia Geantă <horia.geanta@nxp.com>
Cc: Pankaj Gupta <pankaj.gupta@nxp.com>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/crypto/caam/Kconfig | 9 --
drivers/crypto/caam/Makefile | 1 -
drivers/crypto/caam/caamprng.c | 241 ---------------------------------
drivers/crypto/caam/intern.h | 15 --
drivers/crypto/caam/jr.c | 2 -
5 files changed, 268 deletions(-)
delete mode 100644 drivers/crypto/caam/caamprng.c
diff --git a/drivers/crypto/caam/Kconfig b/drivers/crypto/caam/Kconfig
index 05210a0edb8a..ec57bf6aaf6c 100644
--- a/drivers/crypto/caam/Kconfig
+++ b/drivers/crypto/caam/Kconfig
@@ -143,24 +143,15 @@ config CRYPTO_DEV_FSL_CAAM_PKC_API
signature and verification.
config CRYPTO_DEV_FSL_CAAM_RNG_API
bool "Register caam device for hwrng API"
default y
- select CRYPTO_RNG
select HW_RANDOM
help
Selecting this will register the SEC4 hardware rng to
the hw_random API for supplying the kernel entropy pool.
-config CRYPTO_DEV_FSL_CAAM_PRNG_API
- bool "Register Pseudo random number generation implementation with Crypto API"
- default y
- select CRYPTO_RNG
- help
- Selecting this will register the SEC hardware prng to
- the Crypto API.
-
config CRYPTO_DEV_FSL_CAAM_BLOB_GEN
bool
config CRYPTO_DEV_FSL_CAAM_RNG_TEST
bool "Test caam rng"
diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index d2eaf5205b1c..a17e3cf14c61 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -18,11 +18,10 @@ caam-y := ctrl.o
caam_jr-y := jr.o key_gen.o
caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API) += caamalg.o
caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI) += caamalg_qi.o
caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API) += caamhash.o
caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API) += caamrng.o
-caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_PRNG_API) += caamprng.o
caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_PKC_API) += caampkc.o pkc_desc.o
caam_jr-$(CONFIG_CRYPTO_DEV_FSL_CAAM_BLOB_GEN) += blob_gen.o
caam-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI) += qi.o
caam-$(CONFIG_DEBUG_FS) += debugfs.o
diff --git a/drivers/crypto/caam/caamprng.c b/drivers/crypto/caam/caamprng.c
deleted file mode 100644
index 6e4c1191cb28..000000000000
--- a/drivers/crypto/caam/caamprng.c
+++ /dev/null
@@ -1,241 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0+
-/*
- * Driver to expose SEC4 PRNG via crypto RNG API
- *
- * Copyright 2022 NXP
- *
- */
-
-#include <linux/completion.h>
-#include <crypto/internal/rng.h>
-#include <linux/dma-mapping.h>
-#include <linux/kernel.h>
-#include "compat.h"
-#include "regs.h"
-#include "intern.h"
-#include "desc_constr.h"
-#include "jr.h"
-#include "error.h"
-
-/*
- * Length of used descriptors, see caam_init_desc()
- */
-#define CAAM_PRNG_MAX_DESC_LEN (CAAM_CMD_SZ + \
- CAAM_CMD_SZ + \
- CAAM_CMD_SZ + CAAM_PTR_SZ_MAX)
-
-/* prng per-device context */
-struct caam_prng_ctx {
- int err;
- struct completion done;
-};
-
-struct caam_prng_alg {
- struct rng_alg rng;
- bool registered;
-};
-
-static void caam_prng_done(struct device *jrdev, u32 *desc, u32 err,
- void *context)
-{
- struct caam_prng_ctx *jctx = context;
-
- jctx->err = err ? caam_jr_strstatus(jrdev, err) : 0;
-
- complete(&jctx->done);
-}
-
-static u32 *caam_init_reseed_desc(u32 *desc)
-{
- init_job_desc(desc, 0); /* + 1 cmd_sz */
- /* Generate random bytes: + 1 cmd_sz */
- append_operation(desc, OP_TYPE_CLASS1_ALG | OP_ALG_ALGSEL_RNG |
- OP_ALG_AS_FINALIZE);
-
- print_hex_dump_debug("prng reseed desc@: ", DUMP_PREFIX_ADDRESS,
- 16, 4, desc, desc_bytes(desc), 1);
-
- return desc;
-}
-
-static u32 *caam_init_prng_desc(u32 *desc, dma_addr_t dst_dma, u32 len)
-{
- init_job_desc(desc, 0); /* + 1 cmd_sz */
- /* Generate random bytes: + 1 cmd_sz */
- append_operation(desc, OP_ALG_ALGSEL_RNG | OP_TYPE_CLASS1_ALG);
- /* Store bytes: + 1 cmd_sz + caam_ptr_sz */
- append_fifo_store(desc, dst_dma,
- len, FIFOST_TYPE_RNGSTORE);
-
- print_hex_dump_debug("prng job desc@: ", DUMP_PREFIX_ADDRESS,
- 16, 4, desc, desc_bytes(desc), 1);
-
- return desc;
-}
-
-static int caam_prng_generate(struct crypto_rng *tfm,
- const u8 *src, unsigned int slen,
- u8 *dst, unsigned int dlen)
-{
- unsigned int aligned_dlen = ALIGN(dlen, dma_get_cache_alignment());
- struct caam_prng_ctx ctx;
- struct device *jrdev;
- dma_addr_t dst_dma;
- u32 *desc;
- u8 *buf;
- int ret;
-
- if (aligned_dlen < dlen)
- return -EOVERFLOW;
-
- buf = kzalloc(aligned_dlen, GFP_KERNEL);
- if (!buf)
- return -ENOMEM;
-
- jrdev = caam_jr_alloc();
- ret = PTR_ERR_OR_ZERO(jrdev);
- if (ret) {
- pr_err("Job Ring Device allocation failed\n");
- kfree(buf);
- return ret;
- }
-
- desc = kzalloc(CAAM_PRNG_MAX_DESC_LEN, GFP_KERNEL);
- if (!desc) {
- ret = -ENOMEM;
- goto out1;
- }
-
- dst_dma = dma_map_single(jrdev, buf, dlen, DMA_FROM_DEVICE);
- if (dma_mapping_error(jrdev, dst_dma)) {
- dev_err(jrdev, "Failed to map destination buffer memory\n");
- ret = -ENOMEM;
- goto out;
- }
-
- init_completion(&ctx.done);
- ret = caam_jr_enqueue(jrdev,
- caam_init_prng_desc(desc, dst_dma, dlen),
- caam_prng_done, &ctx);
-
- if (ret == -EINPROGRESS) {
- wait_for_completion(&ctx.done);
- ret = ctx.err;
- }
-
- dma_unmap_single(jrdev, dst_dma, dlen, DMA_FROM_DEVICE);
-
- if (!ret)
- memcpy(dst, buf, dlen);
-out:
- kfree(desc);
-out1:
- caam_jr_free(jrdev);
- kfree(buf);
- return ret;
-}
-
-static void caam_prng_exit(struct crypto_tfm *tfm) {}
-
-static int caam_prng_init(struct crypto_tfm *tfm)
-{
- return 0;
-}
-
-static int caam_prng_seed(struct crypto_rng *tfm,
- const u8 *seed, unsigned int slen)
-{
- struct caam_prng_ctx ctx;
- struct device *jrdev;
- u32 *desc;
- int ret;
-
- if (slen) {
- pr_err("Seed length should be zero\n");
- return -EINVAL;
- }
-
- jrdev = caam_jr_alloc();
- ret = PTR_ERR_OR_ZERO(jrdev);
- if (ret) {
- pr_err("Job Ring Device allocation failed\n");
- return ret;
- }
-
- desc = kzalloc(CAAM_PRNG_MAX_DESC_LEN, GFP_KERNEL);
- if (!desc) {
- caam_jr_free(jrdev);
- return -ENOMEM;
- }
-
- init_completion(&ctx.done);
- ret = caam_jr_enqueue(jrdev,
- caam_init_reseed_desc(desc),
- caam_prng_done, &ctx);
-
- if (ret == -EINPROGRESS) {
- wait_for_completion(&ctx.done);
- ret = ctx.err;
- }
-
- kfree(desc);
- caam_jr_free(jrdev);
- return ret;
-}
-
-static struct caam_prng_alg caam_prng_alg = {
- .rng = {
- .generate = caam_prng_generate,
- .seed = caam_prng_seed,
- .seedsize = 0,
- .base = {
- .cra_name = "stdrng",
- .cra_driver_name = "prng-caam",
- .cra_priority = 500,
- .cra_ctxsize = sizeof(struct caam_prng_ctx),
- .cra_module = THIS_MODULE,
- .cra_init = caam_prng_init,
- .cra_exit = caam_prng_exit,
- },
- }
-};
-
-void caam_prng_unregister(void *data)
-{
- if (caam_prng_alg.registered)
- crypto_unregister_rng(&caam_prng_alg.rng);
-}
-
-int caam_prng_register(struct device *ctrldev)
-{
- struct caam_drv_private *priv = dev_get_drvdata(ctrldev);
- u32 rng_inst;
- int ret = 0;
-
- /* Check for available RNG blocks before registration */
- if (priv->era < 10)
- rng_inst = (rd_reg32(&priv->jr[0]->perfmon.cha_num_ls) &
- CHA_ID_LS_RNG_MASK) >> CHA_ID_LS_RNG_SHIFT;
- else
- rng_inst = rd_reg32(&priv->jr[0]->vreg.rng) & CHA_VER_NUM_MASK;
-
- if (!rng_inst) {
- dev_dbg(ctrldev, "RNG block is not available... skipping registering algorithm\n");
- return ret;
- }
-
- ret = crypto_register_rng(&caam_prng_alg.rng);
- if (ret) {
- dev_err(ctrldev,
- "couldn't register rng crypto alg: %d\n",
- ret);
- return ret;
- }
-
- caam_prng_alg.registered = true;
-
- dev_info(ctrldev,
- "rng crypto API alg registered %s\n", caam_prng_alg.rng.base.cra_driver_name);
-
- return 0;
-}
diff --git a/drivers/crypto/caam/intern.h b/drivers/crypto/caam/intern.h
index a88da0d31b23..6e48bf7d6054 100644
--- a/drivers/crypto/caam/intern.h
+++ b/drivers/crypto/caam/intern.h
@@ -210,25 +210,10 @@ static inline int caam_rng_init(struct device *dev)
static inline void caam_rng_exit(struct device *dev) {}
#endif /* CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API */
-#ifdef CONFIG_CRYPTO_DEV_FSL_CAAM_PRNG_API
-
-int caam_prng_register(struct device *dev);
-void caam_prng_unregister(void *data);
-
-#else
-
-static inline int caam_prng_register(struct device *dev)
-{
- return 0;
-}
-
-static inline void caam_prng_unregister(void *data) {}
-#endif /* CONFIG_CRYPTO_DEV_FSL_CAAM_PRNG_API */
-
#ifdef CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI
int caam_qi_algapi_init(struct device *dev);
void caam_qi_algapi_exit(void);
diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 0ef00df9730e..bddeaaaca487 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -38,11 +38,10 @@ static void register_algs(struct caam_drv_private_jr *jrpriv,
caam_algapi_init(dev);
caam_algapi_hash_init(dev);
caam_pkc_init(dev);
jrpriv->hwrng = !caam_rng_init(dev);
- caam_prng_register(dev);
caam_qi_algapi_init(dev);
algs_unlock:
mutex_unlock(&algs_lock);
}
@@ -53,11 +52,10 @@ static void unregister_algs(void)
if (--active_devs != 0)
goto algs_unlock;
caam_qi_algapi_exit();
- caam_prng_unregister(NULL);
caam_pkc_exit();
caam_algapi_hash_exit();
caam_algapi_exit();
algs_unlock:
--
2.54.0
^ permalink raw reply related
* [PATCH 6/7] crypto: sun8i-ss - Remove crypto_rng interface
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers, stable
In-Reply-To: <20260615224131.69370-1-ebiggers@kernel.org>
Since the crypto_rng interface for hardware PRNGs is unused and is
redundant with hwrng and the actual Linux RNG, it's being phased out.
Most drivers for it were already removed. Go ahead and remove the
sun8i-ss support which is one of the only remaining ones.
As usual for crypto_rng, this driver was also buggy: its ->generate()
function had a use-after-free vulnerability due to using
wait_for_completion_interruptible_timeout() without handling shutting
down the DMA operation if a signal is sent. Also, it had a buffer
overread bug in the line 'memcpy(ctx->seed, d + dlen, ctx->slen);'.
There's no point in fixing these bugs separately only to remove the code
anyway, so this commit is marked with Fixes and Cc stable.
Fixes: ac2614d721de ("crypto: sun8i-ss - Add support for the PRNG")
Cc: stable@vger.kernel.org
Cc: Corentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/crypto/allwinner/Kconfig | 8 -
drivers/crypto/allwinner/sun8i-ss/Makefile | 1 -
.../crypto/allwinner/sun8i-ss/sun8i-ss-core.c | 45 -----
.../crypto/allwinner/sun8i-ss/sun8i-ss-prng.c | 177 ------------------
drivers/crypto/allwinner/sun8i-ss/sun8i-ss.h | 23 ---
5 files changed, 254 deletions(-)
delete mode 100644 drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c
diff --git a/drivers/crypto/allwinner/Kconfig b/drivers/crypto/allwinner/Kconfig
index 17bf9ead6ef2..d86ae005fbe2 100644
--- a/drivers/crypto/allwinner/Kconfig
+++ b/drivers/crypto/allwinner/Kconfig
@@ -103,18 +103,10 @@ config CRYPTO_DEV_SUN8I_SS_DEBUG
help
Say y to enable sun8i-ss debug stats.
This will create /sys/kernel/debug/sun8i-ss/stats for displaying
the number of requests per flow and per algorithm.
-config CRYPTO_DEV_SUN8I_SS_PRNG
- bool "Support for Allwinner Security System PRNG"
- depends on CRYPTO_DEV_SUN8I_SS
- select CRYPTO_RNG
- help
- Select this option if you want to provide kernel-side support for
- the Pseudo-Random Number Generator found in the Security System.
-
config CRYPTO_DEV_SUN8I_SS_HASH
bool "Enable support for hash on sun8i-ss"
depends on CRYPTO_DEV_SUN8I_SS
select CRYPTO_MD5
select CRYPTO_SHA1
diff --git a/drivers/crypto/allwinner/sun8i-ss/Makefile b/drivers/crypto/allwinner/sun8i-ss/Makefile
index aabfd893c817..2d6458a42e58 100644
--- a/drivers/crypto/allwinner/sun8i-ss/Makefile
+++ b/drivers/crypto/allwinner/sun8i-ss/Makefile
@@ -1,4 +1,3 @@
obj-$(CONFIG_CRYPTO_DEV_SUN8I_SS) += sun8i-ss.o
sun8i-ss-y += sun8i-ss-core.o sun8i-ss-cipher.o
-sun8i-ss-$(CONFIG_CRYPTO_DEV_SUN8I_SS_PRNG) += sun8i-ss-prng.o
sun8i-ss-$(CONFIG_CRYPTO_DEV_SUN8I_SS_HASH) += sun8i-ss-hash.o
diff --git a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c
index 59c9bc45ec0f..0b22fcddb882 100644
--- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c
+++ b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c
@@ -9,11 +9,10 @@
*
* You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst
*/
#include <crypto/engine.h>
-#include <crypto/internal/rng.h>
#include <crypto/internal/skcipher.h>
#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/dma-mapping.h>
#include <linux/err.h>
@@ -281,29 +280,10 @@ static struct sun8i_ss_alg_template ss_algs[] = {
},
.alg.skcipher.op = {
.do_one_request = sun8i_ss_handle_cipher_request,
},
},
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_PRNG
-{
- .type = CRYPTO_ALG_TYPE_RNG,
- .alg.rng = {
- .base = {
- .cra_name = "stdrng",
- .cra_driver_name = "sun8i-ss-prng",
- .cra_priority = 300,
- .cra_ctxsize = sizeof(struct sun8i_ss_rng_tfm_ctx),
- .cra_module = THIS_MODULE,
- .cra_init = sun8i_ss_prng_init,
- .cra_exit = sun8i_ss_prng_exit,
- },
- .generate = sun8i_ss_prng_generate,
- .seed = sun8i_ss_prng_seed,
- .seedsize = PRNG_SEED_SIZE,
- }
-},
-#endif
#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_HASH
{ .type = CRYPTO_ALG_TYPE_AHASH,
.ss_algo_id = SS_ID_HASH_MD5,
.alg.hash.base = {
.init = sun8i_ss_hash_init,
@@ -499,18 +479,10 @@ static int sun8i_ss_debugfs_show(struct seq_file *seq, void *v)
seq_printf(seq, "\tFallback due to alignment: %lu\n",
ss_algs[i].stat_fb_align);
seq_printf(seq, "\tFallback due to SG numbers: %lu\n",
ss_algs[i].stat_fb_sgnum);
break;
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_PRNG
- case CRYPTO_ALG_TYPE_RNG:
- seq_printf(seq, "%s %s reqs=%lu tsize=%lu\n",
- ss_algs[i].alg.rng.base.cra_driver_name,
- ss_algs[i].alg.rng.base.cra_name,
- ss_algs[i].stat_req, ss_algs[i].stat_bytes);
- break;
-#endif
#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_HASH
case CRYPTO_ALG_TYPE_AHASH:
seq_printf(seq, "%s %s reqs=%lu fallback=%lu\n",
ss_algs[i].alg.hash.base.halg.base.cra_driver_name,
ss_algs[i].alg.hash.base.halg.base.cra_name,
@@ -709,20 +681,10 @@ static int sun8i_ss_register_algs(struct sun8i_ss_dev *ss)
ss_algs[i].alg.skcipher.base.base.cra_name);
ss_algs[i].ss = NULL;
return err;
}
break;
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_PRNG
- case CRYPTO_ALG_TYPE_RNG:
- err = crypto_register_rng(&ss_algs[i].alg.rng);
- if (err) {
- dev_err(ss->dev, "Fail to register %s\n",
- ss_algs[i].alg.rng.base.cra_name);
- ss_algs[i].ss = NULL;
- }
- break;
-#endif
#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_HASH
case CRYPTO_ALG_TYPE_AHASH:
id = ss_algs[i].ss_algo_id;
ss_method = ss->variant->alg_hash[id];
if (ss_method == SS_ID_NOTSUPP) {
@@ -762,17 +724,10 @@ static void sun8i_ss_unregister_algs(struct sun8i_ss_dev *ss)
case CRYPTO_ALG_TYPE_SKCIPHER:
dev_info(ss->dev, "Unregister %d %s\n", i,
ss_algs[i].alg.skcipher.base.base.cra_name);
crypto_engine_unregister_skcipher(&ss_algs[i].alg.skcipher);
break;
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_PRNG
- case CRYPTO_ALG_TYPE_RNG:
- dev_info(ss->dev, "Unregister %d %s\n", i,
- ss_algs[i].alg.rng.base.cra_name);
- crypto_unregister_rng(&ss_algs[i].alg.rng);
- break;
-#endif
#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_HASH
case CRYPTO_ALG_TYPE_AHASH:
dev_info(ss->dev, "Unregister %d %s\n", i,
ss_algs[i].alg.hash.base.halg.base.cra_name);
crypto_engine_unregister_ahash(&ss_algs[i].alg.hash);
diff --git a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c
deleted file mode 100644
index a923cfc6553f..000000000000
--- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c
+++ /dev/null
@@ -1,177 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * sun8i-ss-prng.c - hardware cryptographic offloader for
- * Allwinner A80/A83T SoC
- *
- * Copyright (C) 2015-2020 Corentin Labbe <clabbe@baylibre.com>
- *
- * This file handle the PRNG found in the SS
- *
- * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst
- */
-#include "sun8i-ss.h"
-#include <linux/dma-mapping.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/pm_runtime.h>
-#include <crypto/internal/rng.h>
-
-int sun8i_ss_prng_seed(struct crypto_rng *tfm, const u8 *seed,
- unsigned int slen)
-{
- struct sun8i_ss_rng_tfm_ctx *ctx = crypto_rng_ctx(tfm);
-
- if (ctx->seed && ctx->slen != slen) {
- kfree_sensitive(ctx->seed);
- ctx->slen = 0;
- ctx->seed = NULL;
- }
- if (!ctx->seed)
- ctx->seed = kmalloc(slen, GFP_KERNEL);
- if (!ctx->seed)
- return -ENOMEM;
-
- memcpy(ctx->seed, seed, slen);
- ctx->slen = slen;
-
- return 0;
-}
-
-int sun8i_ss_prng_init(struct crypto_tfm *tfm)
-{
- struct sun8i_ss_rng_tfm_ctx *ctx = crypto_tfm_ctx(tfm);
-
- memset(ctx, 0, sizeof(struct sun8i_ss_rng_tfm_ctx));
- return 0;
-}
-
-void sun8i_ss_prng_exit(struct crypto_tfm *tfm)
-{
- struct sun8i_ss_rng_tfm_ctx *ctx = crypto_tfm_ctx(tfm);
-
- kfree_sensitive(ctx->seed);
- ctx->seed = NULL;
- ctx->slen = 0;
-}
-
-int sun8i_ss_prng_generate(struct crypto_rng *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int dlen)
-{
- struct sun8i_ss_rng_tfm_ctx *ctx = crypto_rng_ctx(tfm);
- struct rng_alg *alg = crypto_rng_alg(tfm);
- struct sun8i_ss_alg_template *algt;
- unsigned int todo_with_padding;
- struct sun8i_ss_dev *ss;
- dma_addr_t dma_iv, dma_dst;
- unsigned int todo;
- int err = 0;
- int flow;
- void *d;
- u32 v;
-
- algt = container_of(alg, struct sun8i_ss_alg_template, alg.rng);
- ss = algt->ss;
-
- if (ctx->slen == 0) {
- dev_err(ss->dev, "The PRNG is not seeded\n");
- return -EINVAL;
- }
-
- /* The SS does not give an updated seed, so we need to get a new one.
- * So we will ask for an extra PRNG_SEED_SIZE data.
- * We want dlen + seedsize rounded up to a multiple of PRNG_DATA_SIZE
- */
- todo = dlen + PRNG_SEED_SIZE + PRNG_DATA_SIZE;
- todo -= todo % PRNG_DATA_SIZE;
-
- todo_with_padding = ALIGN(todo, dma_get_cache_alignment());
- if (todo_with_padding < todo || todo < dlen)
- return -EOVERFLOW;
-
- d = kzalloc(todo_with_padding, GFP_KERNEL);
- if (!d)
- return -ENOMEM;
-
- flow = sun8i_ss_get_engine_number(ss);
-
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_SS_DEBUG
- algt->stat_req++;
- algt->stat_bytes += todo;
-#endif
-
- v = SS_ALG_PRNG | SS_PRNG_CONTINUE | SS_START;
- if (flow)
- v |= SS_FLOW1;
- else
- v |= SS_FLOW0;
-
- dma_iv = dma_map_single(ss->dev, ctx->seed, ctx->slen, DMA_TO_DEVICE);
- if (dma_mapping_error(ss->dev, dma_iv)) {
- dev_err(ss->dev, "Cannot DMA MAP IV\n");
- err = -EFAULT;
- goto err_free;
- }
-
- dma_dst = dma_map_single(ss->dev, d, todo, DMA_FROM_DEVICE);
- if (dma_mapping_error(ss->dev, dma_dst)) {
- dev_err(ss->dev, "Cannot DMA MAP DST\n");
- err = -EFAULT;
- goto err_iv;
- }
-
- err = pm_runtime_resume_and_get(ss->dev);
- if (err < 0)
- goto err_pm;
- err = 0;
-
- mutex_lock(&ss->mlock);
- writel(dma_iv, ss->base + SS_IV_ADR_REG);
- /* the PRNG act badly (failing rngtest) without SS_KEY_ADR_REG set */
- writel(dma_iv, ss->base + SS_KEY_ADR_REG);
- writel(dma_dst, ss->base + SS_DST_ADR_REG);
- writel(todo / 4, ss->base + SS_LEN_ADR_REG);
-
- reinit_completion(&ss->flows[flow].complete);
- ss->flows[flow].status = 0;
- /* Be sure all data is written before enabling the task */
- wmb();
-
- writel(v, ss->base + SS_CTL_REG);
-
- wait_for_completion_interruptible_timeout(&ss->flows[flow].complete,
- msecs_to_jiffies(todo));
- if (ss->flows[flow].status == 0) {
- dev_err(ss->dev, "DMA timeout for PRNG (size=%u)\n", todo);
- err = -EFAULT;
- }
- /* Since cipher and hash use the linux/cryptoengine and that we have
- * a cryptoengine per flow, we are sure that they will issue only one
- * request per flow.
- * Since the cryptoengine wait for completion before submitting a new
- * one, the mlock could be left just after the final writel.
- * But cryptoengine cannot handle crypto_rng, so we need to be sure
- * nothing will use our flow.
- * The easiest way is to grab mlock until the hardware end our requests.
- * We could have used a per flow lock, but this would increase
- * complexity.
- * The drawback is that no request could be handled for the other flow.
- */
- mutex_unlock(&ss->mlock);
-
- pm_runtime_put(ss->dev);
-
-err_pm:
- dma_unmap_single(ss->dev, dma_dst, todo, DMA_FROM_DEVICE);
-err_iv:
- dma_unmap_single(ss->dev, dma_iv, ctx->slen, DMA_TO_DEVICE);
-
- if (!err) {
- memcpy(dst, d, dlen);
- /* Update seed */
- memcpy(ctx->seed, d + dlen, ctx->slen);
- }
-err_free:
- kfree_sensitive(d);
-
- return err;
-}
diff --git a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss.h b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss.h
index 3fc86225edaf..289fb22abfa2 100644
--- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss.h
+++ b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss.h
@@ -6,11 +6,10 @@
* Copyright (C) 2016-2019 Corentin LABBE <clabbe.montjoie@gmail.com>
*/
#include <crypto/aes.h>
#include <crypto/des.h>
#include <crypto/engine.h>
-#include <crypto/rng.h>
#include <crypto/skcipher.h>
#include <linux/atomic.h>
#include <linux/debugfs.h>
#include <linux/crypto.h>
#include <crypto/internal/hash.h>
@@ -25,11 +24,10 @@
#define SS_ALG_AES 0
#define SS_ALG_DES (1 << 2)
#define SS_ALG_3DES (2 << 2)
#define SS_ALG_MD5 (3 << 2)
-#define SS_ALG_PRNG (4 << 2)
#define SS_ALG_SHA1 (6 << 2)
#define SS_ALG_SHA224 (7 << 2)
#define SS_ALG_SHA256 (8 << 2)
#define SS_CTL_REG 0x00
@@ -66,24 +64,19 @@
#define SS_ID_HASH_MAX 4
#define SS_FLOW0 BIT(30)
#define SS_FLOW1 BIT(31)
-#define SS_PRNG_CONTINUE BIT(18)
-
#define MAX_SG 8
#define MAXFLOW 2
#define SS_MAX_CLOCKS 2
#define SS_DIE_ID_SHIFT 20
#define SS_DIE_ID_MASK 0x07
-#define PRNG_DATA_SIZE (160 / 8)
-#define PRNG_SEED_SIZE DIV_ROUND_UP(175, 8)
-
#define MAX_PAD_SIZE 4096
/*
* struct ss_clock - Describe clocks used by sun8i-ss
* @name: Name of clock needed by this variant
@@ -211,20 +204,10 @@ struct sun8i_cipher_tfm_ctx {
u32 keylen;
struct sun8i_ss_dev *ss;
struct crypto_skcipher *fallback_tfm;
};
-/*
- * struct sun8i_ss_prng_ctx - context for PRNG TFM
- * @seed: The seed to use
- * @slen: The size of the seed
- */
-struct sun8i_ss_rng_tfm_ctx {
- void *seed;
- unsigned int slen;
-};
-
/*
* struct sun8i_ss_hash_tfm_ctx - context for an ahash TFM
* @fallback_tfm: pointer to the fallback TFM
* @ss: pointer to the private data of driver handling this TFM
*/
@@ -272,11 +255,10 @@ struct sun8i_ss_alg_template {
u32 ss_algo_id;
u32 ss_blockmode;
struct sun8i_ss_dev *ss;
union {
struct skcipher_engine_alg skcipher;
- struct rng_alg rng;
struct ahash_engine_alg hash;
} alg;
unsigned long stat_req;
unsigned long stat_fb;
unsigned long stat_bytes;
@@ -298,15 +280,10 @@ int sun8i_ss_skdecrypt(struct skcipher_request *areq);
int sun8i_ss_skencrypt(struct skcipher_request *areq);
int sun8i_ss_get_engine_number(struct sun8i_ss_dev *ss);
int sun8i_ss_run_task(struct sun8i_ss_dev *ss, struct sun8i_cipher_req_ctx *rctx, const char *name);
-int sun8i_ss_prng_generate(struct crypto_rng *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int dlen);
-int sun8i_ss_prng_seed(struct crypto_rng *tfm, const u8 *seed, unsigned int slen);
-int sun8i_ss_prng_init(struct crypto_tfm *tfm);
-void sun8i_ss_prng_exit(struct crypto_tfm *tfm);
int sun8i_ss_hash_init_tfm(struct crypto_ahash *tfm);
void sun8i_ss_hash_exit_tfm(struct crypto_ahash *tfm);
int sun8i_ss_hash_init(struct ahash_request *areq);
int sun8i_ss_hash_export(struct ahash_request *areq, void *out);
--
2.54.0
^ permalink raw reply related
* [PATCH 5/7] crypto: sun8i-ce - Remove crypto_rng interface
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers, stable
In-Reply-To: <20260615224131.69370-1-ebiggers@kernel.org>
Since the crypto_rng interface for hardware PRNGs is unused and is
redundant with hwrng and the actual Linux RNG, it's being phased out.
Most drivers for it were already removed. Go ahead and remove the
sun8i-ce support which is one of the only remaining ones.
Note that the sun8i-ce support for hwrng remains in place. That is the
interface that actually matters.
As usual for crypto_rng, this driver was also buggy: its ->generate()
function had a use-after-free vulnerability due to using
wait_for_completion_interruptible_timeout() without handling shutting
down the DMA operation if a signal is sent. There's no point in fixing
this separately only to remove the code anyway, so this commit is marked
with Fixes and Cc stable.
Fixes: 5eb7e9468884 ("crypto: sun8i-ce - Add support for the PRNG")
Cc: stable@vger.kernel.org
Cc: Corentin Labbe <clabbe.montjoie@gmail.com>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/crypto/allwinner/Kconfig | 8 -
drivers/crypto/allwinner/sun8i-ce/Makefile | 1 -
.../crypto/allwinner/sun8i-ce/sun8i-ce-core.c | 63 -------
.../crypto/allwinner/sun8i-ce/sun8i-ce-prng.c | 159 ------------------
drivers/crypto/allwinner/sun8i-ce/sun8i-ce.h | 29 ----
5 files changed, 260 deletions(-)
delete mode 100644 drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c
diff --git a/drivers/crypto/allwinner/Kconfig b/drivers/crypto/allwinner/Kconfig
index 06ea0e9fe6f2..17bf9ead6ef2 100644
--- a/drivers/crypto/allwinner/Kconfig
+++ b/drivers/crypto/allwinner/Kconfig
@@ -68,18 +68,10 @@ config CRYPTO_DEV_SUN8I_CE_HASH
select CRYPTO_SHA256
select CRYPTO_SHA512
help
Say y to enable support for hash algorithms.
-config CRYPTO_DEV_SUN8I_CE_PRNG
- bool "Support for Allwinner Crypto Engine PRNG"
- depends on CRYPTO_DEV_SUN8I_CE
- select CRYPTO_RNG
- help
- Select this option if you want to provide kernel-side support for
- the Pseudo-Random Number Generator found in the Crypto Engine.
-
config CRYPTO_DEV_SUN8I_CE_TRNG
bool "Support for Allwinner Crypto Engine TRNG"
depends on CRYPTO_DEV_SUN8I_CE
select HW_RANDOM
help
diff --git a/drivers/crypto/allwinner/sun8i-ce/Makefile b/drivers/crypto/allwinner/sun8i-ce/Makefile
index 0842eb2d9408..ea708b427e2e 100644
--- a/drivers/crypto/allwinner/sun8i-ce/Makefile
+++ b/drivers/crypto/allwinner/sun8i-ce/Makefile
@@ -1,5 +1,4 @@
obj-$(CONFIG_CRYPTO_DEV_SUN8I_CE) += sun8i-ce.o
sun8i-ce-y += sun8i-ce-core.o sun8i-ce-cipher.o
sun8i-ce-$(CONFIG_CRYPTO_DEV_SUN8I_CE_HASH) += sun8i-ce-hash.o
-sun8i-ce-$(CONFIG_CRYPTO_DEV_SUN8I_CE_PRNG) += sun8i-ce-prng.o
sun8i-ce-$(CONFIG_CRYPTO_DEV_SUN8I_CE_TRNG) += sun8i-ce-trng.o
diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c
index f3b58ed6aed0..c6402e87f8a0 100644
--- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c
+++ b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c
@@ -10,11 +10,10 @@
* You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst
*/
#include <crypto/engine.h>
#include <crypto/internal/hash.h>
-#include <crypto/internal/rng.h>
#include <crypto/internal/skcipher.h>
#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/dma-mapping.h>
#include <linux/err.h>
@@ -47,11 +46,10 @@ static const struct ce_variant ce_h3_variant = {
.ce_clks = {
{ "bus", 0, 200000000 },
{ "mod", 50000000, 0 },
},
.esr = ESR_H3,
- .prng = CE_ALG_PRNG,
.trng = CE_ID_NOTSUPP,
};
static const struct ce_variant ce_h5_variant = {
.alg_cipher = { CE_ALG_AES, CE_ALG_DES, CE_ALG_3DES,
@@ -64,11 +62,10 @@ static const struct ce_variant ce_h5_variant = {
.ce_clks = {
{ "bus", 0, 200000000 },
{ "mod", 300000000, 0 },
},
.esr = ESR_H5,
- .prng = CE_ALG_PRNG,
.trng = CE_ID_NOTSUPP,
};
static const struct ce_variant ce_h6_variant = {
.alg_cipher = { CE_ALG_AES, CE_ALG_DES, CE_ALG_3DES,
@@ -78,19 +75,17 @@ static const struct ce_variant ce_h6_variant = {
},
.op_mode = { CE_OP_ECB, CE_OP_CBC
},
.cipher_t_dlen_in_bytes = true,
.hash_t_dlen_in_bits = true,
- .prng_t_dlen_in_bytes = true,
.trng_t_dlen_in_bytes = true,
.ce_clks = {
{ "bus", 0, 200000000 },
{ "mod", 300000000, 0 },
{ "ram", 0, 400000000 },
},
.esr = ESR_H6,
- .prng = CE_ALG_PRNG_V2,
.trng = CE_ALG_TRNG_V2,
};
static const struct ce_variant ce_h616_variant = {
.alg_cipher = { CE_ALG_AES, CE_ALG_DES, CE_ALG_3DES,
@@ -100,21 +95,19 @@ static const struct ce_variant ce_h616_variant = {
},
.op_mode = { CE_OP_ECB, CE_OP_CBC
},
.cipher_t_dlen_in_bytes = true,
.hash_t_dlen_in_bits = true,
- .prng_t_dlen_in_bytes = true,
.trng_t_dlen_in_bytes = true,
.needs_word_addresses = true,
.ce_clks = {
{ "bus", 0, 200000000 },
{ "mod", 300000000, 0 },
{ "ram", 0, 400000000 },
{ "trng", 0, 0 },
},
.esr = ESR_H6,
- .prng = CE_ALG_PRNG_V2,
.trng = CE_ALG_TRNG_V2,
};
static const struct ce_variant ce_a64_variant = {
.alg_cipher = { CE_ALG_AES, CE_ALG_DES, CE_ALG_3DES,
@@ -127,11 +120,10 @@ static const struct ce_variant ce_a64_variant = {
.ce_clks = {
{ "bus", 0, 200000000 },
{ "mod", 300000000, 0 },
},
.esr = ESR_A64,
- .prng = CE_ALG_PRNG,
.trng = CE_ID_NOTSUPP,
};
static const struct ce_variant ce_d1_variant = {
.alg_cipher = { CE_ALG_AES, CE_ALG_DES, CE_ALG_3DES,
@@ -146,11 +138,10 @@ static const struct ce_variant ce_d1_variant = {
{ "mod", 300000000, 0 },
{ "ram", 0, 400000000 },
{ "trng", 0, 0 },
},
.esr = ESR_D1,
- .prng = CE_ALG_PRNG,
.trng = CE_ALG_TRNG,
};
static const struct ce_variant ce_r40_variant = {
.alg_cipher = { CE_ALG_AES, CE_ALG_DES, CE_ALG_3DES,
@@ -163,11 +154,10 @@ static const struct ce_variant ce_r40_variant = {
.ce_clks = {
{ "bus", 0, 200000000 },
{ "mod", 300000000, 0 },
},
.esr = ESR_R40,
- .prng = CE_ALG_PRNG,
.trng = CE_ID_NOTSUPP,
};
static void sun8i_ce_dump_task_descriptors(struct sun8i_ce_flow *chan)
{
@@ -612,29 +602,10 @@ static struct sun8i_ce_alg_template ce_algs[] = {
.alg.hash.op = {
.do_one_request = sun8i_ce_hash_run,
},
},
#endif
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_CE_PRNG
-{
- .type = CRYPTO_ALG_TYPE_RNG,
- .alg.rng = {
- .base = {
- .cra_name = "stdrng",
- .cra_driver_name = "sun8i-ce-prng",
- .cra_priority = 300,
- .cra_ctxsize = sizeof(struct sun8i_ce_rng_tfm_ctx),
- .cra_module = THIS_MODULE,
- .cra_init = sun8i_ce_prng_init,
- .cra_exit = sun8i_ce_prng_exit,
- },
- .generate = sun8i_ce_prng_generate,
- .seed = sun8i_ce_prng_seed,
- .seedsize = PRNG_SEED_SIZE,
- }
-},
-#endif
};
static int sun8i_ce_debugfs_show(struct seq_file *seq, void *v)
{
struct sun8i_ce_dev *ce __maybe_unused = seq->private;
@@ -691,18 +662,10 @@ static int sun8i_ce_debugfs_show(struct seq_file *seq, void *v)
seq_printf(seq, "\tFallback due to alignment: %lu\n",
ce_algs[i].stat_fb_srcali);
seq_printf(seq, "\tFallback due to SG numbers: %lu\n",
ce_algs[i].stat_fb_maxsg);
break;
-#endif
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_CE_PRNG
- case CRYPTO_ALG_TYPE_RNG:
- seq_printf(seq, "%s %s reqs=%lu bytes=%lu\n",
- ce_algs[i].alg.rng.base.cra_driver_name,
- ce_algs[i].alg.rng.base.cra_name,
- ce_algs[i].stat_req, ce_algs[i].stat_bytes);
- break;
#endif
}
}
#if defined(CONFIG_CRYPTO_DEV_SUN8I_CE_TRNG) && \
defined(CONFIG_CRYPTO_DEV_SUN8I_CE_DEBUG)
@@ -928,29 +891,10 @@ static int sun8i_ce_register_algs(struct sun8i_ce_dev *ce)
ce_algs[i].alg.hash.base.halg.base.cra_name);
ce_algs[i].ce = NULL;
return err;
}
break;
-#endif
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_CE_PRNG
- case CRYPTO_ALG_TYPE_RNG:
- if (ce->variant->prng == CE_ID_NOTSUPP) {
- dev_info(ce->dev,
- "DEBUG: Algo of %s not supported\n",
- ce_algs[i].alg.rng.base.cra_name);
- ce_algs[i].ce = NULL;
- break;
- }
- dev_info(ce->dev, "Register %s\n",
- ce_algs[i].alg.rng.base.cra_name);
- err = crypto_register_rng(&ce_algs[i].alg.rng);
- if (err) {
- dev_err(ce->dev, "Fail to register %s\n",
- ce_algs[i].alg.rng.base.cra_name);
- ce_algs[i].ce = NULL;
- }
- break;
#endif
default:
ce_algs[i].ce = NULL;
dev_err(ce->dev, "ERROR: tried to register an unknown algo\n");
}
@@ -975,17 +919,10 @@ static void sun8i_ce_unregister_algs(struct sun8i_ce_dev *ce)
case CRYPTO_ALG_TYPE_AHASH:
dev_info(ce->dev, "Unregister %d %s\n", i,
ce_algs[i].alg.hash.base.halg.base.cra_name);
crypto_engine_unregister_ahash(&ce_algs[i].alg.hash);
break;
-#endif
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_CE_PRNG
- case CRYPTO_ALG_TYPE_RNG:
- dev_info(ce->dev, "Unregister %d %s\n", i,
- ce_algs[i].alg.rng.base.cra_name);
- crypto_unregister_rng(&ce_algs[i].alg.rng);
- break;
#endif
}
}
}
diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c
deleted file mode 100644
index d0a1ac66738b..000000000000
--- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c
+++ /dev/null
@@ -1,159 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * sun8i-ce-prng.c - hardware cryptographic offloader for
- * Allwinner H3/A64/H5/H2+/H6/R40 SoC
- *
- * Copyright (C) 2015-2020 Corentin Labbe <clabbe@baylibre.com>
- *
- * This file handle the PRNG
- *
- * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst
- */
-#include "sun8i-ce.h"
-#include <linux/dma-mapping.h>
-#include <linux/pm_runtime.h>
-#include <crypto/internal/rng.h>
-
-int sun8i_ce_prng_init(struct crypto_tfm *tfm)
-{
- struct sun8i_ce_rng_tfm_ctx *ctx = crypto_tfm_ctx(tfm);
-
- memset(ctx, 0, sizeof(struct sun8i_ce_rng_tfm_ctx));
- return 0;
-}
-
-void sun8i_ce_prng_exit(struct crypto_tfm *tfm)
-{
- struct sun8i_ce_rng_tfm_ctx *ctx = crypto_tfm_ctx(tfm);
-
- kfree_sensitive(ctx->seed);
- ctx->seed = NULL;
- ctx->slen = 0;
-}
-
-int sun8i_ce_prng_seed(struct crypto_rng *tfm, const u8 *seed,
- unsigned int slen)
-{
- struct sun8i_ce_rng_tfm_ctx *ctx = crypto_rng_ctx(tfm);
-
- if (ctx->seed && ctx->slen != slen) {
- kfree_sensitive(ctx->seed);
- ctx->slen = 0;
- ctx->seed = NULL;
- }
- if (!ctx->seed)
- ctx->seed = kmalloc(slen, GFP_KERNEL | GFP_DMA);
- if (!ctx->seed)
- return -ENOMEM;
-
- memcpy(ctx->seed, seed, slen);
- ctx->slen = slen;
-
- return 0;
-}
-
-int sun8i_ce_prng_generate(struct crypto_rng *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int dlen)
-{
- struct sun8i_ce_rng_tfm_ctx *ctx = crypto_rng_ctx(tfm);
- struct rng_alg *alg = crypto_rng_alg(tfm);
- struct sun8i_ce_alg_template *algt;
- struct sun8i_ce_dev *ce;
- dma_addr_t dma_iv, dma_dst;
- int err = 0;
- int flow = 3;
- unsigned int todo;
- struct sun8i_ce_flow *chan;
- struct ce_task *cet;
- u32 common, sym;
- void *d;
-
- algt = container_of(alg, struct sun8i_ce_alg_template, alg.rng);
- ce = algt->ce;
-
- if (ctx->slen == 0) {
- dev_err(ce->dev, "not seeded\n");
- return -EINVAL;
- }
-
- /* we want dlen + seedsize rounded up to a multiple of PRNG_DATA_SIZE */
- todo = dlen + ctx->slen + PRNG_DATA_SIZE * 2;
- todo -= todo % PRNG_DATA_SIZE;
-
- d = kzalloc(todo, GFP_KERNEL | GFP_DMA);
- if (!d) {
- err = -ENOMEM;
- goto err_mem;
- }
-
- dev_dbg(ce->dev, "%s PRNG slen=%u dlen=%u todo=%u multi=%u\n", __func__,
- slen, dlen, todo, todo / PRNG_DATA_SIZE);
-
-#ifdef CONFIG_CRYPTO_DEV_SUN8I_CE_DEBUG
- algt->stat_req++;
- algt->stat_bytes += todo;
-#endif
-
- dma_iv = dma_map_single(ce->dev, ctx->seed, ctx->slen, DMA_TO_DEVICE);
- if (dma_mapping_error(ce->dev, dma_iv)) {
- dev_err(ce->dev, "Cannot DMA MAP IV\n");
- err = -EFAULT;
- goto err_iv;
- }
-
- dma_dst = dma_map_single(ce->dev, d, todo, DMA_FROM_DEVICE);
- if (dma_mapping_error(ce->dev, dma_dst)) {
- dev_err(ce->dev, "Cannot DMA MAP DST\n");
- err = -EFAULT;
- goto err_dst;
- }
-
- err = pm_runtime_resume_and_get(ce->dev);
- if (err < 0)
- goto err_pm;
-
- mutex_lock(&ce->rnglock);
- chan = &ce->chanlist[flow];
-
- cet = &chan->tl[0];
- memset(cet, 0, sizeof(struct ce_task));
-
- cet->t_id = cpu_to_le32(flow);
- common = ce->variant->prng | CE_COMM_INT;
- cet->t_common_ctl = cpu_to_le32(common);
-
- /* recent CE (H6) need length in bytes, in word otherwise */
- if (ce->variant->prng_t_dlen_in_bytes)
- cet->t_dlen = cpu_to_le32(todo);
- else
- cet->t_dlen = cpu_to_le32(todo / 4);
-
- sym = PRNG_LD;
- cet->t_sym_ctl = cpu_to_le32(sym);
- cet->t_asym_ctl = 0;
-
- cet->t_key = desc_addr_val_le32(ce, dma_iv);
- cet->t_iv = desc_addr_val_le32(ce, dma_iv);
-
- cet->t_dst[0].addr = desc_addr_val_le32(ce, dma_dst);
- cet->t_dst[0].len = cpu_to_le32(todo / 4);
-
- err = sun8i_ce_run_task(ce, 3, "PRNG");
- mutex_unlock(&ce->rnglock);
-
- pm_runtime_put(ce->dev);
-
-err_pm:
- dma_unmap_single(ce->dev, dma_dst, todo, DMA_FROM_DEVICE);
-err_dst:
- dma_unmap_single(ce->dev, dma_iv, ctx->slen, DMA_TO_DEVICE);
-
- if (!err) {
- memcpy(dst, d, dlen);
- memcpy(ctx->seed, d + dlen, ctx->slen);
- }
-err_iv:
- kfree_sensitive(d);
-err_mem:
- return err;
-}
diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce.h b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce.h
index 71f5a0cd3d45..468d99bf5bf6 100644
--- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce.h
+++ b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce.h
@@ -13,11 +13,10 @@
#include <linux/debugfs.h>
#include <linux/crypto.h>
#include <linux/hw_random.h>
#include <crypto/internal/hash.h>
#include <crypto/md5.h>
-#include <crypto/rng.h>
#include <crypto/sha1.h>
#include <crypto/sha2.h>
/* CE Registers */
#define CE_TDQ 0x00
@@ -56,13 +55,11 @@
#define CE_ALG_SHA224 18
#define CE_ALG_SHA256 19
#define CE_ALG_SHA384 20
#define CE_ALG_SHA512 21
#define CE_ALG_TRNG 48
-#define CE_ALG_PRNG 49
#define CE_ALG_TRNG_V2 0x1c
-#define CE_ALG_PRNG_V2 0x1d
/* Used in ce_variant */
#define CE_ID_NOTSUPP 0xFF
#define CE_ID_CIPHER_AES 0
@@ -94,14 +91,10 @@
#define ESR_R40 2
#define ESR_H5 3
#define ESR_H6 4
#define ESR_D1 5
-#define PRNG_DATA_SIZE (160 / 8)
-#define PRNG_SEED_SIZE DIV_ROUND_UP(175, 8)
-#define PRNG_LD BIT(17)
-
#define CE_DIE_ID_SHIFT 16
#define CE_DIE_ID_MASK 0x07
#define MAX_SG 8
@@ -134,31 +127,26 @@ struct ce_clock {
* @op_mode: list of supported block modes
* @cipher_t_dlen_in_bytes: Does the request size for cipher is in
* bytes or words
* @hash_t_dlen_in_bytes: Does the request size for hash is in
* bits or words
- * @prng_t_dlen_in_bytes: Does the request size for PRNG is in
- * bytes or words
* @trng_t_dlen_in_bytes: Does the request size for TRNG is in
* bytes or words
* @ce_clks: list of clocks needed by this variant
* @esr: The type of error register
- * @prng: The CE_ALG_XXX value for the PRNG
* @trng: The CE_ALG_XXX value for the TRNG
*/
struct ce_variant {
char alg_cipher[CE_ID_CIPHER_MAX];
char alg_hash[CE_ID_HASH_MAX];
u32 op_mode[CE_ID_OP_MAX];
bool cipher_t_dlen_in_bytes;
bool hash_t_dlen_in_bits;
- bool prng_t_dlen_in_bytes;
bool trng_t_dlen_in_bytes;
bool needs_word_addresses;
struct ce_clock ce_clks[CE_MAX_CLOCKS];
int esr;
- unsigned char prng;
unsigned char trng;
};
struct sginfo {
__le32 addr;
@@ -325,20 +313,10 @@ struct sun8i_ce_hash_reqctx {
u8 result[CE_MAX_HASH_DIGEST_SIZE] __aligned(CRYPTO_DMA_ALIGN);
u8 pad[2 * CE_MAX_HASH_BLOCK_SIZE];
struct ahash_request fallback_req; // keep at the end
};
-/*
- * struct sun8i_ce_prng_ctx - context for PRNG TFM
- * @seed: The seed to use
- * @slen: The size of the seed
- */
-struct sun8i_ce_rng_tfm_ctx {
- void *seed;
- unsigned int slen;
-};
-
/*
* struct sun8i_ce_alg_template - crypto_alg template
* @type: the CRYPTO_ALG_TYPE for this template
* @ce_algo_id: the CE_ID for this template
* @ce_blockmode: the type of block operation CE_ID
@@ -355,11 +333,10 @@ struct sun8i_ce_alg_template {
u32 ce_blockmode;
struct sun8i_ce_dev *ce;
union {
struct skcipher_engine_alg skcipher;
struct ahash_engine_alg hash;
- struct rng_alg rng;
} alg;
unsigned long stat_req;
unsigned long stat_fb;
unsigned long stat_bytes;
unsigned long stat_fb_maxsg;
@@ -396,13 +373,7 @@ int sun8i_ce_hash_final(struct ahash_request *areq);
int sun8i_ce_hash_update(struct ahash_request *areq);
int sun8i_ce_hash_finup(struct ahash_request *areq);
int sun8i_ce_hash_digest(struct ahash_request *areq);
int sun8i_ce_hash_run(struct crypto_engine *engine, void *breq);
-int sun8i_ce_prng_generate(struct crypto_rng *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int dlen);
-int sun8i_ce_prng_seed(struct crypto_rng *tfm, const u8 *seed, unsigned int slen);
-void sun8i_ce_prng_exit(struct crypto_tfm *tfm);
-int sun8i_ce_prng_init(struct crypto_tfm *tfm);
-
int sun8i_ce_hwrng_register(struct sun8i_ce_dev *ce);
void sun8i_ce_hwrng_unregister(struct sun8i_ce_dev *ce);
--
2.54.0
^ permalink raw reply related
* [PATCH 4/7] hwrng: qcom - Move qcom-rng.c into drivers/char/hw_random/
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers
In-Reply-To: <20260615224131.69370-1-ebiggers@kernel.org>
Since this file just implements a hwrng driver, move it into
drivers/char/hw_random/. Rename the kconfig option accordingly as well.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
arch/arm/configs/multi_v7_defconfig | 2 +-
arch/arm/configs/qcom_defconfig | 2 +-
arch/arm64/configs/defconfig | 2 +-
drivers/char/hw_random/Kconfig | 11 +++++++++++
drivers/char/hw_random/Makefile | 1 +
drivers/{crypto => char/hw_random}/qcom-rng.c | 0
drivers/crypto/Kconfig | 11 -----------
drivers/crypto/Makefile | 1 -
drivers/gpu/drm/ci/arm64.config | 2 +-
9 files changed, 16 insertions(+), 16 deletions(-)
rename drivers/{crypto => char/hw_random}/qcom-rng.c (100%)
diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig
index 3672dd12df60..8e1f70339a47 100644
--- a/arch/arm/configs/multi_v7_defconfig
+++ b/arch/arm/configs/multi_v7_defconfig
@@ -404,10 +404,11 @@ CONFIG_SERIAL_DEV_BUS=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_ASPEED_KCS_IPMI_BMC=m
CONFIG_ASPEED_BT_IPMI_BMC=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_ST=y
+CONFIG_HW_RANDOM_QCOM=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS_I2C_INFINEON=m
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_ARB_GPIO_CHALLENGE=m
CONFIG_I2C_MUX_GPIO=y
@@ -1333,11 +1334,10 @@ CONFIG_CRYPTO_DEV_S5P=m
CONFIG_CRYPTO_DEV_ATMEL_AES=m
CONFIG_CRYPTO_DEV_ATMEL_TDES=m
CONFIG_CRYPTO_DEV_ATMEL_SHA=m
CONFIG_CRYPTO_DEV_MARVELL_CESA=m
CONFIG_CRYPTO_DEV_QCE=m
-CONFIG_CRYPTO_DEV_QCOM_RNG=m
CONFIG_CRYPTO_DEV_ROCKCHIP=m
CONFIG_CRYPTO_DEV_STM32_HASH=m
CONFIG_CRYPTO_DEV_STM32_CRYP=m
CONFIG_CMA_SIZE_MBYTES=64
CONFIG_PRINTK_TIME=y
diff --git a/arch/arm/configs/qcom_defconfig b/arch/arm/configs/qcom_defconfig
index 29a1dea500f0..d57554971c03 100644
--- a/arch/arm/configs/qcom_defconfig
+++ b/arch/arm/configs/qcom_defconfig
@@ -115,10 +115,11 @@ CONFIG_SERIO_LIBPS2=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_MSM=y
CONFIG_SERIAL_MSM_CONSOLE=y
CONFIG_SERIAL_DEV_BUS=y
CONFIG_HW_RANDOM=y
+CONFIG_HW_RANDOM_QCOM=m
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_QUP=y
CONFIG_SPI=y
CONFIG_SPI_QUP=y
@@ -309,11 +310,10 @@ CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_USER_API=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
-CONFIG_CRYPTO_DEV_QCOM_RNG=m
CONFIG_DMA_CMA=y
CONFIG_CMA_SIZE_MBYTES=64
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 6745189b1fbc..7958932a5b2a 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -550,10 +550,11 @@ CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_HW_RANDOM_HISI_TRNG=m
CONFIG_HW_RANDOM_XILINX=m
+CONFIG_HW_RANDOM_QCOM=m
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=m
CONFIG_TCG_TIS_SPI=m
CONFIG_TCG_TIS_SPI_CR50=y
CONFIG_TCG_TIS_I2C_CR50=m
@@ -1953,11 +1954,10 @@ CONFIG_CRYPTO_AES_ARM64_BS=m
CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
CONFIG_CRYPTO_DEV_SUN8I_CE=m
CONFIG_CRYPTO_DEV_FSL_CAAM=m
CONFIG_CRYPTO_DEV_FSL_DPAA2_CAAM=m
CONFIG_CRYPTO_DEV_QCE=m
-CONFIG_CRYPTO_DEV_QCOM_RNG=m
CONFIG_CRYPTO_DEV_TEGRA=m
CONFIG_CRYPTO_DEV_ZYNQMP_AES=m
CONFIG_CRYPTO_DEV_ZYNQMP_SHA3=m
CONFIG_CRYPTO_DEV_CCREE=m
CONFIG_CRYPTO_DEV_HISI_SEC2=m
diff --git a/drivers/char/hw_random/Kconfig b/drivers/char/hw_random/Kconfig
index a5bcef4a54ee..b4c359abc4f9 100644
--- a/drivers/char/hw_random/Kconfig
+++ b/drivers/char/hw_random/Kconfig
@@ -634,10 +634,21 @@ config HW_RANDOM_XILINX
Generator and Pseudo random Number in CTR_DRBG mode as defined in NIST SP800-90A.
To compile this driver as a module, choose M here: the module
will be called xilinx-trng.
+config HW_RANDOM_QCOM
+ tristate "Qualcomm True Random Number Generator Driver"
+ depends on ARCH_QCOM || COMPILE_TEST
+ depends on HW_RANDOM
+ help
+ This driver provides support for the True Random Number
+ Generator hardware found on some Qualcomm SoCs.
+
+ To compile this driver as a module, choose M here. The
+ module will be called qcom-rng. If unsure, say N.
+
endif # HW_RANDOM
config UML_RANDOM
depends on UML
select HW_RANDOM
diff --git a/drivers/char/hw_random/Makefile b/drivers/char/hw_random/Makefile
index 95b5adb49560..8fce2aa6cf7d 100644
--- a/drivers/char/hw_random/Makefile
+++ b/drivers/char/hw_random/Makefile
@@ -52,5 +52,6 @@ obj-$(CONFIG_HW_RANDOM_ARM_SMCCC_TRNG) += arm_smccc_trng.o
obj-$(CONFIG_HW_RANDOM_CN10K) += cn10k-rng.o
obj-$(CONFIG_HW_RANDOM_POLARFIRE_SOC) += mpfs-rng.o
obj-$(CONFIG_HW_RANDOM_ROCKCHIP) += rockchip-rng.o
obj-$(CONFIG_HW_RANDOM_JH7110) += jh7110-trng.o
obj-$(CONFIG_HW_RANDOM_XILINX) += xilinx-trng.o
+obj-$(CONFIG_HW_RANDOM_QCOM) += qcom-rng.o
diff --git a/drivers/crypto/qcom-rng.c b/drivers/char/hw_random/qcom-rng.c
similarity index 100%
rename from drivers/crypto/qcom-rng.c
rename to drivers/char/hw_random/qcom-rng.c
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index eb834d15d614..03a8f7a1f75e 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -635,21 +635,10 @@ config CRYPTO_DEV_QCE_SW_MAX_LEN
Note that 192-bit keys are not supported by the hardware and are
always processed by the software fallback, and all DES requests
are done by the hardware.
-config CRYPTO_DEV_QCOM_RNG
- tristate "Qualcomm Random Number Generator Driver"
- depends on ARCH_QCOM || COMPILE_TEST
- depends on HW_RANDOM
- help
- This driver provides support for the Random Number
- Generator hardware found on Qualcomm SoCs.
-
- To compile this driver as a module, choose M here. The
- module will be called qcom-rng. If unsure, say N.
-
config CRYPTO_DEV_IMGTEC_HASH
tristate "Imagination Technologies hardware hash accelerator"
depends on MIPS || COMPILE_TEST
select CRYPTO_MD5
select CRYPTO_SHA1
diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index 5a950c7abc39..2c33b83f3cfa 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -25,11 +25,10 @@ obj-$(CONFIG_CRYPTO_DEV_OMAP_DES) += omap-des.o
obj-$(CONFIG_CRYPTO_DEV_OMAP_SHAM) += omap-sham.o
obj-$(CONFIG_CRYPTO_DEV_PADLOCK_AES) += padlock-aes.o
obj-$(CONFIG_CRYPTO_DEV_PADLOCK_SHA) += padlock-sha.o
obj-$(CONFIG_CRYPTO_DEV_PPC4XX) += amcc/
obj-$(CONFIG_CRYPTO_DEV_QCE) += qce/
-obj-$(CONFIG_CRYPTO_DEV_QCOM_RNG) += qcom-rng.o
obj-$(CONFIG_CRYPTO_DEV_ROCKCHIP) += rockchip/
obj-$(CONFIG_CRYPTO_DEV_S5P) += s5p-sss.o
obj-$(CONFIG_CRYPTO_DEV_SA2UL) += sa2ul.o
obj-$(CONFIG_CRYPTO_DEV_SAHARA) += sahara.o
obj-$(CONFIG_CRYPTO_DEV_SL3516) += gemini/
diff --git a/drivers/gpu/drm/ci/arm64.config b/drivers/gpu/drm/ci/arm64.config
index 563a69669a7b..c46125c1f80f 100644
--- a/drivers/gpu/drm/ci/arm64.config
+++ b/drivers/gpu/drm/ci/arm64.config
@@ -76,11 +76,10 @@ CONFIG_INTERCONNECT_QCOM_SDM845=y
CONFIG_INTERCONNECT_QCOM_MSM8916=y
CONFIG_INTERCONNECT_QCOM_MSM8996=y
CONFIG_INTERCONNECT_QCOM_OSM_L3=y
CONFIG_INTERCONNECT_QCOM_SC7180=y
CONFIG_INTERCONNECT_QCOM_SM8350=y
-CONFIG_CRYPTO_DEV_QCOM_RNG=y
CONFIG_SC_DISPCC_7180=y
CONFIG_SC_GPUCC_7180=y
CONFIG_SM_GPUCC_8350=y
CONFIG_QCOM_SPMI_ADC5=y
CONFIG_QCOM_SPMI_VADC=y
@@ -187,10 +186,11 @@ CONFIG_PWM_MEDIATEK=y
CONFIG_DRM_MEDIATEK_HDMI=y
CONFIG_GNSS=y
CONFIG_GNSS_MTK_SERIAL=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_MTK=y
+CONFIG_HW_RANDOM_QCOM=y
CONFIG_MTK_DEVAPC=y
CONFIG_PWM_MTK_DISP=y
CONFIG_MTK_CMDQ=y
CONFIG_REGULATOR_DA9211=y
CONFIG_DRM_ANALOGIX_ANX7625=y
--
2.54.0
^ permalink raw reply related
* [PATCH 3/7] crypto: qcom-rng - Remove crypto_rng interface
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers, stable
In-Reply-To: <20260615224131.69370-1-ebiggers@kernel.org>
qcom-rng.c exposes the same hardware through two completely separate
interfaces, crypto_rng and hwrng. However, the implementation of this
is buggy because it permits generation operations from these interfaces
to run concurrently with each other, accessing the same registers. That
is, qcom_rng_generate() synchronizes with itself but not with
qcom_hwrng_read(). This results in potential repetition of output from
the RNG, output of non-random values, etc.
Fortunately, there's actually no point in hardware RNG drivers
implementing the crypto_rng interface. It's not actually used by
anything besides the "rng" algorithm type of AF_ALG, which in turn is
not actually used in practice. Other crypto_rng hardware drivers are
likewise being phased out, leaving just the hwrng support.
Thus, remove it to simplify the code and avoid conflict (and confusion)
with the hwrng interface which is the one that actually matters.
Fixes: f29cd5bb64c2 ("crypto: qcom-rng - Add hw_random interface support")
Cc: stable@vger.kernel.org
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/crypto/Kconfig | 1 -
drivers/crypto/qcom-rng.c | 158 +++++---------------------------------
2 files changed, 19 insertions(+), 140 deletions(-)
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 216a00bad5d7..eb834d15d614 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -639,11 +639,10 @@ config CRYPTO_DEV_QCE_SW_MAX_LEN
config CRYPTO_DEV_QCOM_RNG
tristate "Qualcomm Random Number Generator Driver"
depends on ARCH_QCOM || COMPILE_TEST
depends on HW_RANDOM
- select CRYPTO_RNG
help
This driver provides support for the Random Number
Generator hardware found on Qualcomm SoCs.
To compile this driver as a module, choose M here. The
diff --git a/drivers/crypto/qcom-rng.c b/drivers/crypto/qcom-rng.c
index 7058bd98f9e9..4d046caafe5b 100644
--- a/drivers/crypto/qcom-rng.c
+++ b/drivers/crypto/qcom-rng.c
@@ -1,14 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2017-18 Linaro Limited
//
// Based on msm-rng.c and downstream driver
-#include <crypto/internal/rng.h>
#include <linux/acpi.h>
#include <linux/clk.h>
-#include <linux/crypto.h>
#include <linux/hw_random.h>
#include <linux/io.h>
#include <linux/iopoll.h>
#include <linux/kernel.h>
#include <linux/module.h>
@@ -30,28 +28,19 @@
#define WORD_SZ 4
#define QCOM_TRNG_QUALITY 1024
struct qcom_rng {
- struct mutex lock;
void __iomem *base;
struct clk *clk;
struct hwrng hwrng;
- struct qcom_rng_match_data *match_data;
-};
-
-struct qcom_rng_ctx {
- struct qcom_rng *rng;
};
struct qcom_rng_match_data {
- bool skip_init;
bool hwrng_support;
};
-static struct qcom_rng *qcom_rng_dev;
-
static int qcom_rng_read(struct qcom_rng *rng, u8 *data, unsigned int max)
{
unsigned int currsize = 0;
u32 val;
int ret;
@@ -78,41 +67,10 @@ static int qcom_rng_read(struct qcom_rng *rng, u8 *data, unsigned int max)
} while (currsize < max);
return currsize;
}
-static int qcom_rng_generate(struct crypto_rng *tfm,
- const u8 *src, unsigned int slen,
- u8 *dstn, unsigned int dlen)
-{
- struct qcom_rng_ctx *ctx = crypto_rng_ctx(tfm);
- struct qcom_rng *rng = ctx->rng;
- int ret;
-
- ret = clk_prepare_enable(rng->clk);
- if (ret)
- return ret;
-
- mutex_lock(&rng->lock);
-
- ret = qcom_rng_read(rng, dstn, dlen);
-
- mutex_unlock(&rng->lock);
- clk_disable_unprepare(rng->clk);
-
- if (ret >= 0)
- ret = 0;
-
- return ret;
-}
-
-static int qcom_rng_seed(struct crypto_rng *tfm, const u8 *seed,
- unsigned int slen)
-{
- return 0;
-}
-
static int qcom_hwrng_init(struct hwrng *hwrng)
{
struct qcom_rng *qrng = container_of(hwrng, struct qcom_rng, hwrng);
return clk_prepare_enable(qrng->clk);
@@ -130,135 +88,58 @@ static void qcom_hwrng_cleanup(struct hwrng *hwrng)
struct qcom_rng *qrng = container_of(hwrng, struct qcom_rng, hwrng);
clk_disable_unprepare(qrng->clk);
}
-static int qcom_rng_enable(struct qcom_rng *rng)
-{
- u32 val;
- int ret;
-
- ret = clk_prepare_enable(rng->clk);
- if (ret)
- return ret;
-
- /* Enable PRNG only if it is not already enabled */
- val = readl_relaxed(rng->base + PRNG_CONFIG);
- if (val & PRNG_CONFIG_HW_ENABLE)
- goto already_enabled;
-
- val = readl_relaxed(rng->base + PRNG_LFSR_CFG);
- val &= ~PRNG_LFSR_CFG_MASK;
- val |= PRNG_LFSR_CFG_CLOCKS;
- writel(val, rng->base + PRNG_LFSR_CFG);
-
- val = readl_relaxed(rng->base + PRNG_CONFIG);
- val |= PRNG_CONFIG_HW_ENABLE;
- writel(val, rng->base + PRNG_CONFIG);
-
-already_enabled:
- clk_disable_unprepare(rng->clk);
-
- return 0;
-}
-
-static int qcom_rng_init(struct crypto_tfm *tfm)
-{
- struct qcom_rng_ctx *ctx = crypto_tfm_ctx(tfm);
-
- ctx->rng = qcom_rng_dev;
-
- if (!ctx->rng->match_data->skip_init)
- return qcom_rng_enable(ctx->rng);
-
- return 0;
-}
-
-static struct rng_alg qcom_rng_alg = {
- .generate = qcom_rng_generate,
- .seed = qcom_rng_seed,
- .seedsize = 0,
- .base = {
- .cra_name = "stdrng",
- .cra_driver_name = "qcom-rng",
- .cra_flags = CRYPTO_ALG_TYPE_RNG,
- .cra_priority = 300,
- .cra_ctxsize = sizeof(struct qcom_rng_ctx),
- .cra_module = THIS_MODULE,
- .cra_init = qcom_rng_init,
- }
-};
-
static int qcom_rng_probe(struct platform_device *pdev)
{
+ const struct qcom_rng_match_data *match_data;
struct qcom_rng *rng;
int ret;
+ match_data = device_get_match_data(&pdev->dev);
+ if (match_data == NULL || !match_data->hwrng_support) {
+ dev_info(&pdev->dev, "TRNG support not detected\n");
+ /*
+ * In this case the driver does nothing except the dev_info(),
+ * but bind the device anyway to avoid effects on GCC state.
+ */
+ return 0;
+ }
+
rng = devm_kzalloc(&pdev->dev, sizeof(*rng), GFP_KERNEL);
if (!rng)
return -ENOMEM;
- platform_set_drvdata(pdev, rng);
- mutex_init(&rng->lock);
-
rng->base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(rng->base))
return PTR_ERR(rng->base);
rng->clk = devm_clk_get_optional(&pdev->dev, "core");
if (IS_ERR(rng->clk))
return PTR_ERR(rng->clk);
- rng->match_data = (struct qcom_rng_match_data *)device_get_match_data(&pdev->dev);
-
- qcom_rng_dev = rng;
- ret = crypto_register_rng(&qcom_rng_alg);
- if (ret) {
- dev_err(&pdev->dev, "Register crypto rng failed: %d\n", ret);
- qcom_rng_dev = NULL;
- return ret;
- }
-
- if (rng->match_data->hwrng_support) {
- rng->hwrng.name = "qcom_hwrng";
- rng->hwrng.init = qcom_hwrng_init;
- rng->hwrng.read = qcom_hwrng_read;
- rng->hwrng.cleanup = qcom_hwrng_cleanup;
- rng->hwrng.quality = QCOM_TRNG_QUALITY;
- ret = devm_hwrng_register(&pdev->dev, &rng->hwrng);
- if (ret) {
- dev_err(&pdev->dev, "Register hwrng failed: %d\n", ret);
- qcom_rng_dev = NULL;
- goto fail;
- }
- }
-
- return ret;
-fail:
- crypto_unregister_rng(&qcom_rng_alg);
+ rng->hwrng.name = "qcom_hwrng";
+ rng->hwrng.init = qcom_hwrng_init;
+ rng->hwrng.read = qcom_hwrng_read;
+ rng->hwrng.cleanup = qcom_hwrng_cleanup;
+ rng->hwrng.quality = QCOM_TRNG_QUALITY;
+ ret = devm_hwrng_register(&pdev->dev, &rng->hwrng);
+ if (ret)
+ dev_err(&pdev->dev, "Register hwrng failed: %d\n", ret);
return ret;
}
-static void qcom_rng_remove(struct platform_device *pdev)
-{
- crypto_unregister_rng(&qcom_rng_alg);
-
- qcom_rng_dev = NULL;
-}
-
static struct qcom_rng_match_data qcom_prng_match_data = {
- .skip_init = false,
.hwrng_support = false,
};
static struct qcom_rng_match_data qcom_prng_ee_match_data = {
- .skip_init = true,
.hwrng_support = false,
};
static struct qcom_rng_match_data qcom_trng_match_data = {
- .skip_init = true,
.hwrng_support = true,
};
static const struct acpi_device_id __maybe_unused qcom_rng_acpi_match[] = {
{ .id = "QCOM8160", .driver_data = (kernel_ulong_t)&qcom_prng_ee_match_data },
@@ -274,11 +155,10 @@ static const struct of_device_id __maybe_unused qcom_rng_of_match[] = {
};
MODULE_DEVICE_TABLE(of, qcom_rng_of_match);
static struct platform_driver qcom_rng_driver = {
.probe = qcom_rng_probe,
- .remove = qcom_rng_remove,
.driver = {
.name = KBUILD_MODNAME,
.of_match_table = qcom_rng_of_match,
.acpi_match_table = ACPI_PTR(qcom_rng_acpi_match),
}
--
2.54.0
^ permalink raw reply related
* [PATCH 2/7] crypto: qcom-rng - Allow zero as a random number
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers, stable
In-Reply-To: <20260615224131.69370-1-ebiggers@kernel.org>
Zero is a valid random number and needs to be allowed. Otherwise the
output is distinguishable from random.
Fixes: f29cd5bb64c2 ("crypto: qcom-rng - Add hw_random interface support")
Cc: stable@vger.kernel.org
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/crypto/qcom-rng.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/crypto/qcom-rng.c b/drivers/crypto/qcom-rng.c
index f31a7fe07ba7..7058bd98f9e9 100644
--- a/drivers/crypto/qcom-rng.c
+++ b/drivers/crypto/qcom-rng.c
@@ -63,12 +63,10 @@ static int qcom_rng_read(struct qcom_rng *rng, u8 *data, unsigned int max)
200, 10000);
if (ret)
return ret;
val = readl_relaxed(rng->base + PRNG_DATA_OUT);
- if (!val)
- return -EINVAL;
if ((max - currsize) >= WORD_SZ) {
memcpy(data, &val, WORD_SZ);
data += WORD_SZ;
currsize += WORD_SZ;
--
2.54.0
^ permalink raw reply related
* [PATCH 1/7] crypto: qcom-rng - Enable clock in hwrng case
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers, stable
In-Reply-To: <20260615224131.69370-1-ebiggers@kernel.org>
Fix qcom-rng.c to enable the clock before accessing the hardware.
Fixes: f29cd5bb64c2 ("crypto: qcom-rng - Add hw_random interface support")
Cc: stable@vger.kernel.org
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/crypto/qcom-rng.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/drivers/crypto/qcom-rng.c b/drivers/crypto/qcom-rng.c
index 150e5802e351..f31a7fe07ba7 100644
--- a/drivers/crypto/qcom-rng.c
+++ b/drivers/crypto/qcom-rng.c
@@ -111,17 +111,31 @@ static int qcom_rng_seed(struct crypto_rng *tfm, const u8 *seed,
unsigned int slen)
{
return 0;
}
+static int qcom_hwrng_init(struct hwrng *hwrng)
+{
+ struct qcom_rng *qrng = container_of(hwrng, struct qcom_rng, hwrng);
+
+ return clk_prepare_enable(qrng->clk);
+}
+
static int qcom_hwrng_read(struct hwrng *hwrng, void *data, size_t max, bool wait)
{
struct qcom_rng *qrng = container_of(hwrng, struct qcom_rng, hwrng);
return qcom_rng_read(qrng, data, max);
}
+static void qcom_hwrng_cleanup(struct hwrng *hwrng)
+{
+ struct qcom_rng *qrng = container_of(hwrng, struct qcom_rng, hwrng);
+
+ clk_disable_unprepare(qrng->clk);
+}
+
static int qcom_rng_enable(struct qcom_rng *rng)
{
u32 val;
int ret;
@@ -206,11 +220,13 @@ static int qcom_rng_probe(struct platform_device *pdev)
return ret;
}
if (rng->match_data->hwrng_support) {
rng->hwrng.name = "qcom_hwrng";
+ rng->hwrng.init = qcom_hwrng_init;
rng->hwrng.read = qcom_hwrng_read;
+ rng->hwrng.cleanup = qcom_hwrng_cleanup;
rng->hwrng.quality = QCOM_TRNG_QUALITY;
ret = devm_hwrng_register(&pdev->dev, &rng->hwrng);
if (ret) {
dev_err(&pdev->dev, "Register hwrng failed: %d\n", ret);
qcom_rng_dev = NULL;
--
2.54.0
^ permalink raw reply related
* [PATCH 0/7] Finish removing crypto_rng from drivers/crypto/
From: Eric Biggers @ 2026-06-15 22:41 UTC (permalink / raw)
To: linux-crypto, Herbert Xu
Cc: linux-kernel, Gaurav Jain, Horia Geantă, Pankaj Gupta,
Corentin Labbe, Dmitry Baryshkov, Konrad Dybcio, linux-arm-msm,
Eric Biggers
This series finishes removing the unused, redundant, and frequently
broken crypto_rng support from drivers/crypto/. It applies to
cryptodev/master.
Patches 1-4 are a resend of
https://lore.kernel.org/linux-crypto/20260608175848.2045229-1-ebiggers@kernel.org/
Please consider these patches for 7.2, considering that most of these
drivers had security vulnerabilities which would have needed to be fixed
right away anyway. And the qcom hwrng fixes are important too.
Eric Biggers (7):
crypto: qcom-rng - Enable clock in hwrng case
crypto: qcom-rng - Allow zero as a random number
crypto: qcom-rng - Remove crypto_rng interface
hwrng: qcom - Move qcom-rng.c into drivers/char/hw_random/
crypto: sun8i-ce - Remove crypto_rng interface
crypto: sun8i-ss - Remove crypto_rng interface
crypto: caam - Remove crypto_rng interface
arch/arm/configs/multi_v7_defconfig | 2 +-
arch/arm/configs/qcom_defconfig | 2 +-
arch/arm64/configs/defconfig | 2 +-
drivers/char/hw_random/Kconfig | 11 +
drivers/char/hw_random/Makefile | 1 +
drivers/{crypto => char/hw_random}/qcom-rng.c | 156 ++----------
drivers/crypto/Kconfig | 12 -
drivers/crypto/Makefile | 1 -
drivers/crypto/allwinner/Kconfig | 16 --
drivers/crypto/allwinner/sun8i-ce/Makefile | 1 -
.../crypto/allwinner/sun8i-ce/sun8i-ce-core.c | 63 -----
.../crypto/allwinner/sun8i-ce/sun8i-ce-prng.c | 159 ------------
drivers/crypto/allwinner/sun8i-ce/sun8i-ce.h | 29 ---
drivers/crypto/allwinner/sun8i-ss/Makefile | 1 -
.../crypto/allwinner/sun8i-ss/sun8i-ss-core.c | 45 ----
.../crypto/allwinner/sun8i-ss/sun8i-ss-prng.c | 177 -------------
drivers/crypto/allwinner/sun8i-ss/sun8i-ss.h | 23 --
drivers/crypto/caam/Kconfig | 9 -
drivers/crypto/caam/Makefile | 1 -
drivers/crypto/caam/caamprng.c | 241 ------------------
drivers/crypto/caam/intern.h | 15 --
drivers/crypto/caam/jr.c | 2 -
drivers/gpu/drm/ci/arm64.config | 2 +-
23 files changed, 41 insertions(+), 930 deletions(-)
rename drivers/{crypto => char/hw_random}/qcom-rng.c (53%)
delete mode 100644 drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c
delete mode 100644 drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c
delete mode 100644 drivers/crypto/caam/caamprng.c
base-commit: 6ea0ce3a19f9c37a014099e2b0a46b27fa164564
--
2.54.0
^ permalink raw reply
* Re: [PATCH v3] lib/raid/xor: x86: Add AVX-512 optimized xor_gen()
From: Eric Biggers @ 2026-06-15 21:29 UTC (permalink / raw)
To: Borislav Petkov
Cc: x86, Christoph Hellwig, linux-crypto, David Laight, linux-raid,
Andrew Morton, linux-kernel
In-Reply-To: <255CAE3E-7FD3-4DC2-B3DE-46BE67EF22A8@alien8.de>
On Mon, Jun 15, 2026 at 09:16:55PM +0000, Borislav Petkov wrote:
> On June 15, 2026 8:10:50 PM UTC, Eric Biggers <ebiggers@kernel.org> wrote:
> >
> >But I wanted to ask: do we really care about the case where features are
> >"supported" but their XCR0 bits aren't set? Perhaps the kernel just
> >doesn't/shouldn't support weird cases like "-cpu max,xsave=off"?
> >
>
> Yes, our aim is to support only configurations which are actually
> present in real hardware and not a "oh, it would be good if it did
> that, just because..."
Seems reasonable to me. Would the same apply to UML here?
> >If this case indeed needs to be handled, could we make things easier for
> >the kernel's AVX and AVX-512 optimized code? Currently AVX-512 needs:
> >
> > if (boot_cpu_has(X86_FEATURE_AVX512F) &&
> > cpu_has_xfeatures(XFEATURE_MASK_FP | XFEATURE_MASK_SSE |
> > XFEATURE_MASK_YMM | XFEATURE_MASK_AVX512, NULL))
> >
> >How about we make X86_FEATURE_AVX512F depend on XCR0=111xx111, and
> >X86_FEATURE_AVX depend on XCR0=xxxxx111? Then the cpu_has_xfeatures()
> >check wouldn't be needed. Is there any reason not to do that?
>
> How do you want to accomplish that? Very early during boot on the BSP
> you sanity-check XCR0 and clear feature flags if components are not
> set?
That would be the idea. Something similar to what
arch/x86/kernel/cpu/cpuid-deps.c does. Except that seems to only
enforce the dependencies when the kernel itself is disabling things; if
the hypervisor is broken then it just warns.
In any case, I'd like these to go away:
$ git grep cpu_has_xfeatures | wc -l
31
- Eric
^ permalink raw reply
* Re: [PATCH v3] lib/raid/xor: x86: Add AVX-512 optimized xor_gen()
From: Borislav Petkov @ 2026-06-15 21:16 UTC (permalink / raw)
To: Eric Biggers, x86
Cc: Christoph Hellwig, linux-crypto, David Laight, linux-raid,
Andrew Morton, linux-kernel
In-Reply-To: <20260615201050.GB1764@quark>
On June 15, 2026 8:10:50 PM UTC, Eric Biggers <ebiggers@kernel.org> wrote:
>
>But I wanted to ask: do we really care about the case where features are
>"supported" but their XCR0 bits aren't set? Perhaps the kernel just
>doesn't/shouldn't support weird cases like "-cpu max,xsave=off"?
>
Yes, our aim is to support only configurations which are actually present in real hardware and not a "oh, it would be good if it did that, just because..."
>If this case indeed needs to be handled, could we make things easier for
>the kernel's AVX and AVX-512 optimized code? Currently AVX-512 needs:
>
> if (boot_cpu_has(X86_FEATURE_AVX512F) &&
> cpu_has_xfeatures(XFEATURE_MASK_FP | XFEATURE_MASK_SSE |
> XFEATURE_MASK_YMM | XFEATURE_MASK_AVX512, NULL))
>
>How about we make X86_FEATURE_AVX512F depend on XCR0=111xx111, and
>X86_FEATURE_AVX depend on XCR0=xxxxx111? Then the cpu_has_xfeatures()
>check wouldn't be needed. Is there any reason not to do that?
How do you want to accomplish that? Very early during boot on the BSP you sanity-check XCR0 and clear feature flags if components are not set?
Thx.
--
Small device. Typos and formatting crap
^ permalink raw reply
* [PATCH v2] hw_random: timeriomem-rng: add configurable read width and data mask
From: Jad Keskes @ 2026-06-15 20:13 UTC (permalink / raw)
To: Olivia Mackall, Herbert Xu
Cc: Rob Herring, Krzysztof Kozlowski, Conor Dooley, Alexander Clouter,
linux-crypto, devicetree, linux-kernel, Jad Keskes
The TODO for supporting read sizes other than 32 bits and masking has
been sitting in this driver since 2009. Implement it.
Add width (8, 16, or 32 bits) and mask properties to the platform data
and device tree bindings. The read loop dispatches on width using
readb/readw/readl so a configured 8-bit access doesn't trigger a bus
error on hardware that rejects 32-bit reads to that address. The mask
is ANDed with the value before storing.
These are platform properties, not runtime policy -- width depends on
SoC integration, mask reflects which output bits carry entropy.
The alignment check in probe is updated to verify the resource is
aligned to the configured width instead of hardcoding 4-byte alignment.
Signed-off-by: Jad Keskes <inasj268@gmail.com>
---
v2:
- Remove old timeriomem_rng.yaml to avoid dt_binding_check conflict
- Use IS_ALIGNED() instead of modulo for 32-bit PAE safety
.../bindings/rng/timeriomem-rng.yaml | 76 ++++++++++++++++++
.../bindings/rng/timeriomem_rng.yaml | 48 ------------
drivers/char/hw_random/timeriomem-rng.c | 78 +++++++++++++++----
include/linux/timeriomem-rng.h | 12 +++
4 files changed, 153 insertions(+), 61 deletions(-)
create mode 100644 Documentation/devicetree/bindings/rng/timeriomem-rng.yaml
delete mode 100644 Documentation/devicetree/bindings/rng/timeriomem_rng.yaml
diff --git a/Documentation/devicetree/bindings/rng/timeriomem-rng.yaml b/Documentation/devicetree/bindings/rng/timeriomem-rng.yaml
new file mode 100644
index 000000000000..0d8460e9f916
--- /dev/null
+++ b/Documentation/devicetree/bindings/rng/timeriomem-rng.yaml
@@ -0,0 +1,76 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/rng/timeriomem-rng.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Timer IOMEM Hardware Random Number Generator
+
+description: |
+ This binding covers platforms that have a single IO memory address which
+ provides periodic random data. The driver reads from the address at a
+ fixed interval, returning a configurable-width value masked to the desired
+ bits.
+
+maintainers:
+ - Alexander Clouter <alex@digriz.org.uk>
+
+properties:
+ compatible:
+ enum:
+ - timeriomem_rng
+
+ reg:
+ maxItems: 1
+
+ period:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description:
+ Interval in microseconds between reads. New random data is expected to
+ be available at this rate.
+
+ quality:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ default: 0
+ description:
+ Estimated entropy per 1024 bits of data, in the same scale as the
+ kernel's hwrng core (0-1024).
+
+ width:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ default: 32
+ enum: [8, 16, 32]
+ description:
+ Access width in bits. Determines whether the read is performed as
+ an 8-bit, 16-bit, or 32-bit bus access.
+
+ mask:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ default: 0xFFFFFFFF
+ description:
+ Mask applied to the value read from the register. Bits set to 0 in
+ the mask are cleared in the output data. Default (no mask) passes
+ all bits through.
+
+required:
+ - compatible
+ - reg
+ - period
+
+additionalProperties: false
+
+examples:
+ - |
+ rng@f0001000 {
+ compatible = "timeriomem_rng";
+ reg = <0xf0001000 0x4>;
+ period = <100000>;
+ };
+
+ rng@f0002000 {
+ compatible = "timeriomem_rng";
+ reg = <0xf0002000 0x1>;
+ period = <50000>;
+ width = <8>;
+ mask = <0xFF>;
+ };
diff --git a/Documentation/devicetree/bindings/rng/timeriomem_rng.yaml b/Documentation/devicetree/bindings/rng/timeriomem_rng.yaml
deleted file mode 100644
index 4754174e9849..000000000000
--- a/Documentation/devicetree/bindings/rng/timeriomem_rng.yaml
+++ /dev/null
@@ -1,48 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-only
-%YAML 1.2
----
-$id: http://devicetree.org/schemas/rng/timeriomem_rng.yaml#
-$schema: http://devicetree.org/meta-schemas/core.yaml#
-
-title: TimerIO Random Number Generator
-
-maintainers:
- - Krzysztof Kozlowski <krzk@kernel.org>
-
-properties:
- compatible:
- const: timeriomem_rng
-
- period:
- $ref: /schemas/types.yaml#/definitions/uint32
- description: wait time in microseconds to use between samples
-
- quality:
- $ref: /schemas/types.yaml#/definitions/uint32
- default: 0
- description:
- Estimated number of bits of true entropy per 1024 bits read from the rng.
- Defaults to zero which causes the kernel's default quality to be used
- instead. Note that the default quality is usually zero which disables
- using this rng to automatically fill the kernel's entropy pool.
-
- reg:
- maxItems: 1
- description:
- Base address to sample from. Currently 'reg' must be at least four bytes
- wide and 32-bit aligned.
-
-required:
- - compatible
- - period
- - reg
-
-additionalProperties: false
-
-examples:
- - |
- rng@44 {
- compatible = "timeriomem_rng";
- reg = <0x44 0x04>;
- period = <1000000>;
- };
diff --git a/drivers/char/hw_random/timeriomem-rng.c b/drivers/char/hw_random/timeriomem-rng.c
index e61f06393209..4557326618c9 100644
--- a/drivers/char/hw_random/timeriomem-rng.c
+++ b/drivers/char/hw_random/timeriomem-rng.c
@@ -14,7 +14,9 @@
* has to do is provide the address and 'wait time' that new data becomes
* available.
*
- * TODO: add support for reading sizes other than 32bits and masking
+ * The read width (8, 16, or 32 bits) and an optional data mask can be
+ * configured through platform data or device tree properties. Default is
+ * 32-bit reads with no mask.
*/
#include <linux/completion.h>
@@ -34,6 +36,8 @@ struct timeriomem_rng_private {
void __iomem *io_base;
ktime_t period;
unsigned int present:1;
+ unsigned int width;
+ u32 mask;
struct hrtimer timer;
struct completion completion;
@@ -48,6 +52,7 @@ static int timeriomem_rng_read(struct hwrng *hwrng, void *data,
container_of(hwrng, struct timeriomem_rng_private, rng_ops);
int retval = 0;
int period_us = ktime_to_us(priv->period);
+ int chunk = priv->width / 8;
/*
* There may not have been enough time for new data to be generated
@@ -71,11 +76,28 @@ static int timeriomem_rng_read(struct hwrng *hwrng, void *data,
usleep_range(period_us,
period_us + max(1, period_us / 100));
- *(u32 *)data = readl(priv->io_base);
- retval += sizeof(u32);
- data += sizeof(u32);
- max -= sizeof(u32);
- } while (wait && max > sizeof(u32));
+ switch (priv->width) {
+ case 8: {
+ u8 val = readb(priv->io_base) & priv->mask;
+ *(u8 *)data = val;
+ break;
+ }
+ case 16: {
+ u16 val = readw(priv->io_base) & priv->mask;
+ *(u16 *)data = val;
+ break;
+ }
+ case 32: {
+ u32 val = readl(priv->io_base) & priv->mask;
+ *(u32 *)data = val;
+ break;
+ }
+ }
+
+ retval += chunk;
+ data += chunk;
+ max -= chunk;
+ } while (wait && max > chunk);
/*
* Block any new callers until the RNG has had time to generate new
@@ -125,11 +147,8 @@ static int timeriomem_rng_probe(struct platform_device *pdev)
if (IS_ERR(priv->io_base))
return PTR_ERR(priv->io_base);
- if (res->start % 4 != 0 || resource_size(res) < 4) {
- dev_err(&pdev->dev,
- "address must be at least four bytes wide and 32-bit aligned\n");
- return -EINVAL;
- }
+ priv->width = 32;
+ priv->mask = 0xFFFFFFFF;
if (pdev->dev.of_node) {
int i;
@@ -145,9 +164,42 @@ static int timeriomem_rng_probe(struct platform_device *pdev)
if (!of_property_read_u32(pdev->dev.of_node,
"quality", &i))
priv->rng_ops.quality = i;
+
+ of_property_read_u32(pdev->dev.of_node,
+ "width", &priv->width);
+ of_property_read_u32(pdev->dev.of_node,
+ "mask", &priv->mask);
} else {
period = pdata->period;
priv->rng_ops.quality = pdata->quality;
+
+ if (pdata->width_set)
+ priv->width = pdata->width;
+ if (pdata->mask_set)
+ priv->mask = pdata->mask;
+ }
+
+ if (priv->width == 0)
+ priv->width = 32;
+
+ switch (priv->width) {
+ case 8:
+ case 16:
+ case 32:
+ break;
+ default:
+ dev_err(&pdev->dev, "invalid width %u, must be 8, 16, or 32\n",
+ priv->width);
+ return -EINVAL;
+ }
+
+ if (!IS_ALIGNED(res->start, priv->width / 8) ||
+ resource_size(res) < priv->width / 8) {
+ dev_err(&pdev->dev,
+ "address must be at least %u-bit aligned (%u byte%s)\n",
+ priv->width, priv->width / 8,
+ priv->width / 8 > 1 ? "s" : "");
+ return -EINVAL;
}
priv->period = us_to_ktime(period);
@@ -167,8 +219,8 @@ static int timeriomem_rng_probe(struct platform_device *pdev)
return err;
}
- dev_info(&pdev->dev, "32bits from 0x%p @ %dus\n",
- priv->io_base, period);
+ dev_info(&pdev->dev, "%ubit from %p @ %dus\n",
+ priv->width, priv->io_base, period);
return 0;
}
diff --git a/include/linux/timeriomem-rng.h b/include/linux/timeriomem-rng.h
index 672df7fbf6c1..b4202ad2f507 100644
--- a/include/linux/timeriomem-rng.h
+++ b/include/linux/timeriomem-rng.h
@@ -16,6 +16,18 @@ struct timeriomem_rng_data {
/* bits of entropy per 1024 bits read */
unsigned int quality;
+
+ /* read width (8, 16, or 32), 0 means 32 */
+ unsigned int width;
+
+ /* set to true if width is explicitly provided */
+ bool width_set;
+
+ /* mask applied to raw read value */
+ u32 mask;
+
+ /* set to true if mask is explicitly provided */
+ bool mask_set;
};
#endif /* _LINUX_TIMERIOMEM_RNG_H */
--
2.54.0
^ permalink raw reply related
* Re: [PATCH v3] lib/raid/xor: x86: Add AVX-512 optimized xor_gen()
From: Eric Biggers @ 2026-06-15 20:10 UTC (permalink / raw)
To: x86
Cc: Christoph Hellwig, linux-crypto, David Laight, linux-raid,
Andrew Morton, linux-kernel
In-Reply-To: <20260615190338.26581-1-ebiggers@kernel.org>
On Mon, Jun 15, 2026 at 12:03:38PM -0700, Eric Biggers wrote:
> Note: for now I omitted the cpu_has_xfeatures() check that the AVX-512
> optimized crypto and CRC code does, since it's not implemented on
> User-Mode Linux and it's never been present in the RAID6 code either.
By the way, Sashiko keeps complaining about this decision.
Maybe the x86 maintainers have some advice here?
For context: on x86 processors, executing AVX or AVX512 instructions
requires not just that the CPU supports the feature, but also that the
operating system has set certain bits in XCR0. For example all EVEX
coded instructions (i.e. AVX-512) require XCR0=111xx111b. (See Intel
manual "2.6.11.1 State Dependent #UD".)
Therefore most of the kernel's AVX and AVX512 optimized code checks not
just X86_FEATURE_AVX* but also calls cpu_has_xfeatures() to check XCR0.
But "most" isn't all. The RAID6 code for example doesn't check
cpu_has_xfeatures(). So if you e.g. boot a kernel in QEMU using
"-cpu max,xsave=off", it already crashes when the RAID6 code does its
boot-time benchmark.
Part of the reason for that omission probably is that UML doesn't
provide an implementation of cpu_has_xfeatures(). And the x86 RAID (XOR
and RAID6) code is enabled on UML.
It could be implemented for UML by using the xgetbv instruction, like
what userspace programs do. (We'd also need to copy the XFEATURE_MASK_*
constants, as UML can't include arch/x86/include/asm/fpu/types.h)
But I wanted to ask: do we really care about the case where features are
"supported" but their XCR0 bits aren't set? Perhaps the kernel just
doesn't/shouldn't support weird cases like "-cpu max,xsave=off"?
If this case indeed needs to be handled, could we make things easier for
the kernel's AVX and AVX-512 optimized code? Currently AVX-512 needs:
if (boot_cpu_has(X86_FEATURE_AVX512F) &&
cpu_has_xfeatures(XFEATURE_MASK_FP | XFEATURE_MASK_SSE |
XFEATURE_MASK_YMM | XFEATURE_MASK_AVX512, NULL))
How about we make X86_FEATURE_AVX512F depend on XCR0=111xx111, and
X86_FEATURE_AVX depend on XCR0=xxxxx111? Then the cpu_has_xfeatures()
check wouldn't be needed. Is there any reason not to do that?
- Eric
^ permalink raw reply
* [PATCH v8 7/7] x86/sev: Add debugfs support for RMPOPT
From: Ashish Kalra @ 2026-06-15 19:50 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1781419998.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
Add a debugfs interface to report per-CPU RMPOPT status across all
system RAM.
To dump the per-CPU RMPOPT status for all system RAM:
/sys/kernel/debug/rmpopt# cat rmpopt-table
Memory @ 0GB: CPU(s): none
Memory @ 1GB: CPU(s): none
Memory @ 2GB: CPU(s): 0-1023
Memory @ 3GB: CPU(s): 0-1023
Memory @ 4GB: CPU(s): none
Memory @ 5GB: CPU(s): 0-1023
Memory @ 6GB: CPU(s): 0-1023
Memory @ 7GB: CPU(s): 0-1023
...
Memory @1025GB: CPU(s): 0-1023
Memory @1026GB: CPU(s): 0-1023
Memory @1027GB: CPU(s): 0-1023
Memory @1028GB: CPU(s): 0-1023
Memory @1029GB: CPU(s): 0-1023
Memory @1030GB: CPU(s): 0-1023
Memory @1031GB: CPU(s): 0-1023
Memory @1032GB: CPU(s): 0-1023
Memory @1033GB: CPU(s): 0-1023
Memory @1034GB: CPU(s): 0-1023
Memory @1035GB: CPU(s): 0-1023
Memory @1036GB: CPU(s): 0-1023
Memory @1037GB: CPU(s): 0-1023
Memory @1038GB: CPU(s): none
Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/virt/svm/sev.c | 128 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 128 insertions(+)
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 253a534b9a0d..b8b00c50ce41 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -20,6 +20,8 @@
#include <linux/amd-iommu.h>
#include <linux/nospec.h>
#include <linux/workqueue.h>
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
#include <asm/sev.h>
#include <asm/processor.h>
@@ -145,6 +147,15 @@ static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
static unsigned long snp_nr_leaked_pages;
+/* All users of rmpopt_report_cpumask must hold rmpopt_show_mutex. */
+static cpumask_t rmpopt_report_cpumask;
+static struct dentry *rmpopt_debugfs;
+static DEFINE_MUTEX(rmpopt_show_mutex);
+
+struct seq_paddr {
+ phys_addr_t next_seq_paddr;
+};
+
#undef pr_fmt
#define pr_fmt(fmt) "SEV-SNP: " fmt
@@ -587,6 +598,8 @@ static void rmpopt_cleanup(void)
cancel_delayed_work_sync(&rmpopt_delayed_work);
destroy_workqueue(rmpopt_wq);
+ debugfs_remove_recursive(rmpopt_debugfs);
+ rmpopt_debugfs = NULL;
cpus_read_lock();
@@ -635,6 +648,10 @@ static inline bool __rmpopt(u64 pa_start, u64 op_type)
: "a" (pa_start), "c" (op_type)
: "memory", "cc");
+ if (op_type == RMPOPT_FUNC_REPORT_STATUS)
+ assign_cpu(smp_processor_id(), &rmpopt_report_cpumask,
+ optimized);
+
return optimized;
}
@@ -669,6 +686,115 @@ static long rmpopt_leader_fn(void *arg)
return 0;
}
+/*
+ * 'val' is a system physical address.
+ */
+static void rmpopt_report_status(void *val)
+{
+ u64 pa_start = ALIGN_DOWN((u64)val, SZ_1G);
+ u64 op_type = RMPOPT_FUNC_REPORT_STATUS;
+
+ __rmpopt(pa_start, op_type);
+}
+
+/*
+ * start() can be called multiple times if allocated buffer has overflowed
+ * and bigger buffer is allocated.
+ */
+static void *rmpopt_table_seq_start(struct seq_file *seq, loff_t *pos)
+{
+ phys_addr_t end_paddr = rmpopt_pa_end;
+ struct seq_paddr *p = seq->private;
+
+ if (*pos == 0) {
+ p->next_seq_paddr = rmpopt_pa_start;
+ if (p->next_seq_paddr >= end_paddr)
+ return NULL;
+ return &p->next_seq_paddr;
+ }
+
+ if (p->next_seq_paddr >= end_paddr)
+ return NULL;
+
+ return &p->next_seq_paddr;
+}
+
+static void *rmpopt_table_seq_next(struct seq_file *seq, void *v, loff_t *pos)
+{
+ phys_addr_t end_paddr = rmpopt_pa_end;
+ phys_addr_t *curr_paddr = v;
+
+ (*pos)++;
+ *curr_paddr += SZ_1G;
+ if (*curr_paddr >= end_paddr)
+ return NULL;
+
+ return curr_paddr;
+}
+
+static void rmpopt_table_seq_stop(struct seq_file *seq, void *v)
+{
+}
+
+static int rmpopt_table_seq_show(struct seq_file *seq, void *v)
+{
+ phys_addr_t *curr_paddr = v;
+
+ guard(mutex)(&rmpopt_show_mutex);
+
+ seq_printf(seq, "Memory @%3lluGB: ",
+ *curr_paddr >> (get_order(SZ_1G) + PAGE_SHIFT));
+
+ /*
+ * Query all online CPUs rather than just rmpopt_cpumask (primary
+ * threads only). The RMPOPT instruction only needs to run on one
+ * thread per core for the optimization to take effect, but debugfs
+ * reporting requires the RMPOPT status across all CPUs.
+ * Performance is not a concern for this diagnostic interface.
+ *
+ * This is safe because RMPOPT_BASE MSR is per-core and
+ * snp_prepare() ensures all CPUs are online when the MSR is
+ * programmed during snp_setup_rmpopt().
+ */
+ cpumask_clear(&rmpopt_report_cpumask);
+ on_each_cpu_mask(cpu_online_mask, rmpopt_report_status,
+ (void *)*curr_paddr, true);
+
+ if (cpumask_empty(&rmpopt_report_cpumask))
+ seq_puts(seq, "CPU(s): none\n");
+ else
+ seq_printf(seq, "CPU(s): %*pbl\n", cpumask_pr_args(&rmpopt_report_cpumask));
+
+ return 0;
+}
+
+static const struct seq_operations rmpopt_table_seq_ops = {
+ .start = rmpopt_table_seq_start,
+ .next = rmpopt_table_seq_next,
+ .stop = rmpopt_table_seq_stop,
+ .show = rmpopt_table_seq_show
+};
+
+static int rmpopt_table_open(struct inode *inode, struct file *file)
+{
+ return seq_open_private(file, &rmpopt_table_seq_ops, sizeof(struct seq_paddr));
+}
+
+static const struct file_operations rmpopt_table_fops = {
+ .open = rmpopt_table_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release_private,
+};
+
+static void rmpopt_debugfs_setup(void)
+{
+ rmpopt_debugfs = debugfs_create_dir("rmpopt", arch_debugfs_dir);
+
+ debugfs_create_file("rmpopt-table", 0400, rmpopt_debugfs,
+ NULL, &rmpopt_table_fops);
+}
+
/*
* RMPOPT optimizations skip RMP checks at 1GB granularity if this
* range of memory does not contain any SNP guest memory.
@@ -874,6 +1000,8 @@ void snp_setup_rmpopt(void)
* optimizations on all physical memory.
*/
queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
+
+ rmpopt_debugfs_setup();
}
EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");
--
2.43.0
^ permalink raw reply related
* [PATCH v8 6/7] KVM: SEV: Perform RMP optimizations on SNP guest shutdown
From: Ashish Kalra @ 2026-06-15 19:50 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1781419998.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
Pages are converted from shared to private as SNP guests are launched.
This destroys exisiting RMPOPT optimizations in the regions where
pages are converted.
Conversely, guest pages are converted back to shared during SNP guest
termination and their region may become eligible for RMPOPT
optimization.
To take advantage of this, perform RMPOPT after guest termination.
Do it after a delay so that a single RMPOPT pass can be done if
multiple guests terminate in a short period of time.
Acked-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/kvm/svm/sev.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index e107f368ed2d..29af6f6e603c 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3005,6 +3005,8 @@ void sev_vm_destroy(struct kvm *kvm)
*/
if (snp_decommission_context(kvm))
return;
+
+ snp_rmpopt_all_physmem();
} else {
sev_unbind_asid(kvm, sev->handle);
}
--
2.43.0
^ permalink raw reply related
* [PATCH v8 5/7] x86/sev: Add interface to re-enable RMP optimizations.
From: Ashish Kalra @ 2026-06-15 19:49 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1781419998.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
RMPOPT table is a per-CPU table which indicates if 1GB regions of
physical memory are entirely hypervisor-owned or not.
When performing host memory accesses in hypervisor mode as well as
non-SNP guest mode, the processor may consult the RMPOPT table to
potentially skip an RMP access and improve performance.
Events such as RMPUPDATE can clear RMP optimizations. Add an interface
to re-enable those optimizations.
The interface uses mod_delayed_work() instead of queue_delayed_work()
so that the delay timer is reset on each call. This provides proper
batching semantics: re-optimization runs 10 seconds after the *last*
VM termination rather than after the first. mod_delayed_work() also
re-queues work that is already in-flight, so a re-scan request
during an active scan is not silently dropped.
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/include/asm/sev.h | 2 ++
arch/x86/virt/svm/sev.c | 15 +++++++++++++++
2 files changed, 17 insertions(+)
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 0d662221615a..a11306f25336 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -662,6 +662,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int pages)
__snp_leak_pages(pfn, pages, true);
}
int snp_prepare(void);
+void snp_rmpopt_all_physmem(void);
void snp_setup_rmpopt(void);
void snp_clear_rmpopt_configured(void);
void snp_shutdown(void);
@@ -682,6 +683,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int npages) {}
static inline void kdump_sev_callback(void) { }
static inline void snp_fixup_e820_tables(void) {}
static inline int snp_prepare(void) { return -ENODEV; }
+static inline void snp_rmpopt_all_physmem(void) {}
static inline void snp_setup_rmpopt(void) {}
static inline void snp_clear_rmpopt_configured(void) {}
static inline void snp_shutdown(void) {}
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index b63b639bfc30..253a534b9a0d 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -782,6 +782,21 @@ static void rmpopt_work_handler(struct work_struct *work)
free_cpumask_var(follower_mask);
}
+void snp_rmpopt_all_physmem(void)
+{
+ if (!cpu_feature_enabled(X86_FEATURE_RMPOPT) || !rmpopt_configured)
+ return;
+
+ guard(mutex)(&rmpopt_wq_mutex);
+
+ if (!rmpopt_wq)
+ return;
+
+ mod_delayed_work(rmpopt_wq, &rmpopt_delayed_work,
+ msecs_to_jiffies(RMPOPT_WORK_TIMEOUT));
+}
+EXPORT_SYMBOL_GPL(snp_rmpopt_all_physmem);
+
void snp_setup_rmpopt(void)
{
u64 rmpopt_base;
--
2.43.0
^ permalink raw reply related
* [PATCH v8 4/7] x86/sev: Add support to perform RMP optimizations asynchronously
From: Ashish Kalra @ 2026-06-15 19:49 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1781419998.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
When SEV-SNP is enabled, all writes to memory are checked to ensure
integrity of SNP guest memory. This imposes performance overhead on the
whole system.
RMPOPT is a new instruction that minimizes the performance overhead of
RMP checks on the hypervisor and on non-SNP guests by allowing RMP
checks to be skipped for 1GB regions of memory that are known not to
contain any SEV-SNP guest memory.
Add support for performing RMP optimizations asynchronously using a
dedicated workqueue.
Enable RMPOPT optimizations for up to 2TB of system RAM starting from
the lowest physical memory address aligned down to a 1GB boundary at
RMP initialization time. RMP checks can initially be skipped for 1GB
memory ranges that do not contain SEV-SNP guest memory (excluding
preassigned pages such as the RMP table and firmware pages). As SNP
guests are launched, RMPUPDATE will disable the corresponding RMPOPT
optimizations.
Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/virt/svm/sev.c | 230 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 227 insertions(+), 3 deletions(-)
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 1b5c18408f0b..b63b639bfc30 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -19,6 +19,7 @@
#include <linux/iommu.h>
#include <linux/amd-iommu.h>
#include <linux/nospec.h>
+#include <linux/workqueue.h>
#include <asm/sev.h>
#include <asm/processor.h>
@@ -125,9 +126,20 @@ static void *rmp_bookkeeping __ro_after_init;
static u64 probed_rmp_base, probed_rmp_size;
static cpumask_t rmpopt_cpumask;
-static phys_addr_t rmpopt_pa_start;
+static phys_addr_t rmpopt_pa_start, rmpopt_pa_end;
static bool rmpopt_configured;
+enum rmpopt_function {
+ RMPOPT_FUNC_VERIFY_AND_REPORT_STATUS,
+ RMPOPT_FUNC_REPORT_STATUS
+};
+
+#define RMPOPT_WORK_TIMEOUT 10000
+
+static struct workqueue_struct *rmpopt_wq;
+static struct delayed_work rmpopt_delayed_work;
+static DEFINE_MUTEX(rmpopt_wq_mutex);
+
static LIST_HEAD(snp_leaked_pages_list);
static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
@@ -568,6 +580,14 @@ static void rmpopt_cleanup(void)
{
int cpu;
+ guard(mutex)(&rmpopt_wq_mutex);
+
+ if (!rmpopt_wq)
+ return;
+
+ cancel_delayed_work_sync(&rmpopt_delayed_work);
+ destroy_workqueue(rmpopt_wq);
+
cpus_read_lock();
for_each_cpu(cpu, &rmpopt_cpumask)
@@ -576,7 +596,8 @@ static void rmpopt_cleanup(void)
cpus_read_unlock();
cpumask_clear(&rmpopt_cpumask);
- rmpopt_pa_start = 0;
+ rmpopt_pa_start = rmpopt_pa_end = 0;
+ rmpopt_wq = NULL;
}
void snp_shutdown(void)
@@ -599,6 +620,168 @@ void snp_clear_rmpopt_configured(void)
rmpopt_configured = false;
}
+/*
+ * RMPOPT: F2 0F 01 FC
+ * Input: RAX = system physical address (1GB aligned)
+ * RCX = operation type
+ * Output: CF set if the range was optimized
+ */
+static inline bool __rmpopt(u64 pa_start, u64 op_type)
+{
+ bool optimized;
+
+ asm volatile(".byte 0xf2, 0x0f, 0x01, 0xfc"
+ : "=@ccc" (optimized)
+ : "a" (pa_start), "c" (op_type)
+ : "memory", "cc");
+
+ return optimized;
+}
+
+static void rmpopt(u64 pa)
+{
+ u64 pa_start = ALIGN_DOWN(pa, SZ_1G);
+ u64 op_type = RMPOPT_FUNC_VERIFY_AND_REPORT_STATUS;
+
+ __rmpopt(pa_start, op_type);
+}
+
+/*
+ * 'val' is a system physical address.
+ */
+static void rmpopt_smp(void *val)
+{
+ rmpopt((u64)val);
+}
+
+/*
+ * Leader function for work_on_cpu(): runs the full RMPOPT scan in
+ * process context on a CPU that has RMPOPT_BASE MSR programmed.
+ */
+static long rmpopt_leader_fn(void *arg)
+{
+ phys_addr_t pa;
+
+ for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G) {
+ rmpopt(pa);
+ cond_resched();
+ }
+ return 0;
+}
+
+/*
+ * RMPOPT optimizations skip RMP checks at 1GB granularity if this
+ * range of memory does not contain any SNP guest memory.
+ */
+static void rmpopt_work_handler(struct work_struct *work)
+{
+ cpumask_var_t follower_mask;
+ phys_addr_t pa;
+ int this_cpu;
+
+ pr_info("Attempt RMP optimizations on physical address range @1GB alignment [0x%016llx - 0x%016llx]\n",
+ rmpopt_pa_start, rmpopt_pa_end);
+
+ if (!alloc_cpumask_var(&follower_mask, GFP_KERNEL))
+ return;
+
+ /*
+ * RMPOPT scans the RMP table, stores the result of the scan in the
+ * reserved processor memory. The RMP scan is the most expensive
+ * part. If a second RMPOPT occurs, it can skip the expensive scan
+ * if they can see a cached result in the reserved processor memory.
+ *
+ * Do RMPOPT on one CPU alone. Then, follow that up with RMPOPT
+ * on every other primary thread. Followers are "designed to"
+ * skip the scan if they see the "cached" scan results.
+ */
+ cpumask_copy(follower_mask, &rmpopt_cpumask);
+
+ /*
+ * Pin the worker to the current CPU for the leader loop so that
+ * this_cpu remains valid and the RMPOPT instruction executes on
+ * the correct CPU.
+ *
+ * Use migrate_disable() rather than get_cpu() to prevent
+ * migration while still allowing preemption.
+ */
+ migrate_disable();
+ this_cpu = smp_processor_id();
+
+ if (cpumask_test_cpu(this_cpu, follower_mask)) {
+ /*
+ * Current CPU is a primary thread in rmpopt_cpumask.
+ * Run leader locally and remove from follower mask.
+ */
+ cpumask_clear_cpu(this_cpu, follower_mask);
+
+ for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G) {
+ rmpopt(pa);
+ cond_resched();
+ }
+ } else if (cpumask_intersects(topology_sibling_cpumask(this_cpu),
+ follower_mask)) {
+ /*
+ * Current CPU is a sibling thread whose primary is in
+ * rmpopt_cpumask. RMPOPT_BASE MSR is per-core, so it
+ * is safe to run the leader locally. Remove the sibling's
+ * primary from the follower mask as this core is already
+ * covered by the leader.
+ */
+ cpumask_andnot(follower_mask, follower_mask,
+ topology_sibling_cpumask(this_cpu));
+
+ for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G) {
+ rmpopt(pa);
+ cond_resched();
+ }
+ } else {
+ /*
+ * Current CPU does not have RMPOPT_BASE MSR programmed.
+ * Pick an explicit leader from the cpumask to avoid #UD.
+ * Use work_on_cpu() to run in process context on the leader,
+ * avoiding IPI latency.
+ */
+ int leader_cpu = cpumask_first(follower_mask);
+
+ if (WARN_ON_ONCE(leader_cpu >= nr_cpu_ids)) {
+ migrate_enable();
+ goto out;
+ }
+
+ cpumask_clear_cpu(leader_cpu, follower_mask);
+
+ /* Release migration pin before work_on_cpu(). */
+ migrate_enable();
+
+ work_on_cpu(leader_cpu, rmpopt_leader_fn, NULL);
+
+ goto followers;
+ }
+
+ migrate_enable();
+
+followers:
+ /*
+ * Followers: run RMPOPT on remaining cores.
+ * CPU hotplug is disabled while SNP is active
+ * (cpu_hotplug_disable() in __sev_snp_init_locked()),
+ * so cpus_read_lock() is uncontended.
+ */
+ cpus_read_lock();
+ for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G) {
+ on_each_cpu_mask(follower_mask, rmpopt_smp,
+ (void *)pa, true);
+
+ /* Give a chance for other threads to run */
+ cond_resched();
+ }
+ cpus_read_unlock();
+
+out:
+ free_cpumask_var(follower_mask);
+}
+
void snp_setup_rmpopt(void)
{
u64 rmpopt_base;
@@ -607,11 +790,37 @@ void snp_setup_rmpopt(void)
if (!cpu_feature_enabled(X86_FEATURE_RMPOPT) || !rmpopt_configured)
return;
+ guard(mutex)(&rmpopt_wq_mutex);
+
+ /*
+ * Guard against re-initialization. When SNP_SHUTDOWN_EX is issued
+ * with x86_snp_shutdown=0, snp_shutdown() is not called and
+ * rmpopt_cleanup() is skipped, but snp_initialized is still cleared.
+ * A subsequent __sev_snp_init_locked() would call snp_setup_rmpopt()
+ * again, leaking the existing workqueue, delayed work, debugfs
+ * entries, and cpumask state.
+ */
+ if (rmpopt_wq)
+ return;
+
+ /*
+ * Create an RMPOPT-specific workqueue to avoid scheduling
+ * RMPOPT workitem on the global system workqueue.
+ */
+ rmpopt_wq = alloc_workqueue("rmpopt_wq", WQ_UNBOUND, 1);
+ if (!rmpopt_wq) {
+ pr_err("Failed to allocate RMPOPT workqueue\n");
+ return;
+ }
+
+ INIT_DELAYED_WORK(&rmpopt_delayed_work, rmpopt_work_handler);
+
cpus_read_lock();
/*
* The RMPOPT_BASE MSR is per-core, so only one thread per core needs
- * to set up the RMPOPT_BASE MSR.
+ * to set up the RMPOPT_BASE MSR. Likewise, only one thread per core
+ * needs to issue the RMPOPT instruction.
*
* Note: only online primary threads are included. If a core's
* primary thread is offline, that core is not covered. CPU hotplug
@@ -635,6 +844,21 @@ void snp_setup_rmpopt(void)
WARN_ON_ONCE(wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, rmpopt_base));
cpus_read_unlock();
+
+ rmpopt_pa_end = ALIGN(PFN_PHYS(max_pfn), SZ_1G);
+
+ /* Limit memory scanning to 2TB of RAM */
+ if ((rmpopt_pa_end - rmpopt_pa_start) > SZ_2T) {
+ pr_info("RMPOPT coverage limited to 2TB; memory above 0x%llx not optimized\n",
+ rmpopt_pa_start + SZ_2T);
+ rmpopt_pa_end = rmpopt_pa_start + SZ_2T;
+ }
+
+ /*
+ * Once all per-CPU RMPOPT tables have been configured, enable RMPOPT
+ * optimizations on all physical memory.
+ */
+ queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
}
EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");
--
2.43.0
^ permalink raw reply related
* [PATCH v8 3/7] crypto/ccp: Disable CPU hotplug while SNP is active
From: Ashish Kalra @ 2026-06-15 19:49 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1781419998.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
The SEV firmware enumerates the CPUs at SNP initialization and is not
aware of the OS bringing CPUs online or offline afterwards, so OS CPU
hotplug can diverge from the firmware's expectations and break SNP.
Disable CPU hotplug while SNP is active.
SNP is fully torn down only on the SNP_SHUTDOWN_EX x86_snp_shutdown
path; the legacy path leaves SNP enabled in hardware while clearing
snp_initialized, so __sev_snp_init_locked() can run again. Track the
disable with a flag so it is balanced by a matching enable rather than
stacked, and re-enable hotplug only on the x86_snp_shutdown path, after
snp_shutdown() has cleared the per-core RMPOPT_BASE MSRs with hotplug
still disabled.
This also keeps the CPU set stable for the asynchronous RMPOPT scan
added later in this series, and ensures cpus_read_lock() in the scan
is uncontended.
Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
drivers/crypto/ccp/sev-dev.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c
index 217b6b19802e..c8c3c577463c 100644
--- a/drivers/crypto/ccp/sev-dev.c
+++ b/drivers/crypto/ccp/sev-dev.c
@@ -106,6 +106,9 @@ struct snp_hv_fixed_pages_entry {
static LIST_HEAD(snp_hv_fixed_pages);
+/* Set while SNP has CPU hotplug disabled. */
+static bool snp_cpu_hotplug_disabled;
+
/* Trusted Memory Region (TMR):
* The TMR is a 1MB area that must be 1MB aligned. Use the page allocator
* to allocate the memory, which will return aligned memory for the specified
@@ -1479,6 +1482,17 @@ static int __sev_snp_init_locked(int *error, unsigned int max_snp_asid)
snp_hv_fixed_pages_state_update(sev, HV_FIXED);
+ /*
+ * Disable CPU hotplug while SNP is active. Guard against stacking
+ * the disable count: the legacy SNP_SHUTDOWN_EX path clears
+ * snp_initialized without re-enabling hotplug, so this can run
+ * again while hotplug is already disabled.
+ */
+ if (!snp_cpu_hotplug_disabled) {
+ cpu_hotplug_disable();
+ snp_cpu_hotplug_disabled = true;
+ }
+
snp_setup_rmpopt();
sev->snp_initialized = true;
@@ -2083,8 +2097,21 @@ static int __sev_snp_shutdown_locked(int *error, bool panic)
}
if (data.x86_snp_shutdown) {
- if (!panic)
+ if (!panic) {
snp_shutdown();
+ /*
+ * snp_shutdown() fully tears SNP down (clear_rmp()) and
+ * has already cleared the per-core RMPOPT_BASE MSRs via
+ * rmpopt_cleanup() with hotplug still disabled. Re-enable
+ * CPU hotplug now. On the legacy path SNP stays
+ * enabled in hardware, so hotplug is correctly left
+ * disabled.
+ */
+ if (snp_cpu_hotplug_disabled) {
+ cpu_hotplug_enable();
+ snp_cpu_hotplug_disabled = false;
+ }
+ }
snp_hv_fixed_pages_state_update(sev, ALLOCATED);
} else {
/*
--
2.43.0
^ permalink raw reply related
* [PATCH v8 2/7] x86/sev: Initialize RMPOPT configuration MSRs
From: Ashish Kalra @ 2026-06-15 19:48 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1781419998.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
The new RMPOPT instruction helps manage per-CPU RMP optimization
structures inside the CPU. It takes a 1GB-aligned physical address
and either returns the status of the optimizations or tries to enable
the optimizations.
Per-CPU RMPOPT tables support at most 2 TB of addressable memory for
RMP optimizations.
Initialize the per-CPU RMPOPT table base to the starting physical
address. This enables RMP optimization for up to 2 TB of system RAM on
all CPUs.
Additionally, add support to setup and enable RMPOPT once SNP is
enabled and initialized.
Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/coco/core.c | 2 +
arch/x86/include/asm/msr-index.h | 3 ++
arch/x86/include/asm/sev.h | 4 ++
arch/x86/virt/svm/sev.c | 70 ++++++++++++++++++++++++++++++++
drivers/crypto/ccp/sev-dev.c | 3 ++
5 files changed, 82 insertions(+)
diff --git a/arch/x86/coco/core.c b/arch/x86/coco/core.c
index 989ca9f72ba3..8c1393ddc5df 100644
--- a/arch/x86/coco/core.c
+++ b/arch/x86/coco/core.c
@@ -16,6 +16,7 @@
#include <asm/archrandom.h>
#include <asm/coco.h>
#include <asm/processor.h>
+#include <asm/sev.h>
enum cc_vendor cc_vendor __ro_after_init = CC_VENDOR_NONE;
SYM_PIC_ALIAS(cc_vendor);
@@ -172,6 +173,7 @@ static void amd_cc_platform_clear(enum cc_attr attr)
switch (attr) {
case CC_ATTR_HOST_SEV_SNP:
cc_flags.host_sev_snp = 0;
+ snp_clear_rmpopt_configured();
break;
default:
break;
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 86554de9a3f5..28540744f1eb 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -761,6 +761,9 @@
#define MSR_AMD64_SEG_RMP_ENABLED_BIT 0
#define MSR_AMD64_SEG_RMP_ENABLED BIT_ULL(MSR_AMD64_SEG_RMP_ENABLED_BIT)
#define MSR_AMD64_RMP_SEGMENT_SHIFT(x) (((x) & GENMASK_ULL(13, 8)) >> 8)
+#define MSR_AMD64_RMPOPT_BASE 0xc0010139
+#define MSR_AMD64_RMPOPT_ENABLE_BIT 0
+#define MSR_AMD64_RMPOPT_ENABLE BIT_ULL(MSR_AMD64_RMPOPT_ENABLE_BIT)
#define MSR_SVSM_CAA 0xc001f000
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 594cfa19cbd4..0d662221615a 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -662,6 +662,8 @@ static inline void snp_leak_pages(u64 pfn, unsigned int pages)
__snp_leak_pages(pfn, pages, true);
}
int snp_prepare(void);
+void snp_setup_rmpopt(void);
+void snp_clear_rmpopt_configured(void);
void snp_shutdown(void);
#else
static inline bool snp_probe_rmptable_info(void) { return false; }
@@ -680,6 +682,8 @@ static inline void snp_leak_pages(u64 pfn, unsigned int npages) {}
static inline void kdump_sev_callback(void) { }
static inline void snp_fixup_e820_tables(void) {}
static inline int snp_prepare(void) { return -ENODEV; }
+static inline void snp_setup_rmpopt(void) {}
+static inline void snp_clear_rmpopt_configured(void) {}
static inline void snp_shutdown(void) {}
#endif
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 8bcdce98f6dc..1b5c18408f0b 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -124,6 +124,10 @@ static void *rmp_bookkeeping __ro_after_init;
static u64 probed_rmp_base, probed_rmp_size;
+static cpumask_t rmpopt_cpumask;
+static phys_addr_t rmpopt_pa_start;
+static bool rmpopt_configured;
+
static LIST_HEAD(snp_leaked_pages_list);
static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
@@ -490,7 +494,12 @@ static bool __init setup_rmptable(void)
if (rmp_cfg & MSR_AMD64_SEG_RMP_ENABLED) {
if (!setup_segmented_rmptable())
return false;
+ rmpopt_configured = true;
} else {
+ /*
+ * RMPOPT requires a segmented RMP table, so leave
+ * rmpopt_configured clear on contiguous RMP systems.
+ */
if (!setup_contiguous_rmptable())
return false;
}
@@ -555,6 +564,21 @@ int snp_prepare(void)
}
EXPORT_SYMBOL_FOR_MODULES(snp_prepare, "ccp");
+static void rmpopt_cleanup(void)
+{
+ int cpu;
+
+ cpus_read_lock();
+
+ for_each_cpu(cpu, &rmpopt_cpumask)
+ WARN_ON_ONCE(wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, 0));
+
+ cpus_read_unlock();
+
+ cpumask_clear(&rmpopt_cpumask);
+ rmpopt_pa_start = 0;
+}
+
void snp_shutdown(void)
{
u64 syscfg;
@@ -563,11 +587,57 @@ void snp_shutdown(void)
if (syscfg & MSR_AMD64_SYSCFG_SNP_EN)
return;
+ rmpopt_cleanup();
+
clear_rmp();
on_each_cpu(mfd_reconfigure, NULL, 1);
}
EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
+void snp_clear_rmpopt_configured(void)
+{
+ rmpopt_configured = false;
+}
+
+void snp_setup_rmpopt(void)
+{
+ u64 rmpopt_base;
+ int cpu;
+
+ if (!cpu_feature_enabled(X86_FEATURE_RMPOPT) || !rmpopt_configured)
+ return;
+
+ cpus_read_lock();
+
+ /*
+ * The RMPOPT_BASE MSR is per-core, so only one thread per core needs
+ * to set up the RMPOPT_BASE MSR.
+ *
+ * Note: only online primary threads are included. If a core's
+ * primary thread is offline, that core is not covered. CPU hotplug
+ * is not currently supported with SNP enabled.
+ */
+
+ for_each_online_cpu(cpu)
+ if (topology_is_primary_thread(cpu))
+ cpumask_set_cpu(cpu, &rmpopt_cpumask);
+
+ rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
+ rmpopt_base = rmpopt_pa_start | MSR_AMD64_RMPOPT_ENABLE;
+
+ /*
+ * Per-CPU RMPOPT tables support at most 2 TB of addressable memory
+ * for RMP optimizations. Initialize the per-CPU RMPOPT table base
+ * to the starting physical address to enable RMP optimizations for
+ * up to 2 TB of system RAM on all CPUs.
+ */
+ for_each_cpu(cpu, &rmpopt_cpumask)
+ WARN_ON_ONCE(wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, rmpopt_base));
+
+ cpus_read_unlock();
+}
+EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");
+
/*
* Do the necessary preparations which are verified by the firmware as
* described in the SNP_INIT_EX firmware command description in the SNP
diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c
index 78f98aee7a66..217b6b19802e 100644
--- a/drivers/crypto/ccp/sev-dev.c
+++ b/drivers/crypto/ccp/sev-dev.c
@@ -1478,6 +1478,9 @@ static int __sev_snp_init_locked(int *error, unsigned int max_snp_asid)
}
snp_hv_fixed_pages_state_update(sev, HV_FIXED);
+
+ snp_setup_rmpopt();
+
sev->snp_initialized = true;
dev_dbg(sev->dev, "SEV-SNP firmware initialized, SEV-TIO is %s\n",
data.tio_en ? "enabled" : "disabled");
--
2.43.0
^ permalink raw reply related
* [PATCH v8 1/7] x86/cpufeatures: Add X86_FEATURE_RMPOPT feature flag
From: Ashish Kalra @ 2026-06-15 19:48 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1781419998.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
Add a flag indicating whether RMPOPT instruction is supported.
RMPOPT is a new instruction that reduces the performance overhead of
RMP checks for the hypervisor and non-SNP guests by allowing those
checks to be skipped when 1-GB memory regions are known to contain no
SEV-SNP guest memory.
For more information on the RMPOPT instruction, see the AMD64 RMPOPT
technical documentation.
Suggested-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/include/asm/cpufeatures.h | 2 +-
arch/x86/kernel/cpu/scattered.c | 1 +
tools/arch/x86/include/asm/cpufeatures.h | 2 +-
3 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 1d506e5d6f46..794cc96b8493 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -76,7 +76,7 @@
#define X86_FEATURE_K8 ( 3*32+ 4) /* Opteron, Athlon64 */
#define X86_FEATURE_ZEN5 ( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
#define X86_FEATURE_ZEN6 ( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free ( 3*32+ 7) */
+#define X86_FEATURE_RMPOPT ( 3*32+ 7) /* Support for AMD RMPOPT instruction */
#define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
#define X86_FEATURE_UP ( 3*32+ 9) /* "up" SMP kernel running on UP */
#define X86_FEATURE_ART ( 3*32+10) /* "art" Always running timer (ART) */
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index 937129ce6a96..021c0bf22de2 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -67,6 +67,7 @@ static const struct cpuid_bit cpuid_bits[] = {
{ X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 },
{ X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 },
{ X86_FEATURE_AMD_LBR_PMC_FREEZE, CPUID_EAX, 2, 0x80000022, 0 },
+ { X86_FEATURE_RMPOPT, CPUID_EDX, 0, 0x80000025, 0 },
{ X86_FEATURE_AMD_HTR_CORES, CPUID_EAX, 30, 0x80000026, 0 },
{ 0, 0, 0, 0, 0 }
};
diff --git a/tools/arch/x86/include/asm/cpufeatures.h b/tools/arch/x86/include/asm/cpufeatures.h
index 86d17b195e79..7ce681af1dd7 100644
--- a/tools/arch/x86/include/asm/cpufeatures.h
+++ b/tools/arch/x86/include/asm/cpufeatures.h
@@ -76,7 +76,7 @@
#define X86_FEATURE_K8 ( 3*32+ 4) /* Opteron, Athlon64 */
#define X86_FEATURE_ZEN5 ( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
#define X86_FEATURE_ZEN6 ( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free ( 3*32+ 7) */
+#define X86_FEATURE_RMPOPT ( 3*32+ 7) /* Support for AMD RMPOPT instruction */
#define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
#define X86_FEATURE_UP ( 3*32+ 9) /* "up" SMP kernel running on UP */
#define X86_FEATURE_ART ( 3*32+10) /* "art" Always running timer (ART) */
--
2.43.0
^ permalink raw reply related
* [PATCH v8 0/7] Add RMPOPT support.
From: Ashish Kalra @ 2026-06-15 19:47 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
From: Ashish Kalra <ashish.kalra@amd.com>
In the SEV-SNP architecture, hypervisor and non-SNP guests are subject
to RMP checks on writes to provide integrity of SEV-SNP guest memory.
The RMPOPT architecture enables optimizations whereby the RMP checks
can be skipped if 1GB regions of memory are known to not contain any
SNP guest memory.
RMPOPT is a new instruction designed to minimize the performance
overhead of RMP checks for the hypervisor and non-SNP guests.
RMPOPT instruction currently supports two functions. In case of the
verify and report status function the CPU will read the RMP contents,
verify the entire 1GB region starting at the provided SPA is HV-owned.
For the entire 1GB region it checks that all RMP entries in this region
are HV-owned (i.e, not in assigned state) and then accordingly updates
the RMPOPT table to indicate if optimization has been enabled and
provide indication to software if the optimization was successful.
In case of report status function, the CPU returns the optimization
status for the 1GB region.
The RMPOPT table is managed by a combination of software and hardware.
Software uses the RMPOPT instruction to set bits in the table,
indicating that regions of memory are entirely HV-owned. Hardware
automatically clears bits in the RMPOPT table when RMP contents are
changed during RMPUPDATE instruction.
For more information on the RMPOPT instruction, see the AMD64 RMPOPT
technical documentation.
As SNP is enabled by default the hypervisor and non-SNP guests are
subject to RMP write checks to provide integrity of SNP guest memory.
This patch-series adds support to enable RMP optimizations for up to
2TB of system RAM across the system and allow RMPUPDATE to disable
those optimizations as SNP guests are launched.
Support for RAM larger than 2 TB will be added in follow-on series.
This series also adds support to disable CPU hotplug while SNP is
active, as the SEV firmware enumerates CPUs at SNP initialization and is
not aware of the OS bringing CPUs online or offline afterwards. This
also keeps the set of CPUs stable for the asynchronous RMPOPT scan, so
the per-core RMPOPT_BASE MSRs programmed during setup remain valid.
This series also introduces support to re-enable RMP optimizations
during SNP guest termination, after guest pages have been converted
back to shared.
RMP optimizations are performed asynchronously by queuing work on a
dedicated workqueue after a 10 second delay.
Delaying work allows batching of multiple SNP guest terminations.
Once 1GB hugetlb guest_memfd support is merged, support for
re-enabling RMPOPT optimizations during 1GB page cleanup will be added
in follow-on series.
Additionally add debugfs interface to report per-CPU RMPOPT status
across all system RAM.
v8:
- Add a new patch to disable CPU hotplug while SNP is active, keeping
the CPU set stable for the RMPOPT work handler.
- Drop the setup_clear_cpu_cap(X86_FEATURE_RMPOPT) calls; the
rmpopt_configured bool is the runtime guard.
- WARN_ON_ONCE() on the RMPOPT_BASE MSR writes that previously ignored
their return value.
- Run the RMPOPT leader scan via work_on_cpu() instead of
smp_call_function_single() so it executes in process context. This
fixes the AB-BA deadlock between migrate_disable() and cpus_read_lock()
and avoids running the long RMP scan in IPI context with interrupts
disabled.
- Use mod_delayed_work() in snp_rmpopt_all_physmem() so the batching
delay tracks the last SNP guest termination.
Sashiko AI code review identified several of the above issues.
v7:
- Sync tools/arch/x86/include/asm/cpufeatures.h to mirror the kernel
header for X86_FEATURE_RMPOPT.
- Fix commit title to use X86_FEATURE_RMPOPT to match the code
(was X86_FEATURE_AMD_RMPOPT).
- Add static bool rmpopt_configured, set only when segmented RMP setup
succeeds in setup_rmptable(). Check rmpopt_configured alongside
cpu_feature_enabled(X86_FEATURE_RMPOPT) in snp_setup_rmpopt() and
snp_rmpopt_all_physmem(), because setup_clear_cpu_cap() is unreliable
after alternatives are patched. Add snp_clear_rmpopt_configured()
called from amd_cc_platform_clear() when CC_ATTR_HOST_SEV_SNP is
cleared. Do not use __ro_after_init on rmpopt_configured since the
writer snp_clear_rmpopt_configured() is not __init.
- Add cond_resched() to all three leader loops in rmpopt_work_handler()
to prevent soft lockups on systems with up to 2TB of RAM.
- Add comment above __rmpopt() documenting the RMPOPT instruction
encoding (F2 0F 01 FC) and register interface (RAX = system physical
address input, RCX = operation type input, RFLAGS.CF = output).
Note: RMPOPT does not modify RAX unlike PVALIDATE/RMPUPDATE, so
the existing "a" (input-only) constraint is correct.
Sashiko AI code review identified several of the above issues.
v6:
- Drop wrmsrq_on_cpus() helper; use for_each_cpu() with wrmsrq_on_cpu()
instead, as RMPOPT_BASE MSR programming is not performance-critical.
- Rewrite rmpopt_work_handler() leader selection to use a local
follower_mask copy instead of modifying the global rmpopt_cpumask.
This eliminates the current_cpu_cleared tracking and the restore at
the end, and removes the need for synchronization comments about
transient cpumask inconsistency.
- Add three-way leader selection in rmpopt_work_handler():
1. Current CPU is a primary thread in cpumask: run leader locally.
2. Current CPU is a sibling thread whose primary is in cpumask:
run leader locally (RMPOPT_BASE MSR is per-core), remove the
primary from followers via cpumask_andnot(topology_sibling_cpumask).
3. Current CPU's core has no RMPOPT_BASE MSR programmed: pick an
explicit leader via cpumask_first() + smp_call_function_single()
to avoid #UD, with cpus_read_lock() around the IPI loop.
- Add WARN_ON_ONCE guard for empty cpumask in the explicit leader
fallback path, with migrate_enable() before goto out.
- Add .llseek = seq_lseek to rmpopt_table_fops for consistency with
other seq_file-based debugfs files and to support tools like "less".
- Change debugfs file permissions from 0444 to 0400 to restrict access
to root only.
- Add comment in rmpopt_table_seq_show() explaining why cpu_online_mask
is safe: RMPOPT_BASE MSR is per-core and snp_prepare() ensures all
CPUs are online when the MSR is programmed.
Sashiko AI code review identified several of the above issues.
v5:
- Introduce rmpopt_cleanup() to tear down workqueue, debugfs, cpumask,
and MSR state, called from snp_shutdown().
- Introduce rmpopt_wq_mutex to serialize snp_setup_rmpopt(),
snp_rmpopt_all_physmem(), and rmpopt_cleanup().
- Introduce rmpopt_show_mutex to serialize debugfs reporting of
rmpopt_report_cpumask.
- Move snp_rmpopt_all_physmem() call after SNP DECOMMISSION during
guest shutdown.
- Use migrate_disable()/migrate_enable() for CPU pinning in the
rmpopt_work_handler() leader loop to maintain CPU affinity without
disabling preemption for the entire RMPOPT scan.
- Add cpus_read_lock()/cpus_read_unlock() around the follower
on_each_cpu_mask() loop in rmpopt_work_handler().
- Guard snp_setup_rmpopt() against re-initialization when
SNP_SHUTDOWN_EX with x86_snp_shutdown=0 skips rmpopt_cleanup()
but clears snp_initialized, preventing workqueue and resource
leaks on repeated init/shutdown cycles.
- Replace setup_clear_cpu_cap() with pr_err() on alloc_workqueue()
failure in snp_setup_rmpopt(), as setup_clear_cpu_cap() cannot be
used after alternatives are patched; callers check rmpopt_wq != NULL
as the runtime guard instead.
- Add pr_info() when RMPOPT coverage is capped at 2TB.
- Add comments noting CPU hotplug is not supported with SNP enabled
and only online primary threads are covered by rmpopt_cpumask.
- Add comment in setup_rmptable() noting Segmented RMP must be
enabled to enable RMPOPT.
- Simplify cpumask setup loop to set if primary thread rather than
skip if not primary.
- Improve grammar and clarity in snp_setup_rmpopt() comments.
- Added Reviewed-by's.
Sashiko AI code review identified several of the above issues.
v4:
- Add new wrmsrq_on_cpus() helper to write same u64 value to a
per-CPU MSR across a cpumask without per-cpu struct allocation
overhead.
- Rename configure_and_enable_rmpopt() to snp_setup_rmpopt().
- Use wrmsrq_on_cpus() instead of wrmsrq_on_cpu() loop for
programming RMPOPT_BASE MSRs.
- Add setup_clear_cpu_cap(X86_FEATURE_RMPOPT) if segmented RMP
setup fails or workqueue allocation fails.
- Add X86_FEATURE_RMPOPT feature clear logic in amd_cc_platform_clear()
for CC_ATTR_HOST_SEV_SNP.
- All of the above allow checking for only X86_FEATURE_RMPOPT for both
RMPOPT setup/enable and RMP re-optimizations.
- Rename snp_perform_rmp_optimization() to snp_rmpopt_all_physmem().
- Split rmpopt() into rmpopt() and rmpopt_smp() for SMP callback use.
- Introduce separate rmpopt_report_cpumask for debugfs reporting,
distinct from rmpopt_cpumask used for primary thread tracking.
- Remove snp_perform_rmp_optimization() call from __sev_snp_init_locked()
and instead setup and enable RMPOPT after SNP is enabled and
initialized.
v3:
- Drop all RMPOPT kthread support and introduce adding custom and
dedicated workqueue to schedule delayed and asynchronous RMPOPT work.
- Drop the guest_memfd inode cleanup interface and add support to
re-enable RMP optimizations during guest shutdown using the
asynchronous and delayed workqueue interface.
- Introduce new __rmpopt() helper and rmpopt() and
rmpopt_report_status() wrappers on top which use rax and rcx
parameters to closely match RMPOPT specs.
- Use new optimized RMPOPT loop to issue RMPOPT instructions on all
system RAM upto 2TB and all CPUs, by optimizing each range on one CPU
first, then let other CPUs execute RMPOPT in parallel so they can skip
most work as the range has already been optimized.
- Also add support for running the optimized RMPOPT loop only on
one thread per core.
- Replace all PUD_SIZE references with SZ_1G to conform to 1GB regions
as specified by RMPOPT specifications and not be dependent on PUD_SIZE
which makes the RMPOPT patch-set independent of x86 page table sizes.
- Use wrmsrq_on_cpu() to program the RMPOPT_BASE MSR registers on
all CPUs that removes all ugly casting to use on_each_cpu_mask().
- Fix inline commits and patch commit messages
v2:
- Drop all NUMA and Socket configuration and enablement support and
enable RMPOPT support for up to 2TB of system RAM.
- Drop get_cpumask_of_primary_threads() and enable per-core RMPOPT
base MSRs and issue RMPOPT instruction on all CPUs.
- Drop the configfs interface to manually re-enable RMP optimizations.
- Add new guest_memfd cleanup interface to automatically re-enable
RMP optimizations during guest shutdown.
- Include references to the public RMPOPT documentation.
- Move debugfs directory for RMPOPT under architecuture specific
parent directory.
Ashish Kalra (7):
x86/cpufeatures: Add X86_FEATURE_RMPOPT feature flag
x86/sev: Initialize RMPOPT configuration MSRs
crypto/ccp: Disable CPU hotplug while SNP is active
x86/sev: Add support to perform RMP optimizations asynchronously
x86/sev: Add interface to re-enable RMP optimizations.
KVM: SEV: Perform RMP optimizations on SNP guest shutdown
x86/sev: Add debugfs support for RMPOPT
arch/x86/coco/core.c | 2 +
arch/x86/include/asm/cpufeatures.h | 2 +-
arch/x86/include/asm/msr-index.h | 3 +
arch/x86/include/asm/sev.h | 6 +
arch/x86/kernel/cpu/scattered.c | 1 +
arch/x86/kvm/svm/sev.c | 2 +
arch/x86/virt/svm/sev.c | 437 +++++++++++++++++++++++
drivers/crypto/ccp/sev-dev.c | 32 +-
tools/arch/x86/include/asm/cpufeatures.h | 2 +-
9 files changed, 484 insertions(+), 3 deletions(-)
--
2.43.0
^ permalink raw reply
* [RFC PATCH v1] PCI: Remove pcie_flr() and convert all callers to use pcie_reset_flr()
From: Farhan Ali @ 2026-06-15 19:21 UTC (permalink / raw)
To: linux-kernel, linux-pci
Cc: helgaas, giovanni.cabiddu, alifm, herbert, davem,
dennis.dalessandro, jgg, leon, vikas.gupta, edumazet, kuba,
pabeni, michael.chan, pavan.chebbi, claudiu.manoil,
vladimir.oltean, wei.fang, xiaoning.wang, anthony.l.nguyen,
przemyslaw.kitszel, kys, haiyangz, wei.liu, decui, longli,
richardcochran, Andrew Lunn, Bjorn Helgaas, open list:QAT DRIVER,
open list:CRYPTO API, open list:HFI1 DRIVER,
open list:BROADCOM BNG_EN 800 GIGABIT ETHERNET DRIVER,
open list:FREESCALE ENETC ETHERNET DRIVERS,
moderated list:INTEL ETHERNET DRIVERS,
open list:Hyper-V/Azure CORE AND DRIVERS
The pcie_reset_flr() function includes validation checks to verify FLR
support before performing the reset, while pcie_flr() performs the reset
unconditionally. Having both functions creates unnecessary complexity.
Commit 56f107d7813f ("PCI: Add pcie_reset_flr() with 'probe' argument")
introduced pcie_reset_flr() and removed pcie_has_flr(), converting callers
that previously used the pcie_has_flr() + pcie_flr() to use
pcie_reset_flr() instead. However, it did not convert all pcie_flr()
callers, leaving two different FLR mechanisms in the kernel.
One of the callers of pcie_flr(), the Intel 82599 Virtual Function has a
defect where FLR works despite not advertising FLR support in the PCIe
Device Capability register. Rather than using pcie_flr() to work around
this, enable the FLR capability bit in devcap via an early quirk. This
allows the device to use the standard pcie_reset_flr() path instead of
requiring a device-specific reset method.
Remove pcie_flr() entirely and convert all remaining callers to
pcie_reset_flr(), ensuring consistent validation across the kernel.
Signed-off-by: Farhan Ali <alifm@linux.ibm.com>
---
drivers/crypto/intel/qat/qat_common/adf_aer.c | 2 +-
drivers/infiniband/hw/hfi1/chip.c | 4 +-
.../net/ethernet/broadcom/bnge/bnge_core.c | 2 +-
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 4 +-
.../ethernet/cavium/liquidio/lio_vf_main.c | 2 +-
.../ethernet/cavium/liquidio/octeon_mailbox.c | 3 +-
drivers/net/ethernet/freescale/enetc/enetc.c | 2 +-
.../ethernet/freescale/enetc/enetc_pci_mdio.c | 2 +-
drivers/net/ethernet/intel/ice/ice_main.c | 2 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 2 +-
drivers/net/ethernet/microsoft/mana/mana_en.c | 3 +-
drivers/pci/pci.c | 38 ++++++------------
drivers/pci/quirks.c | 40 +++++++++----------
drivers/ptp/ptp_netc.c | 2 +-
include/linux/pci.h | 1 -
15 files changed, 49 insertions(+), 60 deletions(-)
diff --git a/drivers/crypto/intel/qat/qat_common/adf_aer.c b/drivers/crypto/intel/qat/qat_common/adf_aer.c
index ed01fb9ad74e..a2364a59bc7f 100644
--- a/drivers/crypto/intel/qat/qat_common/adf_aer.c
+++ b/drivers/crypto/intel/qat/qat_common/adf_aer.c
@@ -89,7 +89,7 @@ EXPORT_SYMBOL_GPL(adf_reset_sbr);
void adf_reset_flr(struct adf_accel_dev *accel_dev)
{
- pcie_flr(accel_to_pci_dev(accel_dev));
+ pcie_reset_flr(accel_to_pci_dev(accel_dev), PCI_RESET_DO_RESET);
}
EXPORT_SYMBOL_GPL(adf_reset_flr);
diff --git a/drivers/infiniband/hw/hfi1/chip.c b/drivers/infiniband/hw/hfi1/chip.c
index 44c524e45396..9f53d73e5e76 100644
--- a/drivers/infiniband/hw/hfi1/chip.c
+++ b/drivers/infiniband/hw/hfi1/chip.c
@@ -14042,7 +14042,7 @@ static int init_chip(struct hfi1_devdata *dd)
dd_dev_info(dd, "Resetting CSRs with FLR\n");
/* do the FLR, the DC reset will remain */
- pcie_flr(dd->pcidev);
+ pcie_reset_flr(dd->pcidev, PCI_RESET_DO_RESET);
/* restore command and BARs */
ret = restore_pci_variables(dd);
@@ -14054,7 +14054,7 @@ static int init_chip(struct hfi1_devdata *dd)
if (is_ax(dd)) {
dd_dev_info(dd, "Resetting CSRs with FLR\n");
- pcie_flr(dd->pcidev);
+ pcie_reset_flr(dd->pcidev, PCI_RESET_DO_RESET);
ret = restore_pci_variables(dd);
if (ret) {
dd_dev_err(dd, "%s: Could not restore PCI variables\n",
diff --git a/drivers/net/ethernet/broadcom/bnge/bnge_core.c b/drivers/net/ethernet/broadcom/bnge/bnge_core.c
index 68b74eb2c3a2..4aec01f53e54 100644
--- a/drivers/net/ethernet/broadcom/bnge/bnge_core.c
+++ b/drivers/net/ethernet/broadcom/bnge/bnge_core.c
@@ -274,7 +274,7 @@ static int bnge_probe_one(struct pci_dev *pdev, const struct pci_device_id *ent)
if (is_kdump_kernel()) {
pci_clear_master(pdev);
- pcie_flr(pdev);
+ pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
}
rc = bnge_pci_enable(pdev);
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 35e1f8f663c7..21f8dcbe671e 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -16918,7 +16918,7 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
*/
if (is_kdump_kernel()) {
pci_clear_master(pdev);
- pcie_flr(pdev);
+ pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
}
max_irqs = bnxt_get_max_irq(pdev);
@@ -17203,7 +17203,7 @@ static void bnxt_shutdown(struct pci_dev *pdev)
netif_close(dev);
if (bnxt_hwrm_func_drv_unrgtr(bp)) {
- pcie_flr(pdev);
+ pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
goto shutdown_exit;
}
bnxt_ptp_clear(bp);
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c
index 43c595f3b84e..7f3557d36341 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c
@@ -429,7 +429,7 @@ static void octeon_pci_flr(struct octeon_device *oct)
pci_write_config_word(oct->pci_dev, PCI_COMMAND,
PCI_COMMAND_INTX_DISABLE);
- pcie_flr(oct->pci_dev);
+ pcie_reset_flr(oct->pci_dev, PCI_RESET_DO_RESET);
pci_cfg_access_unlock(oct->pci_dev);
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
index ad685f5d0a13..be08e213aa9a 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
@@ -260,7 +260,8 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox,
dev_info(&oct->pci_dev->dev,
"got a request for FLR from VF that owns DPI ring %u\n",
mbox->q_no);
- pcie_flr(oct->sriov_info.dpiring_to_vfpcidev_lut[mbox->q_no]);
+ pcie_reset_flr(oct->sriov_info.dpiring_to_vfpcidev_lut[mbox->q_no],
+ PCI_RESET_DO_RESET);
break;
case OCTEON_PF_CHANGED_VF_MACADDR:
diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c
index aa8a87124b10..c1c1b523abb5 100644
--- a/drivers/net/ethernet/freescale/enetc/enetc.c
+++ b/drivers/net/ethernet/freescale/enetc/enetc.c
@@ -3635,7 +3635,7 @@ int enetc_pci_probe(struct pci_dev *pdev, const char *name, int sizeof_priv)
size_t alloc_size;
int err, len;
- pcie_flr(pdev);
+ pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
err = pci_enable_device_mem(pdev);
if (err)
return dev_err_probe(&pdev->dev, err, "device enable failed\n");
diff --git a/drivers/net/ethernet/freescale/enetc/enetc_pci_mdio.c b/drivers/net/ethernet/freescale/enetc/enetc_pci_mdio.c
index e108cac8288d..cfccfca1981d 100644
--- a/drivers/net/ethernet/freescale/enetc/enetc_pci_mdio.c
+++ b/drivers/net/ethernet/freescale/enetc/enetc_pci_mdio.c
@@ -73,7 +73,7 @@ static int enetc_pci_mdio_probe(struct pci_dev *pdev,
mdio_priv->mdio_base = ENETC_EMDIO_BASE;
snprintf(bus->id, MII_BUS_ID_SIZE, "%s", dev_name(dev));
- pcie_flr(pdev);
+ pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
err = pci_enable_device_mem(pdev);
if (err) {
dev_err(dev, "device enable failed\n");
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index e2fbe111f849..14b8a90625a8 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -5180,7 +5180,7 @@ ice_probe(struct pci_dev *pdev, const struct pci_device_id __always_unused *ent)
if (is_kdump_kernel()) {
pci_save_state(pdev);
pci_clear_master(pdev);
- err = pcie_flr(pdev);
+ err = pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
if (err)
return err;
pci_restore_state(pdev);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 2646ee6f295f..d8796a68094f 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -8318,7 +8318,7 @@ static void ixgbe_check_for_bad_vf(struct ixgbe_adapter *adapter)
if (status_reg != IXGBE_FAILED_READ_CFG_WORD &&
status_reg & PCI_STATUS_REC_MASTER_ABORT) {
ixgbe_bad_vf_abort(adapter, vf);
- pcie_flr(vfdev);
+ pcie_reset_flr(vfdev, PCI_RESET_DO_RESET);
}
}
}
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index c9b1df1ed109..e51c1170aba7 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -3305,7 +3305,8 @@ static int mana_dealloc_queues(struct net_device *ndev)
}
if (atomic_read(&txq->pending_sends)) {
err =
- pcie_flr(to_pci_dev(gd->gdma_context->dev));
+ pcie_reset_flr(to_pci_dev(gd->gdma_context->dev),
+ PCI_RESET_DO_RESET);
if (err) {
netdev_err(ndev, "flr failed %d with %d pkts pending in txq %u\n",
err,
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index d34266651ad0..878556ea50de 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4321,16 +4321,25 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
EXPORT_SYMBOL(pci_wait_for_pending_transaction);
/**
- * pcie_flr - initiate a PCIe function level reset
+ * pcie_reset_flr - initiate a PCIe function level reset
* @dev: device to reset
+ * @probe: if true, return 0 if device can be reset this way
*
- * Initiate a function level reset unconditionally on @dev without
- * checking any flags and DEVCAP
+ * Initiate a function level reset on @dev.
*/
-int pcie_flr(struct pci_dev *dev)
+int pcie_reset_flr(struct pci_dev *dev, bool probe)
{
int ret;
+ if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
+ return -ENOTTY;
+
+ if (!(dev->devcap & PCI_EXP_DEVCAP_FLR))
+ return -ENOTTY;
+
+ if (probe)
+ return 0;
+
if (!pci_wait_for_pending_transaction(dev))
pci_err(dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
@@ -4357,28 +4366,7 @@ int pcie_flr(struct pci_dev *dev)
done:
pci_dev_reset_iommu_done(dev);
return ret;
-}
-EXPORT_SYMBOL_GPL(pcie_flr);
-
-/**
- * pcie_reset_flr - initiate a PCIe function level reset
- * @dev: device to reset
- * @probe: if true, return 0 if device can be reset this way
- *
- * Initiate a function level reset on @dev.
- */
-int pcie_reset_flr(struct pci_dev *dev, bool probe)
-{
- if (dev->dev_flags & PCI_DEV_FLAGS_NO_FLR_RESET)
- return -ENOTTY;
-
- if (!(dev->devcap & PCI_EXP_DEVCAP_FLR))
- return -ENOTTY;
-
- if (probe)
- return 0;
- return pcie_flr(dev);
}
EXPORT_SYMBOL_GPL(pcie_reset_flr);
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index caaed1a01dc0..564f581599b8 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -2019,6 +2019,23 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_PXH_0, quirk_pc
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_PXH_1, quirk_pcie_pxh);
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_PXHV, quirk_pcie_pxh);
+#define PCI_DEVICE_ID_INTEL_82599_SFP_VF 0x10ed
+static void quirk_intel_82599_sfp_virtfn(struct pci_dev *dev)
+{
+ /*
+ * http://www.intel.com/content/dam/doc/datasheet/82599-10-gbe-controller-datasheet.pdf
+ *
+ * The 82599 supports FLR on VFs, but FLR support is reported only
+ * in the PF DEVCAP (sec 9.3.10.4), not in the VF DEVCAP (sec 9.5).
+ * So enable PCI_EXP_DEVCAP_FLR directly without first checking if it is
+ * supported.
+ */
+
+ dev->devcap |= PCI_EXP_DEVCAP_FLR;
+}
+
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF, quirk_intel_82599_sfp_virtfn);
+
/*
* Some Intel PCI Express chipsets have trouble with downstream device
* power management.
@@ -3944,20 +3961,6 @@ DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL,
* reset a single function if other methods (e.g. FLR, PM D0->D3) are
* not available.
*/
-static int reset_intel_82599_sfp_virtfn(struct pci_dev *dev, bool probe)
-{
- /*
- * http://www.intel.com/content/dam/doc/datasheet/82599-10-gbe-controller-datasheet.pdf
- *
- * The 82599 supports FLR on VFs, but FLR support is reported only
- * in the PF DEVCAP (sec 9.3.10.4), not in the VF DEVCAP (sec 9.5).
- * Thus we must call pcie_flr() directly without first checking if it is
- * supported.
- */
- if (!probe)
- pcie_flr(dev);
- return 0;
-}
#define SOUTH_CHICKEN2 0xc2004
#define PCH_PP_STATUS 0xc7200
@@ -4058,7 +4061,7 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, bool probe)
PCI_MSIX_FLAGS_ENABLE |
PCI_MSIX_FLAGS_MASKALL);
- pcie_flr(dev);
+ pcie_reset_flr(dev, PCI_RESET_DO_RESET);
/*
* Restore the configuration information (BAR values, etc.) including
@@ -4070,7 +4073,6 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, bool probe)
return 0;
}
-#define PCI_DEVICE_ID_INTEL_82599_SFP_VF 0x10ed
#define PCI_DEVICE_ID_INTEL_IVB_M_VGA 0x0156
#define PCI_DEVICE_ID_INTEL_IVB_M2_VGA 0x0166
@@ -4150,7 +4152,7 @@ static int nvme_disable_and_flr(struct pci_dev *dev, bool probe)
pci_iounmap(dev, bar);
- pcie_flr(dev);
+ pcie_reset_flr(dev, PCI_RESET_DO_RESET);
return 0;
}
@@ -4207,7 +4209,7 @@ static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe)
val = val | HINIC_VF_FLR_PROC_BIT;
iowrite32be(val, bar + HINIC_VF_OP);
- pcie_flr(pdev);
+ pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
/*
* The device must recapture its Bus and Device Numbers after FLR
@@ -4238,8 +4240,6 @@ static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe)
}
static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
- { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF,
- reset_intel_82599_sfp_virtfn },
{ PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_IVB_M_VGA,
reset_ivb_igd },
{ PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_IVB_M2_VGA,
diff --git a/drivers/ptp/ptp_netc.c b/drivers/ptp/ptp_netc.c
index 94e952ee6990..24bae237926a 100644
--- a/drivers/ptp/ptp_netc.c
+++ b/drivers/ptp/ptp_netc.c
@@ -802,7 +802,7 @@ static int netc_timer_pci_probe(struct pci_dev *pdev)
if (!priv)
return -ENOMEM;
- pcie_flr(pdev);
+ pcie_reset_flr(pdev, PCI_RESET_DO_RESET);
err = pci_enable_device_mem(pdev);
if (err)
return dev_err_probe(dev, err, "Failed to enable device\n");
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 2c4454583c11..345f0821471a 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1468,7 +1468,6 @@ u32 pcie_bandwidth_available(struct pci_dev *dev, struct pci_dev **limiting_dev,
int pcie_link_speed_mbps(struct pci_dev *pdev);
void pcie_print_link_status(struct pci_dev *dev);
int pcie_reset_flr(struct pci_dev *dev, bool probe);
-int pcie_flr(struct pci_dev *dev);
int __pci_reset_function_locked(struct pci_dev *dev);
int pci_reset_function(struct pci_dev *dev);
int pci_reset_function_locked(struct pci_dev *dev);
--
2.43.0
^ permalink raw reply related
* [PATCH v3] lib/raid/xor: x86: Add AVX-512 optimized xor_gen()
From: Eric Biggers @ 2026-06-15 19:03 UTC (permalink / raw)
To: Andrew Morton, linux-kernel
Cc: Christoph Hellwig, linux-crypto, x86, Eric Biggers, David Laight,
linux-raid
Add an implementation of xor_gen() using AVX-512.
It uses 512-bit vectors, i.e. ZMM registers. It also uses the
vpternlogq instruction to do three-input XORs when applicable.
It's enabled on x86_64 CPUs that have AVX512F && !PREFER_YMM. In
practice that means:
- AMD Zen 4 and later (client and server)
- Intel Sapphire Rapids and later (server)
- Intel Rocket Lake (client)
- Intel Nova Lake and later (client)
The !PREFER_YMM condition excludes the older AVX-512 implementations in
Intel Skylake Server and Intel Ice Lake. They could run this code, but
they're known to have overly-eager downclocking when ZMM registers are
used. This is the same policy that the crypto and CRC code uses.
Benchmark on AMD Ryzen 9 9950X (Zen 5):
src_cnt avx avx512 Improvement
======= ========== ========== ===========
1 56353 MB/s 75388 MB/s 33%
2 54274 MB/s 68409 MB/s 26%
3 44649 MB/s 64042 MB/s 43%
4 41315 MB/s 55002 MB/s 33%
Note: for now I omitted the cpu_has_xfeatures() check that the AVX-512
optimized crypto and CRC code does, since it's not implemented on
User-Mode Linux and it's never been present in the RAID6 code either.
Reviewed-by: David Laight <david.laight.linux@gmail.com>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
Changed in v3:
- Renamed p0-p4 to p1-p5 to match the inline asm indices
- Swapped the base and index registers to the logical order
- Added David's Reviewed-by
Changed in v2:
- Fixed build on UML
- Reworked the implementation
lib/raid/xor/Makefile | 2 +-
lib/raid/xor/x86/xor-avx512.c | 121 ++++++++++++++++++++++++++++++++++
lib/raid/xor/x86/xor_arch.h | 26 ++++----
3 files changed, 137 insertions(+), 12 deletions(-)
create mode 100644 lib/raid/xor/x86/xor-avx512.c
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 4d633dfd5b90..4af945861a51 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -26,11 +26,11 @@ xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
xor-$(CONFIG_RISCV_ISA_V) += riscv/xor.o riscv/xor-glue.o
xor-$(CONFIG_SPARC32) += sparc/xor-sparc32.o
xor-$(CONFIG_SPARC64) += sparc/xor-sparc64.o sparc/xor-sparc64-glue.o
xor-$(CONFIG_S390) += s390/xor.o
xor-$(CONFIG_X86_32) += x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o
-xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o
+xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o x86/xor-avx512.o
obj-y += tests/
CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU)
diff --git a/lib/raid/xor/x86/xor-avx512.c b/lib/raid/xor/x86/xor-avx512.c
new file mode 100644
index 000000000000..17f57900d827
--- /dev/null
+++ b/lib/raid/xor/x86/xor-avx512.c
@@ -0,0 +1,121 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * AVX-512 optimized implementation of xor_gen()
+ *
+ * Copyright 2026 Google LLC
+ */
+
+#include <linux/types.h>
+#include <asm/fpu/api.h>
+#include "xor_impl.h"
+#include "xor_arch.h"
+
+/*
+ * Implementation notes:
+ *
+ * Unrolling by the number of buffers (2-5) is very important.
+ *
+ * Unrolling by length is less important, especially when using register-indexed
+ * addressing with negative indices from the end of the buffers. That approach
+ * results in just two loop control instructions being needed per iteration,
+ * regardless of the number of buffers.
+ *
+ * In fact, benchmarks showed that the 2 and 3 buffer cases require only 2x
+ * unrolling by length, while the 4 and 5 buffer cases don't require any
+ * unrolling by length. Benchmarks also showed that the register-indexed
+ * addressing isn't a bottleneck either; i.e., we can't do any better by
+ * incrementing the pointers as we go along, even with more unrolling.
+ */
+
+static void xor_avx512_2(long bytes, u8 *p1, const u8 *p2)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 64(%1,%0), %%zmm1\n"
+ "vpxorq (%2,%0), %%zmm0, %%zmm0\n"
+ "vpxorq 64(%2,%0), %%zmm1, %%zmm1\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "vmovdqa64 %%zmm1, 64(%1,%0)\n"
+ "add $128, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes)
+ : "memory", "cc");
+}
+
+static void xor_avx512_3(long bytes, u8 *p1, const u8 *p2, const u8 *p3)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 64(%1,%0), %%zmm1\n"
+ "vmovdqa64 (%2,%0), %%zmm2\n"
+ "vmovdqa64 64(%2,%0), %%zmm3\n"
+ "vpternlogq $0x96, (%3,%0), %%zmm2, %%zmm0\n"
+ "vpternlogq $0x96, 64(%3,%0), %%zmm3, %%zmm1\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "vmovdqa64 %%zmm1, 64(%1,%0)\n"
+ "add $128, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes)
+ : "memory", "cc");
+}
+
+static void xor_avx512_4(long bytes, u8 *p1, const u8 *p2, const u8 *p3,
+ const u8 *p4)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 (%2,%0), %%zmm1\n"
+ "vpxorq (%3,%0), %%zmm0, %%zmm0\n"
+ "vpternlogq $0x96, (%4,%0), %%zmm1, %%zmm0\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "add $64, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes),
+ "r"(p4 + bytes)
+ : "memory", "cc");
+}
+
+static void xor_avx512_5(long bytes, u8 *p1, const u8 *p2, const u8 *p3,
+ const u8 *p4, const u8 *p5)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 (%2,%0), %%zmm1\n"
+ "vpternlogq $0x96, (%3,%0), %%zmm1, %%zmm0\n"
+ "vmovdqa64 (%4,%0), %%zmm1\n"
+ "vpternlogq $0x96, (%5,%0), %%zmm1, %%zmm0\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "add $64, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes),
+ "r"(p4 + bytes), "r"(p5 + bytes)
+ : "memory", "cc");
+}
+
+DO_XOR_BLOCKS(avx512_inner, xor_avx512_2, xor_avx512_3, xor_avx512_4,
+ xor_avx512_5);
+
+/*
+ * Preconditions: bytes is a nonzero multiple of 512, and all buffers are
+ * 64-byte aligned.
+ */
+static void xor_gen_avx512(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
+{
+ kernel_fpu_begin();
+ xor_gen_avx512_inner(dest, srcs, src_cnt, bytes);
+ kernel_fpu_end();
+}
+
+struct xor_block_template xor_block_avx512 = {
+ .name = "avx512",
+ .xor_gen = xor_gen_avx512,
+};
diff --git a/lib/raid/xor/x86/xor_arch.h b/lib/raid/xor/x86/xor_arch.h
index 99fe85a213c6..b5d49376fc97 100644
--- a/lib/raid/xor/x86/xor_arch.h
+++ b/lib/raid/xor/x86/xor_arch.h
@@ -4,26 +4,30 @@
extern struct xor_block_template xor_block_pII_mmx;
extern struct xor_block_template xor_block_p5_mmx;
extern struct xor_block_template xor_block_sse;
extern struct xor_block_template xor_block_sse_pf64;
extern struct xor_block_template xor_block_avx;
+extern struct xor_block_template xor_block_avx512;
-/*
- * When SSE is available, use it as it can write around L2. We may also be able
- * to load into the L1 only depending on how the cpu deals with a load to a line
- * that is being prefetched.
- *
- * When AVX2 is available, force using it as it is better by all measures.
- *
- * 32-bit without MMX can fall back to the generic routines.
- */
static __always_inline void __init arch_xor_init(void)
{
- if (boot_cpu_has(X86_FEATURE_AVX) &&
- boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ if (IS_ENABLED(CONFIG_X86_64) && boot_cpu_has(X86_FEATURE_AVX512F) &&
+ boot_cpu_has(X86_FEATURE_OSXSAVE) &&
+ !boot_cpu_has(X86_FEATURE_PREFER_YMM)) {
+ /* AVX-512 will be the best; no need to try others. */
+ /* !PREFER_YMM excludes CPUs with overly-eager downclocking. */
+ xor_force(&xor_block_avx512);
+ } else if (boot_cpu_has(X86_FEATURE_AVX) &&
+ boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ /* AVX will be the best; no need to try others. */
xor_force(&xor_block_avx);
} else if (IS_ENABLED(CONFIG_X86_64) || boot_cpu_has(X86_FEATURE_XMM)) {
+ /*
+ * When SSE is available, use it as it can write around L2. We
+ * may also be able to load into the L1 only depending on how
+ * the cpu deals with a load to a line that is being prefetched.
+ */
xor_register(&xor_block_sse);
xor_register(&xor_block_sse_pf64);
} else if (boot_cpu_has(X86_FEATURE_MMX)) {
xor_register(&xor_block_pII_mmx);
xor_register(&xor_block_p5_mmx);
base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
--
2.54.0
^ permalink raw reply related
* Re: [PATCH v2] lib/raid/xor: x86: Add AVX-512 optimized xor_gen()
From: Eric Biggers @ 2026-06-15 18:44 UTC (permalink / raw)
To: David Laight
Cc: Andrew Morton, linux-kernel, Christoph Hellwig, linux-crypto, x86,
linux-raid
In-Reply-To: <20260614111628.00af46b9@pumpkin>
On Sun, Jun 14, 2026 at 11:16:28AM +0100, David Laight wrote:
> On Sat, 13 Jun 2026 18:03:57 -0700
> Eric Biggers <ebiggers@kernel.org> wrote:
>
> > Add an implementation of xor_gen() using AVX-512.
> >
> > It uses 512-bit vectors, i.e. ZMM registers. It also uses the
> > vpternlogq instruction to do three-input XORs when applicable.
> >
> > It's enabled on x86_64 CPUs that have AVX512F && !PREFER_YMM. In
> > practice that means:
> >
> > - AMD Zen 4 and later (client and server)
>
> Doesn't zen4 only have a 256bit bus between the cpu and cache?
> So avx512 reads take two clocks.
> Since this is memory limited it is unlikely to run faster than the
> avx256 version.
On AMD Genoa (Zen 4 server processor), the AVX-512 code added by this
patch is indeed about the same speed as the existing AVX-2 code.
> OTOH if it doesn't cause down-clocking as well then it won't be slower.
Yes, as far as I know that's not an issue on AMD processors, even Zen 4.
The "avoid AVX-512 due to downclocking" rule is historical guidance for
Intel processors that had a bad implementation of AVX-512. There's no
reason to exclude Zen 4 from executing AVX-512 optimized code. At worst
it will just be the same, as we're seeing here.
> Since I suggested it :-)
>
> Reviewed-By: David Laight <david.laight.linux@gmail.com>
>
> Some 'not very important' comments:
>
> I did wonder whether moving the loop into the asm() would help.
> gcc has a nasty habit of pessimising loops when you try to be clever.
> It is certainly safer for tight loops like these.
I originally tried leaving the loops to the compiler, but gcc unrolled
the 1x ones by 2x, despite it having no visibility into the asm block.
That broke the intent with the indexed addressing, since to achieve the
unrolling it generated code that incremented the pointers.
So I just ended up moving the loop to the asm, which reliably gives us
the code we want.
> That does have the side effect of making p0 be %1 which doesn't improve
> readability. Either used named parameters or possibly just change p0 to p1 (etc)
> so they match.
>
> The code should be limited by the memory reads, so the 3-argument xor and
> the interleave of the unroll may make no difference.
The unroll by 2x in the 2 and 3-buffer cases helped a little bit on
Sapphire Rapids. I don't know exactly why, but it makes sense that
those cases are where the loop overhead is most likely to matter.
> Some cpu do have constraints on the cache alignment in order to do two
> reads per clock, but I've forgotten them and they got better before AVX-512.
> If that were affecting this code (on the tested cpu) then I'd expect the
> interleaved unroll would improve the _4 and -5 functions.
> So it probably doesn't affect this code.
The buffers are always 64-byte aligned here, as documented.
> Using the same loop for the avx-256 and sse (and even smaller) functions could
> well generate code that runs 'pretty much as fast as possible' on older cpu.
> Intel cpu (going back to Sandy bridge) are likely to execute the loop in the
> same number of clocks - but clearly copying half or a quarter of the data.
> But I've no experience of zen1.
>
> Might be worth doing for avx-256, does any care about anything older :-)
Yes, the existing AVX code is probably excessively unrolled. It
generates almost 4 KiB of code.
- Eric
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox