* [PATCH v27 05/12] crypto: DRBG - externalize DRBG functions for LRNG
From: Stephan Müller @ 2020-01-09 8:32 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Greg Kroah-Hartman, linux-crypto, LKML, linux-api,
Eric W. Biederman, Alexander E. Patrakov, Ahmed S. Darwish,
Theodore Y. Ts'o, Willy Tarreau, Matthew Garrett, Vito Caputo,
Andreas Dilger, Jan Kara, Ray Strode, William Jon McCann, zhangjs,
Andy Lutomirski, Florian Weimer, Lennart Poettering,
Nicolai Stange, "Peter, Matthias" <matth>
In-Reply-To: <2641155.iNH938UiKq@positron.chronox.de>
This patch allows several DRBG functions to be called by the LRNG kernel
code paths outside the drbg.c file.
CC: "Eric W. Biederman" <ebiederm@xmission.com>
CC: "Alexander E. Patrakov" <patrakov@gmail.com>
CC: "Ahmed S. Darwish" <darwish.07@gmail.com>
CC: "Theodore Y. Ts'o" <tytso@mit.edu>
CC: Willy Tarreau <w@1wt.eu>
CC: Matthew Garrett <mjg59@srcf.ucam.org>
CC: Vito Caputo <vcaputo@pengaru.com>
CC: Andreas Dilger <adilger.kernel@dilger.ca>
CC: Jan Kara <jack@suse.cz>
CC: Ray Strode <rstrode@redhat.com>
CC: William Jon McCann <mccann@jhu.edu>
CC: zhangjs <zachary@baishancloud.com>
CC: Andy Lutomirski <luto@kernel.org>
CC: Florian Weimer <fweimer@redhat.com>
CC: Lennart Poettering <mzxreary@0pointer.de>
CC: Nicolai Stange <nstange@suse.de>
Reviewed-by: Roman Drahtmueller <draht@schaltsekun.de>
Tested-by: Roman Drahtmüller <draht@schaltsekun.de>
Tested-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Tested-by: Neil Horman <nhorman@redhat.com>
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
crypto/drbg.c | 16 ++++++++++------
include/crypto/drbg.h | 7 +++++++
2 files changed, 17 insertions(+), 6 deletions(-)
diff --git a/crypto/drbg.c b/crypto/drbg.c
index b6929eb5f565..5561dd0f09e1 100644
--- a/crypto/drbg.c
+++ b/crypto/drbg.c
@@ -113,7 +113,7 @@
* the SHA256 / AES 256 over other ciphers. Thus, the favored
* DRBGs are the latest entries in this array.
*/
-static const struct drbg_core drbg_cores[] = {
+const struct drbg_core drbg_cores[] = {
#ifdef CONFIG_CRYPTO_DRBG_CTR
{
.flags = DRBG_CTR | DRBG_STRENGTH128,
@@ -190,6 +190,7 @@ static const struct drbg_core drbg_cores[] = {
},
#endif /* CONFIG_CRYPTO_DRBG_HMAC */
};
+EXPORT_SYMBOL(drbg_cores);
static int drbg_uninstantiate(struct drbg_state *drbg);
@@ -205,7 +206,7 @@ static int drbg_uninstantiate(struct drbg_state *drbg);
* Return: normalized strength in *bytes* value or 32 as default
* to counter programming errors
*/
-static inline unsigned short drbg_sec_strength(drbg_flag_t flags)
+unsigned short drbg_sec_strength(drbg_flag_t flags)
{
switch (flags & DRBG_STRENGTH_MASK) {
case DRBG_STRENGTH128:
@@ -218,6 +219,7 @@ static inline unsigned short drbg_sec_strength(drbg_flag_t flags)
return 32;
}
}
+EXPORT_SYMBOL(drbg_sec_strength);
/*
* FIPS 140-2 continuous self test for the noise source
@@ -1202,7 +1204,7 @@ static int drbg_seed(struct drbg_state *drbg, struct drbg_string *pers,
}
/* Free all substructures in a DRBG state without the DRBG state structure */
-static inline void drbg_dealloc_state(struct drbg_state *drbg)
+void drbg_dealloc_state(struct drbg_state *drbg)
{
if (!drbg)
return;
@@ -1223,12 +1225,13 @@ static inline void drbg_dealloc_state(struct drbg_state *drbg)
drbg->fips_primed = false;
}
}
+EXPORT_SYMBOL(drbg_dealloc_state);
/*
* Allocate all sub-structures for a DRBG state.
* The DRBG state structure must already be allocated.
*/
-static inline int drbg_alloc_state(struct drbg_state *drbg)
+int drbg_alloc_state(struct drbg_state *drbg)
{
int ret = -ENOMEM;
unsigned int sb_size = 0;
@@ -1307,6 +1310,7 @@ static inline int drbg_alloc_state(struct drbg_state *drbg)
drbg_dealloc_state(drbg);
return ret;
}
+EXPORT_SYMBOL(drbg_alloc_state);
/*************************************************************************
* DRBG interface functions
@@ -1874,8 +1878,7 @@ static int drbg_kcapi_sym_ctr(struct drbg_state *drbg,
*
* return: flags
*/
-static inline void drbg_convert_tfm_core(const char *cra_driver_name,
- int *coreref, bool *pr)
+void drbg_convert_tfm_core(const char *cra_driver_name, int *coreref, bool *pr)
{
int i = 0;
size_t start = 0;
@@ -1902,6 +1905,7 @@ static inline void drbg_convert_tfm_core(const char *cra_driver_name,
}
}
}
+EXPORT_SYMBOL(drbg_convert_tfm_core);
static int drbg_kcapi_init(struct crypto_tfm *tfm)
{
diff --git a/include/crypto/drbg.h b/include/crypto/drbg.h
index 8c9af21efce1..c63b20375235 100644
--- a/include/crypto/drbg.h
+++ b/include/crypto/drbg.h
@@ -282,4 +282,11 @@ enum drbg_prefixes {
DRBG_PREFIX3
};
+extern int drbg_alloc_state(struct drbg_state *drbg);
+extern void drbg_dealloc_state(struct drbg_state *drbg);
+extern void drbg_convert_tfm_core(const char *cra_driver_name, int *coreref,
+ bool *pr);
+extern const struct drbg_core drbg_cores[];
+extern unsigned short drbg_sec_strength(drbg_flag_t flags);
+
#endif /* _DRBG_H */
--
2.24.1
^ permalink raw reply related
* [PATCH v27 04/12] LRNG - add switchable DRNG support
From: Stephan Müller @ 2020-01-09 8:32 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Greg Kroah-Hartman, linux-crypto, LKML, linux-api,
Eric W. Biederman, Alexander E. Patrakov, Ahmed S. Darwish,
Theodore Y. Ts'o, Willy Tarreau, Matthew Garrett, Vito Caputo,
Andreas Dilger, Jan Kara, Ray Strode, William Jon McCann, zhangjs,
Andy Lutomirski, Florian Weimer, Lennart Poettering,
Nicolai Stange, "Peter, Matthias" <matth>
In-Reply-To: <2641155.iNH938UiKq@positron.chronox.de>
The DRNG switch support allows replacing the DRNG mechanism of the
LRNG. The switching support rests on the interface definition of
include/linux/lrng.h. A new DRNG is implemented by filling in the
interface defined in this header file.
In addition to the DRNG, the extension also has to provide a hash
implementation that is used to hash the entropy pool for random number
extraction.
Note: It is permissible to implement a DRNG whose operations may sleep.
However, the hash function must not sleep.
The switchable DRNG support allows replacing the DRNG at runtime.
However, only one DRNG extension is allowed to be loaded at any given
time. Before replacing it with another DRNG implementation, the possibly
existing DRNG extension must be unloaded.
The switchable DRNG extension activates the new DRNG during load time.
It is expected, however, that such a DRNG switch would be done only once
by an administrator to load the intended DRNG implementation.
It is permissible to compile DRNG extensions either as kernel modules or
statically. The initialization of the DRNG extension should be performed
with a late_initcall to ensure the extension is available when user
space starts but after all other initialization completed.
The initialization is performed by registering the function call data
structure with the lrng_set_drng_cb function. In order to unload the
DRNG extension, lrng_set_drng_cb must be invoked with the NULL
parameter.
The DRNG extension should always provide a security strength that is at
least as strong as LRNG_DRNG_SECURITY_STRENGTH_BITS.
CC: "Eric W. Biederman" <ebiederm@xmission.com>
CC: "Alexander E. Patrakov" <patrakov@gmail.com>
CC: "Ahmed S. Darwish" <darwish.07@gmail.com>
CC: "Theodore Y. Ts'o" <tytso@mit.edu>
CC: Willy Tarreau <w@1wt.eu>
CC: Matthew Garrett <mjg59@srcf.ucam.org>
CC: Vito Caputo <vcaputo@pengaru.com>
CC: Andreas Dilger <adilger.kernel@dilger.ca>
CC: Jan Kara <jack@suse.cz>
CC: Ray Strode <rstrode@redhat.com>
CC: William Jon McCann <mccann@jhu.edu>
CC: zhangjs <zachary@baishancloud.com>
CC: Andy Lutomirski <luto@kernel.org>
CC: Florian Weimer <fweimer@redhat.com>
CC: Lennart Poettering <mzxreary@0pointer.de>
CC: Nicolai Stange <nstange@suse.de>
Reviewed-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Reviewed-by: Roman Drahtmueller <draht@schaltsekun.de>
Tested-by: Roman Drahtmüller <draht@schaltsekun.de>
Tested-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Tested-by: Neil Horman <nhorman@redhat.com>
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
drivers/char/lrng/Kconfig | 7 ++
drivers/char/lrng/Makefile | 1 +
drivers/char/lrng/lrng_switch.c | 179 ++++++++++++++++++++++++++++++++
3 files changed, 187 insertions(+)
create mode 100644 drivers/char/lrng/lrng_switch.c
diff --git a/drivers/char/lrng/Kconfig b/drivers/char/lrng/Kconfig
index 56f13efd3592..cb701bb0b8b6 100644
--- a/drivers/char/lrng/Kconfig
+++ b/drivers/char/lrng/Kconfig
@@ -64,4 +64,11 @@ config LRNG_POOL_SIZE
default 7 if LRNG_POOL_SIZE_65536
default 8 if LRNG_POOL_SIZE_131072
+menuconfig LRNG_DRNG_SWITCH
+ bool "Support DRNG runtime switching"
+ help
+ The Linux RNG per default uses a ChaCha20 DRNG that is
+ accessible via the external interfaces. With this configuration
+ option other DRNGs can be selected and loaded at runtime.
+
endif # LRNG
diff --git a/drivers/char/lrng/Makefile b/drivers/char/lrng/Makefile
index e69c176f0161..31cfe87c999e 100644
--- a/drivers/char/lrng/Makefile
+++ b/drivers/char/lrng/Makefile
@@ -10,3 +10,4 @@ obj-y += lrng_pool.o lrng_aux.o \
obj-$(CONFIG_NUMA) += lrng_numa.o
obj-$(CONFIG_SYSCTL) += lrng_proc.o
+obj-$(CONFIG_LRNG_DRNG_SWITCH) += lrng_switch.o
diff --git a/drivers/char/lrng/lrng_switch.c b/drivers/char/lrng/lrng_switch.c
new file mode 100644
index 000000000000..2c7468d8de09
--- /dev/null
+++ b/drivers/char/lrng/lrng_switch.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG DRNG switching support
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/lrng.h>
+
+#include "lrng_internal.h"
+
+static int lrng_drng_switch(struct lrng_drng *drng_store,
+ const struct lrng_crypto_cb *cb, int node)
+{
+ const struct lrng_crypto_cb *old_cb;
+ unsigned long flags = 0;
+ int ret;
+ u8 seed[LRNG_DRNG_SECURITY_STRENGTH_BYTES];
+ void *new_drng = cb->lrng_drng_alloc(LRNG_DRNG_SECURITY_STRENGTH_BYTES);
+ void *old_drng, *new_hash, *old_hash;
+ bool sl = false, reset_drng = !lrng_get_available();
+
+ if (IS_ERR(new_drng)) {
+ pr_warn("could not allocate new DRNG for NUMA node %d (%ld)\n",
+ node, PTR_ERR(new_drng));
+ return PTR_ERR(new_drng);
+ }
+
+ new_hash = cb->lrng_hash_alloc(seed, sizeof(seed));
+ if (IS_ERR(new_hash)) {
+ pr_warn("could not allocate new LRNG pool hash (%ld)\n",
+ PTR_ERR(new_hash));
+ cb->lrng_drng_dealloc(new_drng);
+ return PTR_ERR(new_hash);
+ }
+
+ lrng_drng_lock(drng_store, &flags);
+
+ /*
+ * Pull from existing DRNG to seed new DRNG regardless of seed status
+ * of old DRNG -- the entropy state for the DRNG is left unchanged which
+ * implies that als the new DRNG is reseeded when deemed necessary. This
+ * seeding of the new DRNG shall only ensure that the new DRNG has the
+ * same entropy as the old DRNG.
+ */
+ ret = drng_store->crypto_cb->lrng_drng_generate_helper(
+ drng_store->drng, seed, sizeof(seed));
+ lrng_drng_unlock(drng_store, &flags);
+
+ if (ret < 0) {
+ reset_drng = true;
+ pr_warn("getting random data from DRNG failed for NUMA node %d "
+ "(%d)\n", node, ret);
+ } else {
+ /* seed new DRNG with data */
+ ret = cb->lrng_drng_seed_helper(new_drng, seed, ret);
+ if (ret < 0) {
+ reset_drng = true;
+ pr_warn("seeding of new DRNG failed for NUMA node %d "
+ "(%d)\n", node, ret);
+ } else {
+ pr_debug("seeded new DRNG of NUMA node %d instance "
+ "from old DRNG instance\n", node);
+ }
+ }
+
+ mutex_lock(&drng_store->lock);
+ /*
+ * If we switch the DRNG from the initial ChaCha20 DRNG to something
+ * else, there is a lock transition from spin lock to mutex (see
+ * lrng_drng_is_atomic and how the lock is taken in lrng_drng_lock).
+ * Thus, we need to take both locks during the transition phase.
+ */
+ if (lrng_drng_is_atomic(drng_store)) {
+ spin_lock_irqsave(&drng_store->spin_lock, flags);
+ sl = true;
+ }
+
+ if (reset_drng)
+ lrng_drng_reset(drng_store);
+
+ old_drng = drng_store->drng;
+ old_cb = drng_store->crypto_cb;
+ drng_store->drng = new_drng;
+ drng_store->crypto_cb = cb;
+
+ old_hash = drng_store->hash;
+ drng_store->hash = new_hash;
+ pr_info("Entropy pool read-hash allocated for DRNG for NUMA node %d\n",
+ node);
+
+ if (sl)
+ spin_unlock_irqrestore(&drng_store->spin_lock, flags);
+ mutex_unlock(&drng_store->lock);
+
+ /* ChaCha20 serves as atomic instance left untouched. */
+ if (old_drng != &chacha20) {
+ old_cb->lrng_drng_dealloc(old_drng);
+ old_cb->lrng_hash_dealloc(old_hash);
+ }
+
+ pr_info("DRNG of NUMA node %d switched\n", node);
+
+ return 0;
+}
+
+/**
+ * Switch the existing DRNG instances with new using the new crypto callbacks.
+ * The caller must hold the lrng_crypto_cb_update lock.
+ */
+static int lrng_drngs_switch(const struct lrng_crypto_cb *cb)
+{
+ struct lrng_drng **lrng_drng = lrng_drng_instances();
+ struct lrng_drng *lrng_drng_init = lrng_drng_init_instance();
+ int ret = 0;
+
+ /* Update DRNG */
+ if (lrng_drng) {
+ u32 node;
+
+ for_each_online_node(node) {
+ if (lrng_drng[node])
+ ret = lrng_drng_switch(lrng_drng[node], cb,
+ node);
+ }
+ } else {
+ ret = lrng_drng_switch(lrng_drng_init, cb, 0);
+ }
+
+ if (!ret)
+ lrng_set_available();
+
+ return 0;
+}
+
+/**
+ * lrng_set_drng_cb - Register new cryptographic callback functions for DRNG
+ * The registering implies that all old DRNG states are replaced with new
+ * DRNG states.
+ * @cb: Callback functions to be registered -- if NULL, use the default
+ * callbacks pointing to the ChaCha20 DRNG.
+ * @return: 0 on success, < 0 on error
+ */
+int lrng_set_drng_cb(const struct lrng_crypto_cb *cb)
+{
+ struct lrng_drng *lrng_drng_init = lrng_drng_init_instance();
+ int ret;
+
+ if (!cb)
+ cb = &lrng_cc20_crypto_cb;
+
+ mutex_lock(&lrng_crypto_cb_update);
+
+ /*
+ * If a callback other than the default is set, allow it only to be
+ * set back to the default callback. This ensures that multiple
+ * different callbacks can be registered at the same time. If a
+ * callback different from the current callback and the default
+ * callback shall be set, the current callback must be deregistered
+ * (e.g. the kernel module providing it must be unloaded) and the new
+ * implementation can be registered.
+ */
+ if ((cb != &lrng_cc20_crypto_cb) &&
+ (lrng_drng_init->crypto_cb != &lrng_cc20_crypto_cb)) {
+ pr_warn("disallow setting new cipher callbacks, unload the old "
+ "callbacks first!\n");
+ ret = -EINVAL;
+ goto out;
+ }
+
+ ret = lrng_drngs_switch(cb);
+
+out:
+ mutex_unlock(&lrng_crypto_cb_update);
+ return ret;
+}
+EXPORT_SYMBOL(lrng_set_drng_cb);
--
2.24.1
^ permalink raw reply related
* [PATCH v27 03/12] LRNG - sysctls and /proc interface
From: Stephan Müller @ 2020-01-09 8:31 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Greg Kroah-Hartman, linux-crypto, LKML, linux-api,
Eric W. Biederman, Alexander E. Patrakov, Ahmed S. Darwish,
Theodore Y. Ts'o, Willy Tarreau, Matthew Garrett, Vito Caputo,
Andreas Dilger, Jan Kara, Ray Strode, William Jon McCann, zhangjs,
Andy Lutomirski, Florian Weimer, Lennart Poettering,
Nicolai Stange, "Peter, Matthias" <matth>
In-Reply-To: <2641155.iNH938UiKq@positron.chronox.de>
The LRNG sysctl interface provides the same controls as the existing
/dev/random implementation. These sysctls behave identically and are
implemented identically. The goal is to allow a possible merge of the
existing /dev/random implementation with this implementation which
implies that this patch tries have a very close similarity. Yet, all
sysctls are documented at [1].
In addition, it provides the file lrng_type which provides details about
the LRNG:
- the name of the DRNG that produces the random numbers for /dev/random,
/dev/urandom, getrandom(2)
- the hash used to produce random numbers from the entropy pool
- the number of secondary DRNG instances
- indicator whether the LRNG operates SP800-90B compliant
- indicator whether a high-resolution timer is identified - only with a
high-resolution timer the interrupt noise source will deliver sufficient
entropy
- indicator whether the LRNG has been minimally seeded (i.e. is the
secondary DRNG seeded with at least 128 bits of of entropy)
- indicator whether the LRNG has been fully seeded (i.e. is the
secondary DRNG seeded with at least 256 bits of entropy)
[1] https://www.chronox.de/lrng.html
CC: "Eric W. Biederman" <ebiederm@xmission.com>
CC: "Alexander E. Patrakov" <patrakov@gmail.com>
CC: "Ahmed S. Darwish" <darwish.07@gmail.com>
CC: "Theodore Y. Ts'o" <tytso@mit.edu>
CC: Willy Tarreau <w@1wt.eu>
CC: Matthew Garrett <mjg59@srcf.ucam.org>
CC: Vito Caputo <vcaputo@pengaru.com>
CC: Andreas Dilger <adilger.kernel@dilger.ca>
CC: Jan Kara <jack@suse.cz>
CC: Ray Strode <rstrode@redhat.com>
CC: William Jon McCann <mccann@jhu.edu>
CC: zhangjs <zachary@baishancloud.com>
CC: Andy Lutomirski <luto@kernel.org>
CC: Florian Weimer <fweimer@redhat.com>
CC: Lennart Poettering <mzxreary@0pointer.de>
CC: Nicolai Stange <nstange@suse.de>
Reviewed-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Reviewed-by: Roman Drahtmueller <draht@schaltsekun.de>
Tested-by: Roman Drahtmüller <draht@schaltsekun.de>
Tested-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Tested-by: Neil Horman <nhorman@redhat.com>
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
drivers/char/lrng/Makefile | 1 +
drivers/char/lrng/lrng_interfaces.c | 1 -
drivers/char/lrng/lrng_internal.h | 4 +
drivers/char/lrng/lrng_proc.c | 163 ++++++++++++++++++++++++++++
4 files changed, 168 insertions(+), 1 deletion(-)
create mode 100644 drivers/char/lrng/lrng_proc.c
diff --git a/drivers/char/lrng/Makefile b/drivers/char/lrng/Makefile
index 0a32f22c2c1a..e69c176f0161 100644
--- a/drivers/char/lrng/Makefile
+++ b/drivers/char/lrng/Makefile
@@ -9,3 +9,4 @@ obj-y += lrng_pool.o lrng_aux.o \
lrng_interfaces.o \
obj-$(CONFIG_NUMA) += lrng_numa.o
+obj-$(CONFIG_SYSCTL) += lrng_proc.o
diff --git a/drivers/char/lrng/lrng_interfaces.c b/drivers/char/lrng/lrng_interfaces.c
index ff40ed37ddd6..8c4bf53d86ec 100644
--- a/drivers/char/lrng/lrng_interfaces.c
+++ b/drivers/char/lrng/lrng_interfaces.c
@@ -34,7 +34,6 @@ static DECLARE_WAIT_QUEUE_HEAD(lrng_write_wait);
static DECLARE_WAIT_QUEUE_HEAD(lrng_init_wait);
static struct fasync_struct *fasync;
-struct ctl_table random_table[];
/********************************** Helper ***********************************/
/* Is the DRNG seed level too low? */
diff --git a/drivers/char/lrng/lrng_internal.h b/drivers/char/lrng/lrng_internal.h
index c2b0a467548e..349152d3c1b0 100644
--- a/drivers/char/lrng/lrng_internal.h
+++ b/drivers/char/lrng/lrng_internal.h
@@ -116,7 +116,11 @@ void lrng_cc20_init_state(struct chacha20_state *state);
/********************************** /proc *************************************/
+#ifdef CONFIG_SYSCTL
+void lrng_pool_inc_numa_node(void);
+#else
static inline void lrng_pool_inc_numa_node(void) { }
+#endif
/****************************** LRNG interfaces *******************************/
diff --git a/drivers/char/lrng/lrng_proc.c b/drivers/char/lrng/lrng_proc.c
new file mode 100644
index 000000000000..f4ee01f61925
--- /dev/null
+++ b/drivers/char/lrng/lrng_proc.c
@@ -0,0 +1,163 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG proc and sysctl interfaces
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#include <linux/lrng.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#include <linux/sysctl.h>
+#include <linux/uuid.h>
+
+#include "lrng_internal.h"
+
+/*
+ * This function is used to return both the bootid UUID, and random
+ * UUID. The difference is in whether table->data is NULL; if it is,
+ * then a new UUID is generated and returned to the user.
+ *
+ * If the user accesses this via the proc interface, the UUID will be
+ * returned as an ASCII string in the standard UUID format; if via the
+ * sysctl system call, as 16 bytes of binary data.
+ */
+static int lrng_proc_do_uuid(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp, loff_t *ppos)
+{
+ struct ctl_table fake_table;
+ unsigned char buf[64], tmp_uuid[16], *uuid;
+
+ uuid = table->data;
+ if (!uuid) {
+ uuid = tmp_uuid;
+ generate_random_uuid(uuid);
+ } else {
+ static DEFINE_SPINLOCK(bootid_spinlock);
+
+ spin_lock(&bootid_spinlock);
+ if (!uuid[8])
+ generate_random_uuid(uuid);
+ spin_unlock(&bootid_spinlock);
+ }
+
+ sprintf(buf, "%pU", uuid);
+
+ fake_table.data = buf;
+ fake_table.maxlen = sizeof(buf);
+
+ return proc_dostring(&fake_table, write, buffer, lenp, ppos);
+}
+
+static int lrng_proc_do_entropy(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp, loff_t *ppos)
+{
+ struct ctl_table fake_table;
+ int entropy_count;
+
+ entropy_count = lrng_avail_entropy();
+
+ fake_table.data = &entropy_count;
+ fake_table.maxlen = sizeof(entropy_count);
+
+ return proc_dointvec(&fake_table, write, buffer, lenp, ppos);
+}
+
+static int lrng_sysctl_poolsize = LRNG_POOL_SIZE_BITS;
+static int lrng_min_write_thresh;
+static int lrng_max_write_thresh = LRNG_POOL_SIZE_BITS;
+static char lrng_sysctl_bootid[16];
+static int lrng_drng_reseed_max_min;
+
+struct ctl_table random_table[] = {
+ {
+ .procname = "poolsize",
+ .data = &lrng_sysctl_poolsize,
+ .maxlen = sizeof(int),
+ .mode = 0444,
+ .proc_handler = proc_dointvec,
+ },
+ {
+ .procname = "entropy_avail",
+ .maxlen = sizeof(int),
+ .mode = 0444,
+ .proc_handler = lrng_proc_do_entropy,
+ },
+ {
+ .procname = "write_wakeup_threshold",
+ .data = &lrng_write_wakeup_bits,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &lrng_min_write_thresh,
+ .extra2 = &lrng_max_write_thresh,
+ },
+ {
+ .procname = "boot_id",
+ .data = &lrng_sysctl_bootid,
+ .maxlen = 16,
+ .mode = 0444,
+ .proc_handler = lrng_proc_do_uuid,
+ },
+ {
+ .procname = "uuid",
+ .maxlen = 16,
+ .mode = 0444,
+ .proc_handler = lrng_proc_do_uuid,
+ },
+ {
+ .procname = "urandom_min_reseed_secs",
+ .data = &lrng_drng_reseed_max_time,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec,
+ .extra1 = &lrng_drng_reseed_max_min,
+ },
+ { }
+};
+
+/* Number of online DRNGs */
+static u32 numa_drngs = 1;
+
+void lrng_pool_inc_numa_node(void)
+{
+ numa_drngs++;
+}
+
+static int lrng_proc_type_show(struct seq_file *m, void *v)
+{
+ struct lrng_drng *lrng_drng_init = lrng_drng_init_instance();
+ unsigned long flags = 0;
+ unsigned char buf[300];
+
+ lrng_drng_lock(lrng_drng_init, &flags);
+ snprintf(buf, sizeof(buf),
+ "DRNG name: %s\n"
+ "Hash for reading entropy pool: %s\n"
+ "DRNG security strength: %d bits\n"
+ "number of DRNG instances: %u\n"
+ "SP800-90B compliance: %s\n"
+ "High-resolution timer: %s\n"
+ "LRNG minimally seeded: %s\n"
+ "LRNG fully seeded: %s\n",
+ lrng_drng_init->crypto_cb->lrng_drng_name(),
+ lrng_drng_init->crypto_cb->lrng_hash_name(),
+ LRNG_DRNG_SECURITY_STRENGTH_BITS, numa_drngs,
+ lrng_sp80090b_compliant() ? "true" : "false",
+ lrng_pool_highres_timer() ? "true" : "false",
+ lrng_state_min_seeded() ? "true" : "false",
+ lrng_state_fully_seeded() ? "true" : "false");
+ lrng_drng_unlock(lrng_drng_init, &flags);
+
+ seq_write(m, buf, strlen(buf));
+
+ return 0;
+}
+
+static int __init lrng_proc_type_init(void)
+{
+ proc_create_single("lrng_type", 0444, NULL, &lrng_proc_type_show);
+ return 0;
+}
+
+module_init(lrng_proc_type_init);
--
2.24.1
^ permalink raw reply related
* [PATCH v27 02/12] LRNG - allocate one DRNG instance per NUMA node
From: Stephan Müller @ 2020-01-09 8:31 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Greg Kroah-Hartman, linux-crypto, LKML, linux-api,
Eric W. Biederman, Alexander E. Patrakov, Ahmed S. Darwish,
Theodore Y. Ts'o, Willy Tarreau, Matthew Garrett, Vito Caputo,
Andreas Dilger, Jan Kara, Ray Strode, William Jon McCann, zhangjs,
Andy Lutomirski, Florian Weimer, Lennart Poettering,
Nicolai Stange, "Peter, Matthias" <matth>
In-Reply-To: <2641155.iNH938UiKq@positron.chronox.de>
In order to improve NUMA-locality when serving getrandom(2) requests,
allocate one DRNG instance per node.
The DRNG instance that is present right from the start of the kernel is
reused as the first per-NUMA-node DRNG. For all remaining online NUMA
nodes a new DRNG instance is allocated.
During boot time, the multiple DRNG instances are seeded sequentially.
With this, the first DRNG instance (referenced as the initial DRNG
in the code) is completely seeded with 256 bits of entropy before the
next DRNG instance is completely seeded.
When random numbers are requested, the NUMA-node-local DRNG is checked
whether it has been already fully seeded. If this is not the case, the
initial DRNG is used to serve the request.
CC: "Eric W. Biederman" <ebiederm@xmission.com>
CC: "Alexander E. Patrakov" <patrakov@gmail.com>
CC: "Ahmed S. Darwish" <darwish.07@gmail.com>
CC: "Theodore Y. Ts'o" <tytso@mit.edu>
CC: Willy Tarreau <w@1wt.eu>
CC: Matthew Garrett <mjg59@srcf.ucam.org>
CC: Vito Caputo <vcaputo@pengaru.com>
CC: Andreas Dilger <adilger.kernel@dilger.ca>
CC: Jan Kara <jack@suse.cz>
CC: Ray Strode <rstrode@redhat.com>
CC: William Jon McCann <mccann@jhu.edu>
CC: zhangjs <zachary@baishancloud.com>
CC: Andy Lutomirski <luto@kernel.org>
CC: Florian Weimer <fweimer@redhat.com>
CC: Lennart Poettering <mzxreary@0pointer.de>
CC: Nicolai Stange <nstange@suse.de>
Reviewed-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Reviewed-by: Roman Drahtmueller <draht@schaltsekun.de>
Tested-by: Roman Drahtmüller <draht@schaltsekun.de>
Tested-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Tested-by: Neil Horman <nhorman@redhat.com>
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
drivers/char/lrng/Makefile | 2 +
drivers/char/lrng/lrng_internal.h | 5 ++
drivers/char/lrng/lrng_numa.c | 101 ++++++++++++++++++++++++++++++
3 files changed, 108 insertions(+)
create mode 100644 drivers/char/lrng/lrng_numa.c
diff --git a/drivers/char/lrng/Makefile b/drivers/char/lrng/Makefile
index 1d2a0211973d..0a32f22c2c1a 100644
--- a/drivers/char/lrng/Makefile
+++ b/drivers/char/lrng/Makefile
@@ -7,3 +7,5 @@ obj-y += lrng_pool.o lrng_aux.o \
lrng_sw_noise.o lrng_archrandom.o \
lrng_drng.o lrng_chacha20.o \
lrng_interfaces.o \
+
+obj-$(CONFIG_NUMA) += lrng_numa.o
diff --git a/drivers/char/lrng/lrng_internal.h b/drivers/char/lrng/lrng_internal.h
index c9a7bccd14a0..c2b0a467548e 100644
--- a/drivers/char/lrng/lrng_internal.h
+++ b/drivers/char/lrng/lrng_internal.h
@@ -246,8 +246,13 @@ int lrng_drng_get_sleep(u8 *outbuf, u32 outbuflen);
void lrng_drng_force_reseed(void);
void lrng_drng_seed_work(struct work_struct *dummy);
+#ifdef CONFIG_NUMA
+struct lrng_drng **lrng_drng_instances(void);
+void lrng_drngs_numa_alloc(void);
+#else /* CONFIG_NUMA */
static inline struct lrng_drng **lrng_drng_instances(void) { return NULL; }
static inline void lrng_drngs_numa_alloc(void) { return; }
+#endif /* CONFIG_NUMA */
/************************** Health Test linking code **************************/
diff --git a/drivers/char/lrng/lrng_numa.c b/drivers/char/lrng/lrng_numa.c
new file mode 100644
index 000000000000..947c5b3ed517
--- /dev/null
+++ b/drivers/char/lrng/lrng_numa.c
@@ -0,0 +1,101 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG NUMA support
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/lrng.h>
+#include <linux/slab.h>
+
+#include "lrng_internal.h"
+
+static struct lrng_drng **lrng_drng __read_mostly = NULL;
+
+struct lrng_drng **lrng_drng_instances(void)
+{
+ return lrng_drng;
+}
+
+/* Allocate the data structures for the per-NUMA node DRNGs */
+static void _lrng_drngs_numa_alloc(struct work_struct *work)
+{
+ struct lrng_drng **drngs;
+ struct lrng_drng *lrng_drng_init = lrng_drng_init_instance();
+ u32 node;
+ bool init_drng_used = false;
+
+ mutex_lock(&lrng_crypto_cb_update);
+
+ /* per-NUMA-node DRNGs are already present */
+ if (lrng_drng)
+ goto unlock;
+
+ drngs = kcalloc(nr_node_ids, sizeof(void *), GFP_KERNEL|__GFP_NOFAIL);
+ for_each_online_node(node) {
+ struct lrng_drng *drng;
+
+ if (!init_drng_used) {
+ drngs[node] = lrng_drng_init;
+ init_drng_used = true;
+ continue;
+ }
+
+ drng = kmalloc_node(sizeof(struct lrng_drng),
+ GFP_KERNEL|__GFP_NOFAIL, node);
+ memset(drng, 0, sizeof(lrng_drng));
+
+ drng->crypto_cb = lrng_drng_init->crypto_cb;
+ drng->drng = drng->crypto_cb->lrng_drng_alloc(
+ LRNG_DRNG_SECURITY_STRENGTH_BYTES);
+ if (IS_ERR(drng->drng)) {
+ kfree(drng);
+ goto err;
+ }
+
+ mutex_init(&drng->lock);
+ spin_lock_init(&drng->spin_lock);
+
+ /*
+ * No reseeding of NUMA DRNGs from previous DRNGs as this
+ * would complicate the code. Let it simply reseed.
+ */
+ lrng_drng_reset(drng);
+ drngs[node] = drng;
+
+ lrng_pool_inc_numa_node();
+ pr_info("DRNG for NUMA node %d allocated\n", node);
+ }
+
+ /* Ensure that all NUMA nodes receive changed memory here. */
+ mb();
+
+ if (!cmpxchg(&lrng_drng, NULL, drngs))
+ goto unlock;
+
+err:
+ for_each_online_node(node) {
+ struct lrng_drng *drng = drngs[node];
+
+ if (drng == lrng_drng_init)
+ continue;
+
+ if (drng) {
+ drng->crypto_cb->lrng_drng_dealloc(drng->drng);
+ kfree(drng);
+ }
+ }
+ kfree(drngs);
+
+unlock:
+ mutex_unlock(&lrng_crypto_cb_update);
+}
+
+static DECLARE_WORK(lrng_drngs_numa_alloc_work, _lrng_drngs_numa_alloc);
+
+void lrng_drngs_numa_alloc(void)
+{
+ schedule_work(&lrng_drngs_numa_alloc_work);
+}
--
2.24.1
^ permalink raw reply related
* [PATCH v27 01/12] Linux Random Number Generator
From: Stephan Müller @ 2020-01-09 8:30 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Greg Kroah-Hartman, linux-crypto, LKML, linux-api,
Eric W. Biederman, Alexander E. Patrakov, Ahmed S. Darwish,
Theodore Y. Ts'o, Willy Tarreau, Matthew Garrett, Vito Caputo,
Andreas Dilger, Jan Kara, Ray Strode, William Jon McCann, zhangjs,
Andy Lutomirski, Florian Weimer, Lennart Poettering,
Nicolai Stange, "Peter, Matthias" <matth>
In-Reply-To: <2641155.iNH938UiKq@positron.chronox.de>
In an effort to provide a flexible implementation for a random number
generator that also delivers entropy during early boot time, allows
replacement of the deterministic random number generation mechanism,
implement the various components in separate code for easier
maintenance, and provide compliance to SP800-90[A|B|C], introduce
the Linux Random Number Generator (LRNG) framework.
The general design is as follows. Additional implementation details
are given in [1]. The LRNG consists of the following components:
1. The LRNG implements a DRNG. The DRNG always generates the
requested amount of output. When using the SP800-90A terminology
it operates without prediction resistance. The secondary DRNG
maintains a counter of how many bytes were generated since last
re-seed and a timer of the elapsed time since last re-seed. If either
the counter or the timer reaches a threshold, the secondary DRNG is
seeded from the entropy pool.
In case the Linux kernel detects a NUMA system, one secondary DRNG
instance per NUMA node is maintained.
2. The DRNG is seeded by concatenating the data from the
following sources:
(a) the output of the entropy pool,
(b) the Jitter RNG if available and enabled, and
(c) the CPU-based noise source such as Intel RDRAND if available and
enabled.
The entropy estimate of the data of all noise sources are added to
form the entropy estimate of the data used to seed the DRNG with.
The LRNG ensures, however, that the DRNG after seeding is at
maximum the security strength of the DRNG.
The LRNG is designed such that none of these noise sources can dominate
the other noise sources to provide seed data to the DRNG during due to
the following:
(a) During boot time, the amount of received interrupts are the trigger
points to (re)seed the DRNG.
(b) At runtime, the available entropy from the slow noise source is
concatenated with a pre-defined amount of data from the fast noise
sources. In addition, each DRNG reseed operation triggers external
noise source providers to deliver one block of data.
3. The entropy pool accumulates entropy obtained from certain events,
which will henceforth be collectively called "slow noise sources".
The entropy pool collects noise data from slow noise sources. Any data
received by the LRNG from the slow noise sources is inserted into the
entropy pool using an LFSR with a primitive and irreducible polynomial.
The following sources of entropy are used:
(a) When an interrupt occurs, the high-resolution time stamp is mixed
into the LFSR. This time stamp is credited with heuristically implied
entropy.
(b) HID event data like the key stroke or the mouse coordinates are
mixed into the LFSR. This data is not credited with entropy by the LRNG.
(c) Device drivers may provide data that is mixed into the LFSR. This
data is not credited with entropy by the LRNG.
(d) After the entropy pool is ``read'' by the DRNG, the data
used to seed the DRNG is mixed back into the entropy pool to
stir the pool. This data is not credited with entropy by the LRNG.
Any data provided from user space by either writing to /dev/random,
/dev/urandom or the IOCTL of RNDADDENTROPY on both device files
are always injected into the entropy pool.
In addition, when a hardware random number generator covered by the
Linux kernel HW generator framework wants to deliver random numbers,
it is injected into the entropy pool as well. HW generator noise source
is handled separately from the other noise source due to the fact that
the HW generator framework may decide by itself when to deliver data
whereas the other noise sources always requested for data driven by the
LRNG operation. Similarly any user space provided data is inserted into
the entropy pool.
When the DRNG requires data from the entropy pool, the entire
entropy pool is processed with an SP800-90A section 10.3.1 compliant
hash_df function to generate random numbers.
To speed up the interrupt handling code of the LRNG, the time stamp
collected for an interrupt event is truncated to the 8 least
significant bits. 64 truncated time stamps are concatenated and then
jointly inserted into the LFSR. During boot time, until the fully seeded
stage is reached, each time stamp with its 32 least significant bits is
inserted into the LFSR at the time of arrival.
The LRNG allows the DRNG mechanism to be changed at runtime. Per default,
a ChaCha20-based DRNG is used. The ChaCha20-DRNG implemented for the
LRNG is also provided as a stand-alone user space deterministic random
number generator. The LRNG also offers an SP800-90A DRBG based on the
Linux kernel crypto API DRBG implementation.
The processing of entropic data from the noise source before injecting
them into the DRNG is performed with the following mathematical
operations:
1. LFSR: The 8 least significant bits of the time stamp data received
from the interrupts are processed with an LFSR. That LFSR is implemented
identically to the LSFR used in the existing /dev/random implementation
except that it is capable of processing an entire word and that a
different polynomial is used. The reason for the different polynomial
is performance in a performance sensitive code section, the interrupt
handler. The chosen polynomials have 4 taps. Also, this LFSR-approach
is used in the OpenBSD /dev/random equivalent.
2. Concatenation: The temporary seed buffer used to seed the DRNG is
a concatenation of parts of the entropy pool data, and the CPU noise
source output.
The DRNG always tries to seed itself with 256 bits of entropy, except
during boot. In any case, if the noise sources cannot deliver that
amount, the available entropy is used and the DRNG keeps track on how
much entropy it was seeded with. The entropy implied by the LRNG
available in the entropy pool may be too conservative. To ensure
that during boot time all available entropy from the entropy pool is
transferred to the DRNG, the hash_df function always generates 256
data bits during boot to seed the DRNG. During boot, the DRNG is
seeded as follows:
1. The DRNG is reseeded from the entropy pool and potentially the fast
noise sources if the entropy pool has collected at least 32 bits of
entropy from the interrupt noise source. The goal of this step is to
ensure that the DRNG receives some initial entropy as early as
possible. In addition it receives the entropy available from
the fast noise sources.
2. The DRNG is reseeded from the entropy pool and potentially the fast
noise sources if all noise sources collectively can provide at least
128 bits of entropy.
3. The DRNG is reseeded from the entropy pool and potentially the fast
noise sources if all noise sources collectivel can provide at least 256
bits.
At the time of the reseeding steps, the DRNG requests as much entropy as
is available in order to skip certain steps and reach the seeding level
of 256 bits. This may imply that one or more of the aforementioned steps
are skipped.
In all listed steps, the DRNG is (re)seeded with a number of random
bytes from the entropy pool that is at most the amount of entropy
present in the entropy pool. This means that when the entropy pool
contains 128 or 256 bits of entropy, the DRNG is seeded with that
amount of entropy as well.
Before the DRNG is seeded with 256 bits of entropy in step 3,
requests of random data from /dev/random are not processed.
The hash_df operation providing random data from the entropy pool will
always require that all entropy sources collectively can deliver at
least 129 entropy bits as configured with (128 bits of entropy for
seeding plus one bit of entropy that is lost with the post
processing as defined in SP800-90B).
The DRNG operates as deterministic random number generator with the
following properties:
* The maximum number of random bytes that can be generated with one
DRNG generate operation is limited to 4096 bytes. When longer random
numbers are requested, multiple DRNG generate operations are performed.
The ChaCha20 DRNG as well as the SP800-90A DRBGs implement an update of
their state after completing a generate request for backtracking
resistance.
* The secondary DRNG is reseeded with whatever entropy is available –
in the worst case where no additional entropy can be provided by the
noise sources, the DRNG is not re-seeded and continues its operation
to try to reseed again after again the expiry of one of these thresholds:
- If the last reseeding of the secondary DRNG is more than 600 seconds
ago, or
- 2^20 DRNG generate operations are performed, whatever comes first, or
- the secondary DRNG is forced to reseed before the next generation of
random numbers if data has been injected into the LRNG by writing data
into /dev/random or /dev/urandom.
The chosen values prevent high-volume requests from user space to cause
frequent reseeding operations which drag down the performance of the
DRNG.
With the automatic reseeding after 600 seconds, the LRNG is triggered
to reseed itself before the first request after a suspend that put the
hardware to sleep for longer than 600 seconds.
The LRNG uses the following runtime memory using the currently
smallest configuration:
* 572 bytes (512 bytes for the entropy pool and 64 for the entropy pool
meta data) for the entropy pool management
* 64 bytes per CPU for the time stamp array
To support smaller devices including IoT environments, this patch
allows reducing the runtime memory footprint of the LRNG at compile
time by selecting smaller entropy pool sizes.
The entropy pool has support for sizes of 256, 128 and 64 bytes supported
by primitive and irreducible polynomials.
The time stamp array is reduced to one atomic_t variable per CPU, i.e.
4 bytes when CONFIG_BASE_SMALL is selected during kernel
configuration. This implies that after the receipt of 4 interrupts on
one CPU, the data is injected into the LFSR. Depending on the behavior
of the CPU caches, this may imply that the average interrupt handler
execution time increases a bit, since instead of injecting 8 atomic_t
values at one given time into the LFSR, only one is processed which
may incur cache misses.
When selecting the compilation of a kernel for a small environment,
prevent the allocation of a buffer up to 4096 bytes to serve user space
requests. In this case, the stack variable of 64 bytes is used to serve
all user space requests.
The LRNG has the following properties:
* internal noise source: interrupts timing with fast boot time seeding
* high performance of interrupt handling code: The LRNG impact on the
interrupt handling has been reduced to a minimum. On one example
system, the LRNG interrupt handling code executes within an average
of 65 cycles whereas the existing /dev/random on the same device
takes about 97 cycles when measuring the execution time of
add_interrupt_randomness().
* lockless LFSR to collect raw entropy supporing concurrency-free
use of massive parallel systems
* use of standalone ChaCha20 based RNG with the option to use a
different DRNG selectable at compile time
* "atomic" seeding of secondary DRBG to ensure full entropy transport
* instantiate one DRNG per NUMA node
* support for runtime switchable output DRNGs
* use of only well-defined entropy-preserving operations to collect,
compress and forward entropy: concatenation, LFSR, SP800-90A hash_df
function
* compile-time selectable entropy pool size: the choice also
uses the applicable LFSR polynomial to maintain the entropy pool
size
* support of small systems by allowing the reduction of the
runtime memory needs
Further details including the rationale for the design choices and
properties of the LRNG together with testing is provided at [1].
In addition, the documentation explains the conducted regression
tests to verify that the LRNG is API and ABI compatible with the
existing /dev/random implementation.
[1] https://www.chronox.de/lrng.html
CC: "Eric W. Biederman" <ebiederm@xmission.com>
CC: "Alexander E. Patrakov" <patrakov@gmail.com>
CC: "Ahmed S. Darwish" <darwish.07@gmail.com>
CC: "Theodore Y. Ts'o" <tytso@mit.edu>
CC: Willy Tarreau <w@1wt.eu>
CC: Matthew Garrett <mjg59@srcf.ucam.org>
CC: Vito Caputo <vcaputo@pengaru.com>
CC: Andreas Dilger <adilger.kernel@dilger.ca>
CC: Jan Kara <jack@suse.cz>
CC: Ray Strode <rstrode@redhat.com>
CC: William Jon McCann <mccann@jhu.edu>
CC: zhangjs <zachary@baishancloud.com>
CC: Andy Lutomirski <luto@kernel.org>
CC: Florian Weimer <fweimer@redhat.com>
CC: Lennart Poettering <mzxreary@0pointer.de>
CC: Nicolai Stange <nstange@suse.de>
Mathematical aspects Reviewed-by: "Peter, Matthias" <matthias.peter@bsi.bund.de>
Reviewed-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Reviewed-by: Roman Drahtmueller <draht@schaltsekun.de>
Tested-by: Roman Drahtmüller <draht@schaltsekun.de>
Tested-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Tested-by: Neil Horman <nhorman@redhat.com>
Signed-off-by: Stephan Mueller <smueller@chronox.de>
---
MAINTAINERS | 7 +
drivers/char/Kconfig | 2 +
drivers/char/Makefile | 9 +-
drivers/char/lrng/Kconfig | 67 +++
drivers/char/lrng/Makefile | 9 +
drivers/char/lrng/lrng_archrandom.c | 92 ++++
drivers/char/lrng/lrng_aux.c | 148 +++++++
drivers/char/lrng/lrng_chacha20.c | 317 ++++++++++++++
drivers/char/lrng/lrng_chacha20.h | 25 ++
drivers/char/lrng/lrng_drng.c | 397 ++++++++++++++++++
drivers/char/lrng/lrng_interfaces.c | 623 ++++++++++++++++++++++++++++
drivers/char/lrng/lrng_internal.h | 296 +++++++++++++
drivers/char/lrng/lrng_lfsr.h | 152 +++++++
drivers/char/lrng/lrng_pool.c | 585 ++++++++++++++++++++++++++
drivers/char/lrng/lrng_sw_noise.c | 102 +++++
drivers/char/lrng/lrng_sw_noise.h | 57 +++
include/linux/lrng.h | 71 ++++
17 files changed, 2958 insertions(+), 1 deletion(-)
create mode 100644 drivers/char/lrng/Kconfig
create mode 100644 drivers/char/lrng/Makefile
create mode 100644 drivers/char/lrng/lrng_archrandom.c
create mode 100644 drivers/char/lrng/lrng_aux.c
create mode 100644 drivers/char/lrng/lrng_chacha20.c
create mode 100644 drivers/char/lrng/lrng_chacha20.h
create mode 100644 drivers/char/lrng/lrng_drng.c
create mode 100644 drivers/char/lrng/lrng_interfaces.c
create mode 100644 drivers/char/lrng/lrng_internal.h
create mode 100644 drivers/char/lrng/lrng_lfsr.h
create mode 100644 drivers/char/lrng/lrng_pool.c
create mode 100644 drivers/char/lrng/lrng_sw_noise.c
create mode 100644 drivers/char/lrng/lrng_sw_noise.h
create mode 100644 include/linux/lrng.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 8982c6e013b3..f9beac15783c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9606,6 +9606,13 @@ F: Documentation/core-api/atomic_ops.rst
F: Documentation/core-api/refcount-vs-atomic.rst
F: Documentation/memory-barriers.txt
+LINUX RANDOM NUMBER GENERATOR (LRNG) DRIVER
+M: Stephan Mueller <smueller@chronox.de>
+S: Maintained
+W: https://www.chronox.de/lrng.html
+F: drivers/char/lrng/*
+F: include/linux/lrng.h
+
LIS3LV02D ACCELEROMETER DRIVER
M: Eric Piel <eric.piel@tremplin-utc.net>
S: Maintained
diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig
index 26956c006987..1e758735f633 100644
--- a/drivers/char/Kconfig
+++ b/drivers/char/Kconfig
@@ -535,6 +535,8 @@ config ADI
and SSM (Silicon Secured Memory). Intended consumers of this
driver include crash and makedumpfile.
+source "drivers/char/lrng/Kconfig"
+
endmenu
config RANDOM_TRUST_CPU
diff --git a/drivers/char/Makefile b/drivers/char/Makefile
index 7c5ea6f9df14..46ede09fd6d3 100644
--- a/drivers/char/Makefile
+++ b/drivers/char/Makefile
@@ -3,7 +3,14 @@
# Makefile for the kernel character device drivers.
#
-obj-y += mem.o random.o
+obj-y += mem.o
+
+ifeq ($(CONFIG_LRNG),y)
+ obj-y += lrng/
+else
+ obj-y += random.o
+endif
+
obj-$(CONFIG_TTY_PRINTK) += ttyprintk.o
obj-y += misc.o
obj-$(CONFIG_ATARI_DSP56K) += dsp56k.o
diff --git a/drivers/char/lrng/Kconfig b/drivers/char/lrng/Kconfig
new file mode 100644
index 000000000000..56f13efd3592
--- /dev/null
+++ b/drivers/char/lrng/Kconfig
@@ -0,0 +1,67 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Linux Random Number Generator configuration
+#
+
+menuconfig LRNG
+ bool "Linux Random Number Generator"
+ help
+ The Linux Random Number Generator (LRNG) is the replacement
+ of the existing /dev/random provided with drivers/char/random.c.
+ It generates entropy from different noise sources and
+ delivers significant entropy during boot.
+
+if LRNG
+
+choice
+ prompt "LRNG Entropy Pool Size"
+ default LRNG_POOL_SIZE_4096
+ help
+ Select the size of the LRNG entropy pool. The size of the
+ entropy pool is relevant for the amount of entropy that
+ the LRNG can maintain as a maximum. The larger the size
+ of the entropy pool is the more entropy can be maintained
+ but the less often older entropic values are overwritten
+ with new entropy.
+
+ config LRNG_POOL_SIZE_512
+ bool "512 bits"
+
+ config LRNG_POOL_SIZE_1024
+ bool "1024 bits"
+
+ config LRNG_POOL_SIZE_2048
+ bool "2048 bits"
+
+ config LRNG_POOL_SIZE_4096
+ bool "4096 bits (default)"
+
+ config LRNG_POOL_SIZE_8192
+ bool "8192 bits"
+
+ config LRNG_POOL_SIZE_16384
+ bool "16384 bits"
+
+ config LRNG_POOL_SIZE_32768
+ bool "32768 bits"
+
+ config LRNG_POOL_SIZE_65536
+ bool "65536 bits"
+
+ config LRNG_POOL_SIZE_131072
+ bool "131072 bits"
+endchoice
+
+config LRNG_POOL_SIZE
+ int
+ default 0 if LRNG_POOL_SIZE_512
+ default 1 if LRNG_POOL_SIZE_1024
+ default 2 if LRNG_POOL_SIZE_2048
+ default 3 if LRNG_POOL_SIZE_4096
+ default 4 if LRNG_POOL_SIZE_8192
+ default 5 if LRNG_POOL_SIZE_16384
+ default 6 if LRNG_POOL_SIZE_32768
+ default 7 if LRNG_POOL_SIZE_65536
+ default 8 if LRNG_POOL_SIZE_131072
+
+endif # LRNG
diff --git a/drivers/char/lrng/Makefile b/drivers/char/lrng/Makefile
new file mode 100644
index 000000000000..1d2a0211973d
--- /dev/null
+++ b/drivers/char/lrng/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Makefile for the Linux Random Number Generator.
+#
+
+obj-y += lrng_pool.o lrng_aux.o \
+ lrng_sw_noise.o lrng_archrandom.o \
+ lrng_drng.o lrng_chacha20.o \
+ lrng_interfaces.o \
diff --git a/drivers/char/lrng/lrng_archrandom.c b/drivers/char/lrng/lrng_archrandom.c
new file mode 100644
index 000000000000..ec6f17d976e3
--- /dev/null
+++ b/drivers/char/lrng/lrng_archrandom.c
@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG Fast Noise Source: CPU-based noise source
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/random.h>
+
+#include "lrng_internal.h"
+
+/*
+ * Estimated entropy of data is a 32th of LRNG_DRNG_SECURITY_STRENGTH_BITS.
+ * As we have no ability to review the implementation of those noise sources,
+ * it is prudent to have a conservative estimate here.
+ */
+#define LRNG_ARCHRANDOM_DEFAULT_STRENGTH (LRNG_DRNG_SECURITY_STRENGTH_BITS>>5)
+#define LRNG_ARCHRANDOM_TRUST_CPU_STRENGTH LRNG_DRNG_SECURITY_STRENGTH_BITS
+#ifdef CONFIG_RANDOM_TRUST_CPU
+static u32 archrandom = LRNG_ARCHRANDOM_TRUST_CPU_STRENGTH;
+#else
+static u32 archrandom = LRNG_ARCHRANDOM_DEFAULT_STRENGTH;
+#endif
+module_param(archrandom, uint, 0644);
+MODULE_PARM_DESC(archrandom, "Entropy in bits of 256 data bits from CPU noise "
+ "source (e.g. RDRAND)");
+
+static int __init lrng_parse_trust_cpu(char *arg)
+{
+ int ret;
+ bool trust_cpu = false;
+
+ ret = kstrtobool(arg, &trust_cpu);
+ if (ret)
+ return ret;
+
+ if (trust_cpu)
+ archrandom = LRNG_ARCHRANDOM_TRUST_CPU_STRENGTH;
+ else
+ archrandom = LRNG_ARCHRANDOM_DEFAULT_STRENGTH;
+
+ return 0;
+}
+early_param("random.trust_cpu", lrng_parse_trust_cpu);
+
+/**
+ * Get CPU noise source entropy
+ *
+ * @outbuf: buffer to store entropy of size LRNG_DRNG_SECURITY_STRENGTH_BYTES
+ * @return: > 0 on success where value provides the added entropy in bits
+ * 0 if no fast source was available
+ */
+u32 lrng_get_arch(u8 *outbuf)
+{
+ u32 i, ent_bits = archrandom;
+
+ /* operate on full blocks */
+ BUILD_BUG_ON(LRNG_DRNG_SECURITY_STRENGTH_BYTES % sizeof(unsigned long));
+ /* ensure we have aligned buffers */
+ BUILD_BUG_ON(LRNG_KCAPI_ALIGN % sizeof(unsigned long));
+
+ if (!ent_bits)
+ return 0;
+
+ for (i = 0; i < LRNG_DRNG_SECURITY_STRENGTH_BYTES;
+ i += sizeof(unsigned long)) {
+ if (!arch_get_random_seed_long((unsigned long *)(outbuf + i)) &&
+ !arch_get_random_long((unsigned long *)(outbuf + i))) {
+ archrandom = 0;
+ return 0;
+ }
+ }
+
+ /* Obtain entropy statement -- cap entropy to buffer size in bits */
+ ent_bits = min_t(u32, ent_bits, LRNG_DRNG_SECURITY_STRENGTH_BITS);
+ pr_debug("obtained %u bits of entropy from CPU RNG noise source\n",
+ ent_bits);
+ return ent_bits;
+}
+
+u32 lrng_slow_noise_req_entropy(u32 required_entropy_bits)
+{
+ u32 arch_ent_bits = min_t(u32, archrandom,
+ LRNG_DRNG_SECURITY_STRENGTH_BITS);
+ u32 fast_noise_entropy = arch_ent_bits + lrng_jent_entropylevel();
+
+ if (fast_noise_entropy > required_entropy_bits)
+ return 0;
+ return (required_entropy_bits - fast_noise_entropy);
+}
diff --git a/drivers/char/lrng/lrng_aux.c b/drivers/char/lrng/lrng_aux.c
new file mode 100644
index 000000000000..b66c7795d067
--- /dev/null
+++ b/drivers/char/lrng/lrng_aux.c
@@ -0,0 +1,148 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG auxiliary interfaces
+ *
+ * Copyright (C) 2019 Stephan Mueller <smueller@chronox.de>
+ * Copyright (C) 2017 Jason A. Donenfeld <Jason@zx2c4.com>. All
+ * Rights Reserved.
+ * Copyright (C) 2016 Jason Cooper <jason@lakedaemon.net>
+ */
+
+#include <linux/mm.h>
+#include <linux/random.h>
+
+#include "lrng_internal.h"
+
+struct batched_entropy {
+ union {
+ u64 entropy_u64[LRNG_DRNG_BLOCKSIZE / sizeof(u64)];
+ u32 entropy_u32[LRNG_DRNG_BLOCKSIZE / sizeof(u32)];
+ };
+ unsigned int position;
+ spinlock_t batch_lock;
+};
+
+/*
+ * Get a random word for internal kernel use only. The quality of the random
+ * number is either as good as RDRAND or as good as /dev/urandom, with the
+ * goal of being quite fast and not depleting entropy.
+ */
+static DEFINE_PER_CPU(struct batched_entropy, batched_entropy_u64) = {
+ .batch_lock = __SPIN_LOCK_UNLOCKED(batched_entropy_u64.lock),
+};
+
+u64 get_random_u64(void)
+{
+ u64 ret;
+ unsigned long flags;
+ struct batched_entropy *batch;
+
+#if BITS_PER_LONG == 64
+ if (arch_get_random_long((unsigned long *)&ret))
+ return ret;
+#else
+ if (arch_get_random_long((unsigned long *)&ret) &&
+ arch_get_random_long((unsigned long *)&ret + 1))
+ return ret;
+#endif
+
+ lrng_debug_report_seedlevel("get_random_u64");
+
+ batch = raw_cpu_ptr(&batched_entropy_u64);
+ spin_lock_irqsave(&batch->batch_lock, flags);
+ if (batch->position % ARRAY_SIZE(batch->entropy_u64) == 0) {
+ lrng_drng_get_atomic((u8 *)batch->entropy_u64,
+ LRNG_DRNG_BLOCKSIZE);
+ batch->position = 0;
+ }
+ ret = batch->entropy_u64[batch->position++];
+ spin_unlock_irqrestore(&batch->batch_lock, flags);
+ return ret;
+}
+EXPORT_SYMBOL(get_random_u64);
+
+static DEFINE_PER_CPU(struct batched_entropy, batched_entropy_u32) = {
+ .batch_lock = __SPIN_LOCK_UNLOCKED(batched_entropy_u32.lock),
+};
+
+u32 get_random_u32(void)
+{
+ u32 ret;
+ unsigned long flags;
+ struct batched_entropy *batch;
+
+ if (arch_get_random_int(&ret))
+ return ret;
+
+ lrng_debug_report_seedlevel("get_random_u32");
+
+ batch = raw_cpu_ptr(&batched_entropy_u32);
+ spin_lock_irqsave(&batch->batch_lock, flags);
+ if (batch->position % ARRAY_SIZE(batch->entropy_u32) == 0) {
+ lrng_drng_get_atomic((u8 *)batch->entropy_u32,
+ LRNG_DRNG_BLOCKSIZE);
+ batch->position = 0;
+ }
+ ret = batch->entropy_u32[batch->position++];
+ spin_unlock_irqrestore(&batch->batch_lock, flags);
+ return ret;
+}
+EXPORT_SYMBOL(get_random_u32);
+
+/*
+ * It's important to invalidate all potential batched entropy that might
+ * be stored before the crng is initialized, which we can do lazily by
+ * simply resetting the counter to zero so that it's re-extracted on the
+ * next usage.
+ */
+void invalidate_batched_entropy(void)
+{
+ int cpu;
+ unsigned long flags;
+
+ for_each_possible_cpu(cpu) {
+ struct batched_entropy *batched_entropy;
+
+ batched_entropy = per_cpu_ptr(&batched_entropy_u32, cpu);
+ spin_lock_irqsave(&batched_entropy->batch_lock, flags);
+ batched_entropy->position = 0;
+ spin_unlock(&batched_entropy->batch_lock);
+
+ batched_entropy = per_cpu_ptr(&batched_entropy_u64, cpu);
+ spin_lock(&batched_entropy->batch_lock);
+ batched_entropy->position = 0;
+ spin_unlock_irqrestore(&batched_entropy->batch_lock, flags);
+ }
+}
+
+/**
+ * randomize_page - Generate a random, page aligned address
+ * @start: The smallest acceptable address the caller will take.
+ * @range: The size of the area, starting at @start, within which the
+ * random address must fall.
+ *
+ * If @start + @range would overflow, @range is capped.
+ *
+ * NOTE: Historical use of randomize_range, which this replaces, presumed that
+ * @start was already page aligned. We now align it regardless.
+ *
+ * Return: A page aligned address within [start, start + range). On error,
+ * @start is returned.
+ */
+unsigned long randomize_page(unsigned long start, unsigned long range)
+{
+ if (!PAGE_ALIGNED(start)) {
+ range -= PAGE_ALIGN(start) - start;
+ start = PAGE_ALIGN(start);
+ }
+
+ if (start > ULONG_MAX - range)
+ range = ULONG_MAX - start;
+
+ range >>= PAGE_SHIFT;
+
+ if (range == 0)
+ return start;
+
+ return start + (get_random_long() % range << PAGE_SHIFT);
+}
diff --git a/drivers/char/lrng/lrng_chacha20.c b/drivers/char/lrng/lrng_chacha20.c
new file mode 100644
index 000000000000..6b91545893ed
--- /dev/null
+++ b/drivers/char/lrng/lrng_chacha20.c
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * Backend for the LRNG providing the cryptographic primitives using
+ * ChaCha20 cipher implementations.
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <crypto/chacha.h>
+#include <linux/cryptohash.h>
+#include <linux/lrng.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+
+#include "lrng_chacha20.h"
+#include "lrng_internal.h"
+
+/******************************* ChaCha20 DRNG *******************************/
+
+#define CHACHA_BLOCK_WORDS (CHACHA_BLOCK_SIZE / sizeof(u32))
+
+struct chacha20_state {
+ struct chacha20_block block;
+};
+
+/*
+ * Have a static memory blocks for the ChaCha20 DRNG instance to avoid calling
+ * kmalloc too early in the boot cycle. For subsequent allocation requests,
+ * such as per-NUMA-node DRNG instances, kmalloc will be used.
+ */
+struct chacha20_state chacha20;
+
+/**
+ * Update of the ChaCha20 state by either using an unused buffer part or by
+ * generating one ChaCha20 block which is half of the state of the ChaCha20.
+ * The block is XORed into the key part of the state. This shall ensure
+ * backtracking resistance as well as a proper mix of the ChaCha20 state once
+ * the key is injected.
+ */
+static void lrng_chacha20_update(struct chacha20_state *chacha20_state,
+ u32 *buf, u32 used_words)
+{
+ struct chacha20_block *chacha20 = &chacha20_state->block;
+ u32 i, tmp[CHACHA_BLOCK_WORDS];
+
+ BUILD_BUG_ON(sizeof(struct chacha20_block) != CHACHA_BLOCK_SIZE);
+ BUILD_BUG_ON(CHACHA_BLOCK_SIZE != 2 * CHACHA_KEY_SIZE);
+
+ if (used_words > CHACHA_KEY_SIZE_WORDS) {
+ chacha20_block(&chacha20->constants[0], (u8 *)tmp);
+ for (i = 0; i < CHACHA_KEY_SIZE_WORDS; i++)
+ chacha20->key.u[i] ^= tmp[i];
+ memzero_explicit(tmp, sizeof(tmp));
+ } else {
+ for (i = 0; i < CHACHA_KEY_SIZE_WORDS; i++)
+ chacha20->key.u[i] ^= buf[i + used_words];
+ }
+
+ /* Deterministic increment of nonce as required in RFC 7539 chapter 4 */
+ chacha20->nonce[0]++;
+ if (chacha20->nonce[0] == 0)
+ chacha20->nonce[1]++;
+ if (chacha20->nonce[1] == 0)
+ chacha20->nonce[2]++;
+
+ /* Leave counter untouched as it is start value is undefined in RFC */
+}
+
+/**
+ * Seed the ChaCha20 DRNG by injecting the input data into the key part of
+ * the ChaCha20 state. If the input data is longer than the ChaCha20 key size,
+ * perform a ChaCha20 operation after processing of key size input data.
+ * This operation shall spread out the entropy into the ChaCha20 state before
+ * new entropy is injected into the key part.
+ */
+static int lrng_cc20_drng_seed_helper(void *drng, const u8 *inbuf, u32 inbuflen)
+{
+ struct chacha20_state *chacha20_state = (struct chacha20_state *)drng;
+ struct chacha20_block *chacha20 = &chacha20_state->block;
+
+ while (inbuflen) {
+ u32 i, todo = min_t(u32, inbuflen, CHACHA_KEY_SIZE);
+
+ for (i = 0; i < todo; i++)
+ chacha20->key.b[i] ^= inbuf[i];
+
+ /* Break potential dependencies between the inbuf key blocks */
+ lrng_chacha20_update(chacha20_state, NULL,
+ CHACHA_BLOCK_WORDS);
+ inbuf += todo;
+ inbuflen -= todo;
+ }
+
+ return 0;
+}
+
+/**
+ * Chacha20 DRNG generation of random numbers: the stream output of ChaCha20
+ * is the random number. After the completion of the generation of the
+ * stream, the entire ChaCha20 state is updated.
+ *
+ * Note, as the ChaCha20 implements a 32 bit counter, we must ensure
+ * that this function is only invoked for at most 2^32 - 1 ChaCha20 blocks
+ * before a reseed or an update happens. This is ensured by the variable
+ * outbuflen which is a 32 bit integer defining the number of bytes to be
+ * generated by the ChaCha20 DRNG. At the end of this function, an update
+ * operation is invoked which implies that the 32 bit counter will never be
+ * overflown in this implementation.
+ */
+static int lrng_cc20_drng_generate_helper(void *drng, u8 *outbuf, u32 outbuflen)
+{
+ struct chacha20_state *chacha20_state = (struct chacha20_state *)drng;
+ struct chacha20_block *chacha20 = &chacha20_state->block;
+ u32 aligned_buf[CHACHA_BLOCK_WORDS], ret = outbuflen,
+ used = CHACHA_BLOCK_WORDS;
+ int zeroize_buf = 0;
+
+ while (outbuflen >= CHACHA_BLOCK_SIZE) {
+ chacha20_block(&chacha20->constants[0], outbuf);
+ outbuf += CHACHA_BLOCK_SIZE;
+ outbuflen -= CHACHA_BLOCK_SIZE;
+ }
+
+ if (outbuflen) {
+ chacha20_block(&chacha20->constants[0], (u8 *)aligned_buf);
+ memcpy(outbuf, aligned_buf, outbuflen);
+ used = ((outbuflen + sizeof(aligned_buf[0]) - 1) /
+ sizeof(aligned_buf[0]));
+ zeroize_buf = 1;
+ }
+
+ lrng_chacha20_update(chacha20_state, aligned_buf, used);
+
+ if (zeroize_buf)
+ memzero_explicit(aligned_buf, sizeof(aligned_buf));
+
+ return ret;
+}
+
+/**
+ * ChaCha20 DRNG that provides full strength, i.e. the output is capable
+ * of transporting 1 bit of entropy per data bit, provided the DRNG was
+ * seeded with 256 bits of entropy. This is achieved by folding the ChaCha20
+ * block output of 512 bits in half using XOR.
+ *
+ * Other than the output handling, the implementation is conceptually
+ * identical to lrng_drng_generate_helper.
+ */
+static int lrng_cc20_drng_generate_helper_full(void *drng, u8 *outbuf,
+ u32 outbuflen)
+{
+ struct chacha20_state *chacha20_state = (struct chacha20_state *)drng;
+ struct chacha20_block *chacha20 = &chacha20_state->block;
+ u32 aligned_buf[CHACHA_BLOCK_WORDS];
+ u32 ret = outbuflen;
+
+ while (outbuflen >= CHACHA_BLOCK_SIZE) {
+ u32 i;
+
+ chacha20_block(&chacha20->constants[0], outbuf);
+
+ /* fold output in half */
+ for (i = 0; i < (CHACHA_BLOCK_WORDS / 2); i++)
+ outbuf[i] ^= outbuf[i + (CHACHA_BLOCK_WORDS / 2)];
+
+ outbuf += CHACHA_BLOCK_SIZE / 2;
+ outbuflen -= CHACHA_BLOCK_SIZE / 2;
+ }
+
+ while (outbuflen) {
+ u32 i, todo = min_t(u32, CHACHA_BLOCK_SIZE / 2, outbuflen);
+
+ chacha20_block(&chacha20->constants[0], (u8 *)aligned_buf);
+
+ /* fold output in half */
+ for (i = 0; i < (CHACHA_BLOCK_WORDS / 2); i++)
+ aligned_buf[i] ^=
+ aligned_buf[i + (CHACHA_BLOCK_WORDS / 2)];
+
+ memcpy(outbuf, aligned_buf, todo);
+ outbuflen -= todo;
+ outbuf += todo;
+ }
+ memzero_explicit(aligned_buf, sizeof(aligned_buf));
+
+ lrng_chacha20_update(chacha20_state, NULL, CHACHA_BLOCK_WORDS);
+
+ return ret;
+}
+
+void lrng_cc20_init_state(struct chacha20_state *state)
+{
+ struct chacha20_block *chacha20 = &state->block;
+ unsigned long v;
+ u32 i;
+
+ lrng_cc20_init_rfc7539(chacha20);
+
+ for (i = 0; i < CHACHA_KEY_SIZE_WORDS; i++) {
+ chacha20->key.u[i] ^= jiffies;
+ chacha20->key.u[i] ^= random_get_entropy();
+ if (arch_get_random_seed_long(&v) || arch_get_random_long(&v))
+ chacha20->key.u[i] ^= v;
+ }
+
+ for (i = 0; i < 3; i++) {
+ chacha20->nonce[i] ^= jiffies;
+ chacha20->nonce[i] ^= random_get_entropy();
+ if (arch_get_random_seed_long(&v) || arch_get_random_long(&v))
+ chacha20->nonce[i] ^= v;
+ }
+
+ pr_info("ChaCha20 core initialized\n");
+}
+
+/**
+ * Allocation of the DRNG state
+ */
+static void *lrng_cc20_drng_alloc(u32 sec_strength)
+{
+ struct chacha20_state *state = NULL;
+
+ if (sec_strength > CHACHA_KEY_SIZE) {
+ pr_err("Security strength of ChaCha20 DRNG (%u bits) lower "
+ "than requested by LRNG (%u bits)\n",
+ CHACHA_KEY_SIZE * 8, sec_strength * 8);
+ return ERR_PTR(-EINVAL);
+ }
+ if (sec_strength < CHACHA_KEY_SIZE)
+ pr_warn("Security strength of ChaCha20 DRNG (%u bits) higher "
+ "than requested by LRNG (%u bits)\n",
+ CHACHA_KEY_SIZE * 8, sec_strength * 8);
+
+ state = kmalloc(sizeof(struct chacha20_state), GFP_KERNEL);
+ if (!state)
+ return ERR_PTR(-ENOMEM);
+ pr_debug("memory for ChaCha20 core allocated\n");
+
+ lrng_cc20_init_state(state);
+
+ return state;
+}
+
+static void lrng_cc20_drng_dealloc(void *drng)
+{
+ struct chacha20_state *chacha20_state = (struct chacha20_state *)drng;
+
+ if (drng == &chacha20) {
+ memzero_explicit(chacha20_state, sizeof(*chacha20_state));
+ pr_debug("static ChaCha20 core zeroized\n");
+ return;
+ }
+
+ pr_debug("ChaCha20 core zeroized and freed\n");
+ kzfree(chacha20_state);
+}
+
+/******************************* Hash Operation *******************************/
+
+static void *lrng_cc20_hash_alloc(const u8 *key, u32 keylen)
+{
+ pr_info("Hash SHA-1 allocated\n");
+ return NULL;
+}
+
+static void lrng_cc20_hash_dealloc(void *hash)
+{
+}
+
+static u32 lrng_cc20_hash_digestsize(void *hash)
+{
+ return (SHA_DIGEST_WORDS * sizeof(u32));
+}
+
+static int lrng_cc20_hash_buffer(void *hash, const u8 *inbuf, u32 inbuflen,
+ u8 *digest)
+{
+ u32 i;
+ u32 workspace[SHA_WORKSPACE_WORDS];
+
+ WARN_ON(inbuflen % (SHA_WORKSPACE_WORDS * sizeof(u32)));
+
+ sha_init((u32 *)digest);
+ for (i = 0; i < inbuflen; i += (SHA_WORKSPACE_WORDS * sizeof(u32)))
+ sha_transform((u32 *)digest, (inbuf + i), workspace);
+ memzero_explicit(workspace, sizeof(workspace));
+
+ return 0;
+}
+
+static const char *lrng_cc20_drng_name(void)
+{
+ const char *cc20_drng_name = "ChaCha20 DRNG";
+ return cc20_drng_name;
+}
+
+static const char *lrng_cc20_hash_name(void)
+{
+ const char *cc20_hash_name = "SHA-1";
+ return cc20_hash_name;
+}
+
+const struct lrng_crypto_cb lrng_cc20_crypto_cb = {
+ .lrng_drng_name = lrng_cc20_drng_name,
+ .lrng_hash_name = lrng_cc20_hash_name,
+ .lrng_drng_alloc = lrng_cc20_drng_alloc,
+ .lrng_drng_dealloc = lrng_cc20_drng_dealloc,
+ .lrng_drng_seed_helper = lrng_cc20_drng_seed_helper,
+ .lrng_drng_generate_helper = lrng_cc20_drng_generate_helper,
+ .lrng_drng_generate_helper_full = lrng_cc20_drng_generate_helper_full,
+ .lrng_hash_alloc = lrng_cc20_hash_alloc,
+ .lrng_hash_dealloc = lrng_cc20_hash_dealloc,
+ .lrng_hash_digestsize = lrng_cc20_hash_digestsize,
+ .lrng_hash_buffer = lrng_cc20_hash_buffer,
+};
diff --git a/drivers/char/lrng/lrng_chacha20.h b/drivers/char/lrng/lrng_chacha20.h
new file mode 100644
index 000000000000..0387ad59cfbc
--- /dev/null
+++ b/drivers/char/lrng/lrng_chacha20.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * LRNG ChaCha20 definitions
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#include <crypto/chacha.h>
+
+/* State according to RFC 7539 section 2.3 */
+struct chacha20_block {
+ u32 constants[4];
+ union {
+#define CHACHA_KEY_SIZE_WORDS (CHACHA_KEY_SIZE / sizeof(u32))
+ u32 u[CHACHA_KEY_SIZE_WORDS];
+ u8 b[CHACHA_KEY_SIZE];
+ } key;
+ u32 counter;
+ u32 nonce[3];
+};
+
+static inline void lrng_cc20_init_rfc7539(struct chacha20_block *chacha20)
+{
+ memcpy(&chacha20->constants[0], "expand 32-byte k", 16);
+}
diff --git a/drivers/char/lrng/lrng_drng.c b/drivers/char/lrng/lrng_drng.c
new file mode 100644
index 000000000000..b55f4ad7580c
--- /dev/null
+++ b/drivers/char/lrng/lrng_drng.c
@@ -0,0 +1,397 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG DRNG processing
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/lrng.h>
+
+#include "lrng_internal.h"
+
+/*
+ * Maximum number of seconds between DRNG reseed intervals of the DRNG. Note,
+ * this is enforced with the next request of random numbers from the
+ * DRNG. Setting this value to zero implies a reseeding attempt before every
+ * generated random number.
+ */
+int lrng_drng_reseed_max_time = 600;
+
+static atomic_t lrng_avail = ATOMIC_INIT(0);
+
+DEFINE_MUTEX(lrng_crypto_cb_update);
+
+/* DRNG for /dev/urandom, getrandom(2), get_random_bytes */
+static struct lrng_drng lrng_drng_init = {
+ .drng = &chacha20,
+ .crypto_cb = &lrng_cc20_crypto_cb,
+ .lock = __MUTEX_INITIALIZER(lrng_drng_init.lock),
+ .spin_lock = __SPIN_LOCK_UNLOCKED(lrng_drng_init.spin_lock)
+};
+
+/*
+ * DRNG for get_random_bytes when called in atomic context. This
+ * DRNG will always use the ChaCha20 DRNG. It will never benefit from a
+ * DRNG switch like the "regular" DRNG. If there was no DRNG switch, the atomic
+ * DRNG is identical to the "regular" DRNG.
+ *
+ * The reason for having this is due to the fact that DRNGs other than
+ * the ChaCha20 DRNG may sleep.
+ */
+static struct lrng_drng lrng_drng_atomic = {
+ .drng = &chacha20,
+ .crypto_cb = &lrng_cc20_crypto_cb,
+ .spin_lock = __SPIN_LOCK_UNLOCKED(lrng_drng_atomic.spin_lock)
+};
+
+/********************************** Helper ************************************/
+
+bool lrng_get_available(void)
+{
+ return likely(atomic_read(&lrng_avail));
+}
+
+void lrng_set_available(void)
+{
+ atomic_set(&lrng_avail, 1);
+}
+
+struct lrng_drng *lrng_drng_init_instance(void)
+{
+ return &lrng_drng_init;
+}
+
+struct lrng_drng *lrng_drng_atomic_instance(void)
+{
+ return &lrng_drng_atomic;
+}
+
+void lrng_drng_reset(struct lrng_drng *drng)
+{
+ atomic_set(&drng->requests, LRNG_DRNG_RESEED_THRESH);
+ drng->last_seeded = jiffies;
+ drng->fully_seeded = false;
+ drng->force_reseed = true;
+ pr_debug("reset DRNG\n");
+}
+
+/************************* Random Number Generation ***************************/
+
+/* Inject a data buffer into the DRNG */
+static void lrng_drng_inject(struct lrng_drng *drng,
+ const u8 *inbuf, u32 inbuflen)
+{
+ const char *drng_type = unlikely(drng == &lrng_drng_atomic) ?
+ "atomic" : "regular";
+ unsigned long flags = 0;
+
+ BUILD_BUG_ON(LRNG_DRNG_RESEED_THRESH > INT_MAX);
+ pr_debug("seeding %s DRNG with %u bytes\n", drng_type, inbuflen);
+ lrng_drng_lock(drng, &flags);
+ if (drng->crypto_cb->lrng_drng_seed_helper(drng->drng,
+ inbuf, inbuflen) < 0) {
+ pr_warn("seeding of %s DRNG failed\n", drng_type);
+ atomic_set(&drng->requests, 1);
+ } else {
+ pr_debug("%s DRNG stats since last seeding: %lu secs; "
+ "generate calls: %d\n", drng_type,
+ (time_after(jiffies, drng->last_seeded) ?
+ (jiffies - drng->last_seeded) : 0) / HZ,
+ (LRNG_DRNG_RESEED_THRESH -
+ atomic_read(&drng->requests)));
+ drng->last_seeded = jiffies;
+ atomic_set(&drng->requests, LRNG_DRNG_RESEED_THRESH);
+ drng->force_reseed = false;
+
+ if (drng->drng == lrng_drng_atomic.drng) {
+ lrng_drng_atomic.last_seeded = jiffies;
+ atomic_set(&lrng_drng_atomic.requests,
+ LRNG_DRNG_RESEED_THRESH);
+ lrng_drng_atomic.force_reseed = false;
+ }
+ }
+ lrng_drng_unlock(drng, &flags);
+}
+
+/**
+ * Perform the seeding of the DRNG with data from noise source
+ */
+static inline int _lrng_drng_seed(struct lrng_drng *drng)
+{
+ struct entropy_buf seedbuf __aligned(LRNG_KCAPI_ALIGN);
+ unsigned long flags = 0;
+ u32 total_entropy_bits;
+ int ret;
+
+ lrng_drng_lock(drng, &flags);
+ total_entropy_bits = lrng_fill_seed_buffer(drng->crypto_cb, drng->hash,
+ &seedbuf, 0);
+ lrng_drng_unlock(drng, &flags);
+
+ /* Allow the seeding operation to be called again */
+ lrng_pool_unlock();
+ lrng_init_ops(total_entropy_bits);
+ ret = total_entropy_bits >> 3;
+
+ lrng_drng_inject(drng, (u8 *)&seedbuf, sizeof(seedbuf));
+ memzero_explicit(&seedbuf, sizeof(seedbuf));
+
+ return ret;
+}
+
+static int lrng_drng_get(struct lrng_drng *drng, u8 *outbuf, u32 outbuflen);
+static void lrng_drng_seed(struct lrng_drng *drng)
+{
+ int ret = _lrng_drng_seed(drng);
+
+ if (ret >= LRNG_DRNG_SECURITY_STRENGTH_BYTES)
+ drng->fully_seeded = true;
+
+ BUILD_BUG_ON(LRNG_MIN_SEED_ENTROPY_BITS >
+ LRNG_DRNG_SECURITY_STRENGTH_BITS);
+
+ /*
+ * Reseed atomic DRNG from current DRNG,
+ *
+ * We can obtain random numbers from DRNG as the lock type
+ * chosen by lrng_drng_get is usable with the current caller.
+ */
+ if ((drng->drng != lrng_drng_atomic.drng) &&
+ (lrng_drng_atomic.force_reseed ||
+ atomic_read(&lrng_drng_atomic.requests) <= 0 ||
+ time_after(jiffies, lrng_drng_atomic.last_seeded +
+ lrng_drng_reseed_max_time * HZ))) {
+ u8 seedbuf[LRNG_DRNG_SECURITY_STRENGTH_BYTES]
+ __aligned(LRNG_KCAPI_ALIGN);
+
+ ret = lrng_drng_get(drng, seedbuf, sizeof(seedbuf));
+
+ if (ret < 0) {
+ pr_warn("Error generating random numbers for atomic "
+ "DRNG: %d\n", ret);
+ } else {
+ lrng_drng_inject(&lrng_drng_atomic, seedbuf, ret);
+ }
+ memzero_explicit(&seedbuf, sizeof(seedbuf));
+ }
+}
+
+static inline void _lrng_drng_seed_work(struct lrng_drng *drng, u32 node)
+{
+ pr_debug("reseed triggered by interrupt noise source for DRNG on NUMA "
+ "node %d\n", node);
+ lrng_drng_seed(drng);
+ if (drng->fully_seeded) {
+ /* Prevent reseed storm */
+ drng->last_seeded += node * 100 * HZ;
+ /* Prevent draining of pool on idle systems */
+ lrng_drng_reseed_max_time += 100;
+ }
+}
+
+/**
+ * DRNG reseed trigger: Kernel thread handler triggered by the schedule_work()
+ */
+void lrng_drng_seed_work(struct work_struct *dummy)
+{
+ struct lrng_drng **lrng_drng = lrng_drng_instances();
+ u32 node;
+
+ if (lrng_drng) {
+ for_each_online_node(node) {
+ struct lrng_drng *drng = lrng_drng[node];
+
+ if (drng && !drng->fully_seeded) {
+ _lrng_drng_seed_work(drng, node);
+ goto out;
+ }
+ }
+ lrng_pool_all_numa_nodes_seeded();
+ } else {
+ if (!lrng_drng_init.fully_seeded)
+ _lrng_drng_seed_work(&lrng_drng_init, 0);
+ }
+
+out:
+ /* Allow the seeding operation to be called again */
+ lrng_pool_unlock();
+}
+
+/* Force all DRNGs to reseed before next generation */
+void lrng_drng_force_reseed(void)
+{
+ struct lrng_drng **lrng_drng = lrng_drng_instances();
+ u32 node;
+
+ if (!lrng_drng) {
+ lrng_drng_init.force_reseed = true;
+ pr_debug("force reseed of initial DRNG\n");
+ return;
+ }
+ for_each_online_node(node) {
+ struct lrng_drng *drng = lrng_drng[node];
+
+ if (!drng)
+ continue;
+
+ drng->force_reseed = true;
+ pr_debug("force reseed of DRNG on node %u\n", node);
+ }
+ lrng_drng_atomic.force_reseed = true;
+}
+
+/**
+ * Get random data out of the DRNG which is reseeded frequently.
+ *
+ * @outbuf: buffer for storing random data
+ * @outbuflen: length of outbuf
+ * @return: < 0 in error case (DRNG generation or update failed)
+ * >=0 returning the returned number of bytes
+ */
+static int lrng_drng_get(struct lrng_drng *drng, u8 *outbuf, u32 outbuflen)
+{
+ unsigned long flags = 0;
+ u32 processed = 0;
+
+ if (!outbuf || !outbuflen)
+ return 0;
+
+ outbuflen = min_t(size_t, outbuflen, INT_MAX);
+
+ lrng_drngs_init_cc20();
+
+ while (outbuflen) {
+ u32 todo = min_t(u32, outbuflen, LRNG_DRNG_MAX_REQSIZE);
+ int ret;
+
+ /* All but the atomic DRNG are seeded during generation */
+ if (atomic_dec_and_test(&drng->requests) ||
+ drng->force_reseed ||
+ time_after(jiffies, drng->last_seeded +
+ lrng_drng_reseed_max_time * HZ)) {
+ if (likely(drng != &lrng_drng_atomic)) {
+ if (lrng_pool_trylock())
+ atomic_set(&drng->requests, 1);
+ else
+ lrng_drng_seed(drng);
+ }
+ }
+
+ lrng_drng_lock(drng, &flags);
+ ret = drng->crypto_cb->lrng_drng_generate_helper(
+ drng->drng, outbuf + processed, todo);
+ lrng_drng_unlock(drng, &flags);
+ if (ret <= 0) {
+ pr_warn("getting random data from DRNG failed (%d)\n",
+ ret);
+ return -EFAULT;
+ }
+ processed += ret;
+ outbuflen -= ret;
+ }
+
+ return processed;
+}
+
+int lrng_drng_get_atomic(u8 *outbuf, u32 outbuflen)
+{
+ return lrng_drng_get(&lrng_drng_atomic, outbuf, outbuflen);
+}
+
+int lrng_drng_get_sleep(u8 *outbuf, u32 outbuflen)
+{
+ struct lrng_drng **lrng_drng = lrng_drng_instances();
+ struct lrng_drng *drng = &lrng_drng_init;
+ int node = numa_node_id();
+
+ might_sleep();
+
+ if (lrng_drng && lrng_drng[node] && lrng_drng[node]->fully_seeded)
+ drng = lrng_drng[node];
+
+ return lrng_drng_get(drng, outbuf, outbuflen);
+}
+
+/* Initialize the default DRNG during boot */
+void lrng_drngs_init_cc20(void)
+{
+ unsigned long flags = 0;
+
+ if (lrng_get_available())
+ return;
+
+ lrng_drng_lock(&lrng_drng_init, &flags);
+ if (lrng_get_available()) {
+ lrng_drng_unlock(&lrng_drng_init, &flags);
+ return;
+ }
+
+ lrng_drng_reset(&lrng_drng_init);
+ lrng_cc20_init_state(&chacha20);
+ lrng_state_init_seed_work();
+ lrng_drng_unlock(&lrng_drng_init, &flags);
+
+ lrng_drng_lock(&lrng_drng_atomic, &flags);
+ lrng_drng_reset(&lrng_drng_atomic);
+ /*
+ * We do not initialize the state of the atomic DRNG as it is identical
+ * to the DRNG at this point.
+ */
+ lrng_drng_unlock(&lrng_drng_atomic, &flags);
+
+ lrng_set_available();
+}
+
+/* Reset LRNG such that all existing entropy is gone */
+static void _lrng_reset(struct work_struct *work)
+{
+ struct lrng_drng **lrng_drng = lrng_drng_instances();
+ unsigned long flags = 0;
+
+ if (!lrng_drng) {
+ lrng_drng_lock(&lrng_drng_init, &flags);
+ lrng_drng_reset(&lrng_drng_init);
+ lrng_drng_unlock(&lrng_drng_init, &flags);
+ } else {
+ u32 node;
+
+ for_each_online_node(node) {
+ struct lrng_drng *drng = lrng_drng[node];
+
+ if (!drng)
+ continue;
+ lrng_drng_lock(drng, &flags);
+ lrng_drng_reset(drng);
+ lrng_drng_unlock(drng, &flags);
+ }
+ }
+ lrng_set_entropy_thresh(LRNG_INIT_ENTROPY_BITS +
+ LRNG_CONDITIONING_ENTROPY_LOSS);
+
+ lrng_reset_state();
+}
+
+static DECLARE_WORK(lrng_reset_work, _lrng_reset);
+
+void lrng_reset(void)
+{
+ schedule_work(&lrng_reset_work);
+}
+
+/***************************** Initialize LRNG *******************************/
+
+static int __init lrng_init(void)
+{
+ lrng_drngs_init_cc20();
+
+ lrng_drngs_numa_alloc();
+ return 0;
+}
+
+late_initcall(lrng_init);
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_AUTHOR("Stephan Mueller <smueller@chronox.de>");
+MODULE_DESCRIPTION("Linux Random Number Generator");
diff --git a/drivers/char/lrng/lrng_interfaces.c b/drivers/char/lrng/lrng_interfaces.c
new file mode 100644
index 000000000000..ff40ed37ddd6
--- /dev/null
+++ b/drivers/char/lrng/lrng_interfaces.c
@@ -0,0 +1,623 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG User and kernel space interfaces
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/freezer.h>
+#include <linux/fs.h>
+#include <linux/genhd.h>
+#include <linux/kthread.h>
+#include <linux/poll.h>
+#include <linux/preempt.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/syscalls.h>
+#include <linux/timex.h>
+
+#include "lrng_internal.h"
+
+/*
+ * If the entropy count falls under this number of bits, then we
+ * should wake up processes which are selecting or polling on write
+ * access to /dev/random.
+ */
+u32 lrng_write_wakeup_bits = LRNG_WRITE_WAKEUP_ENTROPY;
+
+static LIST_HEAD(lrng_ready_list);
+static DEFINE_SPINLOCK(lrng_ready_list_lock);
+
+static DECLARE_WAIT_QUEUE_HEAD(lrng_write_wait);
+static DECLARE_WAIT_QUEUE_HEAD(lrng_init_wait);
+static struct fasync_struct *fasync;
+
+struct ctl_table random_table[];
+/********************************** Helper ***********************************/
+
+/* Is the DRNG seed level too low? */
+static inline bool lrng_need_entropy(void)
+{
+ return (lrng_avail_entropy() < lrng_write_wakeup_bits);
+}
+
+void lrng_writer_wakeup(void)
+{
+ if (lrng_need_entropy() && wq_has_sleeper(&lrng_write_wait)) {
+ wake_up_interruptible(&lrng_write_wait);
+ kill_fasync(&fasync, SIGIO, POLL_OUT);
+ }
+}
+
+void lrng_init_wakeup(void)
+{
+ wake_up_all(&lrng_init_wait);
+ kill_fasync(&fasync, SIGIO, POLL_IN);
+}
+
+/**
+ * Ping all kernel internal callers waiting until the DRNG is at least minimally
+ * seeded to inform that the DRNG reached that seed level.
+ *
+ * When the SP800-90B testing is enabled, the ping only happens if the SP800-90B
+ * startup health tests are completed. This implies that kernel internal
+ * callers always have an SP800-90B compliant noise source when being
+ * pinged.
+ */
+void lrng_process_ready_list(void)
+{
+ unsigned long flags;
+ struct random_ready_callback *rdy, *tmp;
+
+ if (!lrng_sp80090b_startup_complete())
+ return;
+
+ spin_lock_irqsave(&lrng_ready_list_lock, flags);
+ list_for_each_entry_safe(rdy, tmp, &lrng_ready_list, list) {
+ struct module *owner = rdy->owner;
+
+ list_del_init(&rdy->list);
+ rdy->func(rdy);
+ module_put(owner);
+ }
+ spin_unlock_irqrestore(&lrng_ready_list_lock, flags);
+}
+
+void lrng_debug_report_seedlevel(const char *name)
+{
+#ifdef CONFIG_WARN_ALL_UNSEEDED_RANDOM
+ static void *previous = NULL;
+ void *caller = (void *) _RET_IP_;
+
+ if (READ_ONCE(previous) == caller)
+ return;
+
+ if (!lrng_state_min_seeded())
+ pr_notice("%pS %s called without reaching mimimally seeded "
+ "level (available entropy %u)\n", caller, name,
+ lrng_avail_entropy());
+
+ WRITE_ONCE(previous, caller);
+#endif
+}
+
+/************************ LRNG kernel input interfaces ************************/
+
+/**
+ * Interface for in-kernel drivers of true hardware RNGs.
+ * Those devices may produce endless random bits and will be throttled
+ * when our pool is full.
+ *
+ * @buffer: buffer holding the entropic data from HW noise sources to be used to
+ * insert into entropy pool.
+ * @count: length of buffer
+ * @entropy_bits: amount of entropy in buffer (value is in bits)
+ */
+void add_hwgenerator_randomness(const char *buffer, size_t count,
+ size_t entropy_bits)
+{
+ /*
+ * Suspend writing if we are fully loaded with entropy.
+ * We'll be woken up again once below lrng_write_wakeup_thresh,
+ * or when the calling thread is about to terminate.
+ */
+ wait_event_interruptible(lrng_write_wait,
+ lrng_need_entropy() ||
+ lrng_state_exseed_allow(lrng_noise_source_hw) ||
+ kthread_should_stop());
+ lrng_state_exseed_set(lrng_noise_source_hw, false);
+ lrng_pool_lfsr_nonaligned(buffer, count);
+ lrng_pool_add_entropy(entropy_bits);
+}
+EXPORT_SYMBOL_GPL(add_hwgenerator_randomness);
+
+/**
+ * Handle random seed passed by bootloader.
+ * If the seed is trustworthy, it would be regarded as hardware RNGs. Otherwise
+ * it would be regarded as device data.
+ * The decision is controlled by CONFIG_RANDOM_TRUST_BOOTLOADER.
+ *
+ * @buf: buffer holding the entropic data from HW noise sources to be used to
+ * insert into entropy pool.
+ * @size: length of buffer
+ */
+void add_bootloader_randomness(const void *buf, unsigned int size)
+{
+ if (IS_ENABLED(CONFIG_RANDOM_TRUST_BOOTLOADER))
+ add_hwgenerator_randomness(buf, size, size * 8);
+ else
+ add_device_randomness(buf, size);
+}
+EXPORT_SYMBOL_GPL(add_bootloader_randomness);
+
+/**
+ * Callback for HID layer -- use the HID event values to stir the entropy pool
+ */
+void add_input_randomness(unsigned int type, unsigned int code,
+ unsigned int value)
+{
+ static unsigned char last_value;
+
+ /* ignore autorepeat and the like */
+ if (value == last_value)
+ return;
+
+ last_value = value;
+
+ lrng_pool_lfsr_u32((type << 4) ^ code ^ (code >> 4) ^ value);
+}
+EXPORT_SYMBOL_GPL(add_input_randomness);
+
+/**
+ * Add device- or boot-specific data to the entropy pool to help
+ * initialize it.
+ *
+ * None of this adds any entropy; it is meant to avoid the problem of
+ * the entropy pool having similar initial state across largely
+ * identical devices.
+ *
+ * @buf: buffer holding the entropic data from HW noise sources to be used to
+ * insert into entropy pool.
+ * @size: length of buffer
+ */
+void add_device_randomness(const void *buf, unsigned int size)
+{
+ lrng_pool_lfsr_nonaligned((u8 *)buf, size);
+ lrng_pool_lfsr_u32(random_get_entropy());
+ lrng_pool_lfsr_u32(jiffies);
+}
+EXPORT_SYMBOL(add_device_randomness);
+
+#ifdef CONFIG_BLOCK
+void rand_initialize_disk(struct gendisk *disk) { }
+void add_disk_randomness(struct gendisk *disk) { }
+EXPORT_SYMBOL(add_disk_randomness);
+#endif
+
+/**
+ * Delete a previously registered readiness callback function.
+ *
+ * @rdy: callback definition that was registered initially
+ */
+void del_random_ready_callback(struct random_ready_callback *rdy)
+{
+ unsigned long flags;
+ struct module *owner = NULL;
+
+ spin_lock_irqsave(&lrng_ready_list_lock, flags);
+ if (!list_empty(&rdy->list)) {
+ list_del_init(&rdy->list);
+ owner = rdy->owner;
+ }
+ spin_unlock_irqrestore(&lrng_ready_list_lock, flags);
+
+ module_put(owner);
+}
+EXPORT_SYMBOL(del_random_ready_callback);
+
+/**
+ * Add a callback function that will be invoked when the DRNG is mimimally
+ * seeded.
+ *
+ * @rdy: callback definition to be invoked when the LRNG is seeded
+ * @return: 0 if callback is successfully added
+ * -EALREADY if pool is already initialised (callback not called)
+ * -ENOENT if module for callback is not alive
+ */
+int add_random_ready_callback(struct random_ready_callback *rdy)
+{
+ struct module *owner;
+ unsigned long flags;
+ int err = -EALREADY;
+
+ if (likely(lrng_state_min_seeded()))
+ return err;
+
+ owner = rdy->owner;
+ if (!try_module_get(owner))
+ return -ENOENT;
+
+ spin_lock_irqsave(&lrng_ready_list_lock, flags);
+ if (lrng_state_min_seeded())
+ goto out;
+
+ owner = NULL;
+
+ list_add(&rdy->list, &lrng_ready_list);
+ err = 0;
+
+out:
+ spin_unlock_irqrestore(&lrng_ready_list_lock, flags);
+
+ module_put(owner);
+
+ return err;
+}
+EXPORT_SYMBOL(add_random_ready_callback);
+
+/*********************** LRNG kernel output interfaces ************************/
+
+/**
+ * Provider of cryptographic strong random numbers for kernel-internal usage.
+ * This function is appropriate for all in-kernel use cases. However,
+ * it will always use the ChaCha20 DRNG.
+ *
+ * @buf: buffer to store the random bytes
+ * @nbytes: size of the buffer
+ */
+void get_random_bytes(void *buf, int nbytes)
+{
+ lrng_drng_get_atomic((u8 *)buf, (u32)nbytes);
+ lrng_debug_report_seedlevel("get_random_bytes");
+}
+EXPORT_SYMBOL(get_random_bytes);
+
+/**
+ * Provider of cryptographic strong random numbers for kernel-internal usage.
+ * This function is appropriate only for non-atomic use cases as this
+ * function may sleep. Though, it provides access to the full functionality
+ * of LRNG including the switchable DRNG support, that may support other
+ * DRNGs such as the SP800-90A DRBG.
+ *
+ * @buf: buffer to store the random bytes
+ * @nbytes: size of the buffer
+ */
+void get_random_bytes_full(void *buf, int nbytes)
+{
+ lrng_drng_get_sleep((u8 *)buf, (u32)nbytes);
+ lrng_debug_report_seedlevel("get_random_bytes_full");
+}
+EXPORT_SYMBOL(get_random_bytes_full);
+
+/**
+ * Wait for the LRNG to be seeded and thus guaranteed to supply
+ * cryptographically secure random numbers. This applies to: the /dev/urandom
+ * device, the get_random_bytes function, and the get_random_{u32,u64,int,long}
+ * family of functions. Using any of these functions without first calling
+ * this function forfeits the guarantee of security.
+ *
+ * Returns: 0 if the LRNG has been seeded.
+ * -ERESTARTSYS if the function was interrupted by a signal.
+ */
+int wait_for_random_bytes(void)
+{
+ if (likely(lrng_state_min_seeded()))
+ return 0;
+ return wait_event_interruptible(lrng_init_wait,
+ lrng_state_min_seeded());
+}
+EXPORT_SYMBOL(wait_for_random_bytes);
+
+/**
+ * This function will use the architecture-specific hardware random
+ * number generator if it is available. The arch-specific hw RNG will
+ * almost certainly be faster than what we can do in software, but it
+ * is impossible to verify that it is implemented securely (as
+ * opposed, to, say, the AES encryption of a sequence number using a
+ * key known by the NSA). So it's useful if we need the speed, but
+ * only if we're willing to trust the hardware manufacturer not to
+ * have put in a back door.
+ *
+ * @buf: buffer allocated by caller to store the random data in
+ * @nbytes: length of outbuf
+ *
+ * Return number of bytes filled in.
+ */
+int __must_check get_random_bytes_arch(void *buf, int nbytes)
+{
+ u8 *p = buf;
+
+ while (nbytes) {
+ unsigned long v;
+ int chunk = min_t(int, nbytes, sizeof(unsigned long));
+
+ if (!arch_get_random_long(&v))
+ break;
+
+ memcpy(p, &v, chunk);
+ p += chunk;
+ nbytes -= chunk;
+ }
+
+ if (nbytes)
+ lrng_drng_get_atomic((u8 *)p, (u32)nbytes);
+
+ return nbytes;
+}
+EXPORT_SYMBOL(get_random_bytes_arch);
+
+/************************ LRNG user output interfaces *************************/
+
+static ssize_t lrng_read_common(char __user *buf, size_t nbytes)
+{
+ ssize_t ret = 0;
+ u8 tmpbuf[LRNG_DRNG_BLOCKSIZE] __aligned(LRNG_KCAPI_ALIGN);
+ u8 *tmp_large = NULL, *tmp = tmpbuf;
+ u32 tmplen = sizeof(tmpbuf);
+
+ if (nbytes == 0)
+ return 0;
+
+ /*
+ * Satisfy large read requests -- as the common case are smaller
+ * request sizes, such as 16 or 32 bytes, avoid a kmalloc overhead for
+ * those by using the stack variable of tmpbuf.
+ */
+ if (!IS_ENABLED(CONFIG_BASE_SMALL) && (nbytes > sizeof(tmpbuf))) {
+ tmplen = min_t(u32, nbytes, LRNG_DRNG_MAX_REQSIZE);
+ tmp_large = kmalloc(tmplen + LRNG_KCAPI_ALIGN, GFP_KERNEL);
+ if (!tmp_large)
+ tmplen = sizeof(tmpbuf);
+ else
+ tmp = PTR_ALIGN(tmp_large, LRNG_KCAPI_ALIGN);
+ }
+
+ while (nbytes) {
+ u32 todo = min_t(u32, nbytes, tmplen);
+ int rc = 0;
+
+ /* Reschedule if we received a large request. */
+ if ((tmp_large) && need_resched()) {
+ if (signal_pending(current)) {
+ if (ret == 0)
+ ret = -ERESTARTSYS;
+ break;
+ }
+ schedule();
+ }
+
+ rc = lrng_drng_get_sleep(tmp, todo);
+ if (rc <= 0) {
+ if (rc < 0)
+ ret = rc;
+ break;
+ }
+ if (copy_to_user(buf, tmp, rc)) {
+ ret = -EFAULT;
+ break;
+ }
+
+ nbytes -= rc;
+ buf += rc;
+ ret += rc;
+ }
+
+ /* Wipe data just returned from memory */
+ if (tmp_large)
+ kzfree(tmp_large);
+ else
+ memzero_explicit(tmpbuf, sizeof(tmpbuf));
+
+ return ret;
+}
+
+static ssize_t
+lrng_read_common_block(int nonblock, char __user *buf, size_t nbytes)
+{
+ if (nbytes == 0)
+ return 0;
+
+ if (unlikely(!lrng_state_operational())) {
+ int ret;
+
+ if (nonblock)
+ return -EAGAIN;
+
+ ret = wait_event_interruptible(lrng_init_wait,
+ lrng_state_operational());
+ if (unlikely(ret))
+ return ret;
+ }
+
+ return lrng_read_common(buf, nbytes);
+}
+
+static ssize_t lrng_drng_read_block(struct file *file, char __user *buf,
+ size_t nbytes, loff_t *ppos)
+{
+ return lrng_read_common_block(file->f_flags & O_NONBLOCK, buf, nbytes);
+}
+
+static unsigned int lrng_random_poll(struct file *file, poll_table *wait)
+{
+ __poll_t mask;
+
+ poll_wait(file, &lrng_init_wait, wait);
+ poll_wait(file, &lrng_write_wait, wait);
+ mask = 0;
+ if (lrng_state_operational())
+ mask |= EPOLLIN | EPOLLRDNORM;
+ if (lrng_need_entropy() ||
+ lrng_state_exseed_allow(lrng_noise_source_user))
+ mask |= EPOLLOUT | EPOLLWRNORM;
+ return mask;
+}
+
+static ssize_t lrng_drng_write_common(const char __user *buffer, size_t count,
+ u32 entropy_bits)
+{
+ ssize_t ret = 0;
+ u8 buf[64] __aligned(LRNG_KCAPI_ALIGN);
+ const char __user *p = buffer;
+ u32 orig_entropy_bits = entropy_bits;
+
+ if (!lrng_get_available())
+ return -EAGAIN;
+
+ count = min_t(size_t, count, INT_MAX);
+ while (count > 0) {
+ size_t bytes = min_t(size_t, count, sizeof(buf));
+ u32 ent = min_t(u32, bytes<<3, entropy_bits);
+
+ if (copy_from_user(&buf, p, bytes))
+ return -EFAULT;
+ /* Inject data into entropy pool */
+ lrng_pool_lfsr(buf, bytes);
+ lrng_pool_add_entropy(ent);
+
+ count -= bytes;
+ p += bytes;
+ ret += bytes;
+ entropy_bits -= ent;
+
+ cond_resched();
+ }
+
+ /* Force reseed of DRNG during next data request. */
+ if (!orig_entropy_bits)
+ lrng_drng_force_reseed();
+
+ return ret;
+}
+
+static ssize_t lrng_drng_read(struct file *file, char __user *buf,
+ size_t nbytes, loff_t *ppos)
+{
+ if (!lrng_state_min_seeded())
+ pr_notice_ratelimited("%s - use of insufficiently seeded DRNG "
+ "(%zu bytes read)\n", current->comm,
+ nbytes);
+ else if (!lrng_state_operational())
+ pr_debug_ratelimited("%s - use of not fully seeded DRNG (%zu "
+ "bytes read)\n", current->comm, nbytes);
+
+ return lrng_read_common(buf, nbytes);
+}
+
+static ssize_t lrng_drng_write(struct file *file, const char __user *buffer,
+ size_t count, loff_t *ppos)
+{
+ return lrng_drng_write_common(buffer, count, 0);
+}
+
+static long lrng_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+ int size, ent_count_bits;
+ int __user *p = (int __user *)arg;
+
+ switch (cmd) {
+ case RNDGETENTCNT:
+ ent_count_bits = lrng_avail_entropy();
+ if (put_user(ent_count_bits, p))
+ return -EFAULT;
+ return 0;
+ case RNDADDTOENTCNT:
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+ if (get_user(ent_count_bits, p))
+ return -EFAULT;
+ ent_count_bits = (int)lrng_avail_entropy() + ent_count_bits;
+ if (ent_count_bits < 0)
+ ent_count_bits = 0;
+ if (ent_count_bits > LRNG_POOL_SIZE_BITS)
+ ent_count_bits = LRNG_POOL_SIZE_BITS;
+ lrng_pool_set_entropy(ent_count_bits);
+ return 0;
+ case RNDADDENTROPY:
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+ if (get_user(ent_count_bits, p++))
+ return -EFAULT;
+ if (ent_count_bits < 0)
+ return -EINVAL;
+ if (get_user(size, p++))
+ return -EFAULT;
+ if (size < 0)
+ return -EINVAL;
+ lrng_state_exseed_set(lrng_noise_source_user, false);
+ /* there cannot be more entropy than data */
+ ent_count_bits = min(ent_count_bits, size<<3);
+ return lrng_drng_write_common((const char __user *)p, size,
+ ent_count_bits);
+ case RNDZAPENTCNT:
+ case RNDCLEARPOOL:
+ /* Clear the entropy pool counter. */
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+ lrng_pool_set_entropy(0);
+ return 0;
+ case RNDRESEEDCRNG:
+ /*
+ * We leave the capability check here since it is present
+ * in the upstream's RNG implementation. Yet, user space
+ * can trigger a reseed as easy as writing into /dev/random
+ * or /dev/urandom where no privilege is needed.
+ */
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+ /* Force a reseed of all DRNGs */
+ lrng_drng_force_reseed();
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int lrng_fasync(int fd, struct file *filp, int on)
+{
+ return fasync_helper(fd, filp, on, &fasync);
+}
+
+const struct file_operations random_fops = {
+ .read = lrng_drng_read_block,
+ .write = lrng_drng_write,
+ .poll = lrng_random_poll,
+ .unlocked_ioctl = lrng_ioctl,
+ .compat_ioctl = compat_ptr_ioctl,
+ .fasync = lrng_fasync,
+ .llseek = noop_llseek,
+};
+
+const struct file_operations urandom_fops = {
+ .read = lrng_drng_read,
+ .write = lrng_drng_write,
+ .unlocked_ioctl = lrng_ioctl,
+ .compat_ioctl = compat_ptr_ioctl,
+ .fasync = lrng_fasync,
+ .llseek = noop_llseek,
+};
+
+SYSCALL_DEFINE3(getrandom, char __user *, buf, size_t, count,
+ unsigned int, flags)
+{
+ if (flags & ~(GRND_NONBLOCK|GRND_RANDOM|GRND_INSECURE))
+ return -EINVAL;
+
+ /*
+ * Requesting insecure and blocking randomness at the same time makes
+ * no sense.
+ */
+ if ((flags &
+ (GRND_INSECURE|GRND_RANDOM)) == (GRND_INSECURE|GRND_RANDOM))
+ return -EINVAL;
+
+ if (count > INT_MAX)
+ count = INT_MAX;
+
+ if (flags & GRND_INSECURE)
+ return lrng_drng_read(NULL, buf, count, NULL);
+
+ return lrng_read_common_block(flags & GRND_NONBLOCK, buf, count);
+}
diff --git a/drivers/char/lrng/lrng_internal.h b/drivers/char/lrng/lrng_internal.h
new file mode 100644
index 000000000000..c9a7bccd14a0
--- /dev/null
+++ b/drivers/char/lrng/lrng_internal.h
@@ -0,0 +1,296 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright (C) 2018 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#ifndef _LRNG_INTERNAL_H
+#define _LRNG_INTERNAL_H
+
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/spinlock.h>
+
+/*************************** General LRNG parameter ***************************/
+
+/* Entropy pool parameter
+ *
+ * The LRNG_POOL_SIZE cannot be smaller than 64 bytes as the SHA-1 operation
+ * in lrng_chacha20.c requires multiples of 64 bytes
+ */
+#define LRNG_POOL_SIZE (16 << CONFIG_LRNG_POOL_SIZE)
+#define LRNG_POOL_WORD_BYTES (4) /* (sizeof(atomic_t)) */
+#define LRNG_POOL_SIZE_BYTES (LRNG_POOL_SIZE * LRNG_POOL_WORD_BYTES)
+#define LRNG_POOL_SIZE_BITS (LRNG_POOL_SIZE_BYTES * 8)
+#define LRNG_POOL_WORD_BITS (LRNG_POOL_WORD_BYTES * 8)
+
+/* Security strength of LRNG -- this must match DRNG security strength */
+#define LRNG_DRNG_SECURITY_STRENGTH_BYTES 32
+#define LRNG_DRNG_SECURITY_STRENGTH_BITS (LRNG_DRNG_SECURITY_STRENGTH_BYTES * 8)
+#define LRNG_DRNG_BLOCKSIZE 64 /* Maximum of DRNG block sizes */
+
+/*
+ * SP800-90A defines a maximum request size of 1<<16 bytes. The given value is
+ * considered a safer margin.
+ *
+ * This value is allowed to be changed.
+ */
+#define LRNG_DRNG_MAX_REQSIZE (1<<12)
+
+/*
+ * SP800-90A defines a maximum number of requests between reseeds of 2^48.
+ * The given value is considered a much safer margin, balancing requests for
+ * frequent reseeds with the need to conserve entropy. This value MUST NOT be
+ * larger than INT_MAX because it is used in an atomic_t.
+ *
+ * This value is allowed to be changed.
+ */
+#define LRNG_DRNG_RESEED_THRESH (1<<20)
+
+/*
+ * Number of interrupts to be recorded to assume that DRNG security strength
+ * bits of entropy are received.
+ * Note: a value below the DRNG security strength should not be defined as this
+ * may imply the DRNG can never be fully seeded in case other noise
+ * sources are unavailable.
+ *
+ * This value is allowed to be changed.
+ */
+#define LRNG_IRQ_ENTROPY_BITS LRNG_DRNG_SECURITY_STRENGTH_BITS
+
+/*
+ * Min required seed entropy is 128 bits covering the minimum entropy
+ * requirement of SP800-131A and the German BSI's TR02102.
+ *
+ * This value is allowed to be changed.
+ */
+#define LRNG_FULL_SEED_ENTROPY_BITS LRNG_DRNG_SECURITY_STRENGTH_BITS
+#define LRNG_MIN_SEED_ENTROPY_BITS 128
+#define LRNG_INIT_ENTROPY_BITS 32
+
+/*
+ * Amount of entropy that is lost with the conditioning functions of LFSR and
+ * hash_df as shown with the entropy analysis compliant to SP800-90B.
+ */
+#define LRNG_CONDITIONING_ENTROPY_LOSS 1
+
+/*
+ * Wakeup value
+ *
+ * This value is allowed to be changed.
+ */
+#if (LRNG_POOL_SIZE_BITS <= (LRNG_DRNG_SECURITY_STRENGTH_BITS * 2))
+# define LRNG_WRITE_WAKEUP_ENTROPY (LRNG_DRNG_SECURITY_STRENGTH_BITS + \
+ LRNG_CONDITIONING_ENTROPY_LOSS)
+#else
+# define LRNG_WRITE_WAKEUP_ENTROPY (LRNG_DRNG_SECURITY_STRENGTH_BITS * 2)
+#endif
+
+/*
+ * Oversampling factor of IRQ events to obtain
+ * LRNG_DRNG_SECURITY_STRENGTH_BYTES. This factor is used when a
+ * high-resolution time stamp is not available. In this case, jiffies and
+ * register contents are used to fill the entropy pool. These noise sources
+ * are much less entropic than the high-resolution timer. The entropy content
+ * is the entropy content assumed with LRNG_IRQ_ENTROPY_BITS divided by
+ * LRNG_IRQ_OVERSAMPLING_FACTOR.
+ *
+ * This value is allowed to be changed.
+ */
+#define LRNG_IRQ_OVERSAMPLING_FACTOR 10
+
+/*
+ * Alignmask which should cover all cipher implementations
+ * WARNING: If this is changed to a value larger than 8, manual
+ * alignment is necessary as older versions of GCC may not be capable
+ * of aligning stack variables at boundaries greater than 8.
+ * In this case, PTR_ALIGN must be used.
+ */
+#define LRNG_KCAPI_ALIGN 8
+
+/************************ Default DRNG implementation *************************/
+
+extern struct chacha20_state chacha20;
+extern const struct lrng_crypto_cb lrng_cc20_crypto_cb;
+void lrng_cc20_init_state(struct chacha20_state *state);
+
+/********************************** /proc *************************************/
+
+static inline void lrng_pool_inc_numa_node(void) { }
+
+/****************************** LRNG interfaces *******************************/
+
+extern u32 lrng_write_wakeup_bits;
+extern int lrng_drng_reseed_max_time;
+
+void lrng_writer_wakeup(void);
+void lrng_init_wakeup(void);
+void lrng_debug_report_seedlevel(const char *name);
+void lrng_process_ready_list(void);
+
+/************************** Entropy pool management ***************************/
+
+enum lrng_external_noise_source {
+ lrng_noise_source_hw,
+ lrng_noise_source_user
+};
+
+bool lrng_state_exseed_allow(enum lrng_external_noise_source source);
+void lrng_state_exseed_set(enum lrng_external_noise_source source, bool type);
+void lrng_state_init_seed_work(void);
+u32 lrng_avail_entropy(void);
+void lrng_set_entropy_thresh(u32 new);
+int lrng_pool_trylock(void);
+void lrng_pool_unlock(void);
+void lrng_reset_state(void);
+void lrng_pool_all_numa_nodes_seeded(void);
+bool lrng_state_min_seeded(void);
+bool lrng_state_fully_seeded(void);
+bool lrng_state_operational(void);
+bool lrng_pool_highres_timer(void);
+void lrng_pool_set_entropy(u32 entropy_bits);
+void lrng_pool_lfsr(const u8 *buf, u32 buflen);
+void lrng_pool_lfsr_nonaligned(const u8 *buf, u32 buflen);
+void lrng_pool_lfsr_u32(u32 value);
+void lrng_pool_add_irq(u32 irq_num);
+void lrng_pool_add_entropy(u32 entropy_bits);
+
+struct entropy_buf {
+ u8 a[LRNG_DRNG_SECURITY_STRENGTH_BYTES];
+ u8 b[LRNG_DRNG_SECURITY_STRENGTH_BYTES];
+ u8 c[LRNG_DRNG_SECURITY_STRENGTH_BYTES];
+ u32 now;
+};
+
+int lrng_fill_seed_buffer(const struct lrng_crypto_cb *crypto_cb, void *hash,
+ struct entropy_buf *entropy_buf, u32 entropy_retain);
+void lrng_init_ops(u32 seed_bits);
+
+/************************** Jitter RNG Noise Source ***************************/
+
+#ifdef CONFIG_LRNG_JENT
+u32 lrng_get_jent(u8 *outbuf, unsigned int outbuflen);
+u32 lrng_jent_entropylevel(void);
+#else /* CONFIG_CRYPTO_JITTERENTROPY */
+static inline u32 lrng_get_jent(u8 *outbuf, unsigned int outbuflen) {return 0; }
+static inline u32 lrng_jent_entropylevel(void) { return 0; }
+#endif /* CONFIG_CRYPTO_JITTERENTROPY */
+
+/*************************** CPU-based Noise Source ***************************/
+
+u32 lrng_get_arch(u8 *outbuf);
+u32 lrng_slow_noise_req_entropy(u32 required_entropy_bits);
+
+/****************************** DRNG processing *******************************/
+
+/* Secondary DRNG state handle */
+struct lrng_drng {
+ void *drng; /* DRNG handle */
+ void *hash; /* Hash handle */
+ const struct lrng_crypto_cb *crypto_cb; /* Crypto callbacks */
+ atomic_t requests; /* Number of DRNG requests */
+ unsigned long last_seeded; /* Last time it was seeded */
+ bool fully_seeded; /* Is DRNG fully seeded? */
+ bool force_reseed; /* Force a reseed */
+ struct mutex lock;
+ spinlock_t spin_lock;
+};
+
+extern struct mutex lrng_crypto_cb_update;
+
+struct lrng_drng *lrng_drng_init_instance(void);
+struct lrng_drng *lrng_drng_atomic_instance(void);
+
+static __always_inline bool lrng_drng_is_atomic(struct lrng_drng *drng)
+{
+ return (drng->drng == lrng_drng_atomic_instance()->drng);
+}
+
+/* Lock the DRNG */
+static __always_inline void lrng_drng_lock(struct lrng_drng *drng,
+ unsigned long *flags)
+{
+ /* Use spin lock in case the atomic DRNG context is used */
+ if (lrng_drng_is_atomic(drng)) {
+ spin_lock_irqsave(&drng->spin_lock, *flags);
+
+ /*
+ * In case a lock transition happened while we were spinning,
+ * catch this case and use the new lock type.
+ */
+ if (unlikely(!lrng_drng_is_atomic(drng))) {
+ spin_unlock_irqrestore(&drng->spin_lock, *flags);
+ mutex_lock(&drng->lock);
+ }
+ } else {
+ mutex_lock(&drng->lock);
+ }
+}
+
+/* Unlock the DRNG */
+static __always_inline void lrng_drng_unlock(struct lrng_drng *drng,
+ unsigned long *flags)
+{
+ if (lrng_drng_is_atomic(drng))
+ spin_unlock_irqrestore(&drng->spin_lock, *flags);
+ else
+ mutex_unlock(&drng->lock);
+}
+
+bool lrng_get_available(void);
+void lrng_set_available(void);
+void lrng_drngs_init_cc20(void);
+void lrng_drng_reset(struct lrng_drng *drng);
+int lrng_drng_get_atomic(u8 *outbuf, u32 outbuflen);
+int lrng_drng_get_sleep(u8 *outbuf, u32 outbuflen);
+void lrng_drng_force_reseed(void);
+void lrng_drng_seed_work(struct work_struct *dummy);
+
+static inline struct lrng_drng **lrng_drng_instances(void) { return NULL; }
+static inline void lrng_drngs_numa_alloc(void) { return; }
+
+/************************** Health Test linking code **************************/
+
+enum lrng_health_res {
+ lrng_health_pass, /* Health test passes on time stamp */
+ lrng_health_fail_use, /* Time stamp unhealthy, but mix in */
+ lrng_health_fail_drop /* Time stamp unhealthy, drop it */
+};
+
+#ifdef CONFIG_LRNG_HEALTH_TESTS
+bool lrng_sp80090b_startup_complete(void);
+bool lrng_sp80090b_compliant(void);
+
+enum lrng_health_res lrng_health_test(u32 now_time);
+void lrng_health_disable(void);
+
+void lrng_reset(void);
+#else /* CONFIG_LRNG_HEALTH_TESTS */
+static inline bool lrng_sp80090b_startup_complete(void) { return true; }
+static inline bool lrng_sp80090b_compliant(void) { return false; }
+
+static inline enum lrng_health_res
+lrng_health_test(u32 now_time) { return lrng_health_pass; }
+static inline void lrng_health_disable(void) { }
+#endif /* CONFIG_LRNG_HEALTH_TESTS */
+
+/****************************** Helper code ***********************************/
+
+static inline u32 atomic_read_u32(atomic_t *v)
+{
+ return (u32)atomic_read(v);
+}
+
+/*************************** Auxiliary functions ******************************/
+
+void invalidate_batched_entropy(void);
+
+/***************************** Testing code ***********************************/
+
+#ifdef CONFIG_LRNG_TESTING
+bool lrng_raw_entropy_store(u32 value);
+#else /* CONFIG_LRNG_TESTING */
+static inline bool lrng_raw_entropy_store(u32 value) { return false; }
+#endif /* CONFIG_LRNG_TESTING */
+
+#endif /* _LRNG_INTERNAL_H */
diff --git a/drivers/char/lrng/lrng_lfsr.h b/drivers/char/lrng/lrng_lfsr.h
new file mode 100644
index 000000000000..f0e38a0aadfd
--- /dev/null
+++ b/drivers/char/lrng/lrng_lfsr.h
@@ -0,0 +1,152 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * LRNG Linear Feedback Shift Register
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+/* Status information about IRQ noise source */
+struct lrng_irq_info {
+ atomic_t num_events; /* Number of healthy IRQs since last read */
+ atomic_t num_events_thresh; /* Reseed threshold */
+ atomic_t reseed_in_progress; /* Flag for on executing reseed */
+ bool irq_highres_timer; /* Is high-resolution timer available? */
+ u32 irq_entropy_bits; /* LRNG_IRQ_ENTROPY_BITS? */
+};
+
+/*
+ * This is the entropy pool used by the slow noise source. Its size should
+ * be at least as large as LRNG_DRNG_SECURITY_STRENGTH_BITS.
+ *
+ * The pool array is aligned to 8 bytes to comfort the kernel crypto API cipher
+ * implementations of the hash functions used to read the pool: for some
+ * accelerated implementations, we need an alignment to avoid a realignment
+ * which involves memcpy(). The alignment to 8 bytes should satisfy all crypto
+ * implementations.
+ *
+ * LRNG_POOL_SIZE is allowed to be changed only if the taps of the polynomial
+ * used for the LFSR are changed as well. The size must be in powers of 2 due
+ * to the mask handling in lrng_pool_lfsr_u32 which uses AND instead of modulo.
+ */
+struct lrng_pool {
+ union {
+ struct {
+ /*
+ * hash_df implementation: counter, requested_bits and
+ * pool form a linear buffer that is used in the
+ * hash_df function specified in SP800-90A section
+ * 10.3.1
+ */
+ unsigned char counter;
+ __be32 requested_bits;
+
+ /* Pool */
+ atomic_t pool[LRNG_POOL_SIZE];
+ /* Ptr into pool for next IRQ word injection */
+ atomic_t pool_ptr;
+ /* rotate for LFSR */
+ atomic_t input_rotate;
+ /* All NUMA DRNGs seeded? */
+ bool all_online_numa_node_seeded;
+ /* IRQ noise source status info */
+ struct lrng_irq_info irq_info;
+ /* Serialize read of entropy pool */
+ spinlock_t lock;
+ };
+ /*
+ * Static SHA-1 implementation in lrng_cc20_hash_buffer
+ * processes data 64-byte-wise. Hence, ensure proper size
+ * of LRNG entropy pool data structure.
+ */
+ u8 hash_input_buf[LRNG_POOL_SIZE_BYTES + 64];
+ };
+};
+
+/*
+ * Implement a (modified) twisted Generalized Feedback Shift Register. (See M.
+ * Matsumoto & Y. Kurita, 1992. Twisted GFSR generators. ACM Transactions on
+ * Modeling and Computer Simulation 2(3):179-194. Also see M. Matsumoto & Y.
+ * Kurita, 1994. Twisted GFSR generators II. ACM Transactions on Modeling and
+ * Computer Simulation 4:254-266).
+ */
+static u32 const lrng_twist_table[8] = {
+ 0x00000000, 0x3b6e20c8, 0x76dc4190, 0x4db26158,
+ 0xedb88320, 0xd6d6a3e8, 0x9b64c2b0, 0xa00ae278 };
+
+/*
+ * The polynomials for the LFSR are taken from the document "Table of Linear
+ * Feedback Shift Registers" by Roy Ward, Tim Molteno, October 26, 2007.
+ * The first polynomial is from "Primitive Binary Polynomials" by Wayne
+ * Stahnke (1973) and is primitive as well as irreducible.
+ *
+ * Note, the tap values are smaller by one compared to the documentation because
+ * they are used as an index into an array where the index starts by zero.
+ *
+ * All polynomials were also checked to be primitive and irreducible with magma
+ * which ensures that the key property of the LFSR providing a compression
+ * function for entropy is guaranteed.
+ */
+static u32 const lrng_lfsr_polynomial[][4] = {
+ { 15, 13, 12, 10 }, /* 16 words */
+ { 31, 29, 25, 24 }, /* 32 words */
+ { 63, 62, 60, 59 }, /* 64 words */
+ { 127, 28, 26, 1 }, /* 128 words by Stahnke */
+ { 255, 253, 250, 245 }, /* 256 words */
+ { 511, 509, 506, 503 }, /* 512 words */
+ { 1023, 1014, 1001, 1000 }, /* 1024 words */
+ { 2047, 2034, 2033, 2028 }, /* 2048 words */
+ { 4095, 4094, 4080, 4068 }, /* 4096 words */
+};
+
+static inline void _lrng_pool_lfsr_u32(struct lrng_pool *pool, u32 value)
+{
+ /*
+ * Process the LFSR by altering not adjacent words but rather
+ * more spaced apart words. Using a prime number ensures that all words
+ * are processed evenly. As some the LFSR polynomials taps are close
+ * together, processing adjacent words with the LSFR taps may be
+ * inappropriate as the data just mixed-in at these taps may be not
+ * independent from the current data to be mixed in.
+ */
+ u32 ptr = (u32)atomic_add_return_relaxed(67, &pool->pool_ptr) &
+ (LRNG_POOL_SIZE - 1);
+ /*
+ * Add 7 bits of rotation to the pool. At the beginning of the
+ * pool, add an extra 7 bits rotation, so that successive passes
+ * spread the input bits across the pool evenly.
+ *
+ * Note, there is a race between getting ptr and calculating
+ * input_rotate when ptr is is obtained on two or more CPUs at the
+ * same time. This race is irrelevant as it may only come into effect
+ * if 3 or more CPUs race at the same time which is very unlikely. If
+ * the race happens, it applies to one event only. As this rolling
+ * supports the LFSR without being strictly needed, we accept this
+ * race.
+ */
+ u32 input_rotate = (u32)atomic_add_return_relaxed((ptr ? 7 : 14),
+ &pool->input_rotate) & 31;
+ u32 word = rol32(value, input_rotate);
+
+ BUILD_BUG_ON(LRNG_POOL_WORD_BYTES != sizeof(pool->pool[0]));
+ BUILD_BUG_ON(LRNG_POOL_SIZE - 1 !=
+ lrng_lfsr_polynomial[CONFIG_LRNG_POOL_SIZE][0]);
+ word ^= atomic_read_u32(&pool->pool[ptr]);
+ word ^= atomic_read_u32(&pool->pool[
+ (ptr + lrng_lfsr_polynomial[CONFIG_LRNG_POOL_SIZE][0]) &
+ (LRNG_POOL_SIZE - 1)]);
+ word ^= atomic_read_u32(&pool->pool[
+ (ptr + lrng_lfsr_polynomial[CONFIG_LRNG_POOL_SIZE][1]) &
+ (LRNG_POOL_SIZE - 1)]);
+ word ^= atomic_read_u32(&pool->pool[
+ (ptr + lrng_lfsr_polynomial[CONFIG_LRNG_POOL_SIZE][2]) &
+ (LRNG_POOL_SIZE - 1)]);
+ word ^= atomic_read_u32(&pool->pool[
+ (ptr + lrng_lfsr_polynomial[CONFIG_LRNG_POOL_SIZE][3]) &
+ (LRNG_POOL_SIZE - 1)]);
+
+ word = (word >> 3) ^ lrng_twist_table[word & 7];
+ atomic_set(&pool->pool[ptr], word);
+}
+
+u32 __lrng_pool_hash_df(const struct lrng_crypto_cb *crypto_cb, void *hash,
+ struct lrng_pool *pool, u8 *outbuf, u32 requested_bits);
diff --git a/drivers/char/lrng/lrng_pool.c b/drivers/char/lrng/lrng_pool.c
new file mode 100644
index 000000000000..1547aa14dbc6
--- /dev/null
+++ b/drivers/char/lrng/lrng_pool.c
@@ -0,0 +1,585 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG Entropy pool management
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <asm/irq_regs.h>
+#include <linux/lrng.h>
+#include <linux/percpu.h>
+#include <linux/random.h>
+#include <linux/utsname.h>
+#include <linux/workqueue.h>
+
+#include "lrng_internal.h"
+#include "lrng_lfsr.h"
+
+struct lrng_state {
+ bool lrng_operational; /* Is DRNG operational? */
+ bool lrng_fully_seeded; /* Is DRNG fully seeded? */
+ bool lrng_min_seeded; /* Is DRNG minimally seeded? */
+
+ /*
+ * To ensure that external entropy providers cannot dominate the
+ * internal noise sources but yet cannot be dominated by internal
+ * noise sources, the following booleans are intended to allow
+ * external to provide seed once when a DRNG reseed occurs. This
+ * triggering of external noise source is performed even when the
+ * entropy pool has sufficient entropy.
+ */
+ bool lrng_seed_hw; /* Allow HW to provide seed */
+ bool lrng_seed_user; /* Allow user space to provide seed */
+
+ struct work_struct lrng_seed_work; /* (re)seed work queue */
+};
+
+static struct lrng_pool lrng_pool __aligned(LRNG_KCAPI_ALIGN) = {
+ .irq_info = {
+ .irq_entropy_bits = LRNG_IRQ_ENTROPY_BITS,
+ .num_events_thresh = ATOMIC_INIT(LRNG_INIT_ENTROPY_BITS +
+ LRNG_CONDITIONING_ENTROPY_LOSS),
+ /* Sample IRQ pointer data at least during boot */
+ .irq_highres_timer = false },
+ .lock = __SPIN_LOCK_UNLOCKED(lrng_pool.lock)
+};
+
+static struct lrng_state lrng_state = { false, false, false, true, true };
+
+/********************************** Helper ***********************************/
+
+/* External entropy provider is allowed to provide seed data */
+bool lrng_state_exseed_allow(enum lrng_external_noise_source source)
+{
+ if (source == lrng_noise_source_hw)
+ return lrng_state.lrng_seed_hw;
+ return lrng_state.lrng_seed_user;
+}
+
+/* Enable / disable external entropy provider to furnish seed */
+void lrng_state_exseed_set(enum lrng_external_noise_source source, bool type)
+{
+ if (source == lrng_noise_source_hw)
+ lrng_state.lrng_seed_hw = type;
+ else
+ lrng_state.lrng_seed_user = type;
+}
+
+static inline void lrng_state_exseed_allow_all(void)
+{
+ lrng_state_exseed_set(lrng_noise_source_hw, true);
+ lrng_state_exseed_set(lrng_noise_source_user, true);
+}
+
+void lrng_state_init_seed_work(void)
+{
+ INIT_WORK(&lrng_state.lrng_seed_work, lrng_drng_seed_work);
+}
+
+static inline u32 lrng_entropy_to_data(u32 entropy_bits)
+{
+ return ((entropy_bits * lrng_pool.irq_info.irq_entropy_bits) /
+ LRNG_DRNG_SECURITY_STRENGTH_BITS);
+}
+
+static inline u32 lrng_data_to_entropy(u32 irqnum)
+{
+ return ((irqnum * LRNG_DRNG_SECURITY_STRENGTH_BITS) /
+ lrng_pool.irq_info.irq_entropy_bits);
+}
+
+u32 lrng_avail_entropy(void)
+{
+ return min_t(u32, LRNG_POOL_SIZE_BITS, lrng_data_to_entropy(
+ atomic_read_u32(&lrng_pool.irq_info.num_events)));
+}
+
+void lrng_set_entropy_thresh(u32 new)
+{
+ atomic_set(&lrng_pool.irq_info.num_events_thresh,
+ lrng_entropy_to_data(new));
+}
+
+/*
+ * Reading of the LRNG pool is only allowed by one caller. The reading is
+ * only performed to (re)seed DRNGs. Thus, if this "lock" is already taken,
+ * the reseeding operation is in progress. The caller is not intended to wait
+ * but continue with its other operation.
+ */
+int lrng_pool_trylock(void)
+{
+ return atomic_cmpxchg(&lrng_pool.irq_info.reseed_in_progress, 0, 1);
+}
+
+void lrng_pool_unlock(void)
+{
+ atomic_set(&lrng_pool.irq_info.reseed_in_progress, 0);
+}
+
+void lrng_reset_state(void)
+{
+ struct lrng_irq_info *irq_info = &lrng_pool.irq_info;
+
+ atomic_set(&irq_info->num_events, 0);
+ lrng_state.lrng_operational = false;
+ lrng_state.lrng_fully_seeded = false;
+ lrng_state.lrng_min_seeded = false;
+ lrng_pool.all_online_numa_node_seeded = false;
+ pr_debug("reset LRNG\n");
+}
+
+void lrng_pool_all_numa_nodes_seeded(void)
+{
+ lrng_pool.all_online_numa_node_seeded = true;
+}
+
+bool lrng_state_min_seeded(void)
+{
+ return lrng_state.lrng_min_seeded;
+}
+
+bool lrng_state_fully_seeded(void)
+{
+ return lrng_state.lrng_fully_seeded;
+}
+
+bool lrng_state_operational(void)
+{
+ return lrng_state.lrng_operational;
+}
+
+bool lrng_pool_highres_timer(void)
+{
+ return lrng_pool.irq_info.irq_highres_timer;
+}
+
+void lrng_pool_set_entropy(u32 entropy_bits)
+{
+ atomic_set(&lrng_pool.irq_info.num_events,
+ lrng_entropy_to_data(entropy_bits));
+}
+
+static void lrng_pool_configure(bool highres_timer, u32 irq_entropy_bits)
+{
+ struct lrng_irq_info *irq_info = &lrng_pool.irq_info;
+
+ irq_info->irq_highres_timer = highres_timer;
+ if (irq_info->irq_entropy_bits != irq_entropy_bits) {
+ irq_info->irq_entropy_bits = irq_entropy_bits;
+ /* Reset the threshold based on new oversampling factor. */
+ lrng_set_entropy_thresh(atomic_read_u32(
+ &irq_info->num_events_thresh));
+ }
+}
+
+static int __init lrng_init_time_source(void)
+{
+ if (random_get_entropy() || random_get_entropy()) {
+ /*
+ * As the highres timer is identified here, previous interrupts
+ * obtained during boot time are treated like a lowres-timer
+ * would have been present.
+ */
+ lrng_pool_configure(true, LRNG_IRQ_ENTROPY_BITS);
+ } else {
+ lrng_health_disable();
+ lrng_pool_configure(false, LRNG_IRQ_ENTROPY_BITS *
+ LRNG_IRQ_OVERSAMPLING_FACTOR);
+ pr_warn("operating without high-resolution timer and applying "
+ "IRQ oversampling factor %u\n",
+ LRNG_IRQ_OVERSAMPLING_FACTOR);
+ }
+
+ return 0;
+}
+
+core_initcall(lrng_init_time_source);
+
+/* invoke function with buffer aligned to 4 bytes */
+void lrng_pool_lfsr(const u8 *buf, u32 buflen)
+{
+ u32 *p_buf = (u32 *)buf;
+
+ for (; buflen >= 4; buflen -= 4)
+ lrng_pool_lfsr_u32(*p_buf++);
+
+ buf = (u8 *)p_buf;
+ while (buflen--)
+ lrng_pool_lfsr_u32(*buf++);
+}
+
+void lrng_pool_lfsr_nonaligned(const u8 *buf, u32 buflen)
+{
+ while (buflen) {
+ if (!((unsigned long)buf & (sizeof(u32) - 1))) {
+ lrng_pool_lfsr(buf, buflen);
+ return;
+ }
+
+ lrng_pool_lfsr_u32(*buf++);
+ buflen--;
+ }
+}
+
+/**************************** Interrupt processing ****************************/
+
+/**
+ * Hot code path - inject data into entropy pool using LFSR
+ */
+void lrng_pool_lfsr_u32(u32 value)
+{
+ _lrng_pool_lfsr_u32(&lrng_pool, value);
+}
+
+/**
+ * Hot code path - mix data into entropy pool
+ */
+void lrng_pool_add_irq(u32 irq_num)
+{
+ struct lrng_irq_info *irq_info = &lrng_pool.irq_info;
+
+ atomic_add(irq_num, &irq_info->num_events);
+
+ /*
+ * Once all DRNGs are fully seeded, the interrupt noise
+ * sources will not trigger any reseeding any more.
+ */
+ if (likely(lrng_pool.all_online_numa_node_seeded))
+ return;
+
+ /* Only try to reseed if the DRNG is alive. */
+ if (!lrng_get_available())
+ return;
+
+ /* Only trigger the DRNG reseed if we have collected enough IRQs. */
+ if (atomic_read_u32(&lrng_pool.irq_info.num_events) <
+ atomic_read_u32(&lrng_pool.irq_info.num_events_thresh))
+ return;
+
+ /* Ensure that the seeding only occurs once at any given time. */
+ if (lrng_pool_trylock())
+ return;
+
+ /* Seed the DRNG with IRQ noise. */
+ schedule_work(&lrng_state.lrng_seed_work);
+}
+
+void lrng_pool_add_entropy(u32 entropy_bits)
+{
+ lrng_pool_add_irq(lrng_entropy_to_data(entropy_bits));
+}
+
+/**
+ * Generate a hashed output of pool using the SP800-90A section 10.3.1 hash_df
+ * function
+ */
+u32 __lrng_pool_hash_df(const struct lrng_crypto_cb *crypto_cb, void *hash,
+ struct lrng_pool *pool, u8 *outbuf, u32 requested_bits)
+{
+ u32 digestsize, requested_bytes = requested_bits >> 3,
+ generated_bytes = 0;
+ u8 digest[64] __aligned(LRNG_KCAPI_ALIGN);
+
+ digestsize = crypto_cb->lrng_hash_digestsize(hash);
+ if (digestsize > sizeof(digest)) {
+ pr_err("Digest buffer too small\n");
+ return 0;
+ }
+
+ pool->counter = 1;
+ pool->requested_bits = cpu_to_be32(requested_bytes << 3);
+
+ while (requested_bytes) {
+ u32 tocopy = min_t(u32, requested_bytes, digestsize);
+
+ /* The counter must not wrap */
+ if (pool->counter == 0)
+ goto out;
+
+ if (crypto_cb->lrng_hash_buffer(hash, (u8 *)pool,
+ LRNG_POOL_SIZE_BYTES + 64,
+ digest))
+ goto out;
+
+ /* Copy the data out to the caller */
+ memcpy(outbuf + generated_bytes, digest, tocopy);
+ requested_bytes -= tocopy;
+ generated_bytes += tocopy;
+ pool->counter++;
+ }
+
+out:
+ /* Mix read data back into pool for backtracking resistance */
+ if (generated_bytes)
+ lrng_pool_lfsr(outbuf, generated_bytes);
+ memzero_explicit(digest, digestsize);
+ return (generated_bytes<<3);
+}
+
+static inline u32 lrng_pool_hash_df(const struct lrng_crypto_cb *crypto_cb,
+ void *hash, u8 *outbuf, u32 requested_bits)
+{
+ return __lrng_pool_hash_df(crypto_cb, hash, &lrng_pool, outbuf,
+ requested_bits);
+}
+
+/**
+ * Read the entropy pool out for use.
+ *
+ * This function handles the translation from the number of received interrupts
+ * into an entropy statement. The conversion depends on LRNG_IRQ_ENTROPY_BITS
+ * which defines how many interrupts must be received to obtain 256 bits of
+ * entropy. With this value, the function lrng_data_to_entropy converts a given
+ * data size (received interrupts, requested amount of data, etc.) into an
+ * entropy statement. lrng_entropy_to_data does the reverse.
+ *
+ * Both functions are agnostic about the type of data: when the number of
+ * interrupts is processed by these functions, the resulting entropy value is in
+ * bits as we assume the entropy of interrupts is measured in bits. When data is
+ * processed, the entropy value is in bytes as the data is measured in bytes.
+ *
+ * @outbuf: buffer to store data in with size LRNG_DRNG_SECURITY_STRENGTH_BYTES
+ * @requested_entropy_bits: requested bits of entropy -- the function will
+ * return at least this amount of entropy if available
+ * @entropy_retain: amount of entropy in bits that should be left in the pool
+ * @return: estimated entropy from the IRQs that was obtained
+ */
+static u32 lrng_get_pool(const struct lrng_crypto_cb *crypto_cb, void *hash,
+ u8 *outbuf, u32 requested_entropy_bits,
+ u32 entropy_retain)
+{
+ struct lrng_pool *pool = &lrng_pool;
+ struct lrng_state *state = &lrng_state;
+ struct lrng_irq_info *irq_info = &pool->irq_info;
+ unsigned long flags;
+
+ u32 irq_num_events_used, irq_num_events, avail_entropy_bits;
+
+ /* This get_pool operation must only be called once at a given time! */
+ spin_lock_irqsave(&pool->lock, flags);
+
+ /* How many unused interrupts are in entropy pool? */
+ irq_num_events = atomic_read_u32(&irq_info->num_events);
+ /* Convert available interrupts into entropy statement */
+ avail_entropy_bits = lrng_data_to_entropy(irq_num_events);
+
+ /* Cap available entropy to pool size */
+ avail_entropy_bits =
+ min_t(u32, avail_entropy_bits, LRNG_POOL_SIZE_BITS);
+
+ /* How much entropy we need to and can we use? */
+ if (unlikely(!state->lrng_fully_seeded)) {
+ /*
+ * During boot time, we read 256 bits data with
+ * avail_entropy_bits entropy. In case our conservative
+ * entropy estimate underestimates the available entropy
+ * we can transport as much available entropy as
+ * possible. The entropy pool does not operate compliant to
+ * the German AIS 21/31 NTG.1 yet.
+ */
+ requested_entropy_bits =
+ LRNG_DRNG_SECURITY_STRENGTH_BITS;
+ } else {
+ /* Provide all entropy above retaining level */
+ if (avail_entropy_bits < entropy_retain) {
+ requested_entropy_bits = 0;
+ goto out;
+ }
+ avail_entropy_bits -= entropy_retain;
+ requested_entropy_bits = min_t(u32, avail_entropy_bits,
+ requested_entropy_bits);
+ }
+
+ /* Hash is a compression function: we generate entropy amount of data */
+ requested_entropy_bits = round_down(requested_entropy_bits, 8);
+
+ requested_entropy_bits = lrng_pool_hash_df(crypto_cb, hash, outbuf,
+ requested_entropy_bits);
+
+ /* Boot time: After getting the full buffer adjust the entropy value. */
+ requested_entropy_bits = min_t(u32, avail_entropy_bits,
+ requested_entropy_bits);
+
+out:
+ /* Convert used entropy into interrupt number for subtraction */
+ irq_num_events_used = lrng_entropy_to_data(requested_entropy_bits);
+
+ /*
+ * The hash_df operation entropy assessment shows that the output
+ * entropy is one bit smaller than the input entropy. Therefore we
+ * account for this one bit of entropy here: if we have sufficient
+ * entropy in the LFSR, we say we used one bit of entropy more.
+ * Otherwise we reduce the amount of entropy we say we generated with
+ * the hash_df.
+ */
+ if (irq_num_events_used) {
+ if ((irq_num_events_used + LRNG_CONDITIONING_ENTROPY_LOSS) <=
+ lrng_entropy_to_data(avail_entropy_bits)) {
+ irq_num_events_used += LRNG_CONDITIONING_ENTROPY_LOSS;
+ } else {
+ if (unlikely(requested_entropy_bits <
+ LRNG_CONDITIONING_ENTROPY_LOSS))
+ requested_entropy_bits = 0;
+ else
+ requested_entropy_bits -=
+ LRNG_CONDITIONING_ENTROPY_LOSS;
+ }
+ }
+
+ /*
+ * New events might have arrived in the meanwhile and we don't
+ * want to throw them away unconditionally. On the other hand,
+ * these new events might have been mixed in before
+ * lrng_hash_df_pool() had been able to draw any entropy
+ * from the pool and thus, the pool capacity might have been
+ * exceeded at some point. Note that in theory, some events
+ * might get lost inbetween the atomic_read() and
+ * atomic_set() below. But that's fine, because it's no real
+ * concern while code preventing this would come at the cost of
+ * additional complexity. Likewise, some events which arrived
+ * after full or partial completion of the __lrng_hash_df_pool()
+ * above might get unnecessarily thrown away by the min()
+ * operation below; the same argument applies there.
+ */
+ irq_num_events = atomic_read_u32(&irq_info->num_events);
+ irq_num_events = min_t(u32, irq_num_events,
+ lrng_entropy_to_data(LRNG_POOL_SIZE_BITS));
+ irq_num_events -= irq_num_events_used;
+ atomic_set(&irq_info->num_events, irq_num_events);
+
+ spin_unlock_irqrestore(&pool->lock, flags);
+
+ /* Obtain entropy statement in bits from the used entropy */
+ pr_debug("obtained %u bits of entropy from %u newly collected "
+ "interrupts - not using %u interrupts\n",
+ requested_entropy_bits, irq_num_events_used,
+ irq_num_events);
+
+ return requested_entropy_bits;
+}
+
+/* Fill the seed buffer with data from the noise sources */
+int lrng_fill_seed_buffer(const struct lrng_crypto_cb *crypto_cb, void *hash,
+ struct entropy_buf *entropy_buf, u32 entropy_retain)
+{
+ struct lrng_state *state = &lrng_state;
+ u32 total_entropy_bits = 0;
+
+ /* Require at least 128 bits of entropy for any reseed. */
+ if (state->lrng_fully_seeded &&
+ (lrng_avail_entropy() <
+ lrng_slow_noise_req_entropy(LRNG_MIN_SEED_ENTROPY_BITS +
+ LRNG_CONDITIONING_ENTROPY_LOSS) +
+ entropy_retain))
+ goto wakeup;
+
+ /*
+ * Concatenate the output of the noise sources. This would be the
+ * spot to add an entropy extractor logic if desired. Note, this
+ * has the ability to collect entropy equal or larger than the DRNG
+ * strength.
+ */
+ total_entropy_bits = lrng_get_pool(crypto_cb, hash, entropy_buf->a,
+ LRNG_DRNG_SECURITY_STRENGTH_BITS,
+ entropy_retain);
+ total_entropy_bits += lrng_get_arch(entropy_buf->b);
+ total_entropy_bits += lrng_get_jent(entropy_buf->c,
+ LRNG_DRNG_SECURITY_STRENGTH_BYTES);
+
+ /* also reseed the DRNG with the current time stamp */
+ entropy_buf->now = random_get_entropy();
+
+ /* allow external entropy provider to provide seed */
+ lrng_state_exseed_allow_all();
+
+wakeup:
+ /*
+ * Shall we wake up user space writers? This location covers
+ * ensures that the user space provider does not dominate the internal
+ * noise sources since in case the first call of this function finds
+ * sufficient entropy in the entropy pool, it will not trigger the
+ * wakeup. This implies that when the next /dev/urandom read happens,
+ * the entropy pool is drained.
+ */
+ lrng_writer_wakeup();
+
+ return total_entropy_bits;
+}
+
+/**
+ * Set the slow noise source reseed trigger threshold. The initial threshold
+ * is set to the minimum data size that can be read from the pool: a word. Upon
+ * reaching this value, the next seed threshold of 128 bits is set followed
+ * by 256 bits.
+ *
+ * @entropy_bits: size of entropy currently injected into DRNG
+ */
+void lrng_init_ops(u32 seed_bits)
+{
+ struct lrng_state *state = &lrng_state;
+
+ if (state->lrng_operational)
+ return;
+
+ /* DRNG is seeded with full security strength */
+ if (state->lrng_fully_seeded) {
+ state->lrng_operational = lrng_sp80090b_startup_complete();
+ lrng_process_ready_list();
+ lrng_init_wakeup();
+ } else if (seed_bits >= LRNG_FULL_SEED_ENTROPY_BITS) {
+ invalidate_batched_entropy();
+ state->lrng_fully_seeded = true;
+ state->lrng_operational = lrng_sp80090b_startup_complete();
+ state->lrng_min_seeded = true;
+ pr_info("LRNG fully seeded with %u bits of entropy\n",
+ seed_bits);
+ lrng_set_entropy_thresh(LRNG_FULL_SEED_ENTROPY_BITS +
+ LRNG_CONDITIONING_ENTROPY_LOSS);
+ lrng_process_ready_list();
+ lrng_init_wakeup();
+
+ } else if (!state->lrng_min_seeded) {
+
+ /* DRNG is seeded with at least 128 bits of entropy */
+ if (seed_bits >= LRNG_MIN_SEED_ENTROPY_BITS) {
+ invalidate_batched_entropy();
+ state->lrng_min_seeded = true;
+ pr_info("LRNG minimally seeded with %u bits of "
+ "entropy\n", seed_bits);
+ lrng_set_entropy_thresh(
+ lrng_slow_noise_req_entropy(
+ LRNG_FULL_SEED_ENTROPY_BITS +
+ LRNG_CONDITIONING_ENTROPY_LOSS));
+ lrng_process_ready_list();
+ lrng_init_wakeup();
+
+ /* DRNG is seeded with at least LRNG_INIT_ENTROPY_BITS bits */
+ } else if (seed_bits >= LRNG_INIT_ENTROPY_BITS) {
+ pr_info("LRNG initial entropy level %u bits of "
+ "entropy\n", seed_bits);
+ lrng_set_entropy_thresh(
+ lrng_slow_noise_req_entropy(
+ LRNG_MIN_SEED_ENTROPY_BITS +
+ LRNG_CONDITIONING_ENTROPY_LOSS));
+ }
+ }
+}
+
+int __init rand_initialize(void)
+{
+ ktime_t now_time = ktime_get_real();
+ unsigned int i, rand;
+
+ lrng_pool_lfsr_u32(now_time);
+ for (i = 0; i < LRNG_POOL_SIZE; i++) {
+ if (!arch_get_random_seed_int(&rand) &&
+ !arch_get_random_int(&rand))
+ rand = random_get_entropy();
+ lrng_pool_lfsr_u32(rand);
+ }
+ lrng_pool_lfsr_nonaligned((u8 *)utsname(), sizeof(*(utsname())));
+
+ return 0;
+}
diff --git a/drivers/char/lrng/lrng_sw_noise.c b/drivers/char/lrng/lrng_sw_noise.c
new file mode 100644
index 000000000000..d153371cb25b
--- /dev/null
+++ b/drivers/char/lrng/lrng_sw_noise.c
@@ -0,0 +1,102 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * LRNG Slow Noise Source: Interrupt data collection
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#include <asm/irq_regs.h>
+#include <asm/ptrace.h>
+#include <linux/random.h>
+
+#include "lrng_internal.h"
+#include "lrng_sw_noise.h"
+
+/* Holder of time stamps before mixing them into the entropy pool */
+static DEFINE_PER_CPU(u32 [LRNG_TIME_ARRAY_SIZE], lrng_time);
+static DEFINE_PER_CPU(u32, lrng_time_ptr) = 0;
+static DEFINE_PER_CPU(u8, lrng_time_irqs) = 0;
+
+/**
+ * Batching up of entropy in per-CPU array before injecting into entropy pool.
+ */
+static inline void lrng_time_process(void)
+{
+ u32 i, ptr, now_time = random_get_entropy() &
+ (likely(lrng_state_fully_seeded()) ?
+ LRNG_TIME_SLOTSIZE_MASK : (u32)-1);
+ enum lrng_health_res health_test;
+
+ /* Ensure sufficient space in lrng_time_irqs */
+ BUILD_BUG_ON(LRNG_TIME_NUM_VALUES >= (1 << (sizeof(u8) << 3)));
+ BUILD_BUG_ON(LRNG_TIME_ARRAY_MEMBER_BITS % LRNG_TIME_SLOTSIZE_BITS);
+ /* Ensure consistency of values */
+ BUILD_BUG_ON(LRNG_TIME_ARRAY_MEMBER_BITS != sizeof(lrng_time[0]) << 3);
+
+ if (lrng_raw_entropy_store(now_time))
+ return;
+
+ health_test = lrng_health_test(now_time);
+ if (health_test > lrng_health_fail_use)
+ return;
+
+ /* During boot time, we mix the full time stamp directly into LFSR */
+ if (unlikely(!lrng_state_fully_seeded())) {
+ lrng_pool_lfsr_u32(now_time);
+ if (health_test == lrng_health_pass)
+ lrng_pool_add_irq(1);
+ return;
+ }
+
+ ptr = this_cpu_inc_return(lrng_time_ptr) & LRNG_TIME_WORD_MASK;
+ this_cpu_or(lrng_time[lrng_time_idx2array(ptr)],
+ lrng_time_slot_val(now_time & LRNG_TIME_SLOTSIZE_MASK,
+ lrng_time_idx2slot(ptr)));
+
+ /* Interrupt delivers entropy if health test passes */
+ if (health_test == lrng_health_pass)
+ this_cpu_inc(lrng_time_irqs);
+
+ /* Only mix the buffer of time stamps into LFSR when wrapping */
+ if (ptr < LRNG_TIME_WORD_MASK)
+ return;
+
+ for (i = 0; i < LRNG_TIME_ARRAY_SIZE; i++) {
+ lrng_pool_lfsr_u32(this_cpu_read(lrng_time[i]));
+ this_cpu_write(lrng_time[i], 0);
+ }
+ lrng_pool_add_irq(this_cpu_read(lrng_time_irqs));
+ this_cpu_write(lrng_time_irqs, 0);
+}
+
+/**
+ * Hot code path - Callback for interrupt handler
+ */
+void add_interrupt_randomness(int irq, int irq_flags)
+{
+ lrng_time_process();
+
+ if (!lrng_pool_highres_timer()) {
+ struct pt_regs *regs = get_irq_regs();
+ static atomic_t reg_idx = ATOMIC_INIT(0);
+ u64 ip;
+
+ lrng_pool_lfsr_u32(jiffies);
+ lrng_pool_lfsr_u32(irq);
+ lrng_pool_lfsr_u32(irq_flags);
+
+ if (regs) {
+ u32 *ptr = (u32 *)regs;
+ int reg_ptr = atomic_add_return_relaxed(1, ®_idx);
+ size_t n = (sizeof(struct pt_regs) / sizeof(u32));
+
+ ip = instruction_pointer(regs);
+ lrng_pool_lfsr_u32(*(ptr + (reg_ptr % n)));
+ } else
+ ip = _RET_IP_;
+
+ lrng_pool_lfsr_u32(ip >> 32);
+ lrng_pool_lfsr_u32(ip);
+ }
+}
+EXPORT_SYMBOL(add_interrupt_randomness);
diff --git a/drivers/char/lrng/lrng_sw_noise.h b/drivers/char/lrng/lrng_sw_noise.h
new file mode 100644
index 000000000000..107c6f9153bc
--- /dev/null
+++ b/drivers/char/lrng/lrng_sw_noise.h
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * LRNG Slow Noise Source: Time stamp array handling
+ *
+ * Copyright (C) 2016 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+/*
+ * To limit the impact on the interrupt handling, the LRNG concatenates
+ * entropic LSB parts of the time stamps in a per-CPU array and only
+ * injects them into the entropy pool when the array is full.
+ */
+
+/* Store multiple integers in one u32 */
+#define LRNG_TIME_SLOTSIZE_BITS (8)
+#define LRNG_TIME_SLOTSIZE_MASK ((1 << LRNG_TIME_SLOTSIZE_BITS) - 1)
+#define LRNG_TIME_ARRAY_MEMBER_BITS (4 << 3)
+#define LRNG_TIME_SLOTS_PER_UINT (LRNG_TIME_ARRAY_MEMBER_BITS / \
+ LRNG_TIME_SLOTSIZE_BITS)
+
+/*
+ * Number of time values to store in the array - in small environments
+ * only one atomic_t variable per CPU is used.
+ */
+#define LRNG_TIME_NUM_VALUES (CONFIG_BASE_SMALL ? \
+ LRNG_TIME_SLOTS_PER_UINT : 64)
+/* Mask of LSB of time stamp to store */
+#define LRNG_TIME_WORD_MASK (LRNG_TIME_NUM_VALUES - 1)
+
+#define LRNG_TIME_SLOTS_MASK (LRNG_TIME_SLOTS_PER_UINT - 1)
+#define LRNG_TIME_ARRAY_SIZE (LRNG_TIME_NUM_VALUES / \
+ LRNG_TIME_SLOTS_PER_UINT)
+
+/* Starting bit index of slot */
+static inline unsigned int lrng_time_slot2bitindex(unsigned int slot)
+{
+ return (LRNG_TIME_SLOTSIZE_BITS * slot);
+}
+
+/* Convert index into the array index */
+static inline unsigned int lrng_time_idx2array(unsigned int idx)
+{
+ return idx / LRNG_TIME_SLOTS_PER_UINT;
+}
+
+/* Convert index into the slot of a given array index */
+static inline unsigned int lrng_time_idx2slot(unsigned int idx)
+{
+ return idx & LRNG_TIME_SLOTS_MASK;
+}
+
+/* Convert value into slot value */
+static inline unsigned int lrng_time_slot_val(unsigned int val,
+ unsigned int slot)
+{
+ return val << lrng_time_slot2bitindex(slot);
+}
diff --git a/include/linux/lrng.h b/include/linux/lrng.h
new file mode 100644
index 000000000000..72abb575eb72
--- /dev/null
+++ b/include/linux/lrng.h
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright (C) 2018 - 2020, Stephan Mueller <smueller@chronox.de>
+ */
+
+#ifndef _LRNG_H
+#define _LRNG_H
+
+#include <linux/errno.h>
+#include <linux/types.h>
+
+/**
+ * struct lrng_crypto_cb - cryptographic callback functions
+ * @lrng_drng_name Name of DRNG
+ * @lrng_hash_name Name of Hash used for reading entropy pool
+ * @lrng_drng_alloc: Allocate DRNG -- the provided integer should be
+ * used for sanity checks.
+ * return: allocated data structure or PTR_ERR on
+ * error
+ * @lrng_drng_dealloc: Deallocate DRNG
+ * @lrng_drng_seed_helper: Seed the DRNG with data of arbitrary length
+ * drng: is pointer to data structure allocated
+ * with lrng_drng_alloc
+ * return: >= 0 on success, < 0 on error
+ * @lrng_drng_generate_helper: Generate random numbers from the DRNG with
+ * arbitrary length
+ * @lrng_drng_generate_helper_full: Generate random numbers from the DRNG with
+ * arbitrary length where the output is
+ * capable of providing 1 bit of entropy per
+ * data bit.
+ * return: generated number of bytes,
+ * < 0 on error
+ * @lrng_hash_alloc: Allocate the hash for reading the entropy pool
+ * return: allocated data structure (NULL is
+ * success too) or ERR_PTR on error
+ * @lrng_hash_dealloc: Deallocate Hash
+ * @lrng_hash_digestsize: Return the digestsize for the used hash to read
+ * out entropy pool
+ * hash: is pointer to data structure allocated
+ * with lrng_hash_alloc
+ * return: size of digest of hash in bytes
+ * @lrng_hash_buffer: Generate hash
+ * hash: is pointer to data structure allocated
+ * with lrng_hash_alloc
+ * return: 0 on success, < 0 on error
+ */
+struct lrng_crypto_cb {
+ const char *(*lrng_drng_name)(void);
+ const char *(*lrng_hash_name)(void);
+ void *(*lrng_drng_alloc)(u32 sec_strength);
+ void (*lrng_drng_dealloc)(void *drng);
+ int (*lrng_drng_seed_helper)(void *drng, const u8 *inbuf, u32 inbuflen);
+ int (*lrng_drng_generate_helper)(void *drng, u8 *outbuf, u32 outbuflen);
+ int (*lrng_drng_generate_helper_full)(void *drng, u8 *outbuf,
+ u32 outbuflen);
+ void *(*lrng_hash_alloc)(const u8 *key, u32 keylen);
+ void (*lrng_hash_dealloc)(void *hash);
+ u32 (*lrng_hash_digestsize)(void *hash);
+ int (*lrng_hash_buffer)(void *hash, const u8 *inbuf, u32 inbuflen,
+ u8 *digest);
+};
+
+/* Register cryptographic backend */
+#ifdef CONFIG_LRNG_DRNG_SWITCH
+int lrng_set_drng_cb(const struct lrng_crypto_cb *cb);
+#else /* CONFIG_LRNG_DRNG_SWITCH */
+static inline int
+lrng_set_drng_cb(const struct lrng_crypto_cb *cb) { return -EOPNOTSUPP; }
+#endif /* CONFIG_LRNG_DRNG_SWITCH */
+
+#endif /* _LRNG_H */
--
2.24.1
^ permalink raw reply related
* [PATCH v27 00/12] /dev/random - a new approach with full SP800-90B
From: Stephan Müller @ 2020-01-09 8:29 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Greg Kroah-Hartman, linux-crypto, LKML, linux-api,
Eric W. Biederman, Alexander E. Patrakov, Ahmed S. Darwish,
Theodore Y. Ts'o, Willy Tarreau, Matthew Garrett, Vito Caputo,
Andreas Dilger, Jan Kara, Ray Strode, William Jon McCann, zhangjs,
Andy Lutomirski, Florian Weimer, Lennart Poettering,
Nicolai Stange, "Peter, Matthias" <matth>
In-Reply-To: <2722222.P16TYeLAVu@positron.chronox.de>
Hi,
The following patch set provides a different approach to /dev/random which is
called Linux Random Number Generator (LRNG) to collect entropy within the Linux
kernel. The main improvements compared to the existing /dev/random is to provide
sufficient entropy during boot time as well as in virtual environments and when
using SSDs. A secondary design goal is to limit the impact of the entropy
collection on massive parallel systems and also allow the use accelerated
cryptographic primitives. Also, all steps of the entropic data processing are
testable.
The LRNG patch set allows a user to select use of the existing /dev/random or
the LRNG during compile time. As the LRNG provides API and ABI compatible
interfaces to the existing /dev/random implementation, the user can freely chose
the RNG implementation without affecting kernel or user space operations.
This patch set provides early boot-time entropy which implies that no
additional flags to the getrandom(2) system call discussed recently on
the LKML is considered to be necessary. Yet, if additional flags are
introduced to cover special hardware, the LRNG implementation will also
provide them to be fully ABI and API compliant as already discussed on
LKML.
The LRNG is fully compliant to SP800-90B requirements and is shipped with a
full SP800-90B assessment and all required test tools. The existing /dev/random
implementation on the other hand has architectural limitations which
does not easily allow to bring the implementation in compliance with
SP800-90B. The key statement that causes concern is SP800-90B section
3.1.6. This section denies crediting entropy to multiple similar noise
sources. This section explicitly references different noise sources resting
on the timing of events and their derivatives (i.e. it is a direct complaint
to the existing existing /dev/random implementation). Therefore, SP800-90B
now denies the very issue mentioned in [1] with the existing /dev/random
implementation for a long time: crediting entropy to interrupts as well as
crediting entropy to derivatives of interrupts (HID and disk events). This is
not permissible with SP800-90B.
SP800-90B specifies various requirements for the noise source(s) that seed any
DRNG including SP800-90A DRBGs. In about a year from now, SP800-90B will be
mandated for all noise sources that provide entropy to DRBGs as part of a FIPS
140-[2|3] validation or other evaluation types. That means, if we there are no
solutions to comply with the requirements of SP800-90B found till one year
from now, any random number generation and ciphers based on random numbers
on Linux will be considered and treated as not applicable and delivering
no entropy! As /dev/urandom, getrandom(2) and /dev/random are the most
common and prevalent noise sources for DRNGs, all these DRNGs are affected.
This applies across the board for all validations of cryptography executing on
Linux (kernel and user space modules).
For users that are not interested in SP800-90B, the entire code for the
compliance as well as test interfaces can be deselected at compile time.
The design and implementation is driven by a set of goals described in [1]
that the LRNG completely implements. Furthermore, [1] includes the full
assessment of the SP800-90B compliance as well as a comparison with RNG
design suggestions of SP800-90C, and AIS20/31.
The LRNG provides a complete separation of the noise source maintenance
and the collection of entropy into an entropy pool from the post-processing
using a pseudo-random number generator. Different DRNGs are supported,
including:
* The LRNG can be compile-time enabled to replace the existing /dev/random
implementation. When not selecting the LRNG at compile time (default), the
existing /dev/random implementation is built.
* Built-in ChaCha20 DRNG which has no dependency to other kernel
frameworks.
* SP800-90A DRBG using the kernel crypto API including its accelerated
raw cipher implementations. This implies that the output of /dev/random,
getrandom(2), /dev/urandom or get_random_bytes is fully compliant to
SP800-90A.
* Arbitrary DRNGs registered with the kernel crypto API
* Full compliance with SP800-90B which covers the startup and runtime health
tests mandated by SP800-90B as well as providing the test tools and test
interfaces to obtain raw noise data securely. The test tools are provided at
[1].
Booting the patch with the kernel command line option
"dyndbg=file drivers/char/lrng/* +p" generates logs indicating the operation
of the LRNG. Each log is pre-pended with "lrng".
The LRNG has a flexible design by allowing an easy replacement of the
deterministic random number generator component.
Compared to the existing /dev/random implementation, the compiled binary
is smaller when the LRNG is compiled with all options equal to the
existing /dev/random (i.e. only CONFIG_LRNG is set): random.o is 52.5 kBytes
whereas all LRNG object files are in 49 kBytes in size. The fully
SP800-90A/SP800-90B compliant binary code (CONFIG_LRNG,
CONFIG_LRNG_DRNG_SWITCH, CONFIG_LRNG_DRBG, CONFIG_LRNG_HEALTH_TESTS)
uses some 61 kBytes. In addition, the LRNG is about 50% faster in the
performance critical interrupt handler code path compared to the existing
/dev/random implementation.
[1] http://www.chronox.de/lrng.html - If the patch is accepted, I would
be volunteering to convert the documentation into RST format and
contribute it to the Linux kernel documentation directory.
Changes (compared to the previous patch set):
* fix function prototype of lrng_init_time_source
* fix indentation in getrandom
* add unlock to error code path as reported by Julia Lawall
* integrate 0da522107e5d9c000a4871d52e570912aa1225a2 from Arnd Bergmann and
supplemental patch 4aa37c463764052c68c5c430af2a67b5d784c1e0 from
Jason A. Donenfeld
* use new jitterentropy.h header file
* add power-on self-tests of security critical functions of hash_df, LFSR,
ChaCha20 DRNG, and time stamp array management
* add support for significantly reduced runtime memory footprint as outlined
in patch 1
* drop TRNG support - the LRNG has the same user experience as random.c
with Andy Lutomirski's recent patch removing the blocking_pool - rename
all *sdrng* symbols to *drng*
* ensure that external noise sources can provide seed at least once when a
DRNG is reseeded to ensure internal and external noise sources are balanced
* add full documentation to all API calls provided to the remainder of the
kernel
* ensure that after a SP800-90B health test failure the interrupt handler
triggers reseeds like during boot time
This patch requires the presence of patch
75551dbf112c992bc6c99a972990b3f272247e23 from Ted Tso's kernel tree
(specifically the addition of GRND_INSECURE to random.h)
As a side node: With the switchable DRNG support offered in this patch set,
the following areas could be removed. As the existing /dev/random has no support
for switchable DRNGs, however, this is not yet feasible though.
* remove lrng_ready_list and all code around it in lrng_interfaces.c
* remove the kernel crypto API RNG API to avoid having two random number
providing APIs - this would imply that all RNGs developed for this API would
be converted to the LRNG interface
CC: "Eric W. Biederman" <ebiederm@xmission.com>
CC: "Alexander E. Patrakov" <patrakov@gmail.com>
CC: "Ahmed S. Darwish" <darwish.07@gmail.com>
CC: "Theodore Y. Ts'o" <tytso@mit.edu>
CC: Willy Tarreau <w@1wt.eu>
CC: Matthew Garrett <mjg59@srcf.ucam.org>
CC: Vito Caputo <vcaputo@pengaru.com>
CC: Andreas Dilger <adilger.kernel@dilger.ca>
CC: Jan Kara <jack@suse.cz>
CC: Ray Strode <rstrode@redhat.com>
CC: William Jon McCann <mccann@jhu.edu>
CC: zhangjs <zachary@baishancloud.com>
CC: Andy Lutomirski <luto@kernel.org>
CC: Florian Weimer <fweimer@redhat.com>
CC: Lennart Poettering <mzxreary@0pointer.de>
CC: Nicolai Stange <nstange@suse.de>
Tested-by: Roman Drahtmüller <draht@schaltsekun.de>
Tested-by: Marcelo Henrique Cerri <marcelo.cerri@canonical.com>
Stephan Mueller (12):
Linux Random Number Generator
LRNG - allocate one DRNG instance per NUMA node
LRNG - sysctls and /proc interface
LRNG - add switchable DRNG support
crypto: DRBG - externalize DRBG functions for LRNG
LRNG - add SP800-90A DRBG extension
LRNG - add kernel crypto API PRNG extension
crypto: provide access to a static Jitter RNG state
LRNG - add Jitter RNG fast noise source
LRNG - add SP800-90B compliant health tests
LRNG - add interface for gathering of raw entropy
LRNG - add power-on and runtime self-tests
MAINTAINERS | 7 +
crypto/drbg.c | 16 +-
crypto/jitterentropy-kcapi.c | 3 +-
crypto/jitterentropy.c | 25 +-
drivers/char/Kconfig | 2 +
drivers/char/Makefile | 9 +-
drivers/char/lrng/Kconfig | 203 ++++++
drivers/char/lrng/Makefile | 19 +
drivers/char/lrng/lrng_archrandom.c | 92 +++
drivers/char/lrng/lrng_aux.c | 148 +++++
drivers/char/lrng/lrng_chacha20.c | 317 +++++++++
drivers/char/lrng/lrng_chacha20.h | 25 +
drivers/char/lrng/lrng_drbg.c | 261 ++++++++
drivers/char/lrng/lrng_drng.c | 397 +++++++++++
drivers/char/lrng/lrng_health.c | 409 ++++++++++++
drivers/char/lrng/lrng_interfaces.c | 622 ++++++++++++++++++
drivers/char/lrng/lrng_internal.h | 305 +++++++++
drivers/char/lrng/lrng_jent.c | 87 +++
drivers/char/lrng/lrng_kcapi.c | 328 +++++++++
drivers/char/lrng/lrng_lfsr.h | 152 +++++
drivers/char/lrng/lrng_numa.c | 101 +++
drivers/char/lrng/lrng_pool.c | 585 ++++++++++++++++
drivers/char/lrng/lrng_proc.c | 163 +++++
drivers/char/lrng/lrng_selftest.c | 418 ++++++++++++
drivers/char/lrng/lrng_sw_noise.c | 102 +++
drivers/char/lrng/lrng_sw_noise.h | 57 ++
drivers/char/lrng/lrng_switch.c | 179 +++++
drivers/char/lrng/lrng_testing.c | 271 ++++++++
include/crypto/drbg.h | 7 +
.../crypto/internal}/jitterentropy.h | 3 +
include/linux/lrng.h | 71 ++
31 files changed, 5374 insertions(+), 10 deletions(-)
create mode 100644 drivers/char/lrng/Kconfig
create mode 100644 drivers/char/lrng/Makefile
create mode 100644 drivers/char/lrng/lrng_archrandom.c
create mode 100644 drivers/char/lrng/lrng_aux.c
create mode 100644 drivers/char/lrng/lrng_chacha20.c
create mode 100644 drivers/char/lrng/lrng_chacha20.h
create mode 100644 drivers/char/lrng/lrng_drbg.c
create mode 100644 drivers/char/lrng/lrng_drng.c
create mode 100644 drivers/char/lrng/lrng_health.c
create mode 100644 drivers/char/lrng/lrng_interfaces.c
create mode 100644 drivers/char/lrng/lrng_internal.h
create mode 100644 drivers/char/lrng/lrng_jent.c
create mode 100644 drivers/char/lrng/lrng_kcapi.c
create mode 100644 drivers/char/lrng/lrng_lfsr.h
create mode 100644 drivers/char/lrng/lrng_numa.c
create mode 100644 drivers/char/lrng/lrng_pool.c
create mode 100644 drivers/char/lrng/lrng_proc.c
create mode 100644 drivers/char/lrng/lrng_selftest.c
create mode 100644 drivers/char/lrng/lrng_sw_noise.c
create mode 100644 drivers/char/lrng/lrng_sw_noise.h
create mode 100644 drivers/char/lrng/lrng_switch.c
create mode 100644 drivers/char/lrng/lrng_testing.c
rename {crypto => include/crypto/internal}/jitterentropy.h (84%)
create mode 100644 include/linux/lrng.h
--
2.24.1
^ permalink raw reply
* Re: [PATCH RFC 0/1] mount: universally disallow mounting over symlinks
From: Al Viro @ 2020-01-08 21:34 UTC (permalink / raw)
To: Linus Torvalds
Cc: Aleksa Sarai, David Howells, Eric Biederman, stable,
Christian Brauner, Serge Hallyn, dev, Linux Containers, Linux API,
linux-fsdevel, Linux Kernel Mailing List, Ian Kent
In-Reply-To: <CAHk-=wgQ3yOBuK8mxpnntD8cfX-+10ba81f86BYg8MhvwpvOMg@mail.gmail.com>
On Tue, Jan 07, 2020 at 07:54:02PM -0800, Linus Torvalds wrote:
> > Another interesting question is whether we want O_PATH open
> > to trigger automounts.
>
> It does sound like they shouldn't, but as you say:
>
> > The thing is, we do *NOT* trigger them
> > (or traverse mountpoints) at the starting point of lookups.
> > I believe it's a mistake (and mine, at that), but I doubt that
> > there's anything that can be done about it at that point.
> > It's a user-visible behaviour [..]
>
> Hmm. I wonder how set in stone that is. We may have two decades of
> history of not doing it at start point of lookups, but we do *not*
> have two decades of history of O_PATH.
>
> So what I think we agree would be sane behavior would be for O_PATH
> opens to not trigger automounts (unless there's a slash at the end,
> whatever), but _do_ add the mount-point traversal to the beginning of
> lookups.
>
> But only do it for the actual O_PATH fd case, not the cwd/root/non-O_PATH case.
>
> That way we maintain original behavior: if somebody overmounts your
> cwd, you still see the pre-mount directory on lookups, because your
> cwd is "under" the mount.
>
> But if you open a file with O_PATH, and somebody does a mount
> _afterwards_, the openat() will see that later mount and/or do the
> automount.
>
> Don't you think that would be the more sane/obvious semantics of how
> O_PATH should work?
Maybe, but... note that we do not (and AFAICS never had) follow mounts
on /proc/self/cwd, /proc/self/fd/42, etc. And there are very good
reasons for that. First of all, if your stdin is from /tmp/foo,
you'd better get that file when you open /dev/stdin, even if somebody
has done mount --bind /tmp/bar /tmp/foo; another issue is with
the use of stat("/proc/self/fd/42", &buf) - it should be an equivalent
of fstat(42, &buf), even if somebody has overmounted that. BTW, for
similar reason after
link(".", "foo");
fd = open("foo", O_PATH); // return 42
we really should (and do) have resolution of /proc/self/fd/42 stop at
foo, not . Reason: consistency of stat() behaviour...
The point is, we'd never followed mounts on /proc/self/cwd et.al.
I hadn't checked 2.0, but 2.1.100 ('97, before any changes from me)
is that way. Actually, scratch that - 2.0 behaves the same way
(mountpoint crossing is done in iget() there; is that Minix influence
or straight from the Lions' book?)
Hmm... Looking through the history, we have
(for reference) v7: mount traversal in iget()
(forward) and namei() (back); due to the way it's done, forward
traversal happens
* at starting point
* after any component (. and .. included)
* on results of forward traversal (due to a loop in iget()).
Back traversal (to covered on .. from root directory) is also
to unlimited depth.
0.01: no mount handling
0.10: forward traversal in iget(), back traversal in fs/namei.c:find_entry()
(not by Lions' Book, then - v6 didn't do back traversals at all).
Forward traversal
* after any component (. and .. included)
No traversal on starting point, no traversal on result of traversal.
OTOH, mount(2) refuses to mount on top of root, so the lack of the last
one is not an issue.
0.12: symlinks added; no mount traversal on starting point of those either.
We start at the process' root for absolute ones, even if it happens to be
overmounted, and we start from parent for relative ones. The latter matters
only if we were in the beginning of the pathwalk, since anything else would've
traversed mounts back when we'd picked said parent. Mount traversal takes
precedence over symlink traversal, but that's not an issue since mount follows
links on mountpoint. It does not, at that point, reject fs image with
symlink for root, but that actually more or less works.
0.97.3: same, with addition of procfs symlinks. No mount crossing on their
targets (for normal symlinks we don't do mount crossing in the beginning
and any component inside triggers mount crossing as usual; for procfs ones
there's no components inside)
Situation remains essentially unchanged until 2.1.42. Next few kernels
are in flux, to put it politely - initial merge had been insane and it
took until 2.1.44 or so for the things to get more or less working.
At 2.1.44: forward traversal in fs/namei.c:lookup(), back traversal in
fs/namei.c:reserved_lookup(). Otherwise the same behaviour as pre-dcache
(wrt mount traversals, that is).
2.1.51pre1: forward traversal moved into real_lookup() and __d_lookup().
Forward traversal happens *ONLY* after normal components - not after . or ..
2.1.61: forward traversal moved into follow_mount(), behaviour reverted to
pre-dcache one.
Previous is from reading through the historical trees; my involvement started
circa 2.1.120-something.
2.3.50pre3: call of follow_mount() moved a bit, reverting to 2.1.51pre1
behaviour (nor traversal on . or ..) *again*. Not sure whose idea had that
been - might've been mine, but unlike the other patch that went into fs/namei.c
in the same release, I hadn't been able to find anything related to that
one. If your memories (or mail archives) are better...
2.3.99pre4-5: massive surgery in there. Preparations to allowing mount on top
of mount; forward traversal adjusted accordingly, back traversal still isn't.
2.3.99pre7-1: more surgery, back traversals are also to unlimited depth now
and mount on top of mount has been allowed.
2.3.99pre9-4: mount --bind taught to mount non-directories on top of
non-directories. At that point it does *NOT* follow trailing symlinks, so
mounting of symlinks and mounting on top of symlinks becomes possible.
Mount traversal still takes precedence over symlink traversal, symlink traversal
of mount traversal result still generally works, even though it's not something
I considered at the time.
v2.4.5.2: mount --bind started to follow symlinks. So that source of mounting
of and on the symlinks was no more.
2.5.0.5: forward mount traversal is done after .. (in handle_dotdot()).
That brings back the pre-dcache behaviour for those suckers. Still no
forward traversal after ., though.
At about the same time I'd been getting rid of the early-boot incestous
relationships with fs/namespace.c (initramfs work) and that was probably
the last time we could realistically switch to following mounts at starting
point; I considered trying to do that, but decided not to. Pity, that...
2.6.5-rc2: normal mount now checks for corrupt fs with symlink for root.
Since it has always been following symlinks for mountpoint, the remaining
source of mounting of and on symlinks was gone; that lasted until
after O_PATH introduction.
2.6.39-rc1: mount traps support - instead of abusing ->follow_link()
for automounting, we have an explicit pair of methods that can be
called at the same places where we traverse mounts. None too consistent -
we don't do that on .. results. That was Dave Howells and Ian Kent.
2.6.39-rc1: O_PATH introduced and, later in the same series, allowed for
symlinks. That has changed things - now procfs symlink targets could
be symlinks themselves. Originally an attempt to follow those would
blow up with -ELOOP (there's simply no good way to follow such beast;
it's either "stop even if we are asked to follow" or "give an error").
3.6.0-rc1: nd_jump_link() introduction (hch) had unnoticed side effects -
we'd switched from "fail traversal with -ELOOP" to "stop there". Mostly it
doesn't change behaviour, but it has opened a way to mount symlinks and
mount on top of symlinks. Which generally worked.
circa 3.8--3.9: side effects had been noticed; my first reaction had been
"let's make nd_jump_link() return an error, then", but I hadn't been
able to find good reasons when challenged to do so. Did an audit,
found no obvious problems, went "oh, well - whether it works by accident
or by design, it doesn't break anything".
3.12.0-rc1: lookups for umount(2) are different - we don't want
revalidate on the last component. Which had been handled by
introduction of path_umountat()/umount_lookup_last(), parallel to
path_lookupat(). Which has gotten quite a few things wrong -
it *did* try to follow symlinks obtained by following procfs
ones (and blew up big way) and it didn't follow mounts on
overmounted trailing symlinks. Nobody noticed for 6 years,
until folks actually tried to play with mount-on-symlink...
Patches were by Jeff Layton, neither he nor I have spotted the
problem back then. And I should have, since it had been only
a few months since the audit for exactly that kind of problems...
AFAICS, there'd been no serious semantical changes since then. What we
have right now:
* no mount traversal on the starting point
* mount traversal after any component other than "."
* symlink traversal consists of possibly jumping to given
point plus following a given (possibly empty) series of components.
It can be both - e.g. symlink to "/foo/bar" is 'jump to root,
then traverse "foo", then traverse "bar"'. Procfs "magic" symlinks
are not really magical - they behave as symlinks to "/" as far as
the pathwalk semantics is concerned. The only differences is that
jump might be not to process' root.
* mount traversal takes precedence over symlink traversal.
* jump (if any) in symlink traversal is treated the same
as the starting point - it's not followed by mount traversal.
It's also not followed by symlink traversal, even if we are jumping
into a symlink. Of course, in any position other than the end of
pathname that's an instant error. That's also not different from
the starting point treatment - if ...at(2) is given a symlink for
starting point, it leaves it as-is if AT_EMPTY_PATH is given and
fails with -ENOTDIR otherwise.
* umount(2) handles the final component differently -
for one thing, it does not do revalidate, for another - its
mount traversal (if any) does not include automount-related
parts. And there we *do* want mount traversal at the final
point, for obvious reasons.
> > I think the easiest way to handle that is to have O_PATH
> > turn LOOKUP_AUTOMOUNT, same as the normal open() does. That's
> > trivial to do, but that changes user-visible behaviour. OTOH,
> > with the current behaviour nobody can rely upon automount not
> > getting triggered by somebody else just as they are entering
> > their open(dir, O_PATH), so I think that's not a problem.
> >
> > Linus, do you have any objections to such O_PATH semantics
> > change?
>
> See above: I think I'd prefer the O_PATH behavior the other way
> around. That seems to be more of a consistent behavior of what
> "O_PATH" means - it means "don't really open, we'll do it only when
> you use it as a directory".
How would your proposal deal with access("/proc/self/fd/42/foo", MAY_READ)
vs. faccessat(42, "foo", MAY_READ)? The latter would trigger automount,
the former would not... Or would you extend that to "traverse mounts
upon following procfs links, if the file in question had been opened with
O_PATH"? We could do that (give nd_jump_link() an extra argument telling
if we want mount traversal), but I'm not sure if the resulting semantics
is sane...
Note, BTW, that O_PATH users really can't rely upon automounts _not_
being triggered - all it takes is a lookup on bogus path with such prefix
by anybody who can reach that place... We are not opening anything,
really, but we are not able to ignore automounts triggered by somebody
else.
^ permalink raw reply
* Re: [RESEND RFC PATCH 1/1] Selectively allow CAP_SYS_NICE capability inside user namespaces
From: prakash.sangappa @ 2020-01-08 21:23 UTC (permalink / raw)
To: Eric W. Biederman; +Cc: linux-kernel, linux-api, tglx, peterz, serge
In-Reply-To: <0d7fb84d-e7e8-c442-37a3-23b036fdf12c@oracle.com>
On 11/21/2019 05:45 PM, Prakash Sangappa wrote:
>
>
> On 11/21/19 1:27 PM, ebiederm@xmission.com wrote:
>> Prakash Sangappa <prakash.sangappa@oracle.com> writes:
<..>
>> 2) If I read the other thread correctly there was talk about setting the
>> nice levels of processes in other containers. Ouch!
>
> No not in other containers. Only on processes within the container
> which has this capability. The use case is to use it in a container
> with user namespace and pid namespace. So no processes from other
> containers should be visible. Necessary checks should be added?.
>
>
>>
>> The only thing I can think that makes any sense at all is to allow
>> setting the nice levels of the processes in your own container.
>
> Yes that is the intended use.
>
>>
>> I can totally see having a test to see if a processes credentials
>> are
>> in the caller's user namespace or a child of caller's user namespace
>> and allowing admin level access if the caller has the appropriate
>> caps in their user namespace.
>
> Ok
>
>> But in this case I don't see anything preventing the admin in a
>> container from using the ordinary nice levels on a task. You are
>> unlocking the nice levels reserved for the system administrator
>> for special occassions. I don't see how that makes any sense
>> to do from inside a container.
>
> But this is what seems to be lacking. A container could have some
> critical processes running which need to run at a higher priority.
Any comments about this? What would be the recommendation for dealing
with such a requirement?
^ permalink raw reply
* Re: [PATCH v2 0/6] introduce configfd as generalisation of fsconfig
From: James Bottomley @ 2020-01-08 18:42 UTC (permalink / raw)
To: Christian Brauner
Cc: David Howells, Al Viro, linux-fsdevel, Christian Brauner,
Miklos Szeredi, linux-kernel, linux-api
In-Reply-To: <20200108170703.zhcuohzdp22y5yma@wittgenstein>
On Wed, 2020-01-08 at 18:07 +0100, Christian Brauner wrote:
> [extending the Cc a bit]
>
> On Sun, Jan 05, 2020 at 10:02:08AM -0800, James Bottomley wrote:
> > On Sun, 2020-01-05 at 17:23 +0100, Christian Brauner wrote:
> > > On Sat, Jan 04, 2020 at 12:14:26PM -0800, James Bottomley wrote:
> > > > fsconfig is a very powerful configuration mechanism except that
> > > > it only works for filesystems with superblocks. This patch
> > > > series generalises the useful concept of a multiple step
> > > > configurationalmechanism carried by a file descriptor. The
> > > > object of this patch series is to get bind mounts to be
> > > > configurable in the same way that superblock based ones are,
> > > > but it should have utility beyond the filesytem realm. Patch 4
> > > > also reimplements fsconfig in terms of configfd, but that's not
> > > > a strictly necessary patch, it is merely a useful demonstration
> > > > that configfd is a superset of the properties of fsconfig.
> > >
> > > Thanks for the patch. I'm glad fsconfig() is picked back up;
> > > either by you or by David. We will need this for sure.
> > > But the configfd approach does not strike me as a great idea.
> > > Anonymous inode fds provide an abstraction mechanism for kernel
> > > objects which we built around fds such as timerfd, pidfd, mountfd
> > > and so on. When you stat an anonfd you get ANON_INODE_FS_MAGIC
> > > and you get the actual type by looking at fdinfo, or - more
> > > common - by parsing out /proc/<pid>/fd/<nr> and discovering
> > > "[fscontext]". So it's already a pretty massive abstraction layer
> > > we have. But configfd would be yet another fd abstraction based
> > > on anonfds. The idea has been that a new fd type based on
> > > anonfds comes with an api specific to that type of fd. That seems
> > > way nicer from an api design perspective than implementing new
> > > apis as part of yet another generic configfd layer.
> >
> > Really, it's just a fd that gathers config information and can
> > reserve specific errors (and we should really work out the i18n
> > implications of
>
> It's rather a complex multiplexer intended to go beyond the realm of
> filesystems/mount api and that's something we have been burned by
> before.
>
> > the latter). Whether it's a new fd type or an anonfd with a
> > specific name doesn't seem to be that significant, so the name
> > could be set by the type.
> >
> > > Another problem is that these syscalls here would be massive
> > > multiplexing syscalls. If they are ever going to be used outside
> > > of filesystem use-cases (which is doubtful) they will quickly
> > > rival prctl(), seccomp(), and ptrace().
> >
> > Actually, that's partly the point. We do have several systemcalls
> > with
>
> Actually I think that's the problem. The keyctl api itself suffers
> from the problem that it already has a complex multiplexer. That
> could either point to bad api design (sorry, David :)) or it's just a
> very complex use-case like the mount api.
I do really think it's a fairly pattern exact use case. The mount API
has a complex per-fs-type configuration but simple use via the VFS
abstractions, which is why we use a single mount system call instead of
one system call per fs. The keyctl API exactly mirrors this: creating
keys is complex and highly type dependent but once they're created
they're more generically usable. This pattern replicates itself across
a lot of subsystems. Storage being an excellent example. Once we have
the things configured, we can hide most of the configurational
complexity behind a simple use API: the block abstraction.
And the point that annoys me as someone who has to interact with and
maintain a few of these systems is that we keep reinventing similar but
slightly different solutions for them all.
> The good thing is that it's restricted to a single domain: keys. And
> that's good. Plumbing both e.g. keys and (parts of) mounts on top of
> another generic api is what strikes me as a bad idea.
Actually, I don't think this holds up to examination given the fact
that the pattern isn't specific to mount. You can take the view that
the configfd approach is wrong ... but that would mean it's wrong for
the mount use case as well. I have actually seen people advocate a
per-config type system call approach, but that always sank in the
plumbing complexity. You could take the view that each problem is a
separate domain and needs a separate solution, which is essentially
what we do today but that solution then tends to leak (using netlink in
SCSI for instance). We've also tried the configuration as filesystem
(that's the configfs one that iSCSI and USB gadget does) as an approach
to this.
I think the broader question is given the pattern is replicated across
subsystems, can we get a solution that works for all the patterns
instead of our current patchwork. fsconfig has certain features that
make it an interesting solution. It also has the problem that the way
its currently constructed means it doesn't apply to a part of its
design coverage (bind mounts). Configfd was an attempt to extract the
generic part and apply it (initially just on the bit that the coverage
was missing for).
I don't disagree that configuration multiplexors are a user space
annoyance, but we put up with them because we get a simple and very
generic API for the configured object. Given that they're a necessary
evil and a widespread pattern, I think examining the question of
whether we could cover them all with a single API and what properties
it should have is a useful one.
For instance, I think the cardinal annoying missing property in all our
above attempts is the lack of introspectability. The inability to
enumerate what types and interfaces are available before you start
configuring. Even configfs, which is our most introspectable one, has
that problem.
So given that we have to have things like this and they do get spread
widely, what are the desirable properties to make them more manageable?
The reason I liked fsconfig initially was error return: complex
configuration means complex errors and very few of our other solutions
have the ability to send them back. The other was the observation that
even though fsconfig isn't introspectable, it could be made so.
James
^ permalink raw reply
* Re: [PATCH v2 2/3] clone3: allow spawning processes into cgroups
From: Christian Brauner @ 2020-01-08 18:10 UTC (permalink / raw)
To: Michal Koutný
Cc: linux-api, linux-kernel, Tejun Heo, Ingo Molnar, Oleg Nesterov,
Johannes Weiner, Li Zefan, Peter Zijlstra, cgroups
In-Reply-To: <20200108160102.GA17415@blackbody.suse.cz>
On Wed, Jan 08, 2020 at 05:01:02PM +0100, Michal Koutný wrote:
> On Mon, Dec 23, 2019 at 07:15:03AM +0100, Christian Brauner <christian.brauner@ubuntu.com> wrote:
> > This adds support for creating a process in a different cgroup than its
> > parent.
> Binding fork and migration together looks useful.
>
> > --- a/kernel/cgroup/cgroup.c
> > +++ b/kernel/cgroup/cgroup.c
> > @@ -5882,21 +5882,176 @@ void cgroup_fork(struct task_struct *child)
> > INIT_LIST_HEAD(&child->cg_list);
> Just a nitpick, I noticed the comment for cgroup_fork should be updated
> too (generic migration happens in cgroup_post_fork).
Thanks.
>
> > --- a/kernel/fork.c
> > +++ b/kernel/fork.c
> > [...]
> > @@ -2279,8 +2278,7 @@ static __latent_entropy struct task_struct *copy_process(
> > write_unlock_irq(&tasklist_lock);
> >
> > proc_fork_connector(p);
> > - cgroup_post_fork(p);
> > - cgroup_threadgroup_change_end(current);
> > + cgroup_post_fork(current, p, args);
> I can see that when CLONE_INTO_CGROUP | CLONE_NEWCGROUP is passed, then
> the child's cgroup NS will be rooted at parent's css set
> (copy_namespaces precedes cgroup_post_fork).
>
> Wouldn't it make better sense if this flags combination resulted in
> child's NS rooted in its css set?
I need to take a closer look but it sounds like we should move the
copying of the cgroup namespace to a later point; but again I need to
look into this.
Thanks!
Christian
^ permalink raw reply
* Re: [PATCH v2 2/3] clone3: allow spawning processes into cgroups
From: Christian Brauner @ 2020-01-08 18:09 UTC (permalink / raw)
To: Tejun Heo
Cc: linux-api, linux-kernel, Ingo Molnar, Oleg Nesterov,
Johannes Weiner, Li Zefan, Peter Zijlstra, cgroups
In-Reply-To: <20200107163204.GB2677547@devbig004.ftw2.facebook.com>
On Tue, Jan 07, 2020 at 08:32:04AM -0800, Tejun Heo wrote:
> On Mon, Dec 23, 2019 at 07:15:03AM +0100, Christian Brauner wrote:
> > +static struct cgroup *cgroup_get_from_file(struct file *f)
> > +{
> > + struct cgroup_subsys_state *css;
> > + struct cgroup *cgrp;
> > +
> > + css = css_tryget_online_from_dir(f->f_path.dentry, NULL);
> > + if (IS_ERR(css))
> > + return ERR_CAST(css);
> > +
> > + cgrp = css->cgroup;
> > + if (!cgroup_on_dfl(cgrp)) {
> > + cgroup_put(cgrp);
> > + return ERR_PTR(-EBADF);
> > + }
> > +
> > + return cgrp;
> > +}
>
> It's minor but can you put this refactoring into a separate patch?
Yep, will do.
>
> ...
> > +static int cgroup_css_set_fork(struct task_struct *parent,
> > + struct kernel_clone_args *kargs)
> > + __acquires(&cgroup_mutex) __acquires(&cgroup_threadgroup_rwsem)
> > +{
> > + int ret;
> > + struct cgroup *dst_cgrp = NULL, *src_cgrp;
> > + struct css_set *cset;
> > + struct super_block *sb;
> > + struct file *f;
> > +
> > + if (kargs->flags & CLONE_INTO_CGROUP) {
> > + ret = mutex_lock_killable(&cgroup_mutex);
> > + if (ret)
> > + return ret;
> > + }
>
> I don't think this is necessary. cgroup_mutex should always only be
> held for a finite enough time; otherwise, processes would get stuck on
> random cgroupfs accesses or even /proc/self/cgroup.
Ok, so a simple mutex_lock() should suffice then.
>
> ...
> > + spin_lock_irq(&css_set_lock);
> > + src_cgrp = task_cgroup_from_root(parent, &cgrp_dfl_root);
> > + spin_unlock_irq(&css_set_lock);
>
> You can simply do cset->dfl_root here, which is consistent with other
> code paths which know that they want the dfl cgroup.
Ah, great!
>
> > + ret = cgroup_attach_permissions(src_cgrp, dst_cgrp, sb,
> > + !!(kargs->flags & CLONE_THREAD));
> > + if (ret)
> > + goto err;
>
> So, the existing perm check depends on the fact that for the write
> operation to have started, it already should have passed write perm
> check on the destination cgroup.procs file. We're missing that here,
> so we prolly need to check that explicitly.
I need to look into this before I can say yay or nay. :)
>
> > @@ -214,13 +215,21 @@ static void pids_cancel_attach(struct cgroup_taskset *tset)
> > +static int pids_can_fork(struct task_struct *parent, struct task_struct *child,
> > + struct kernel_clone_args *args)
> > {
> > + struct css_set *new_cset = NULL;
> > struct cgroup_subsys_state *css;
> > struct pids_cgroup *pids;
> > int err;
> >
> > - css = task_css_check(current, pids_cgrp_id, true);
> > + if (args)
> > + new_cset = args->cset;
> > +
> > + if (!new_cset)
> > + css = task_css_check(current, pids_cgrp_id, true);
> > + else
> > + css = new_cset->subsys[pids_cgrp_id];
>
> Heh, this kinda sucks. Would it be better to pass in the new css into
> the callbacks rather than clone args?
Hm, maybe. My reasoning was that the can_fork callbacks are really only
ever used when - well - fork()ing/clone{3}()ing. Additionally, I was
trying to make sure that struct css_set doesn't show up in too many
places outside of cgroup core. But I'm fine with changing this to just
take the css_set directly. Let's try that...
>
> > diff --git a/kernel/fork.c b/kernel/fork.c
> > index 2508a4f238a3..1604552f7cd3 100644
> > --- a/kernel/fork.c
> > +++ b/kernel/fork.c
> > @@ -2165,16 +2165,15 @@ static __latent_entropy struct task_struct *copy_process(
> > INIT_LIST_HEAD(&p->thread_group);
> > p->task_works = NULL;
> >
> > - cgroup_threadgroup_change_begin(current);
> > /*
> > * Ensure that the cgroup subsystem policies allow the new process to be
> > * forked. It should be noted the the new process's css_set can be changed
> > * between here and cgroup_post_fork() if an organisation operation is in
> > * progress.
> > */
> > - retval = cgroup_can_fork(p);
> > + retval = cgroup_can_fork(current, p, args);
> > if (retval)
> > - goto bad_fork_cgroup_threadgroup_change_end;
> > + goto bad_fork_put_pidfd;
> >
> > /*
> > * From this point on we must avoid any synchronous user-space
>
> Maybe we can move these changes into a prep patch together with the
> get_from_file change so that this patch only contains the actual
> feature implementation?
Should be doable!
>
> Other than that, looks good to me. Once the above review points are
> addressed and Oleg is okay with it, I'll be happy to route this
> through the cgroup tree.
>
> Thanks so much for working on this. This is really cool.
Thanks and I agree! :)
Christian
^ permalink raw reply
* Re: [PATCH v2 0/6] introduce configfd as generalisation of fsconfig
From: Christian Brauner @ 2020-01-08 17:07 UTC (permalink / raw)
To: James Bottomley
Cc: David Howells, Al Viro, linux-fsdevel, Christian Brauner,
Miklos Szeredi, linux-kernel, linux-api
In-Reply-To: <1578247328.3310.36.camel@HansenPartnership.com>
[extending the Cc a bit]
On Sun, Jan 05, 2020 at 10:02:08AM -0800, James Bottomley wrote:
> On Sun, 2020-01-05 at 17:23 +0100, Christian Brauner wrote:
> > On Sat, Jan 04, 2020 at 12:14:26PM -0800, James Bottomley wrote:
> > > fsconfig is a very powerful configuration mechanism except that it
> > > only works for filesystems with superblocks. This patch series
> > > generalises the useful concept of a multiple step configurational
> > > mechanism carried by a file descriptor. The object of this patch
> > > series is to get bind mounts to be configurable in the same way
> > > that superblock based ones are, but it should have utility beyond
> > > the filesytem realm. Patch 4 also reimplements fsconfig in terms
> > > of configfd, but that's not a strictly necessary patch, it is
> > > merely a useful demonstration that configfd is a superset of the
> > > properties of fsconfig.
> >
> > Thanks for the patch. I'm glad fsconfig() is picked back up; either
> > by you or by David. We will need this for sure.
> > But the configfd approach does not strike me as a great idea.
> > Anonymous inode fds provide an abstraction mechanism for kernel
> > objects which we built around fds such as timerfd, pidfd, mountfd and
> > so on. When you stat an anonfd you get ANON_INODE_FS_MAGIC and you
> > get the actual type by looking at fdinfo, or - more common - by
> > parsing out /proc/<pid>/fd/<nr> and discovering "[fscontext]". So
> > it's already a pretty massive abstraction layer we have. But configfd
> > would be yet another fd abstraction based on anonfds.
> > The idea has been that a new fd type based on anonfds comes with an
> > api specific to that type of fd. That seems way nicer from an api
> > design perspective than implementing new apis as part of yet another
> > generic configfd layer.
>
> Really, it's just a fd that gathers config information and can reserve
> specific errors (and we should really work out the i18n implications of
It's rather a complex multiplexer intended to go beyond the realm of
filesystems/mount api and that's something we have been burned by before.
> the latter). Whether it's a new fd type or an anonfd with a specific
> name doesn't seem to be that significant, so the name could be set by
> the type.
>
> > Another problem is that these syscalls here would be massive
> > multiplexing syscalls. If they are ever going to be used outside of
> > filesystem use-cases (which is doubtful) they will quickly rival
> > prctl(), seccomp(), and ptrace().
>
> Actually, that's partly the point. We do have several systemcalls with
Actually I think that's the problem. The keyctl api itself suffers
from the problem that it already has a complex multiplexer. That could
either point to bad api design (sorry, David :)) or it's just a very
complex use-case like the mount api. The good thing is that it's
restricted to a single domain: keys. And that's good. Plumbing both e.g.
keys and (parts of) mounts on top of another generic api is what strikes
me as a bad idea.
> variable argument parsing that would benefit from an approach like
> this. keyctl springs immediately to mind.
>
> > That's not a great thing. Especially, since we recently (a few
> > months ago with Linus chiming in too) had long discussions with the
> > conclusion that multiplexing syscalls are discouraged, from a
> > security and api design perspective. Especially when they are not
> > tied to a specific API (e.g. seccomp() and bpf() are at least tied to
> > a specific API). libcs such as glibc and musl had reservations in
> > that regard as well.
> >
> > This would also spread the mount api across even more fd types than
> > it already does now which is cumbersome for userspace.
> >
> > A generic API like that also makes it hard to do interception in
> > userspace which is important for brokers such as e.g. used in Firefox
> > or what we do in various container use-cases.
> >
> > So I have strong reservations about configfd and would strongly favor
> > the revival of the original fsconfig() patchset.
>
> Ah well, I did have plans for configfd to be self describing, so the
> arguments accepted by each type would be typed and pre-registered and
> thus parseable generically, so instead of being the usual anonymous
> multiplex sink, it would at least be an introspectable multiplexed
> sink. The problem there was I can't make fsconfig fit into that
We already have fsconfig() to configure mounts so it seems odd to now
spread the mount api onto configfd imho.
Christian
^ permalink raw reply
* Re: [PATCH v2 2/3] clone3: allow spawning processes into cgroups
From: Michal Koutný @ 2020-01-08 16:01 UTC (permalink / raw)
To: Christian Brauner
Cc: linux-api, linux-kernel, Tejun Heo, Ingo Molnar, Oleg Nesterov,
Johannes Weiner, Li Zefan, Peter Zijlstra, cgroups
In-Reply-To: <20191223061504.28716-3-christian.brauner@ubuntu.com>
[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]
On Mon, Dec 23, 2019 at 07:15:03AM +0100, Christian Brauner <christian.brauner@ubuntu.com> wrote:
> This adds support for creating a process in a different cgroup than its
> parent.
Binding fork and migration together looks useful.
> --- a/kernel/cgroup/cgroup.c
> +++ b/kernel/cgroup/cgroup.c
> @@ -5882,21 +5882,176 @@ void cgroup_fork(struct task_struct *child)
> INIT_LIST_HEAD(&child->cg_list);
Just a nitpick, I noticed the comment for cgroup_fork should be updated
too (generic migration happens in cgroup_post_fork).
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> [...]
> @@ -2279,8 +2278,7 @@ static __latent_entropy struct task_struct *copy_process(
> write_unlock_irq(&tasklist_lock);
>
> proc_fork_connector(p);
> - cgroup_post_fork(p);
> - cgroup_threadgroup_change_end(current);
> + cgroup_post_fork(current, p, args);
I can see that when CLONE_INTO_CGROUP | CLONE_NEWCGROUP is passed, then
the child's cgroup NS will be rooted at parent's css set
(copy_namespaces precedes cgroup_post_fork).
Wouldn't it make better sense if this flags combination resulted in
child's NS rooted in its css set?
Michal
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* [RFC PATCH glibc 5/8] glibc: sched_getcpu(): use rseq cpu_id TLS on Linux (v5)
From: Mathieu Desnoyers @ 2020-01-08 14:58 UTC (permalink / raw)
To: Carlos O'Donell
Cc: Florian Weimer, Joseph Myers, Szabolcs Nagy, libc-alpha,
Mathieu Desnoyers, Thomas Gleixner, Ben Maurer, Peter Zijlstra,
Paul E. McKenney, Boqun Feng, Will Deacon, Dave Watson,
Paul Turner, linux-kernel, linux-api
In-Reply-To: <20200108145851.13594-1-mathieu.desnoyers@efficios.com>
When available, use the cpu_id field from __rseq_abi on Linux to
implement sched_getcpu(). Fall-back on the vgetcpu vDSO if unavailable.
Benchmarks:
x86-64: Intel E5-2630 v3@2.40GHz, 16-core, hyperthreading
glibc sched_getcpu(): 13.7 ns (baseline)
glibc sched_getcpu() using rseq: 2.5 ns (speedup: 5.5x)
inline load cpuid from __rseq_abi TLS: 0.8 ns (speedup: 17.1x)
* sysdeps/unix/sysv/linux/sched_getcpu.c: use rseq cpu_id TLS on
Linux.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC: Carlos O'Donell <carlos@redhat.com>
CC: Florian Weimer <fweimer@redhat.com>
CC: Joseph Myers <joseph@codesourcery.com>
CC: Szabolcs Nagy <szabolcs.nagy@arm.com>
CC: Thomas Gleixner <tglx@linutronix.de>
CC: Ben Maurer <bmaurer@fb.com>
CC: Peter Zijlstra <peterz@infradead.org>
CC: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
CC: Boqun Feng <boqun.feng@gmail.com>
CC: Will Deacon <will.deacon@arm.com>
CC: Dave Watson <davejwatson@fb.com>
CC: Paul Turner <pjt@google.com>
CC: libc-alpha@sourceware.org
CC: linux-kernel@vger.kernel.org
CC: linux-api@vger.kernel.org
---
Changes since v1:
- rseq is only used if both __NR_rseq and RSEQ_SIG are defined.
Changes since v2:
- remove duplicated __rseq_abi extern declaration.
Changes since v3:
- update ChangeLog.
Changes since v4:
- Use atomic_load_relaxed to load the __rseq_abi.cpu_id field, a
consequence of the fact that __rseq_abi is not volatile anymore.
- Include atomic.h which provides atomic_load_relaxed.
---
sysdeps/unix/sysv/linux/sched_getcpu.c | 29 ++++++++++++++++++++++++--
1 file changed, 27 insertions(+), 2 deletions(-)
diff --git a/sysdeps/unix/sysv/linux/sched_getcpu.c b/sysdeps/unix/sysv/linux/sched_getcpu.c
index c019cfb3cf..6d3541333d 100644
--- a/sysdeps/unix/sysv/linux/sched_getcpu.c
+++ b/sysdeps/unix/sysv/linux/sched_getcpu.c
@@ -18,10 +18,15 @@
#include <errno.h>
#include <sched.h>
#include <sysdep.h>
+#include <atomic.h>
#include <sysdep-vdso.h>
-int
-sched_getcpu (void)
+#ifdef HAVE_GETCPU_VSYSCALL
+# define HAVE_VSYSCALL
+#endif
+
+static int
+vsyscall_sched_getcpu (void)
{
unsigned int cpu;
int r = -1;
@@ -32,3 +37,23 @@ sched_getcpu (void)
#endif
return r == -1 ? r : cpu;
}
+
+#ifdef __NR_rseq
+#include <sys/rseq.h>
+#endif
+
+#if defined __NR_rseq && defined RSEQ_SIG
+int
+sched_getcpu (void)
+{
+ int cpu_id = atomic_load_relaxed (&__rseq_abi.cpu_id);
+
+ return cpu_id >= 0 ? cpu_id : vsyscall_sched_getcpu ();
+}
+#else
+int
+sched_getcpu (void)
+{
+ return vsyscall_sched_getcpu ();
+}
+#endif
--
2.17.1
^ permalink raw reply related
* [RFC PATCH glibc 4/8] glibc: Perform rseq(2) registration at C startup and thread creation (v14)
From: Mathieu Desnoyers @ 2020-01-08 14:58 UTC (permalink / raw)
To: Carlos O'Donell
Cc: Florian Weimer, Joseph Myers, Szabolcs Nagy, libc-alpha,
Mathieu Desnoyers, Thomas Gleixner, Ben Maurer, Peter Zijlstra,
Paul E. McKenney, Boqun Feng, Will Deacon, Dave Watson,
Paul Turner, Rich Felker, linux-kernel, linux-api
In-Reply-To: <20200108145851.13594-1-mathieu.desnoyers@efficios.com>
Register rseq(2) TLS for each thread (including main), and unregister
for each thread (excluding main). "rseq" stands for Restartable
Sequences.
See the rseq(2) man page proposed here:
https://lkml.org/lkml/2018/9/19/647
This patch is based on glibc-2.30. The rseq(2) system call was merged
into Linux 4.18.
* NEWS: Add Restartable Sequences feature description.
* elf/libc_early_init.c: Perform rseq(2) registration at C startup.
startup for shared libc.
* nptl/pthread_create.c: Perform rseq(2) registration at thread
creation.
* manual/threads.texi: Document __rseq_abi, RSEQ_SIG, sys/rseq.h.
* sysdeps/unix/sysv/linux/Makefile: Add rseq-sym, sys/rseq.h,
bits/rseq.h.
* sysdeps/unix/sysv/linux/Versions: Export __rseq_abi from libc.
* sysdeps/unix/sysv/linux/aarch64/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/alpha/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/arm/be/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/arm/le/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/csky/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/hppa/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/i386/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/ia64/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/m68k/coldfire/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/m68k/m680x0/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/microblaze/be/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/microblaze/le/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/fpu/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n32/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n64/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/nios2/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/be/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/le/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/sh/be/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/sh/le/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/x86_64/64/libc.abilist: Likewise.
* sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist: Likewise.
* misc/rseq-internal.h: New file.
* sysdeps/unix/sysv/linux/rseq-internal.h: Likewise.
* sysdeps/unix/sysv/linux/rseq-sym.c: Likewise.
* sysdeps/unix/sysv/linux/sys/rseq.h: Likewise.
* sysdeps/unix/sysv/linux/bits/rseq.h: Likewise.
* sysdeps/unix/sysv/linux/aarch64/bits/rseq.h: Likewise.
* sysdeps/unix/sysv/linux/arm/bits/rseq.h: Likewise.
* sysdeps/unix/sysv/linux/mips/bits/rseq.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/bits/rseq.h: Likewise.
* sysdeps/unix/sysv/linux/s390/bits/rseq.h: Likewise.
* sysdeps/unix/sysv/linux/x86/bits/rseq.h: Likewise.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC: Carlos O'Donell <carlos@redhat.com>
CC: Florian Weimer <fweimer@redhat.com>
CC: Joseph Myers <joseph@codesourcery.com>
CC: Szabolcs Nagy <szabolcs.nagy@arm.com>
CC: Thomas Gleixner <tglx@linutronix.de>
CC: Ben Maurer <bmaurer@fb.com>
CC: Peter Zijlstra <peterz@infradead.org>
CC: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
CC: Boqun Feng <boqun.feng@gmail.com>
CC: Will Deacon <will.deacon@arm.com>
CC: Dave Watson <davejwatson@fb.com>
CC: Paul Turner <pjt@google.com>
CC: Rich Felker <dalias@libc.org>
CC: libc-alpha@sourceware.org
CC: linux-kernel@vger.kernel.org
CC: linux-api@vger.kernel.org
---
Changes since v1:
- Move __rseq_refcount to an extra field at the end of __rseq_abi to
eliminate one symbol.
All libraries/programs which try to register rseq (glibc,
early-adopter applications, early-adopter libraries) should use the
rseq refcount. It becomes part of the ABI within a user-space
process, but it's not part of the ABI shared with the kernel per se.
- Restructure how this code is organized so glibc keeps building on
non-Linux targets.
- Use non-weak symbol for __rseq_abi.
- Move rseq registration/unregistration implementation into its own
nptl/rseq.c compile unit.
- Move __rseq_abi symbol under GLIBC_2.29.
Changes since v2:
- Move __rseq_refcount to its own symbol, which is less ugly than
trying to play tricks with the rseq uapi.
- Move __rseq_abi from nptl to csu (C start up), so it can be used
across glibc, including memory allocator and sched_getcpu(). The
__rseq_refcount symbol is kept in nptl, because there is no reason
to use it elsewhere in glibc.
Changes since v3:
- Set __rseq_refcount TLS to 1 on register/set to 0 on unregister
because glibc is the first/last user.
- Unconditionally register/unregister rseq at thread start/exit, because
glibc is the first/last user.
- Add missing abilist items.
- Rebase on glibc master commit a502c5294.
- Add NEWS entry.
Changes since v4:
- Do not use "weak" symbols for __rseq_abi and __rseq_refcount. Based on
"System V Application Binary Interface", weak only affects the link
editor, not the dynamic linker.
- Install a new sys/rseq.h system header on Linux, which contains the
RSEQ_SIG definition, __rseq_abi declaration and __rseq_refcount
declaration. Move those definition/declarations from rseq-internal.h
to the installed sys/rseq.h header.
- Considering that rseq is only available on Linux, move csu/rseq.c to
sysdeps/unix/sysv/linux/rseq-sym.c.
- Move __rseq_refcount from nptl/rseq.c to
sysdeps/unix/sysv/linux/rseq-sym.c, so it is only defined on Linux.
- Move both ABI definitions for __rseq_abi and __rseq_refcount to
sysdeps/unix/sysv/linux/Versions, so they only appear on Linux.
- Document __rseq_abi and __rseq_refcount volatile.
- Document the RSEQ_SIG signature define.
- Move registration functions from rseq.c to rseq-internal.h static
inline functions. Introduce empty stubs in misc/rseq-internal.h,
which can be overridden by architecture code in
sysdeps/unix/sysv/linux/rseq-internal.h.
- Rename __rseq_register_current_thread and __rseq_unregister_current_thread
to rseq_register_current_thread and rseq_unregister_current_thread,
now that those are only visible as internal static inline functions.
- Invoke rseq_register_current_thread() from libc-start.c LIBC_START_MAIN
rather than nptl init, so applications not linked against
libpthread.so have rseq registered for their main() thread. Note that
it is invoked separately for SHARED and !SHARED builds.
Changes since v5:
- Replace __rseq_refcount by __rseq_lib_abi, which contains two
uint32_t: register_state and refcount. The "register_state" field
allows inhibiting rseq registration from signal handlers nested on top
of glibc registration and occuring after rseq unregistration by glibc.
- Introduce enum rseq_register_state, which contains the states allowed
for the struct rseq_lib_abi register_state field.
Changes since v6:
- Introduce bits/rseq.h to define RSEQ_SIG for each architecture.
The generic bits/rseq.h does not define RSEQ_SIG, meaning that each
architecture implementing rseq needs to implement bits/rseq.h.
- Rename enum item RSEQ_REGISTER_NESTED to RSEQ_REGISTER_ONGOING.
- Port to glibc-2.29.
Changes since v7:
- Remove __rseq_lib_abi symbol, including refcount and register_state
fields.
- Remove reference counting and nested signals handling from
registration/unregistration functions.
- Introduce new __rseq_handled exported symbol, which is set to 1
by glibc on C startup when it handles restartable sequences.
This allows glibc to coexist with early adopter libraries and
applications wishing to register restartable sequences when it
is not handled by glibc.
- Introduce rseq_init (), which sets __rseq_handled to 1 from
C startup.
- Update NEWS entry.
- Update comments at the beginning of new files.
- Registration depends on both __NR_rseq and RSEQ_SIG.
- Remove ARM, powerpc, MIPS RSEQ_SIG until we agree with maintainers
on the signature choice.
- Update x86, s390 RSEQ_SIG based on discussion with arch maintainers.
- Remove rseq-internal.h from headers list of misc/Makefile, so it
it not installed by make install.
Changes since v8:
- Introduce RSEQ_SIG_CODE and RSEQ_SIG_DATA on aarch64 to handle
compiling with -mbig-endian.
Changes since v9:
- Update Changelog.
- Remove unneeded new file comment header newlines.
Changes since v10:
- Remove volatile from __rseq_abi declaration.
- Document that __rseq_handled is about library managing rseq
registration, independently of whether rseq is available or not.
- Move __rseq_handled symbol to ld.so, initialize this symbol within
the dynamic linker initialization for both shared (rtld.c) and static
(dl-support.c) builds.
- Only register the rseq TLS on initialization once in multiple-libc
scenarios. Use rtld_active () for this purpose.
- In the static libc case, register the rseq TLS after LD_PRELOAD
constructors are run, so it matches the order of this initialization
vs LD_PRELOAD contructors execution for the shared libc.
- Agreed on signature choice with powerpc and MIPS maintainers,
re-adding those signatures,
- The main architecture still left out signature-wise is ARM32.
Changes since v11:
- Rebase on glibc 2.30.
- Re-introduce ARM RSEQ_SIG following feedback from Will Deacon.
Changes since v12:
- Remove __rseq_handled,
- Rely on OS implicit rseq unregistration on thread teardown,
- Register main thread in __libc_early_init ().
- Add Restartable Sequences entry to threads manual.
Changes since v13:
- Update following be/le abilist split for arm, microblaze, and sh.
- Update manual to add the __rseq_abi variable and RSEQ_SIG macro to
generate manual index entries, and add missing "Restartable Sequences"
menu entry to the threads chapter.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC: Carlos O'Donell <carlos@redhat.com>
CC: Florian Weimer <fweimer@redhat.com>
CC: Joseph Myers <joseph@codesourcery.com>
CC: Szabolcs Nagy <szabolcs.nagy@arm.com>
CC: Thomas Gleixner <tglx@linutronix.de>
CC: Ben Maurer <bmaurer@fb.com>
CC: Peter Zijlstra <peterz@infradead.org>
CC: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
CC: Boqun Feng <boqun.feng@gmail.com>
CC: Will Deacon <will.deacon@arm.com>
CC: Dave Watson <davejwatson@fb.com>
CC: Paul Turner <pjt@google.com>
CC: Rich Felker <dalias@libc.org>
CC: libc-alpha@sourceware.org
CC: linux-kernel@vger.kernel.org
CC: linux-api@vger.kernel.org
---
NEWS | 10 +++
elf/libc_early_init.c | 3 +
manual/threads.texi | 30 ++++++-
misc/rseq-internal.h | 33 ++++++++
nptl/pthread_create.c | 12 +++
sysdeps/unix/sysv/linux/Makefile | 5 +-
sysdeps/unix/sysv/linux/Versions | 3 +
sysdeps/unix/sysv/linux/aarch64/bits/rseq.h | 43 ++++++++++
sysdeps/unix/sysv/linux/aarch64/libc.abilist | 1 +
sysdeps/unix/sysv/linux/alpha/libc.abilist | 1 +
sysdeps/unix/sysv/linux/arm/be/libc.abilist | 1 +
sysdeps/unix/sysv/linux/arm/bits/rseq.h | 83 +++++++++++++++++++
sysdeps/unix/sysv/linux/arm/le/libc.abilist | 1 +
sysdeps/unix/sysv/linux/bits/rseq.h | 29 +++++++
sysdeps/unix/sysv/linux/csky/libc.abilist | 1 +
sysdeps/unix/sysv/linux/hppa/libc.abilist | 1 +
sysdeps/unix/sysv/linux/i386/libc.abilist | 1 +
sysdeps/unix/sysv/linux/ia64/libc.abilist | 1 +
.../sysv/linux/m68k/coldfire/libc.abilist | 1 +
.../unix/sysv/linux/m68k/m680x0/libc.abilist | 1 +
.../sysv/linux/microblaze/be/libc.abilist | 1 +
.../sysv/linux/microblaze/le/libc.abilist | 1 +
sysdeps/unix/sysv/linux/mips/bits/rseq.h | 62 ++++++++++++++
.../sysv/linux/mips/mips32/fpu/libc.abilist | 1 +
.../sysv/linux/mips/mips32/nofpu/libc.abilist | 1 +
.../sysv/linux/mips/mips64/n32/libc.abilist | 1 +
.../sysv/linux/mips/mips64/n64/libc.abilist | 1 +
sysdeps/unix/sysv/linux/nios2/libc.abilist | 1 +
sysdeps/unix/sysv/linux/powerpc/bits/rseq.h | 37 +++++++++
.../linux/powerpc/powerpc32/fpu/libc.abilist | 1 +
.../powerpc/powerpc32/nofpu/libc.abilist | 1 +
.../linux/powerpc/powerpc64/be/libc.abilist | 1 +
.../linux/powerpc/powerpc64/le/libc.abilist | 1 +
.../unix/sysv/linux/riscv/rv64/libc.abilist | 1 +
sysdeps/unix/sysv/linux/rseq-internal.h | 77 +++++++++++++++++
sysdeps/unix/sysv/linux/rseq-sym.c | 43 ++++++++++
sysdeps/unix/sysv/linux/s390/bits/rseq.h | 37 +++++++++
.../unix/sysv/linux/s390/s390-32/libc.abilist | 1 +
.../unix/sysv/linux/s390/s390-64/libc.abilist | 1 +
sysdeps/unix/sysv/linux/sh/be/libc.abilist | 1 +
sysdeps/unix/sysv/linux/sh/le/libc.abilist | 1 +
.../sysv/linux/sparc/sparc32/libc.abilist | 1 +
.../sysv/linux/sparc/sparc64/libc.abilist | 1 +
sysdeps/unix/sysv/linux/sys/rseq.h | 30 +++++++
sysdeps/unix/sysv/linux/x86/bits/rseq.h | 30 +++++++
.../unix/sysv/linux/x86_64/64/libc.abilist | 1 +
.../unix/sysv/linux/x86_64/x32/libc.abilist | 1 +
47 files changed, 593 insertions(+), 4 deletions(-)
create mode 100644 misc/rseq-internal.h
create mode 100644 sysdeps/unix/sysv/linux/aarch64/bits/rseq.h
create mode 100644 sysdeps/unix/sysv/linux/arm/bits/rseq.h
create mode 100644 sysdeps/unix/sysv/linux/bits/rseq.h
create mode 100644 sysdeps/unix/sysv/linux/mips/bits/rseq.h
create mode 100644 sysdeps/unix/sysv/linux/powerpc/bits/rseq.h
create mode 100644 sysdeps/unix/sysv/linux/rseq-internal.h
create mode 100644 sysdeps/unix/sysv/linux/rseq-sym.c
create mode 100644 sysdeps/unix/sysv/linux/s390/bits/rseq.h
create mode 100644 sysdeps/unix/sysv/linux/sys/rseq.h
create mode 100644 sysdeps/unix/sysv/linux/x86/bits/rseq.h
diff --git a/NEWS b/NEWS
index b85989ec3d..42f9b1c134 100644
--- a/NEWS
+++ b/NEWS
@@ -49,6 +49,16 @@ Major new features:
responses, indicating a lack of DNSSEC validation. (Therefore, the name
servers and the network path to them are treated as untrusted.)
+* Support for automatically registering threads with the Linux rseq(2)
+ system call has been added. This system call is implemented starting
+ from Linux 4.18. The Restartable Sequences ABI accelerates user-space
+ operations on per-cpu data. It allows user-space to perform updates
+ on per-cpu data without requiring heavy-weight atomic operations.
+ Automatically registering threads allows all libraries, including libc,
+ to make immediate use of the rseq(2) support by using the documented ABI.
+ See 'man 2 rseq' for the details of the ABI shared between libc and the
+ kernel.
+
Deprecated and removed features, and other changes affecting compatibility:
* The totalorder and totalordermag functions, and the corresponding
diff --git a/elf/libc_early_init.c b/elf/libc_early_init.c
index 1ac66d895d..30466afea0 100644
--- a/elf/libc_early_init.c
+++ b/elf/libc_early_init.c
@@ -18,10 +18,13 @@
#include <ctype.h>
#include <libc-early-init.h>
+#include <rseq-internal.h>
void
__libc_early_init (void)
{
/* Initialize ctype data. */
__ctype_init ();
+ /* Register rseq ABI to the kernel. */
+ (void) rseq_register_current_thread ();
}
diff --git a/manual/threads.texi b/manual/threads.texi
index 0858ef8f92..59f634e432 100644
--- a/manual/threads.texi
+++ b/manual/threads.texi
@@ -9,8 +9,10 @@ This chapter describes functions used for managing threads.
POSIX threads.
@menu
-* ISO C Threads:: Threads based on the ISO C specification.
-* POSIX Threads:: Threads based on the POSIX specification.
+* ISO C Threads:: Threads based on the ISO C specification.
+* POSIX Threads:: Threads based on the POSIX specification.
+* Restartable Sequences:: Linux-specific Restartable Sequences
+ integration.
@end menu
@@ -881,3 +883,27 @@ Behaves like @code{pthread_timedjoin_np} except that the absolute time in
@c pthread_spin_unlock
@c pthread_testcancel
@c pthread_yield
+
+@node Restartable Sequences
+@section Restartable Sequences
+@cindex rseq
+
+This section describes @theglibc{} Restartable Sequences integration.
+
+@deftypevar {struct rseq} __rseq_abi
+@standards{GNU, sys/rseq.h}
+@Theglibc{} implements a @code{__rseq_abi} TLS symbol to interact with the
+Restartable Sequences system call (Linux-specific). The layout of this
+structure is defined by the Linux kernel @file{linux/rseq.h} UAPI.
+Registration of each thread's @code{__rseq_abi} is performed by
+@theglibc{} at libc initialization and pthread creation.
+@end deftypevar
+
+@deftypevr Macro int RSEQ_SIG
+@standards{GNU, sys/rseq.h}
+Each supported architecture provide a @code{RSEQ_SIG} macro in
+@file{sys/rseq.h} which contains a signature. That signature is expected to be
+present in the code before each Restartable Sequences abort handler. Failure
+to provide the expected signature may terminate the process with a Segmentation
+fault.
+@end deftypevr
diff --git a/misc/rseq-internal.h b/misc/rseq-internal.h
new file mode 100644
index 0000000000..df8fc6c006
--- /dev/null
+++ b/misc/rseq-internal.h
@@ -0,0 +1,33 @@
+/* Restartable Sequences internal API. Stub version.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef RSEQ_INTERNAL_H
+#define RSEQ_INTERNAL_H
+
+static inline int
+rseq_register_current_thread (void)
+{
+ return -1;
+}
+
+static inline int
+rseq_unregister_current_thread (void)
+{
+ return -1;
+}
+
+#endif /* rseq-internal.h */
diff --git a/nptl/pthread_create.c b/nptl/pthread_create.c
index 5b3d4382b3..94dd30b173 100644
--- a/nptl/pthread_create.c
+++ b/nptl/pthread_create.c
@@ -33,6 +33,7 @@
#include <default-sched.h>
#include <futex-internal.h>
#include <tls-setup.h>
+#include <rseq-internal.h>
#include "libioP.h"
#include <shlib-compat.h>
@@ -384,6 +385,9 @@ START_THREAD_DEFN
/* Initialize pointers to locale data. */
__ctype_init ();
+ /* Register rseq TLS to the kernel. */
+ (void) rseq_register_current_thread ();
+
#ifdef __NR_set_robust_list
# ifndef __ASSUME_SET_ROBUST_LIST
if (__set_robust_list_avail >= 0)
@@ -581,6 +585,14 @@ START_THREAD_DEFN
process is really dead since 'clone' got passed the CLONE_CHILD_CLEARTID
flag. The 'tid' field in the TCB will be set to zero.
+ rseq TLS is still registered at this point. Rely on implicit unregistration
+ performed by the kernel on thread teardown. This is not a problem because the
+ rseq TLS lives on the stack, and the stack outlives the thread. If TCB
+ allocation is ever changed, additional steps may be required, such as
+ performing explicit rseq unregistration before reclaiming the rseq TLS area
+ memory. It is NOT sufficient to block signals because the kernel may write
+ to the rseq area even without signals.
+
The exit code is zero since in case all threads exit by calling
'pthread_exit' the exit status must be 0 (zero). */
__exit_thread ();
diff --git a/sysdeps/unix/sysv/linux/Makefile b/sysdeps/unix/sysv/linux/Makefile
index f12b7b1a2d..36a0fd231b 100644
--- a/sysdeps/unix/sysv/linux/Makefile
+++ b/sysdeps/unix/sysv/linux/Makefile
@@ -41,7 +41,7 @@ update-syscall-lists: arch-syscall.h
endif
ifeq ($(subdir),csu)
-sysdep_routines += errno-loc
+sysdep_routines += errno-loc rseq-sym
endif
ifeq ($(subdir),assert)
@@ -91,7 +91,8 @@ sysdep_headers += sys/mount.h sys/acct.h sys/sysctl.h \
bits/termios-baud.h bits/termios-c_cflag.h \
bits/termios-c_lflag.h bits/termios-tcflow.h \
bits/termios-misc.h \
- bits/ipc-perm.h
+ bits/ipc-perm.h \
+ sys/rseq.h bits/rseq.h
tests += tst-clone tst-clone2 tst-clone3 tst-fanotify tst-personality \
tst-quota tst-sync_file_range tst-sysconf-iov_max tst-ttyname \
diff --git a/sysdeps/unix/sysv/linux/Versions b/sysdeps/unix/sysv/linux/Versions
index d385085c61..7f0da50580 100644
--- a/sysdeps/unix/sysv/linux/Versions
+++ b/sysdeps/unix/sysv/linux/Versions
@@ -177,6 +177,9 @@ libc {
GLIBC_2.30 {
getdents64; gettid; tgkill;
}
+ GLIBC_2.31 {
+ __rseq_abi;
+ }
GLIBC_PRIVATE {
# functions used in other libraries
__syscall_rt_sigqueueinfo;
diff --git a/sysdeps/unix/sysv/linux/aarch64/bits/rseq.h b/sysdeps/unix/sysv/linux/aarch64/bits/rseq.h
new file mode 100644
index 0000000000..35fcc41f1e
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/aarch64/bits/rseq.h
@@ -0,0 +1,43 @@
+/* Restartable Sequences Linux aarch64 architecture header.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+# error "Never use <bits/rseq.h> directly; include <sys/rseq.h> instead."
+#endif
+
+/* RSEQ_SIG is a signature required before each abort handler code.
+
+ It is a 32-bit value that maps to actual architecture code compiled
+ into applications and libraries. It needs to be defined for each
+ architecture. When choosing this value, it needs to be taken into
+ account that generating invalid instructions may have ill effects on
+ tools like objdump, and may also have impact on the CPU speculative
+ execution efficiency in some cases.
+
+ aarch64 -mbig-endian generates mixed endianness code vs data:
+ little-endian code and big-endian data. Ensure the RSEQ_SIG signature
+ matches code endianness. */
+
+#define RSEQ_SIG_CODE 0xd428bc00 /* BRK #0x45E0. */
+
+#ifdef __AARCH64EB__
+#define RSEQ_SIG_DATA 0x00bc28d4 /* BRK #0x45E0. */
+#else
+#define RSEQ_SIG_DATA RSEQ_SIG_CODE
+#endif
+
+#define RSEQ_SIG RSEQ_SIG_DATA
diff --git a/sysdeps/unix/sysv/linux/aarch64/libc.abilist b/sysdeps/unix/sysv/linux/aarch64/libc.abilist
index a4c31932cb..6784f13c09 100644
--- a/sysdeps/unix/sysv/linux/aarch64/libc.abilist
+++ b/sysdeps/unix/sysv/linux/aarch64/libc.abilist
@@ -2145,3 +2145,4 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
diff --git a/sysdeps/unix/sysv/linux/alpha/libc.abilist b/sysdeps/unix/sysv/linux/alpha/libc.abilist
index e7f2174ac2..71db8422a2 100644
--- a/sysdeps/unix/sysv/linux/alpha/libc.abilist
+++ b/sysdeps/unix/sysv/linux/alpha/libc.abilist
@@ -2225,6 +2225,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _IO_fprintf F
GLIBC_2.4 _IO_printf F
GLIBC_2.4 _IO_sprintf F
diff --git a/sysdeps/unix/sysv/linux/arm/be/libc.abilist b/sysdeps/unix/sysv/linux/arm/be/libc.abilist
index b152c0e24a..eecebc908f 100644
--- a/sysdeps/unix/sysv/linux/arm/be/libc.abilist
+++ b/sysdeps/unix/sysv/linux/arm/be/libc.abilist
@@ -133,6 +133,7 @@ GLIBC_2.30 twalk_r F
GLIBC_2.31 msgctl F
GLIBC_2.31 semctl F
GLIBC_2.31 shmctl F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _Exit F
GLIBC_2.4 _IO_2_1_stderr_ D 0xa0
GLIBC_2.4 _IO_2_1_stdin_ D 0xa0
diff --git a/sysdeps/unix/sysv/linux/arm/bits/rseq.h b/sysdeps/unix/sysv/linux/arm/bits/rseq.h
new file mode 100644
index 0000000000..cd00513bfb
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/arm/bits/rseq.h
@@ -0,0 +1,83 @@
+/* Restartable Sequences Linux arm architecture header.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+# error "Never use <bits/rseq.h> directly; include <sys/rseq.h> instead."
+#endif
+
+/*
+ RSEQ_SIG is a signature required before each abort handler code.
+
+ It is a 32-bit value that maps to actual architecture code compiled
+ into applications and libraries. It needs to be defined for each
+ architecture. When choosing this value, it needs to be taken into
+ account that generating invalid instructions may have ill effects on
+ tools like objdump, and may also have impact on the CPU speculative
+ execution efficiency in some cases.
+
+ - ARM little endian
+
+ RSEQ_SIG uses the udf A32 instruction with an uncommon immediate operand
+ value 0x5de3. This traps if user-space reaches this instruction by mistake,
+ and the uncommon operand ensures the kernel does not move the instruction
+ pointer to attacker-controlled code on rseq abort.
+
+ The instruction pattern in the A32 instruction set is:
+
+ e7f5def3 udf #24035 ; 0x5de3
+
+ This translates to the following instruction pattern in the T16 instruction
+ set:
+
+ little endian:
+ def3 udf #243 ; 0xf3
+ e7f5 b.n <7f5>
+
+ - ARMv6+ big endian (BE8):
+
+ ARMv6+ -mbig-endian generates mixed endianness code vs data: little-endian
+ code and big-endian data. The data value of the signature needs to have its
+ byte order reversed to generate the trap instruction:
+
+ Data: 0xf3def5e7
+
+ Translates to this A32 instruction pattern:
+
+ e7f5def3 udf #24035 ; 0x5de3
+
+ Translates to this T16 instruction pattern:
+
+ def3 udf #243 ; 0xf3
+ e7f5 b.n <7f5>
+
+ - Prior to ARMv6 big endian (BE32):
+
+ Prior to ARMv6, -mbig-endian generates big-endian code and data
+ (which match), so the endianness of the data representation of the
+ signature should not be reversed. However, the choice between BE32
+ and BE8 is done by the linker, so we cannot know whether code and
+ data endianness will be mixed before the linker is invoked. So rather
+ than try to play tricks with the linker, the rseq signature is simply
+ data (not a trap instruction) prior to ARMv6 on big endian. This is
+ why the signature is expressed as data (.word) rather than as
+ instruction (.inst) in assembler. */
+
+#ifdef __ARMEB__
+#define RSEQ_SIG 0xf3def5e7 /* udf #24035 ; 0x5de3 (ARMv6+) */
+#else
+#define RSEQ_SIG 0xe7f5def3 /* udf #24035 ; 0x5de3 */
+#endif
diff --git a/sysdeps/unix/sysv/linux/arm/le/libc.abilist b/sysdeps/unix/sysv/linux/arm/le/libc.abilist
index 9371927927..3e7434d3d5 100644
--- a/sysdeps/unix/sysv/linux/arm/le/libc.abilist
+++ b/sysdeps/unix/sysv/linux/arm/le/libc.abilist
@@ -130,6 +130,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _Exit F
GLIBC_2.4 _IO_2_1_stderr_ D 0xa0
GLIBC_2.4 _IO_2_1_stdin_ D 0xa0
diff --git a/sysdeps/unix/sysv/linux/bits/rseq.h b/sysdeps/unix/sysv/linux/bits/rseq.h
new file mode 100644
index 0000000000..a3c023f5c7
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/bits/rseq.h
@@ -0,0 +1,29 @@
+/* Restartable Sequences architecture header. Stub version.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+# error "Never use <bits/rseq.h> directly; include <sys/rseq.h> instead."
+#endif
+
+/* RSEQ_SIG is a signature required before each abort handler code.
+
+ It is a 32-bit value that maps to actual architecture code compiled
+ into applications and libraries. It needs to be defined for each
+ architecture. When choosing this value, it needs to be taken into
+ account that generating invalid instructions may have ill effects on
+ tools like objdump, and may also have impact on the CPU speculative
+ execution efficiency in some cases. */
diff --git a/sysdeps/unix/sysv/linux/csky/libc.abilist b/sysdeps/unix/sysv/linux/csky/libc.abilist
index 9b3cee65bb..b7ed346b1c 100644
--- a/sysdeps/unix/sysv/linux/csky/libc.abilist
+++ b/sysdeps/unix/sysv/linux/csky/libc.abilist
@@ -2089,3 +2089,4 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
diff --git a/sysdeps/unix/sysv/linux/hppa/libc.abilist b/sysdeps/unix/sysv/linux/hppa/libc.abilist
index df6d96fbae..d55b153c48 100644
--- a/sysdeps/unix/sysv/linux/hppa/libc.abilist
+++ b/sysdeps/unix/sysv/linux/hppa/libc.abilist
@@ -2046,6 +2046,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/i386/libc.abilist b/sysdeps/unix/sysv/linux/i386/libc.abilist
index fcb625b6bf..c9061600f6 100644
--- a/sysdeps/unix/sysv/linux/i386/libc.abilist
+++ b/sysdeps/unix/sysv/linux/i386/libc.abilist
@@ -2212,6 +2212,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/ia64/libc.abilist b/sysdeps/unix/sysv/linux/ia64/libc.abilist
index cb556c5998..f794303f0e 100644
--- a/sysdeps/unix/sysv/linux/ia64/libc.abilist
+++ b/sysdeps/unix/sysv/linux/ia64/libc.abilist
@@ -2078,6 +2078,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/m68k/coldfire/libc.abilist b/sysdeps/unix/sysv/linux/m68k/coldfire/libc.abilist
index 5e3cdea246..e5e545f3af 100644
--- a/sysdeps/unix/sysv/linux/m68k/coldfire/libc.abilist
+++ b/sysdeps/unix/sysv/linux/m68k/coldfire/libc.abilist
@@ -134,6 +134,7 @@ GLIBC_2.30 twalk_r F
GLIBC_2.31 msgctl F
GLIBC_2.31 semctl F
GLIBC_2.31 shmctl F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _Exit F
GLIBC_2.4 _IO_2_1_stderr_ D 0x98
GLIBC_2.4 _IO_2_1_stdin_ D 0x98
diff --git a/sysdeps/unix/sysv/linux/m68k/m680x0/libc.abilist b/sysdeps/unix/sysv/linux/m68k/m680x0/libc.abilist
index ea5e7a41af..3fc1223e2c 100644
--- a/sysdeps/unix/sysv/linux/m68k/m680x0/libc.abilist
+++ b/sysdeps/unix/sysv/linux/m68k/m680x0/libc.abilist
@@ -2158,6 +2158,7 @@ GLIBC_2.30 twalk_r F
GLIBC_2.31 msgctl F
GLIBC_2.31 semctl F
GLIBC_2.31 shmctl F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/microblaze/be/libc.abilist b/sysdeps/unix/sysv/linux/microblaze/be/libc.abilist
index ac55b0acd7..81db30b543 100644
--- a/sysdeps/unix/sysv/linux/microblaze/be/libc.abilist
+++ b/sysdeps/unix/sysv/linux/microblaze/be/libc.abilist
@@ -2140,3 +2140,4 @@ GLIBC_2.30 twalk_r F
GLIBC_2.31 msgctl F
GLIBC_2.31 semctl F
GLIBC_2.31 shmctl F
+GLIBC_2.31 __rseq_abi T 0x20
diff --git a/sysdeps/unix/sysv/linux/microblaze/le/libc.abilist b/sysdeps/unix/sysv/linux/microblaze/le/libc.abilist
index f7ced487f7..a2ce147dde 100644
--- a/sysdeps/unix/sysv/linux/microblaze/le/libc.abilist
+++ b/sysdeps/unix/sysv/linux/microblaze/le/libc.abilist
@@ -2137,3 +2137,4 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
diff --git a/sysdeps/unix/sysv/linux/mips/bits/rseq.h b/sysdeps/unix/sysv/linux/mips/bits/rseq.h
new file mode 100644
index 0000000000..8c75f107e7
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/mips/bits/rseq.h
@@ -0,0 +1,62 @@
+/* Restartable Sequences Linux mips architecture header.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+# error "Never use <bits/rseq.h> directly; include <sys/rseq.h> instead."
+#endif
+
+/* RSEQ_SIG is a signature required before each abort handler code.
+
+ It is a 32-bit value that maps to actual architecture code compiled
+ into applications and libraries. It needs to be defined for each
+ architecture. When choosing this value, it needs to be taken into
+ account that generating invalid instructions may have ill effects on
+ tools like objdump, and may also have impact on the CPU speculative
+ execution efficiency in some cases.
+
+ RSEQ_SIG uses the break instruction. The instruction pattern is:
+
+ On MIPS:
+ 0350000d break 0x350
+
+ On nanoMIPS:
+ 00100350 break 0x350
+
+ On microMIPS:
+ 0000d407 break 0x350
+
+ For nanoMIPS32 and microMIPS, the instruction stream is encoded as
+ 16-bit halfwords, so the signature halfwords need to be swapped
+ accordingly for little-endian. */
+
+#if defined(__nanomips__)
+# ifdef __MIPSEL__
+# define RSEQ_SIG 0x03500010
+# else
+# define RSEQ_SIG 0x00100350
+# endif
+#elif defined(__mips_micromips)
+# ifdef __MIPSEL__
+# define RSEQ_SIG 0xd4070000
+# else
+# define RSEQ_SIG 0x0000d407
+# endif
+#elif defined(__mips__)
+# define RSEQ_SIG 0x0350000d
+#else
+/* Unknown MIPS architecture. */
+#endif
diff --git a/sysdeps/unix/sysv/linux/mips/mips32/fpu/libc.abilist b/sysdeps/unix/sysv/linux/mips/mips32/fpu/libc.abilist
index 06c2e64edd..c0040ddd4e 100644
--- a/sysdeps/unix/sysv/linux/mips/mips32/fpu/libc.abilist
+++ b/sysdeps/unix/sysv/linux/mips/mips32/fpu/libc.abilist
@@ -2129,6 +2129,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc.abilist b/sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc.abilist
index bdfd073b86..61f19076eb 100644
--- a/sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc.abilist
+++ b/sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc.abilist
@@ -2127,6 +2127,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/mips/mips64/n32/libc.abilist b/sysdeps/unix/sysv/linux/mips/mips64/n32/libc.abilist
index 3d61d4974a..df4f3a3c04 100644
--- a/sysdeps/unix/sysv/linux/mips/mips64/n32/libc.abilist
+++ b/sysdeps/unix/sysv/linux/mips/mips64/n32/libc.abilist
@@ -2135,6 +2135,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/mips/mips64/n64/libc.abilist b/sysdeps/unix/sysv/linux/mips/mips64/n64/libc.abilist
index 675acca5db..a96de2e467 100644
--- a/sysdeps/unix/sysv/linux/mips/mips64/n64/libc.abilist
+++ b/sysdeps/unix/sysv/linux/mips/mips64/n64/libc.abilist
@@ -2129,6 +2129,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/nios2/libc.abilist b/sysdeps/unix/sysv/linux/nios2/libc.abilist
index 7fec0c9670..7b2ccbe953 100644
--- a/sysdeps/unix/sysv/linux/nios2/libc.abilist
+++ b/sysdeps/unix/sysv/linux/nios2/libc.abilist
@@ -2178,3 +2178,4 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
diff --git a/sysdeps/unix/sysv/linux/powerpc/bits/rseq.h b/sysdeps/unix/sysv/linux/powerpc/bits/rseq.h
new file mode 100644
index 0000000000..bae8f4aaa1
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/powerpc/bits/rseq.h
@@ -0,0 +1,37 @@
+/* Restartable Sequences Linux powerpc architecture header.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+# error "Never use <bits/rseq.h> directly; include <sys/rseq.h> instead."
+#endif
+
+/* RSEQ_SIG is a signature required before each abort handler code.
+
+ It is a 32-bit value that maps to actual architecture code compiled
+ into applications and libraries. It needs to be defined for each
+ architecture. When choosing this value, it needs to be taken into
+ account that generating invalid instructions may have ill effects on
+ tools like objdump, and may also have impact on the CPU speculative
+ execution efficiency in some cases.
+
+ RSEQ_SIG uses the following trap instruction:
+
+ powerpc-be: 0f e5 00 0b twui r5,11
+ powerpc64-le: 0b 00 e5 0f twui r5,11
+ powerpc64-be: 0f e5 00 0b twui r5,11 */
+
+#define RSEQ_SIG 0x0fe5000b
diff --git a/sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc.abilist b/sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc.abilist
index 1e8ff6f83e..6f4c6515dc 100644
--- a/sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc.abilist
+++ b/sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc.abilist
@@ -2185,6 +2185,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _IO_fprintf F
GLIBC_2.4 _IO_printf F
GLIBC_2.4 _IO_sprintf F
diff --git a/sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc.abilist b/sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc.abilist
index b5a0751d90..f9875b4e22 100644
--- a/sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc.abilist
+++ b/sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc.abilist
@@ -2218,6 +2218,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _IO_fprintf F
GLIBC_2.4 _IO_printf F
GLIBC_2.4 _IO_sprintf F
diff --git a/sysdeps/unix/sysv/linux/powerpc/powerpc64/be/libc.abilist b/sysdeps/unix/sysv/linux/powerpc/powerpc64/be/libc.abilist
index 0c86217fc6..db06080db8 100644
--- a/sysdeps/unix/sysv/linux/powerpc/powerpc64/be/libc.abilist
+++ b/sysdeps/unix/sysv/linux/powerpc/powerpc64/be/libc.abilist
@@ -2048,6 +2048,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _IO_fprintf F
GLIBC_2.4 _IO_printf F
GLIBC_2.4 _IO_sprintf F
diff --git a/sysdeps/unix/sysv/linux/powerpc/powerpc64/le/libc.abilist b/sysdeps/unix/sysv/linux/powerpc/powerpc64/le/libc.abilist
index 2229a1dcc0..608ad49593 100644
--- a/sysdeps/unix/sysv/linux/powerpc/powerpc64/le/libc.abilist
+++ b/sysdeps/unix/sysv/linux/powerpc/powerpc64/le/libc.abilist
@@ -2247,3 +2247,4 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist
index 31010e6cf7..c7657ce7f6 100644
--- a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist
+++ b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist
@@ -2107,3 +2107,4 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
diff --git a/sysdeps/unix/sysv/linux/rseq-internal.h b/sysdeps/unix/sysv/linux/rseq-internal.h
new file mode 100644
index 0000000000..1dd3b9a968
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/rseq-internal.h
@@ -0,0 +1,77 @@
+/* Restartable Sequences internal API. Linux implementation.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef RSEQ_INTERNAL_H
+#define RSEQ_INTERNAL_H
+
+#include <sysdep.h>
+#include <errno.h>
+
+#ifdef __NR_rseq
+#include <sys/rseq.h>
+#endif
+
+#if defined __NR_rseq && defined RSEQ_SIG
+
+static inline int
+rseq_register_current_thread (void)
+{
+ int rc, ret = 0;
+ INTERNAL_SYSCALL_DECL (err);
+
+ if (__rseq_abi.cpu_id == RSEQ_CPU_ID_REGISTRATION_FAILED)
+ return -1;
+ rc = INTERNAL_SYSCALL_CALL (rseq, err, &__rseq_abi, sizeof (struct rseq),
+ 0, RSEQ_SIG);
+ if (!rc)
+ goto end;
+ if (INTERNAL_SYSCALL_ERRNO (rc, err) != EBUSY)
+ __rseq_abi.cpu_id = RSEQ_CPU_ID_REGISTRATION_FAILED;
+ ret = -1;
+end:
+ return ret;
+}
+
+static inline int
+rseq_unregister_current_thread (void)
+{
+ int rc, ret = 0;
+ INTERNAL_SYSCALL_DECL (err);
+
+ rc = INTERNAL_SYSCALL_CALL (rseq, err, &__rseq_abi, sizeof (struct rseq),
+ RSEQ_FLAG_UNREGISTER, RSEQ_SIG);
+ if (!rc)
+ goto end;
+ ret = -1;
+end:
+ return ret;
+}
+#else
+static inline int
+rseq_register_current_thread (void)
+{
+ return -1;
+}
+
+static inline int
+rseq_unregister_current_thread (void)
+{
+ return -1;
+}
+#endif
+
+#endif /* rseq-internal.h */
diff --git a/sysdeps/unix/sysv/linux/rseq-sym.c b/sysdeps/unix/sysv/linux/rseq-sym.c
new file mode 100644
index 0000000000..f86869a380
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/rseq-sym.c
@@ -0,0 +1,43 @@
+/* Restartable Sequences exported symbols. Linux Implementation.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <sys/syscall.h>
+#include <stdint.h>
+
+#ifdef __NR_rseq
+#include <sys/rseq.h>
+#else
+
+enum rseq_cpu_id_state {
+ RSEQ_CPU_ID_UNINITIALIZED = -1,
+ RSEQ_CPU_ID_REGISTRATION_FAILED = -2,
+};
+
+/* linux/rseq.h defines struct rseq as aligned on 32 bytes. The kernel ABI
+ size is 20 bytes. */
+struct rseq {
+ uint32_t cpu_id_start;
+ uint32_t cpu_id;
+ uint64_t rseq_cs;
+ uint32_t flags;
+} __attribute__ ((aligned(4 * sizeof(uint64_t))));
+
+#endif
+
+__thread struct rseq __rseq_abi = {
+ .cpu_id = RSEQ_CPU_ID_UNINITIALIZED,
+};
diff --git a/sysdeps/unix/sysv/linux/s390/bits/rseq.h b/sysdeps/unix/sysv/linux/s390/bits/rseq.h
new file mode 100644
index 0000000000..453250d761
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/s390/bits/rseq.h
@@ -0,0 +1,37 @@
+/* Restartable Sequences Linux s390 architecture header.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+# error "Never use <bits/rseq.h> directly; include <sys/rseq.h> instead."
+#endif
+
+/* RSEQ_SIG is a signature required before each abort handler code.
+
+ It is a 32-bit value that maps to actual architecture code compiled
+ into applications and libraries. It needs to be defined for each
+ architecture. When choosing this value, it needs to be taken into
+ account that generating invalid instructions may have ill effects on
+ tools like objdump, and may also have impact on the CPU speculative
+ execution efficiency in some cases.
+
+ RSEQ_SIG uses the trap4 instruction. As Linux does not make use of the
+ access-register mode nor the linkage stack this instruction will always
+ cause a special-operation exception (the trap-enabled bit in the DUCT
+ is and will stay 0). The instruction pattern is
+ b2 ff 0f ff trap4 4095(%r0) */
+
+#define RSEQ_SIG 0xB2FF0FFF
diff --git a/sysdeps/unix/sysv/linux/s390/s390-32/libc.abilist b/sysdeps/unix/sysv/linux/s390/s390-32/libc.abilist
index 4feca641b0..9c29ec0d2d 100644
--- a/sysdeps/unix/sysv/linux/s390/s390-32/libc.abilist
+++ b/sysdeps/unix/sysv/linux/s390/s390-32/libc.abilist
@@ -2183,6 +2183,7 @@ GLIBC_2.30 twalk_r F
GLIBC_2.31 msgctl F
GLIBC_2.31 semctl F
GLIBC_2.31 shmctl F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _IO_fprintf F
GLIBC_2.4 _IO_printf F
GLIBC_2.4 _IO_sprintf F
diff --git a/sysdeps/unix/sysv/linux/s390/s390-64/libc.abilist b/sysdeps/unix/sysv/linux/s390/s390-64/libc.abilist
index efe588a072..99424ceac9 100644
--- a/sysdeps/unix/sysv/linux/s390/s390-64/libc.abilist
+++ b/sysdeps/unix/sysv/linux/s390/s390-64/libc.abilist
@@ -2084,6 +2084,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _IO_fprintf F
GLIBC_2.4 _IO_printf F
GLIBC_2.4 _IO_sprintf F
diff --git a/sysdeps/unix/sysv/linux/sh/be/libc.abilist b/sysdeps/unix/sysv/linux/sh/be/libc.abilist
index 6bfc2b7439..80198de5d6 100644
--- a/sysdeps/unix/sysv/linux/sh/be/libc.abilist
+++ b/sysdeps/unix/sysv/linux/sh/be/libc.abilist
@@ -2053,6 +2053,7 @@ GLIBC_2.30 twalk_r F
GLIBC_2.31 msgctl F
GLIBC_2.31 semctl F
GLIBC_2.31 shmctl F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/sh/le/libc.abilist b/sysdeps/unix/sysv/linux/sh/le/libc.abilist
index 4b057bf4a2..916aa0b7f0 100644
--- a/sysdeps/unix/sysv/linux/sh/le/libc.abilist
+++ b/sysdeps/unix/sysv/linux/sh/le/libc.abilist
@@ -2050,6 +2050,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/sparc/sparc32/libc.abilist b/sysdeps/unix/sysv/linux/sparc/sparc32/libc.abilist
index 49cd597fd6..9a27df8e43 100644
--- a/sysdeps/unix/sysv/linux/sparc/sparc32/libc.abilist
+++ b/sysdeps/unix/sysv/linux/sparc/sparc32/libc.abilist
@@ -2174,6 +2174,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 _IO_fprintf F
GLIBC_2.4 _IO_printf F
GLIBC_2.4 _IO_sprintf F
diff --git a/sysdeps/unix/sysv/linux/sparc/sparc64/libc.abilist b/sysdeps/unix/sysv/linux/sparc/sparc64/libc.abilist
index 95e68e0ba1..32908666c4 100644
--- a/sysdeps/unix/sysv/linux/sparc/sparc64/libc.abilist
+++ b/sysdeps/unix/sysv/linux/sparc/sparc64/libc.abilist
@@ -2101,6 +2101,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/sys/rseq.h b/sysdeps/unix/sysv/linux/sys/rseq.h
new file mode 100644
index 0000000000..e675219ace
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/sys/rseq.h
@@ -0,0 +1,30 @@
+/* Restartable Sequences exported symbols. Linux header.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+#define _SYS_RSEQ_H 1
+
+/* We use the structures declarations from the kernel headers. */
+#include <linux/rseq.h>
+/* Architecture-specific rseq signature. */
+#include <bits/rseq.h>
+#include <stdint.h>
+
+extern __thread struct rseq __rseq_abi
+__attribute__ ((tls_model ("initial-exec")));
+
+#endif /* sys/rseq.h */
diff --git a/sysdeps/unix/sysv/linux/x86/bits/rseq.h b/sysdeps/unix/sysv/linux/x86/bits/rseq.h
new file mode 100644
index 0000000000..a2918c4617
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/x86/bits/rseq.h
@@ -0,0 +1,30 @@
+/* Restartable Sequences Linux x86 architecture header.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_RSEQ_H
+# error "Never use <bits/rseq.h> directly; include <sys/rseq.h> instead."
+#endif
+
+/* RSEQ_SIG is a signature required before each abort handler code.
+
+ RSEQ_SIG is used with the following reserved undefined instructions, which
+ trap in user-space:
+
+ x86-32: 0f b9 3d 53 30 05 53 ud1 0x53053053,%edi
+ x86-64: 0f b9 3d 53 30 05 53 ud1 0x53053053(%rip),%edi */
+
+#define RSEQ_SIG 0x53053053
diff --git a/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist b/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist
index 1f2dbd1451..7366565608 100644
--- a/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist
+++ b/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist
@@ -2059,6 +2059,7 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
GLIBC_2.4 __confstr_chk F
GLIBC_2.4 __fgets_chk F
GLIBC_2.4 __fgets_unlocked_chk F
diff --git a/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist b/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist
index 59da85a5d8..c1aa86f06e 100644
--- a/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist
+++ b/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist
@@ -2158,3 +2158,4 @@ GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
GLIBC_2.30 twalk_r F
+GLIBC_2.31 __rseq_abi T 0x20
--
2.17.1
^ permalink raw reply related
* Re: [PATCH v6 00/10] proc: modernize proc to support multiple private instances
From: Alexey Gladkov @ 2020-01-08 10:37 UTC (permalink / raw)
To: Alexey Dobriyan
Cc: LKML, Kernel Hardening, Linux API, Linux FS Devel,
Linux Security Module, Akinobu Mita, Alexander Viro,
Andrew Morton, Andy Lutomirski, Daniel Micay, Djalal Harouni,
Dmitry V . Levin, Eric W . Biederman, Greg Kroah-Hartman,
Ingo Molnar, J . Bruce Fields, Jeff Layton, Jonathan Corbet
In-Reply-To: <20200106151514.GA382@avx2>
On Mon, Jan 06, 2020 at 06:15:14PM +0300, Alexey Dobriyan wrote:
> > hidepid= Set /proc/<pid>/ access mode.
> > gid= Set the group authorized to learn processes information.
> > + pidonly= Show only task related subset of procfs.
>
> I'd rather have
>
> mount -t proc -o set=pid
This is a great idea.
> so that is can be naturally extended to
>
> mount -t proc -o set=pid,sysctl,misc
>
> > +static int proc_dir_open(struct inode *inode, struct file *file)
> > +{
> > + struct proc_fs_info *fs_info = proc_sb_info(inode->i_sb);
> > +
> > + if (proc_fs_pidonly(fs_info) == PROC_PIDONLY_ON)
> > + return -ENOENT;
> > +
> > + return 0;
> > +}
> > +
> > /*
> > * These are the generic /proc directory operations. They
> > * use the in-memory "struct proc_dir_entry" tree to parse
> > @@ -338,6 +357,7 @@ static const struct file_operations proc_dir_operations = {
> > .llseek = generic_file_llseek,
> > .read = generic_read_dir,
> > .iterate_shared = proc_readdir,
> > + .open = proc_dir_open,
>
> This should not be necessary: if lookup and readdir filters work
> then ->open can't happen.
Yes you are right.
> > --- a/include/linux/proc_fs.h
> > +++ b/include/linux/proc_fs.h
> > +/* definitions for hide_pid field */
> > +enum {
> > + HIDEPID_OFF = 0,
> > + HIDEPID_NO_ACCESS = 1,
> > + HIDEPID_INVISIBLE = 2,
> > + HIDEPID_NOT_PTRACABLE = 3, /* Limit pids to only ptracable pids */
> > +};
>
> These should live in uapi/ as they _are_ user interface to mount().
OK.
What do you think, maybe it's better to make these values a mask ?
I mean:
#define HIDEPID_OFF 0
#define HIDEPID_NO_ACCESS 1
#define HIDEPID_INVISIBLE 2
#define HIDEPID_NOT_PTRACABLE 4
In this case, if in the future there appear values that can be combined,
then there will be no need to make additional parameters.
--
Rgrds, legion
^ permalink raw reply
* Re: [PATCH RFC 1/1] mount: universally disallow mounting over symlinks
From: Andy Lutomirski @ 2020-01-08 4:39 UTC (permalink / raw)
To: Aleksa Sarai
Cc: Linus Torvalds, Al Viro, David Howells, Eric Biederman, stable,
Christian Brauner, Serge Hallyn, dev, Linux Containers, Linux API,
linux-fsdevel, Linux Kernel Mailing List
In-Reply-To: <20191230082847.dkriyisvu7wwxqqu@yavin.dot.cyphar.com>
On Mon, Dec 30, 2019 at 12:29 AM Aleksa Sarai <cyphar@cyphar.com> wrote:
>
> On 2019-12-29, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> > On Sun, Dec 29, 2019 at 9:21 PM Aleksa Sarai <cyphar@cyphar.com> wrote:
>
> If allowing bind-mounts over symlinks is allowed (which I don't have a
> problem with really), it just means we'll need a few more kernel pieces
> to get this hardening to work. But these features would be useful
> outside of the problems I'm dealing with (O_EMPTYPATH and some kind of
> pidfd-based interface to grab the equivalent of /proc/self/exe and a few
> other such magic-link targets).
As one data point, I would use this ability in virtme: this would
allow me to more reliably mount over /etc/resolve.conf even when it's
a symlink.
(Perhaps I should use overlayfs instead. Hmm.)
^ permalink raw reply
* Re: [PATCH RFC 0/1] mount: universally disallow mounting over symlinks
From: Linus Torvalds @ 2020-01-08 3:54 UTC (permalink / raw)
To: Al Viro
Cc: Aleksa Sarai, David Howells, Eric Biederman, stable,
Christian Brauner, Serge Hallyn, dev, Linux Containers, Linux API,
linux-fsdevel, Linux Kernel Mailing List, Ian Kent
In-Reply-To: <20200108031314.GE8904@ZenIV.linux.org.uk>
On Tue, Jan 7, 2020 at 7:13 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> FWIW, I suspect that we want to do something along the following lines:
>
> 1) make build_open_flags() treat O_CREAT | O_EXCL as if there had been
> O_NOFOLLOW in the mix.
My reaction to that is "Whee, that's a big change".
But:
> Benefit: this fragment in do_last()
you're right.
That's the semantics we have right now (and I think it's the correct
safe semantics when I think about it). But when I first looked at your
email I without thinking more about it actually thought we followed
the symlink, and then did the O_CREAT | O_EXCL on the target (and
potentially succeeded).
So I agree - making O_CREAT | O_EXCL imply O_NOFOLLOW seems to be the
right thing to do, and not only should simplify our code, it's much
more descriptive of what the real semantics are.
Even if my first reaction was that it would act differently.
Slash-and-burn approach to your explanatory subsequent steps:
> 2) make follow_managed() take &inode and &seq.
> 3) have the followup to failing __follow_mount_rcu() taken into it.
> 4) fold __follow_mount_rcu() into follow_managed(), using the latter both in
> RCU and non-RCU cases.
> 5) take the calls of follow_managed() out of lookup_fast() into its callers.
> 6) after that we have 3 callers of step_into(); [..]
> So if we manage to convert the damn thing in mountpoint_last() into
> follow_managed(), we could fold follow_managed() into step_into().
I think that all makes sense. I didn't go to look at the source, but
from the email contents your steps look reasonable to me.
> Another interesting question is whether we want O_PATH open
> to trigger automounts.
It does sound like they shouldn't, but as you say:
> The thing is, we do *NOT* trigger them
> (or traverse mountpoints) at the starting point of lookups.
> I believe it's a mistake (and mine, at that), but I doubt that
> there's anything that can be done about it at that point.
> It's a user-visible behaviour [..]
Hmm. I wonder how set in stone that is. We may have two decades of
history of not doing it at start point of lookups, but we do *not*
have two decades of history of O_PATH.
So what I think we agree would be sane behavior would be for O_PATH
opens to not trigger automounts (unless there's a slash at the end,
whatever), but _do_ add the mount-point traversal to the beginning of
lookups.
But only do it for the actual O_PATH fd case, not the cwd/root/non-O_PATH case.
That way we maintain original behavior: if somebody overmounts your
cwd, you still see the pre-mount directory on lookups, because your
cwd is "under" the mount.
But if you open a file with O_PATH, and somebody does a mount
_afterwards_, the openat() will see that later mount and/or do the
automount.
Don't you think that would be the more sane/obvious semantics of how
O_PATH should work?
> I think the easiest way to handle that is to have O_PATH
> turn LOOKUP_AUTOMOUNT, same as the normal open() does. That's
> trivial to do, but that changes user-visible behaviour. OTOH,
> with the current behaviour nobody can rely upon automount not
> getting triggered by somebody else just as they are entering
> their open(dir, O_PATH), so I think that's not a problem.
>
> Linus, do you have any objections to such O_PATH semantics
> change?
See above: I think I'd prefer the O_PATH behavior the other way
around. That seems to be more of a consistent behavior of what
"O_PATH" means - it means "don't really open, we'll do it only when
you use it as a directory".
But I don't have any _strong_ opinions. If you have a good reason to
tell me that I'm being stupid, go ahead and do so and override my
stupidity.
Linus
^ permalink raw reply
* Re: [PATCH RFC 0/1] mount: universally disallow mounting over symlinks
From: Al Viro @ 2020-01-08 3:13 UTC (permalink / raw)
To: Aleksa Sarai
Cc: David Howells, Eric Biederman, Linus Torvalds, stable,
Christian Brauner, Serge Hallyn, dev, containers, linux-api,
linux-fsdevel, linux-kernel, Ian Kent
In-Reply-To: <20200103014901.GC8904@ZenIV.linux.org.uk>
On Fri, Jan 03, 2020 at 01:49:01AM +0000, Al Viro wrote:
> It's a mess, again in mountpoint_last(). FWIW, at some point I proposed
> to have nd_jump_link() to fail with -ELOOP if the target was a symlink;
> Linus asked for reasons deeper than my dislike of the semantics, I looked
> around and hadn't spotted anything. And there hadn't been at the time,
> but when four months later umount_lookup_last() went in I failed to look
> for that source of potential problems in it ;-/
>
> I've looked at that area again now. Aside of usual cursing at do_last()
> horrors (yes, its control flow is a horror; yes, it needs serious massage;
> no, it's not a good idea to get sidetracked into that right now), there
> are several fun questions:
> * d_manage() and d_automount(). We almost certainly don't
> want those for autofs on the final component of pathname in umount,
> including the trailing symlinks. But do we want those on usual access
> via /proc/*/fd/*? I.e. suppose somebody does open() (O_PATH or not)
> in autofs; do we want ->d_manage()/->d_automount() called when
> resolving /proc/self/fd/<whatever>/foo/bar? We do not; is that
> correct from autofs point of view? I suspect that refusing to
> do ->d_automount() is correct, but I don't understand ->d_manage()
> purpose well enough to tell.
> * I really hope that the weird "trailing / forces automount
> even in cases when we normally wouldn't trigger it" (stat /mnt/foo
> vs. stat /mnt/foo/) is not meant to extend to umount. I'd like
> Ian's confirmation, though.
> * do we want ->d_manage() on following .. into overmounted
> directory? Again, autofs question...
FWIW, I suspect that we want to do something along the following lines:
1) make build_open_flags() treat O_CREAT | O_EXCL as if there had been
O_NOFOLLOW in the mix. Reason: if there is a trailing symlink, we want
to fail with EEXIST anyway. Benefit: this fragment in do_last()
error = follow_managed(&path, nd);
if (unlikely(error < 0))
return error;
/*
* create/update audit record if it already exists.
*/
audit_inode(nd->name, path.dentry, 0);
if (unlikely((open_flag & (O_EXCL | O_CREAT)) == (O_EXCL | O_CREAT))) {
path_to_nameidata(&path, nd);
return -EEXIST;
}
seq = 0; /* out of RCU mode, so the value doesn't matter */
inode = d_backing_inode(path.dentry);
finish_lookup:
error = step_into(nd, &path, 0, inode, seq);
if (unlikely(error))
return error;
can become
error = follow_managed(&path, nd);
if (unlikely(error < 0))
return error;
seq = 0; /* out of RCU mode, so the value doesn't matter */
inode = d_backing_inode(path.dentry);
finish_lookup:
error = step_into(nd, &path, 0, inode, seq);
if (unlikely(error))
return error;
if (unlikely((open_flag & (O_EXCL | O_CREAT)) == (O_EXCL | O_CREAT))) {
audit_inode(nd->name, nd->path.dentry, 0);
return -EEXIST;
}
Equivalent transformation, since the the only goto finish_lookup; is under
if (!(open_flag & O_CREAT)). What it buys us is more regular structure
of follow_managed() callers.
2) make follow_managed() take &inode and &seq. Look: follow_managed() never
returns 0 (we have
if (ret == -EISDIR || !ret)
ret = 1;
on the way to the only return in it) and the callers are
err = follow_managed(path, nd);
if (likely(err > 0))
*inode = d_backing_inode(path->dentry);
return err;
in lookup_fast(),
err = follow_managed(&path, nd);
if (unlikely(err < 0))
return err;
seq = 0; /* we are already out of RCU mode */
inode = d_backing_inode(path.dentry);
in walk_component(),
err = follow_managed(&path, nd);
if (unlikely(err < 0))
return err;
inode = d_backing_inode(path.dentry);
seq = 0;
in handle_lookup_down() and (after the previous change)
error = follow_managed(&path, nd);
if (unlikely(error < 0))
return error;
seq = 0; /* out of RCU mode, so the value doesn't matter */
inode = d_backing_inode(path.dentry);
in do_last(). That's begging to fold those followups into follow_managed()
itself, doesn't it? And having *seqp = 0; equivalent added in lookup_fast()
is not going to hurt the performance - in all callers it's an address of
local variable, right next to the one whose address is passed as inodep.
Which we'd just dirtied, and the cacheline is not going to have been shared
anyway.
Note that after that the arguments for follow_managed() become identical
to those for __follow_mount_rcu(). Which makes a lot of sense, since
the latter is RCU-mode counterpart of the former.
3) have the followup to failing __follow_mount_rcu() taken into it.
After (2) we have this in lookup_fast():
*seqp = seq;
status = d_revalidate(dentry, nd->flags);
if (likely(status > 0)) {
/*
* Note: do negative dentry check after revalidation in
* case that drops it.
*/
if (unlikely(negative))
return -ENOENT;
path->mnt = mnt;
path->dentry = dentry;
if (likely(__follow_mount_rcu(nd, path, inode, seqp)))
return 1;
}
if (unlazy_child(nd, dentry, seq))
return -ECHILD;
if (unlikely(status == -ECHILD))
/* we'd been told to redo it in non-rcu mode */
status = d_revalidate(dentry, nd->flags);
} else {
...
}
if (unlikely(status <= 0)) {
if (!status)
d_invalidate(dentry);
dput(dentry);
return status;
}
path->mnt = mnt;
path->dentry = dentry;
return follow_managed(path, nd, inode, seqp);
Suppose __follow_mount_rcu() returns false; what follows is
if (unlazy_child(nd, dentry, seq))
return -ECHILD;
seq here is equal to *seqp here, dentry - the value of path->dentry at the
time of __follow_mount_rcu() call.
if (unlikely(status == -ECHILD))
....
not taken - we know that status must have been positive
if (unlikely(status <= 0)) {
...
}
ditto
path->mnt = mnt;
path->dentry = dentry;
return follow_managed(path, nd, inode, seqp);
we return *path to original and call follow_managed(). IOW, we could bloody
well do all of that in the __follow_mount_rcu() itself, having it return 1
when the original would've returned true and doing that "revert *path,
call unlazy_child() and fall back to follow_mount_rcu() in case of success"
in cases when the original would've returned false. The caller turns into
/*
* Note: do negative dentry check after revalidation in
* case that drops it.
*/
if (unlikely(negative))
return -ENOENT;
path->mnt = mnt;
path->dentry = dentry;
return __follow_mount_rcu(nd, path, inode, seqp);
4) fold __follow_mount_rcu() into follow_managed(), using the latter both in
RCU and non-RCU cases.
5) take the calls of follow_managed() out of lookup_fast() into its callers.
That would be
err = lookup_fast(nd, &path, &inode, &seq);
if (unlikely(err <= 0)) {
if (err < 0)
return err;
path.dentry = lookup_slow(&nd->last, nd->path.dentry,
nd->flags);
if (IS_ERR(path.dentry))
return PTR_ERR(path.dentry);
path.mnt = nd->path.mnt;
err = follow_managed(&path, nd, &inode, &seq);
if (unlikely(err < 0))
return err;
}
turning into
err = lookup_fast(nd, &path, &inode, &seq);
if (unlikely(err <= 0)) {
if (err < 0)
return err;
path.dentry = lookup_slow(&nd->last, nd->path.dentry,
nd->flags);
if (IS_ERR(path.dentry))
return PTR_ERR(path.dentry);
path.mnt = nd->path.mnt;
}
err = follow_managed(&path, nd, &inode, &seq);
if (unlikely(err < 0))
return err;
in walk_component() and
error = lookup_fast(nd, &path, &inode, &seq);
if (likely(error > 0))
goto finish_lookup;
...
error = follow_managed(&path, nd, &inode, &seq);
if (unlikely(error < 0))
return error;
finish_lookup:
turning into
error = lookup_fast(nd, &path, &inode, &seq);
if (likely(error > 0))
goto finish_lookup;
...
finish_lookup:
error = follow_managed(&path, nd, &inode, &seq);
if (unlikely(error < 0))
return error;
in do_last().
6) after that we have 3 callers of step_into(); the ones in
walk_component() and in do_last() would be immediately preceded
by the calls of follow_managed(). The last one is in
mountpoint_last(). That's
if (d_flags_negative(smp_load_acquire(&path.dentry->d_flags))) {
dput(path.dentry);
return -ENOENT;
}
path.mnt = nd->path.mnt;
return step_into(nd, &path, 0, d_backing_inode(path.dentry), 0);
And that's where we are missing the mountpoint traversal in symlink case -
sure, the caller does follow_mount(), but it doesn't catch the case when
we have a symlink overmounted - we run into step_into() before that.
Note that smp_load_acquire + d_flags_negative is what we would've done
in follow_managed(), as well as getting d_backing_inode(). So here
we also have an open-coded bastardized variant of follow_managed().
The difference is, we don't want to trigger ->d_automount() and ->d_manage()
in that one.
And at that point the only call of follow_managed() *NOT* followed by
step_into() is in handle_lookup_down(). What it is followed by is
path_to_nameidata(&path, nd);
nd->inode = inode;
nd->seq = seq;
And that's a piece of step_into():
if (likely(!d_is_symlink(path->dentry)) ||
!(flags & WALK_FOLLOW || nd->flags & LOOKUP_FOLLOW)) {
/* not a symlink or should not follow */
path_to_nameidata(path, nd);
nd->inode = inode;
nd->seq = seq;
return 0;
}
is the normal path through that sucker. What's more, we are guaranteed
that this will _not_ be a symlink (it's the starting point of pathwalk,
and path_init() would've told us to sod off were it not a directory).
So if we manage to convert the damn thing in mountpoint_last() into
follow_managed(), we could fold follow_managed() into step_into().
Which suggests the way to do that - not that step_into() takes an
argument containing ORed WALK_... constants. So we can simply add
WALK_NOAUTOMOUNT and put a check for it into
if (flags & DCACHE_MANAGE_TRANSIT) {
and
if (flags & DCACHE_NEED_AUTOMOUNT) {
bodies, so that they would be ignored if that's passed to
follow_mount()/step_into() hybrid.
At that point we have one primitive for moving into child, handling
both the mountpoint traversals and keeping track of symlinks. Moreover,
there's a fairly strong argument for using it in case of .. as well.
As it is, if the parent is overmounted, we cross into whatever is
mounted on top of it. And we ignore ->d_manage/->d_automount on
the damn thing. Which is not an issue for anything other than
autofs (nobody else has ->d_manage() and nfs/afs/cifs automount
points don't have children) and for autofs we *want* those called;
that's not something likely to be encountered, but it's an impossible
setup (autofs direct mount set on an ancestor of somebody's current
directory) and autofs does count upon not walking into something
being set up by the daemon.
I'll put together such series and see how well does it work; it would
fix the idiocies in user_path_mountpoint_at() and make the pathwalk
machinery easier to follow - the boilerplate around mountpoint
crossing and symlink handling is demonstrably easy to get wrong.
If that works and doesn't cause observable slowdown, I'll put it
into -next, either stepping around the changes done by openat2()
series, or rebasing it on top of that.
Another interesting question is whether we want O_PATH open
to trigger automounts. The thing is, we do *NOT* trigger them
(or traverse mountpoints) at the starting point of lookups.
I believe it's a mistake (and mine, at that), but I doubt that
there's anything that can be done about it at that point.
It's a user-visible behaviour and I can easily imagine
a custom /init that ends up relying upon it ;-/ mkdir /root,
mount the final root there, chdir /root, mount --move . /,
remove everything on initramfs using absolute pathnames
and chroot to "." to finish... Traversing mounts at the
beginning of pathwalk would break the hell out of that,
potentially with root filesystem contents wiped out... ;-/
I wish we could change that, but I'm afraid that's cast
in stone by now (and had been for 20 years or so). As it is,
we have an unpleasant side effect - O_PATH open does *NOT*
trigger automounts. So if you do that to e.g. referral point
and try to do ...at() syscalls with that as the origin, you'll
get an unpleasant surprise - automount won't trigger at all.
I think the easiest way to handle that is to have O_PATH
turn LOOKUP_AUTOMOUNT, same as the normal open() does. That's
trivial to do, but that changes user-visible behaviour. OTOH,
with the current behaviour nobody can rely upon automount not
getting triggered by somebody else just as they are entering
their open(dir, O_PATH), so I think that's not a problem.
Linus, do you have any objections to such O_PATH semantics
change?
PS: I think I see how to untangle the control flow horrors
in do_last() with this massage done, but I'm not going there
until this is sorted out - by previous experience touching
the damn thing can easily turn into several weeks of digging
through the nfs/gfs2/etc. guts trying to verify something,
with a couple of detours into fixing something in there
found in process... ;-/
^ permalink raw reply
* Re: [PATCH v3 8/8] random: Remove kernel.random.read_wakeup_threshold
From: Theodore Y. Ts'o @ 2020-01-07 21:04 UTC (permalink / raw)
To: Andy Lutomirski
Cc: LKML, Linux API, Kees Cook, Jason A. Donenfeld, Ahmed S. Darwish,
Lennart Poettering, Eric W. Biederman, Alexander E. Patrakov,
Michael Kerrisk, Willy Tarreau, Matthew Garrett,
Ext4 Developers List, linux-man, Stephan Mueller
In-Reply-To: <a74ed2cf0b5a5451428a246a9239f5bc4e29358f.1577088521.git.luto@kernel.org>
On Mon, Dec 23, 2019 at 12:20:51AM -0800, Andy Lutomirski wrote:
> It has no effect any more, so remove it. We can revert this if
> there is some user code that expects to be able to set this sysctl.
>
> Signed-off-by: Andy Lutomirski <luto@kernel.org>
Applied, thanks.
- Ted
^ permalink raw reply
* Re: [PATCH v3 7/8] random: Delete code to pull data into pools
From: Theodore Y. Ts'o @ 2020-01-07 21:03 UTC (permalink / raw)
To: Andy Lutomirski
Cc: LKML, Linux API, Kees Cook, Jason A. Donenfeld, Ahmed S. Darwish,
Lennart Poettering, Eric W. Biederman, Alexander E. Patrakov,
Michael Kerrisk, Willy Tarreau, Matthew Garrett,
Ext4 Developers List, linux-man, Stephan Mueller
In-Reply-To: <4a05fe0c7a5c831389ef4aea51d24528ac8682c7.1577088521.git.luto@kernel.org>
On Mon, Dec 23, 2019 at 12:20:50AM -0800, Andy Lutomirski wrote:
> There is no pool that pulls, so it was just dead code.
>
> Signed-off-by: Andy Lutomirski <luto@kernel.org>
Applied, thanks.
- Ted
^ permalink raw reply
* Re: [PATCH v3 6/8] random: Remove the blocking pool
From: Theodore Y. Ts'o @ 2020-01-07 21:03 UTC (permalink / raw)
To: Andy Lutomirski
Cc: LKML, Linux API, Kees Cook, Jason A. Donenfeld, Ahmed S. Darwish,
Lennart Poettering, Eric W. Biederman, Alexander E. Patrakov,
Michael Kerrisk, Willy Tarreau, Matthew Garrett,
Ext4 Developers List, linux-man, Stephan Mueller
In-Reply-To: <511225a224bf0a291149d3c0b8b45393cd03ab96.1577088521.git.luto@kernel.org>
On Mon, Dec 23, 2019 at 12:20:49AM -0800, Andy Lutomirski wrote:
> There is no longer any interface to read data from the blocking
> pool, so remove it.
>
> This enables quite a bit of code deletion, much of which will be
> done in subsequent patches.
>
> Signed-off-by: Andy Lutomirski <luto@kernel.org>
Applied, thanks.
- Ted
^ permalink raw reply
* Re: [PATCH v3 5/8] random: Make /dev/random be almost like /dev/urandom
From: Theodore Y. Ts'o @ 2020-01-07 21:02 UTC (permalink / raw)
To: Andy Lutomirski
Cc: LKML, Linux API, Kees Cook, Jason A. Donenfeld, Ahmed S. Darwish,
Lennart Poettering, Eric W. Biederman, Alexander E. Patrakov,
Michael Kerrisk, Willy Tarreau, Matthew Garrett,
Ext4 Developers List, linux-man, Stephan Mueller
In-Reply-To: <5e6ac8831c6cf2e56a7a4b39616d1732b2bdd06c.1577088521.git.luto@kernel.org>
On Mon, Dec 23, 2019 at 12:20:48AM -0800, Andy Lutomirski wrote:
> This patch changes the read semantics of /dev/random to be the same
> as /dev/urandom except that reads will block until the CRNG is
> ready.
>
> None of the cleanups that this enables have been done yet. As a
> result, this gives a warning about an unused function.
>
> Signed-off-by: Andy Lutomirski <luto@kernel.org>
Applied, thanks.
- Ted
^ permalink raw reply
* Re: [PATCH v9 0/4] Add pidfd_getfd syscall
From: Christian Brauner @ 2020-01-07 20:54 UTC (permalink / raw)
To: Sargun Dhillon
Cc: linux-kernel, containers, linux-api, linux-fsdevel, tycho, jannh,
cyphar, oleg, luto, viro, gpascutto, ealvarez, fweimer, jld, arnd
In-Reply-To: <20200107175927.4558-1-sargun@sargun.me>
On Tue, Jan 07, 2020 at 09:59:23AM -0800, Sargun Dhillon wrote:
> This patchset introduces a mechanism (pidfd_getfd syscall) to get file
> descriptors from other processes via pidfd. Although this can be achieved
> using SCM_RIGHTS, and parasitic code injection, this offers a more
> straightforward mechanism, with less overhead and complexity. The process
> under manipulation's fd still remains valid, and unmodified by the
> copy operation.
>
> It introduces a flags field. The flags field is reserved a the moment,
> but the intent is to extend it with the following capabilities:
> * Close the remote FD when copying it
> * Drop the cgroup data if it's a fd pointing a socket when copying it
>
> The syscall numbers were chosen to be one greater than openat2.
>
> Summary of history:
> This initially started as a ptrace command. It did not require the process
> to be stopped, and felt like kind of an awkward fit for ptrace. After that,
> it moved to an ioctl on the pidfd. Given the core functionality, it made
> sense to make it a syscall which did not require the process to be stopped.
>
> Previous versions:
> V8: https://lore.kernel.org/lkml/20200103162928.5271-1-sargun@sargun.me/
> V7: https://lore.kernel.org/lkml/20191226180227.GA29389@ircssh-2.c.rugged-nimbus-611.internal/
> V6: https://lore.kernel.org/lkml/20191223210823.GA25083@ircssh-2.c.rugged-nimbus-611.internal/
> V5: https://lore.kernel.org/lkml/20191220232746.GA20215@ircssh-2.c.rugged-nimbus-611.internal/
> V4: https://lore.kernel.org/lkml/20191218235310.GA17259@ircssh-2.c.rugged-nimbus-611.internal/
> V3: https://lore.kernel.org/lkml/20191217005842.GA14379@ircssh-2.c.rugged-nimbus-611.internal/
> V2: https://lore.kernel.org/lkml/20191209070446.GA32336@ircssh-2.c.rugged-nimbus-611.internal/
> RFC V1: https://lore.kernel.org/lkml/20191205234450.GA26369@ircssh-2.c.rugged-nimbus-611.internal/
I don't see anything wrong with this series anymore:
Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
Other Acked-bys/Reviewed-bys and reviews of course strongly encouraged!
Christian
^ permalink raw reply
* Re: [PATCH v3 4/8] random: Ignore GRND_RANDOM in getentropy(2)
From: Theodore Y. Ts'o @ 2020-01-07 20:44 UTC (permalink / raw)
To: Andy Lutomirski
Cc: LKML, Linux API, Kees Cook, Jason A. Donenfeld, Ahmed S. Darwish,
Lennart Poettering, Eric W. Biederman, Alexander E. Patrakov,
Michael Kerrisk, Willy Tarreau, Matthew Garrett,
Ext4 Developers List, linux-man, Stephan Mueller
In-Reply-To: <705c5a091b63cc5da70c99304bb97e0109be0a26.1577088521.git.luto@kernel.org>
On Mon, Dec 23, 2019 at 12:20:47AM -0800, Andy Lutomirski wrote:
> The separate blocking pool is going away. Start by ignoring
> GRND_RANDOM in getentropy(2).
>
> This should not materially break any API. Any code that worked
> without this change should work at least as well with this change.
>
> Signed-off-by: Andy Lutomirski <luto@kernel.org>
Applied, thanks.
- Ted
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox