Re: [patch 09/10] rseq: Reenable performance optimizations conditionally

Dmitry Vyukov
2026-04-29T09:35:40Z

On Wed, 29 Apr 2026 at 01:34, Thomas Gleixner <tglx@kernel.org> wrote:
>
> Due to the incompatibility with TCMalloc the RSEQ optimizations and
> extended features (time slice extensions) have been disabled and made
> run-time conditional.
>
> The original RSEQ implementation, which TCMalloc depends on, registers a 32
> byte region (ORIG_RSEG_SIZE). This region has a 32 byte alignment
> requirement.
>
> The extension safe newer variant exposes the kernel RSEQ feature size via
> getauxval(AT_RSEQ_FEATURE_SIZE) and the alignment requirement via
> getauxval(AT_RSEQ_ALIGN). The alignment requirement is that the registered
> RSEQ region is aligned to the next power of two of the feature size. The
> kernel currently has a feature size of 33 bytes, which means the alignment
> requirement is 64 bytes.
>
> The TCMalloc RSEQ region is embedded into a cache line aligned data
> structure starting at offset 32 bytes so that bytes 28-31 and the
> cpu_id_start field at bytes 32-35 form a 64-bit little endian pointer with
> the top-most bit (63 set) to check whether the kernel has overwritten
> cpu_id_start with an actual CPU id value, which is guaranteed to not have
> the top most bit set.
>
> As this is part of their performance tuned magic, it's a pretty safe
> assumption, that TCMalloc won't use a larger RSEQ size.
>
> This allows the kernel to declare that registrations with a size greater
> than the original size of 32 bytes, which is the cases since time slice
> extensions got introduced, as RSEQ ABI v2 with the following differences to
> the original behaviour:
>
> 1) Unconditional updates of the user read only fields (CPU, node, MMCID)
> are removed. Those fields are only updated on registration, task
> migration and MMCID changes.
>
> 2) Unconditional evaluation of the criticial section pointer is
> removed. It's only evaluated when user space was interrupted and was
> scheduled out or before delivering a signal in the interrupted
> context.
>
> 3) The read/only requirement of the ID fields is enforced. When the
> kernel detects that userspace manipulated the fields, the process is
> terminated. This ensures that multiple entities (libraries) can
> utilize RSEQ without interfering.
>
> 4) Todays extended RSEQ feature (time slice extensions) and future
> extensions are only enabled in the v2 enabled mode.
>
> Registrations with the original size of 32 bytes operate in backwards
> compatible legacy mode without performance improvements and extended
> features.
>
> Unfortunately that also affects users of older GLIBC versions which
> register the original size of 32 bytes and do not evaluate the kernel
> required size in the auxiliary vector AT_RSEQ_FEATURE_SIZE.
>
> That's the result of the lack of enforcement in the original implementation
> and the unwillingness of a single entity to cooperate with the larger
> ecosystem for many years.
>
> Implement the required registration changes by restructuring the spaghetti
> code and adding the size/version check. Also add documentation about the
> differences of legacy and optimized RSEQ V2 mode.
>
> Thanks to Mathieu for pointing out the ORIG_RSEQ_SIZE constraints!
>
> Fixes: d6200245c75e ("rseq: Allow registering RSEQ with slice extension")
> Signed-off-by: Thomas Gleixner <tglx@kernel.org>
> Cc: stable@vger.kernel.org

Reviewed-by: Dmitry Vyukov <dvyukov@google.com>

> ---
> Documentation/userspace-api/rseq.rst | 94 ++++++++++++++++++++++
> kernel/rseq.c | 144 ++++++++++++++++++++---------------
> 2 files changed, 178 insertions(+), 60 deletions(-)
>
> --- a/Documentation/userspace-api/rseq.rst
> +++ b/Documentation/userspace-api/rseq.rst
> @@ -24,6 +24,97 @@ Quick access to CPU number, node ID
> Allows to implement per CPU data efficiently. Documentation is in code and
> selftests. :(
>
> +Optimized RSEQ V2
> +-----------------
> +
> +On architectures which utilize the generic entry code and generic TIF bits
> +the kernel supports runtime optimizations for RSEQ, which also enable
> +enhanced features like scheduler time slice extensions.
> +
> +To enable them a task has to register the RSEQ region with at least the
> +length advertised by getauxval(AT_RSEQ_FEATURE_SIZE).
> +
> +If existing binaries register with RSEQ_ORIG_SIZE (32 bytes), the kernel
> +keeps the legacy low performance mode enabled to fulfil the expectations
> +of existing users regarding the original RSEQ implementation behaviour.
> +
> +The following table documents the ABI and behavioral guarantees of the
> +legacy and the optimized V2 mode.
> +
> +.. list-table:: RSEQ modes
> + :header-rows: 1
> +
> + * - Nr
> + - What
> +
> + - Legacy
> + - Optimized V2
> +
> + * - 1
> + - The cpu_id_start, cpu_id, node_id and mm_cid fields (User mode read
> + only)
> + .. Legacy
> + - Updated by the kernel unconditionally after each context switch and
> + before signal delivery
> + .. Optimized V2
> + - Updated by the kernel if and only if they change, i.e. if the task
> + is migrated or mm_cid changes
> +
> + * - 2
> + - The rseq_cs critical section field
> + .. Legacy
> + - Evaluated and handled unconditionally after each context switch and
> + before signal delivery
> + .. Optimized V2
> + - Evaluated and handled conditionally only when user space was
> + interrupted and was scheduled out or before delivering a signal in
> + the interrupted context.
> +
> + * - 3
> + - Read only fields
> + .. Legacy
> + - No strict enforcement except in debug mode
> + .. Optimized V2
> + - Strict enforcement
> +
> + * - 4
> + - membarrier(...RSEQ)
> + .. Legacy
> + - All running threads of the process are interrupted and the ID fields
> + are rewritten and eventually active critical sections are aborted
> + before they return to user space. All threads which are scheduled
> + out whether voluntary or not are covered by #1/#2 above.
> + .. Optimized V2
> + - All running threads of the process are interrupted and eventually
> + active critical sections are aborted before these threads return to
> + user space. The ID fields are only updated if changed as a
> + consequence of the interrupt. All threads which are scheduled out
> + whether voluntary or not are covered by #1/#2 above.
> +
> + * - 5
> + - Time slice extensions
> + .. Legacy
> + - Not supported
> + .. Optimized V2
> + - Supported
> +
> +The legacy mode is obviously less performant as it does unconditional
> +updates and critical section checks even if not strictly required by the
> +ABI contract. That can't be changed anymore as some users depend on that
> +observed behavior, which in turn enables them to violate the ABI and
> +overwrite the cpu_id_start field for their own purposes. This is obviously
> +discouraged as it renders RSEQ incompatible with the intended usage and
> +breaks the expectation of other libraries in the same application.
> +
> +The ABI compliant optimized v2 mode, which respects the read only fields,
> +does not require unconditional updates and therefore is way more
> +performant. The kernel validates the read only fields for compliance. If
> +user space modifies them, the process is killed. Compliant usage allows
> +multiple libraries in the same application to benefit from the RSEQ
> +functionality without disturbing each other. The ABI compliant optimized v2
> +mode also enables extended RSEQ features like time slice extensions.
> +
> +
> Scheduler time slice extensions
> -------------------------------
>
> @@ -37,7 +128,8 @@ scheduled out inside of the critical sec
>
> * Enabled at boot time (default is enabled)
>
> - * A rseq userspace pointer has been registered for the thread
> + * A rseq userspace pointer has been registered for the thread in
> + optimized V2 mode
>
> The thread has to enable the functionality via prctl(2)::
>
> --- a/kernel/rseq.c
> +++ b/kernel/rseq.c
> @@ -413,70 +413,23 @@ static bool rseq_reset_ids(void)
> /* The original rseq structure size (including padding) is 32 bytes. */
> #define ORIG_RSEQ_SIZE 32
>
> -/*
> - * sys_rseq - setup restartable sequences for caller thread.
> - */
> -SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len, int, flags, u32, sig)
> +static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u32 sig)
> {
> u32 rseqfl = 0;
> u8 version = 1;
>
> - if (flags & RSEQ_FLAG_UNREGISTER) {
> - if (flags & ~RSEQ_FLAG_UNREGISTER)
> - return -EINVAL;
> - /* Unregister rseq for current thread. */
> - if (current->rseq.usrptr != rseq || !current->rseq.usrptr)
> - return -EINVAL;
> - if (rseq_len != current->rseq.len)
> - return -EINVAL;
> - if (current->rseq.sig != sig)
> - return -EPERM;
> - if (!rseq_reset_ids())
> - return -EFAULT;
> - rseq_reset(current);
> - return 0;
> - }
> -
> - if (unlikely(flags & ~(RSEQ_FLAG_SLICE_EXT_DEFAULT_ON)))
> - return -EINVAL;
> -
> - if (current->rseq.usrptr) {
> - /*
> - * If rseq is already registered, check whether
> - * the provided address differs from the prior
> - * one.
> - */
> - if (current->rseq.usrptr != rseq || rseq_len != current->rseq.len)
> - return -EINVAL;
> - if (current->rseq.sig != sig)
> - return -EPERM;
> - /* Already registered. */
> - return -EBUSY;
> - }
> -
> - /*
> - * If there was no rseq previously registered, ensure the provided rseq
> - * is properly aligned, as communcated to user-space through the ELF
> - * auxiliary vector AT_RSEQ_ALIGN. If rseq_len is the original rseq
> - * size, the required alignment is the original struct rseq alignment.
> - *
> - * The rseq_len is required to be greater or equal to the original rseq
> - * size. In order to be valid, rseq_len is either the original rseq size,
> - * or large enough to contain all supported fields, as communicated to
> - * user-space through the ELF auxiliary vector AT_RSEQ_FEATURE_SIZE.
> - */
> - if (rseq_len < ORIG_RSEQ_SIZE ||
> - (rseq_len == ORIG_RSEQ_SIZE && !IS_ALIGNED((unsigned long)rseq, ORIG_RSEQ_SIZE)) ||
> - (rseq_len != ORIG_RSEQ_SIZE && (!IS_ALIGNED((unsigned long)rseq, rseq_alloc_align()) ||
> - rseq_len < offsetof(struct rseq, end))))
> - return -EINVAL;
> if (!access_ok(rseq, rseq_len))
> return -EFAULT;
>
> /*
> - * The version check effectivly disables time slice extensions until the
> - * RSEQ ABI V2 registration are implemented.
> + * Architectures, which use the generic IRQ entry code (at least) enable
> + * registrations with a size greater than the original v1 fixed sized
> + * @rseq_len, which has been validated already to utilize the optimized
> + * v2 ABI mode which also enables extended RSEQ features beyond MMCID.
> */
> + if (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY) && rseq_len > ORIG_RSEQ_SIZE)
> + version = 2;
> +
> if (IS_ENABLED(CONFIG_RSEQ_SLICE_EXTENSION) && version > 1) {
> if (rseq_slice_extension_enabled()) {
> rseqfl |= RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE;
> @@ -524,11 +477,10 @@ SYSCALL_DEFINE4(rseq, struct rseq __user
> #endif
>
> /*
> - * If rseq was previously inactive, and has just been
> - * registered, ensure the cpu_id_start and cpu_id fields
> - * are updated before returning to user-space.
> + * Ensure the cpu_id_start and cpu_id fields are updated before
> + * returning to user-space.
> */
> - current->rseq.event.has_rseq = true;
> + current->rseq.event.has_rseq = version;
> rseq_force_update();
> return 0;
>
> @@ -536,6 +488,80 @@ SYSCALL_DEFINE4(rseq, struct rseq __user
> return -EFAULT;
> }
>
> +static long rseq_unregister(struct rseq __user * rseq, u32 rseq_len, int flags, u32 sig)
> +{
> + if (flags & ~RSEQ_FLAG_UNREGISTER)
> + return -EINVAL;
> + if (current->rseq.usrptr != rseq || !current->rseq.usrptr)
> + return -EINVAL;
> + if (rseq_len != current->rseq.len)
> + return -EINVAL;
> + if (current->rseq.sig != sig)
> + return -EPERM;
> + if (!rseq_reset_ids())
> + return -EFAULT;
> + rseq_reset(current);
> + return 0;
> +}
> +
> +static long rseq_reregister(struct rseq __user * rseq, u32 rseq_len, u32 sig)
> +{
> + /*
> + * If rseq is already registered, check whether the provided address
> + * differs from the prior one.
> + */
> + if (current->rseq.usrptr != rseq || rseq_len != current->rseq.len)
> + return -EINVAL;
> + if (current->rseq.sig != sig)
> + return -EPERM;
> + /* Already registered. */
> + return -EBUSY;
> +}
> +
> +static bool rseq_length_valid(struct rseq __user *rseq, unsigned int rseq_len)
> +{
> + /*
> + * Ensure the provided rseq is properly aligned, as communicated to
> + * user-space through the ELF auxiliary vector AT_RSEQ_ALIGN. If
> + * rseq_len is the original rseq size, the required alignment is the
> + * original struct rseq alignment.
> + *
> + * In order to be valid, rseq_len is either the original rseq size, or
> + * large enough to contain all supported fields, as communicated to
> + * user-space through the ELF auxiliary vector AT_RSEQ_FEATURE_SIZE.
> + */
> + if (rseq_len < ORIG_RSEQ_SIZE)
> + return false;
> +
> + if (rseq_len == ORIG_RSEQ_SIZE)
> + return IS_ALIGNED((unsigned long)rseq, ORIG_RSEQ_SIZE);
> +
> + return IS_ALIGNED((unsigned long)rseq, rseq_alloc_align()) &&
> + rseq_len >= offsetof(struct rseq, end);
> +}
> +
> +#define RSEQ_FLAGS_SUPPORTED (RSEQ_FLAG_SLICE_EXT_DEFAULT_ON)
> +
> +/*
> + * sys_rseq - Register or unregister restartable sequences for the caller thread.
> + */
> +SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, rseq_len, int, flags, u32, sig)
> +{
> + if (flags & RSEQ_FLAG_UNREGISTER)
> + return rseq_unregister(rseq, rseq_len, flags, sig);
> +
> + if (unlikely(flags & ~RSEQ_FLAGS_SUPPORTED))
> + return -EINVAL;
> +
> + if (current->rseq.usrptr)
> + return rseq_reregister(rseq, rseq_len, sig);
> +
> + if (!rseq_length_valid(rseq, rseq_len))
> + return -EINVAL;
> +
> + return rseq_register(rseq, rseq_len, flags, sig);
> +}
> +
> #ifdef CONFIG_RSEQ_SLICE_EXTENSION
> struct slice_timer {
> struct hrtimer timer;
>