Re: [PATCH v3 2/3] riscv: hwprobe: export the availability of vector to user

[Mark Harris <mark.hsj@gmail.com>]

Andy Chiu wrote:
> Userland IFUNC resolvers use hwprobe to decide whether to dispatch to
> vectorized routines. But RISCV_HWPROBE_KEY_IMA_EXT_0 only reports what
> is present in hardware, not what the calling process may actually use:
> when Vector is disabled for a process via
> prctl(PR_RISCV_V_SET_CONTROL, PR_RISCV_V_VSTATE_CTRL_OFF), it is still
> reported as present. A resolver that trusts this and runs a vector
> instruction crashes with SIGILL.
> 
> Add RISCV_HWPROBE_KEY_EXT_ENABLED, a positional modifier key that carries
> no value of its own. Within a single request, keys placed after it report
> extensions that are both present and enabled for the calling process,
> while keys before it keep reporting hardware presence. This masks out V
> and its V-dependent sub-extensions when V is disabled for the process, and
> lets userland obtain both views in one query:
> 
>       [ {IMA_EXT_0}, {EXT_ENABLED}, {IMA_EXT_0} ]
>           present       modifier       enabled
> 
> The enabled view depends on per-process state, so it cannot be served from
> the vDSO's process-independent cache; requests carrying the modifier are
> deferred to the syscall. Unknown keys are still reported as -1, so the
> feature is detectable and existing users are unaffected.

Currently, each key has a value and the order of the keys does not
matter.  If the interface needed to be extended to support
process-specific values, or thread-specific values (the prctl() is
thread-specific), I would expect either new keys to retrieve those
process-specific or thread-specific values, or a new flag bit to
indicate that existing keys should be reinterpreted in a new manner.
Using a key as a modifier with no value and changing the keys to
be order-dependent seems like an unnecessarily confusing ugly hack
that we would have to live with for decades to come, just to slightly
simplify code needed in the short term to handle older kernels.

Additionally, the approach of falling back to the syscall whenever
the modifier key is used could lead to each check for the vector
extension, in an ifunc resolver or almost anywhere else, having to
perform a syscall.  Callers wanting to use other extensions may
even use this modifier when checking for them, because given the
choice of knowing whether the extension is potentially available
or is available, the latter sounds like what they should be using
to be future-proof, even if there is not currently a way to disable
the extension.  That could lead to a syscall for every extension
check, defeating the vDSO cache.

On some non-RISC-V platforms glibc allows its own use of specific
CPU capabilities to be disabled through tunables (e.g.,
GLIBC_TUNABLES=glibc.cpu.hwcaps=-AVX2 ./my_x86_64_program), and as
of glibc 2.44 tunables can also be set in /etc/tunables.conf and
applied system-wide, even to specific programs.  This is more
convenient than a prctl and it would be nice if RISC-V extensions
could also be disabled for specific programs through tunables, for
any use where its availability is checked beforehand (in an ifunc
resolver or elsewhere), without the need to go to the kernel for
each check.

Because ifunc resolvers may not have access to external symbols
beyond __riscv_hwprobe(), it is really attractive to be able to
obtain extension availability information using the same function.
But that doesn't mean that the kernel has to be involved.  The
function is in libc, so it could call the vDSO function and then
optionally modify the resulting values according to its own
process-specific information about enabled or disabled extensions.
This extra information would ideally come from the initial hwcaps
(which reflect the vector prctl) and any tunables, so it is
process-wide, can support arbitrary extensions, and can be checked
without any kernel syscalls and without callers having to know or
care whether a particular extension can be disabled.  If the prctl
was used to re-enable vector in some thread that may not affect it,
but that seems like the desired behavior, at least for ifunc
resolvers, since they are assumed to produce the same result in any
thread and at any point during the process lifetime.

If implemented using a new flag, the libc __riscv_hwprobe() function
could just call the existing vDSO function but with the new flag
masked out, and then if the flag is set, modify the resulting values.
That would probably be the simplest and easiest to understand API
and would not require kernel syscalls.

 - Mark