[PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

From: Drew Fustini <dfustini@tenstorrent.com>

Clobbering the vector registers can significantly increase system call
latency for some implementations. To mitigate this performance impact, a
sysctl knob is provided that controls whether the vector state is
discarded in the syscall path:

/proc/sys/abi/riscv_v_vstate_discard

Valid values are:

0: Vector state is not always clobbered in all syscalls
1: Mandatory clobbering of vector state in all syscalls

The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.

Fixes: 9657e9b7d253 ("riscv: Discard vector state on syscalls")
Signed-off-by: Drew Fustini <dfustini@tenstorrent.com>
---
Changes in v2:
 - Reword the description of the abi.riscv_v_vstate_discard sysctl to
   clarify that option '0' does not preserve the vector state - it just
   means that vector state will not always be clobbered in the syscall
   path.
 - Add clarification suggested by Palmer in v1 to the "Vector Register
   State Across System Calls" documentation section.
 - v1: https://lore.kernel.org/linux-riscv/20250719033912.1313955-1-fustini@kernel.org/

Test results:
-------------
I've tested the impact of riscv_v_vstate_discard() on the SiFive X280
cores [1] in the Tenstorrent Blackhole SoC [2]. The results from the
Blackhole P100 [3] card show that discarding the vector registers
increases null syscall latency by 25%.

The null syscall program [4] executes vsetvli and then calls getppid()
in a loop. The average duration of getppid() is 198 ns when registers
are clobbered in riscv_v_vstate_discard(). The average duration drops
to 149 ns when riscv_v_vstate_discard() skips clobbering the registers
becaise riscv_v_vstate_discard is set to 0.

$ sudo sysctl abi.riscv_v_vstate_discard=1
abi.riscv_v_vstate_discard = 1

$ ./null_syscall --vsetvli
vsetvli complete
 iterations: 1000000000
   duration: 198 seconds
avg latency: 198.73 ns

$ sudo sysctl abi.riscv_v_vstate_discard=0
abi.riscv_v_vstate_discard = 0

$ ./null_syscall --vsetvli
vsetvli complete
 iterations: 1000000000
   duration: 149 seconds
avg latency: 149.89 ns

I'm testing on the tt-blackhole-v6.16-rc1_vstate_discard [5] branch that
has 13 patches, including this one, on top of v6.16-rc1. Most are simple
yaml patches for dt bindings along with dts files and a bespoke network
driver. I don't think the other patches are relevant to this discussion.

This patch applies clean on its own to riscv/for-next.

[1] https://www.sifive.com/cores/intelligence-x200-series
[2] https://tenstorrent.com/en/hardware/blackhole
[3] https://github.com/tenstorrent/tt-bh-linux
[4] https://gist.github.com/tt-fustini/ab9b217756912ce75522b3cce11d0d58
[5] https://github.com/tenstorrent/linux/tree/tt-blackhole-v6.16-rc1_vstate_discard

Signed-off-by: Drew Fustini <fustini@kernel.org>
---
 Documentation/arch/riscv/vector.rst | 22 ++++++++++++++++++++--
 arch/riscv/Kconfig                  | 10 ++++++++++
 arch/riscv/include/asm/vector.h     |  4 ++++
 arch/riscv/kernel/vector.c          | 16 +++++++++++++++-
 4 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
index 3987f5f76a9deb0824e53a72df4c3bf90ac2bee1..b702c00351617165a4d8897c7df68eadcd2d562e 100644
--- a/Documentation/arch/riscv/vector.rst
+++ b/Documentation/arch/riscv/vector.rst
@@ -134,7 +134,25 @@ processes in form of sysctl knob:
 3.  Vector Register State Across System Calls
 ---------------------------------------------
 
-As indicated by version 1.0 of the V extension [1], vector registers are
-clobbered by system calls.
+Linux adopts the syscall ABI proposed by version 1.0 of the V extension [1],
+where vector registers are clobbered by system calls. Specifically:
+
+    Executing a system call causes all caller-saved vector registers
+    (v0-v31, vl, vtype) and vstart to become unspecied.
+
+However, clobbering the vector registers can significantly increase system call
+latency for some implementations. To mitigate this performance impact, a sysctl
+knob is provided that controls whether vector state is always discarded in the
+syscall path:
+
+* /proc/sys/abi/riscv_v_vstate_discard
+
+    Valid values are:
+
+    * 0: Vector state is not always clobbered in all syscalls
+    * 1: Mandatory clobbering of vector state in all syscalls
+
+    Reading this file returns the current discard behavior. The initial state is
+    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
 
 1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 36061f4732b7496a9c68a9a10f9959849dc2a95c..7bb8a8513135cbc105bd94d273012486a886f724 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -656,6 +656,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
 
 	  If you don't know what to do here, say Y.
 
+config RISCV_ISA_V_VSTATE_DISCARD
+	bool "Enable Vector state discard by default"
+	depends on RISCV_ISA_V
+	default n
+	help
+	  Say Y here if you want to always discard vector state in syscalls.
+	  Otherwise, userspace has to enable it via the sysctl interface.
+
+	  If you don't know what to do here, say N.
+
 config RISCV_ISA_V_UCOPY_THRESHOLD
 	int "Threshold size for vectorized user copies"
 	depends on RISCV_ISA_V
diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
index 45c9b426fcc52321d55d1a4a42030c3b988e53c0..77991013216b9aea1744540caef38589338717ff 100644
--- a/arch/riscv/include/asm/vector.h
+++ b/arch/riscv/include/asm/vector.h
@@ -40,6 +40,7 @@
 	_res;								\
 })
 
+extern bool riscv_v_vstate_discard_ctl;
 extern unsigned long riscv_v_vsize;
 int riscv_v_setup_vsize(void);
 bool insn_is_vector(u32 insn_buf);
@@ -270,6 +271,9 @@ static inline void __riscv_v_vstate_discard(void)
 {
 	unsigned long vl, vtype_inval = 1UL << (BITS_PER_LONG - 1);
 
+	if (READ_ONCE(riscv_v_vstate_discard_ctl) == 0)
+		return;
+
 	riscv_v_enable();
 	if (has_xtheadvector())
 		asm volatile (THEAD_VSETVLI_T4X0E8M8D1 : : : "t4");
diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c
index 901e67adf57608385e6815be1518e70216236eda..d81dcc86e794896dd36803d6e7540aad1dc37d79 100644
--- a/arch/riscv/kernel/vector.c
+++ b/arch/riscv/kernel/vector.c
@@ -26,6 +26,7 @@ static struct kmem_cache *riscv_v_user_cachep;
 static struct kmem_cache *riscv_v_kernel_cachep;
 #endif
 
+bool riscv_v_vstate_discard_ctl = IS_ENABLED(CONFIG_RISCV_ISA_V_VSTATE_DISCARD);
 unsigned long riscv_v_vsize __read_mostly;
 EXPORT_SYMBOL_GPL(riscv_v_vsize);
 
@@ -307,11 +308,24 @@ static const struct ctl_table riscv_v_default_vstate_table[] = {
 	},
 };
 
+static const struct ctl_table riscv_v_vstate_discard_table[] = {
+	{
+		.procname       = "riscv_v_vstate_discard",
+		.data           = &riscv_v_vstate_discard_ctl,
+		.maxlen         = sizeof(riscv_v_vstate_discard_ctl),
+		.mode           = 0644,
+		.proc_handler   = proc_dobool,
+	},
+};
+
 static int __init riscv_v_sysctl_init(void)
 {
-	if (has_vector() || has_xtheadvector())
+	if (has_vector() || has_xtheadvector()) {
 		if (!register_sysctl("abi", riscv_v_default_vstate_table))
 			return -EINVAL;
+		if (!register_sysctl("abi", riscv_v_vstate_discard_table))
+			return -EINVAL;
+	}
 	return 0;
 }
 

---
base-commit: fda589c286040d9ba2d72a0eaf0a13945fc48026
change-id: 20250805-riscv_v_vstate_discard-23ba1c1d1b68

Best regards,
-- 
Drew Fustini <fustini@kernel.org>

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Darius Rad]

On Wed, Aug 06, 2025 at 07:03:28AM -0700, Drew Fustini wrote:
[...]
> 
> diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
> index 3987f5f76a9deb0824e53a72df4c3bf90ac2bee1..b702c00351617165a4d8897c7df68eadcd2d562e 100644
> --- a/Documentation/arch/riscv/vector.rst
> +++ b/Documentation/arch/riscv/vector.rst
> @@ -134,7 +134,25 @@ processes in form of sysctl knob:
>  3.  Vector Register State Across System Calls
>  ---------------------------------------------
>  
> -As indicated by version 1.0 of the V extension [1], vector registers are
> -clobbered by system calls.
> +Linux adopts the syscall ABI proposed by version 1.0 of the V extension [1],
> +where vector registers are clobbered by system calls. Specifically:
> +
> +    Executing a system call causes all caller-saved vector registers
> +    (v0-v31, vl, vtype) and vstart to become unspecied.
> +

Perhaps:

Clobbering the vector registers may prevent leaking information to user
space and aid in debugging, but can significantly increase system call
latency for some implementations.  [...]

> +However, clobbering the vector registers can significantly increase system call
> +latency for some implementations. To mitigate this performance impact, a sysctl
> +knob is provided that controls whether vector state is always discarded in the
> +syscall path:
> +
> +* /proc/sys/abi/riscv_v_vstate_discard
> +
> +    Valid values are:
> +
> +    * 0: Vector state is not always clobbered in all syscalls
> +    * 1: Mandatory clobbering of vector state in all syscalls
> +
> +    Reading this file returns the current discard behavior. The initial state is
> +    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
>  
>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index 36061f4732b7496a9c68a9a10f9959849dc2a95c..7bb8a8513135cbc105bd94d273012486a886f724 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -656,6 +656,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
>  
>  	  If you don't know what to do here, say Y.
>  
> +config RISCV_ISA_V_VSTATE_DISCARD
> +	bool "Enable Vector state discard by default"
> +	depends on RISCV_ISA_V
> +	default n
> +	help

Perhaps add the following paragraph:

Discarding vector state is more robust, but has negative performance
implications in certain implementations.


> +	  Say Y here if you want to always discard vector state in syscalls.
> +	  Otherwise, userspace has to enable it via the sysctl interface.
> +
> +	  If you don't know what to do here, say N.
> +
>  config RISCV_ISA_V_UCOPY_THRESHOLD
>  	int "Threshold size for vectorized user copies"
>  	depends on RISCV_ISA_V

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Vivian Wang]

My previous comment on v1 on prefering clobbering with VS = Initial
handling aside...

On 8/8/25 20:36, Darius Rad wrote:
> On Wed, Aug 06, 2025 at 07:03:28AM -0700, Drew Fustini wrote:
> [...]
>> diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
>> index 3987f5f76a9deb0824e53a72df4c3bf90ac2bee1..b702c00351617165a4d8897c7df68eadcd2d562e 100644
>> --- a/Documentation/arch/riscv/vector.rst
>> +++ b/Documentation/arch/riscv/vector.rst
>> @@ -134,7 +134,25 @@ processes in form of sysctl knob:
>>  3.  Vector Register State Across System Calls
>>  ---------------------------------------------
>>  
>> -As indicated by version 1.0 of the V extension [1], vector registers are
>> -clobbered by system calls.
>> +Linux adopts the syscall ABI proposed by version 1.0 of the V extension [1],
>> +where vector registers are clobbered by system calls. Specifically:
>> +
>> +    Executing a system call causes all caller-saved vector registers
>> +    (v0-v31, vl, vtype) and vstart to become unspecied.
>> +
> Perhaps:
>
> Clobbering the vector registers may prevent leaking information to user

No... Not clobbering does not "leak" anything. If you find that it leaks
information, please report - that's a bug.

> space and aid in debugging, but can significantly increase system call
> latency for some implementations.  [...]
>
>> +However, clobbering the vector registers can significantly increase system call
>> +latency for some implementations. To mitigate this performance impact, a sysctl
>> +knob is provided that controls whether vector state is always discarded in the
>> +syscall path:
>> +
>> +* /proc/sys/abi/riscv_v_vstate_discard
>> +
>> +    Valid values are:
>> +
>> +    * 0: Vector state is not always clobbered in all syscalls
>> +    * 1: Mandatory clobbering of vector state in all syscalls
>> +
>> +    Reading this file returns the current discard behavior. The initial state is
>> +    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
>>  
>>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>> index 36061f4732b7496a9c68a9a10f9959849dc2a95c..7bb8a8513135cbc105bd94d273012486a886f724 100644
>> --- a/arch/riscv/Kconfig
>> +++ b/arch/riscv/Kconfig
>> @@ -656,6 +656,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
>>  
>>  	  If you don't know what to do here, say Y.
>>  
>> +config RISCV_ISA_V_VSTATE_DISCARD
>> +	bool "Enable Vector state discard by default"
>> +	depends on RISCV_ISA_V
>> +	default n
>> +	help
> Perhaps add the following paragraph:
>
> Discarding vector state is more robust, but has negative performance
> implications in certain implementations.

"Robust" is too vague... I don't think this word is helpful for anyone
trying to understand what this does.

Vivian "dramforever" Wang

>
>> +	  Say Y here if you want to always discard vector state in syscalls.
>> +	  Otherwise, userspace has to enable it via the sysctl interface.
>> +
>> +	  If you don't know what to do here, say N.
>> +
>>  config RISCV_ISA_V_UCOPY_THRESHOLD
>>  	int "Threshold size for vectorized user copies"
>>  	depends on RISCV_ISA_V

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Sat, Aug 09, 2025 at 11:58:24AM +0800, Vivian Wang wrote:
> My previous comment on v1 on prefering clobbering with VS = Initial
> handling aside...

I found that in the discard vector state patch discussion 2 years ago
that Andy and Bjorn discussed how Initial could cause a problem [1]:

  It's not a racy, but you're correct that setting the state to Initial,
  will cause issues. When get/set_regs is called, the tracee will be
  stopped, and a schedule() has been done.

In the v3 series, Bjorn notes [2]:

  Set state to Dirty after discard, for proper ptrace() handling (Andy)

Also, I would like the ability to have the ability to switch off
__riscv_v_vstate_discard() and not loose any cycles to it, so I think
this sysctl is a good fit for that.

> 
> On 8/8/25 20:36, Darius Rad wrote:
> > On Wed, Aug 06, 2025 at 07:03:28AM -0700, Drew Fustini wrote:
> > [...]
> >> diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
> >> index 3987f5f76a9deb0824e53a72df4c3bf90ac2bee1..b702c00351617165a4d8897c7df68eadcd2d562e 100644
> >> --- a/Documentation/arch/riscv/vector.rst
> >> +++ b/Documentation/arch/riscv/vector.rst
> >> @@ -134,7 +134,25 @@ processes in form of sysctl knob:
> >>  3.  Vector Register State Across System Calls
> >>  ---------------------------------------------
> >>  
> >> -As indicated by version 1.0 of the V extension [1], vector registers are
> >> -clobbered by system calls.
> >> +Linux adopts the syscall ABI proposed by version 1.0 of the V extension [1],
> >> +where vector registers are clobbered by system calls. Specifically:
> >> +
> >> +    Executing a system call causes all caller-saved vector registers
> >> +    (v0-v31, vl, vtype) and vstart to become unspecied.
> >> +
> > Perhaps:
> >
> > Clobbering the vector registers may prevent leaking information to user
> 
> No... Not clobbering does not "leak" anything. If you find that it leaks
> information, please report - that's a bug.

Thanks Darius and Vivian for your comments. I think it is a good idea
for me to write about the possible advantages of mandatory clobbering on
syscall entry. However, I am also uncertain how clobbering on syscall
entry helps prevent leaking information.

> > space and aid in debugging, but can significantly increase system call
> > latency for some implementations.  [...]

I think that is a good idea for me to call out that this is can be
useful for debugging and testing.

> >
> >> +However, clobbering the vector registers can significantly increase system call
> >> +latency for some implementations. To mitigate this performance impact, a sysctl
> >> +knob is provided that controls whether vector state is always discarded in the
> >> +syscall path:
> >> +
> >> +* /proc/sys/abi/riscv_v_vstate_discard
> >> +
> >> +    Valid values are:
> >> +
> >> +    * 0: Vector state is not always clobbered in all syscalls
> >> +    * 1: Mandatory clobbering of vector state in all syscalls
> >> +
> >> +    Reading this file returns the current discard behavior. The initial state is
> >> +    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> >>  
> >>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
> >> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> >> index 36061f4732b7496a9c68a9a10f9959849dc2a95c..7bb8a8513135cbc105bd94d273012486a886f724 100644
> >> --- a/arch/riscv/Kconfig
> >> +++ b/arch/riscv/Kconfig
> >> @@ -656,6 +656,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
> >>  
> >>  	  If you don't know what to do here, say Y.
> >>  
> >> +config RISCV_ISA_V_VSTATE_DISCARD
> >> +	bool "Enable Vector state discard by default"
> >> +	depends on RISCV_ISA_V
> >> +	default n
> >> +	help
> > Perhaps add the following paragraph:
> >
> > Discarding vector state is more robust, but has negative performance
> > implications in certain implementations.
> 
> "Robust" is too vague... I don't think this word is helpful for anyone
> trying to understand what this does.

I agree that I should add more description to the Kconfig option as I
think what I wrote assumes too much prior knowledge of the code. Maybe
something like this:

  Discarding vector state on syscall entry can help identify userpace
  programs that are mistakenly relying on vector state being preserved
  across syscalls. This can be useful for debugging and test suites.
  However, this behavior can negatively impact performance on some
  RISC-V implementations.

  Say Y here if you want mandatory clobbering of vector state before
  entering all syscalls. If you select N, then userspace can still
  eanble it via the abi.riscv_v_vstate_discard sysctl knob.

  If you don't know what to do here, then select N.

Thanks,
Drew


[1] https://lore.kernel.org/linux-riscv/87r0pug6hb.fsf@all.your.base.are.belong.to.us/
[2] https://lore.kernel.org/linux-riscv/20230629062730.985184-1-bjorn@kernel.org/

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Vivian Wang]

On 8/9/25 15:54, Drew Fustini wrote:
> On Sat, Aug 09, 2025 at 11:58:24AM +0800, Vivian Wang wrote:
>> My previous comment on v1 on prefering clobbering with VS = Initial
>> handling aside...
> I found that in the discard vector state patch discussion 2 years ago
> that Andy and Bjorn discussed how Initial could cause a problem [1]:
>
>   It's not a racy, but you're correct that setting the state to Initial,
>   will cause issues. When get/set_regs is called, the tracee will be
>   stopped, and a schedule() has been done.
>
> In the v3 series, Bjorn notes [2]:
>
>   Set state to Dirty after discard, for proper ptrace() handling (Andy)

Ahh... That's more complicated than I thought.

I have some rough ideas on fixing ptrace handling, and I personally
still vastly prefer VS=Initial, but given the complexity, I think it
makes sense to defer this work until later and just get this performance
fix in first. And also...

> Also, I would like the ability to have the ability to switch off
> __riscv_v_vstate_discard() and not loose any cycles to it, so I think
> this sysctl is a good fit for that.

I was initially (ha!) worried about the sysctl knob getting in, and
immediately going away after we implement VS=Initial (correctly), but if
you think the right thing is to have this knob always, that wouldn't be
a problem.

>> On 8/8/25 20:36, Darius Rad wrote:
>>> On Wed, Aug 06, 2025 at 07:03:28AM -0700, Drew Fustini wrote:
>>> [...]
>>>> diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
>>>> index 3987f5f76a9deb0824e53a72df4c3bf90ac2bee1..b702c00351617165a4d8897c7df68eadcd2d562e 100644
>>>> --- a/Documentation/arch/riscv/vector.rst
>>>> +++ b/Documentation/arch/riscv/vector.rst
>>>> @@ -134,7 +134,25 @@ processes in form of sysctl knob:
>>>>  3.  Vector Register State Across System Calls
>>>>  ---------------------------------------------
>>>>  
>>>> -As indicated by version 1.0 of the V extension [1], vector registers are
>>>> -clobbered by system calls.
>>>> +Linux adopts the syscall ABI proposed by version 1.0 of the V extension [1],
>>>> +where vector registers are clobbered by system calls. Specifically:
>>>> +
>>>> +    Executing a system call causes all caller-saved vector registers
>>>> +    (v0-v31, vl, vtype) and vstart to become unspecied.
>>>> +
>>> Perhaps:
>>>
>>> Clobbering the vector registers may prevent leaking information to user
>> No... Not clobbering does not "leak" anything. If you find that it leaks
>> information, please report - that's a bug.
> Thanks Darius and Vivian for your comments. I think it is a good idea
> for me to write about the possible advantages of mandatory clobbering on
> syscall entry. However, I am also uncertain how clobbering on syscall
> entry helps prevent leaking information.
>
>>> space and aid in debugging, but can significantly increase system call
>>> latency for some implementations.  [...]
> I think that is a good idea for me to call out that this is can be
> useful for debugging and testing.
>
Yeah, I agree that this second part of the description is good.

>>>> +However, clobbering the vector registers can significantly increase system call
>>>> +latency for some implementations. To mitigate this performance impact, a sysctl
>>>> +knob is provided that controls whether vector state is always discarded in the
>>>> +syscall path:
>>>> +
>>>> +* /proc/sys/abi/riscv_v_vstate_discard
>>>> +
>>>> +    Valid values are:
>>>> +
>>>> +    * 0: Vector state is not always clobbered in all syscalls
>>>> +    * 1: Mandatory clobbering of vector state in all syscalls
>>>> +
>>>> +    Reading this file returns the current discard behavior. The initial state is
>>>> +    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
>>>>  
>>>>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
>>>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>>>> index 36061f4732b7496a9c68a9a10f9959849dc2a95c..7bb8a8513135cbc105bd94d273012486a886f724 100644
>>>> --- a/arch/riscv/Kconfig
>>>> +++ b/arch/riscv/Kconfig
>>>> @@ -656,6 +656,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
>>>>  
>>>>  	  If you don't know what to do here, say Y.
>>>>  
>>>> +config RISCV_ISA_V_VSTATE_DISCARD
>>>> +	bool "Enable Vector state discard by default"
>>>> +	depends on RISCV_ISA_V
>>>> +	default n
>>>> +	help
>>> Perhaps add the following paragraph:
>>>
>>> Discarding vector state is more robust, but has negative performance
>>> implications in certain implementations.
>> "Robust" is too vague... I don't think this word is helpful for anyone
>> trying to understand what this does.
> I agree that I should add more description to the Kconfig option as I
> think what I wrote assumes too much prior knowledge of the code. Maybe
> something like this:
>
>   Discarding vector state on syscall entry can help identify userpace
>   programs that are mistakenly relying on vector state being preserved
>   across syscalls. This can be useful for debugging and test suites.
>   However, this behavior can negatively impact performance on some
>   RISC-V implementations.
>
>   Say Y here if you want mandatory clobbering of vector state before
>   entering all syscalls. If you select N, then userspace can still
>   eanble it via the abi.riscv_v_vstate_discard sysctl knob.

Typo: eanble -> enable

Also, I think it kinda makes the false impression that the sysctl knob
is only available with this set to N? I think this should be structured
to say something like this being the default value, and also say that
userspace can override this behavior system-wide via the sysctl knob.

Otherwise mostly looks good to me, assuming that this matches the
intentions of the original design.

Thanks,
Vivian "dramforever" Wang

>   If you don't know what to do here, then select N.
>
> Thanks,
> Drew
>
>
> [1] https://lore.kernel.org/linux-riscv/87r0pug6hb.fsf@all.your.base.are.belong.to.us/
> [2] https://lore.kernel.org/linux-riscv/20230629062730.985184-1-bjorn@kernel.org/

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Sun, Aug 10, 2025 at 05:52:54PM +0800, Vivian Wang wrote:
> >   Discarding vector state on syscall entry can help identify userpace
> >   programs that are mistakenly relying on vector state being preserved
> >   across syscalls. This can be useful for debugging and test suites.
> >   However, this behavior can negatively impact performance on some
> >   RISC-V implementations.
> >
> >   Say Y here if you want mandatory clobbering of vector state before
> >   entering all syscalls. If you select N, then userspace can still
> >   eanble it via the abi.riscv_v_vstate_discard sysctl knob.
> 
> Typo: eanble -> enable
> 
> Also, I think it kinda makes the false impression that the sysctl knob
> is only available with this set to N? I think this should be structured
> to say something like this being the default value, and also say that
> userspace can override this behavior system-wide via the sysctl knob.

Thanks, that is a good point that I should clarify that the user can
still override via the sysctl knob regardless of whether choosing Y or N
for this Kconfig option.

-Drew

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Darius Rad]

On Sat, Aug 09, 2025 at 11:58:24AM +0800, Vivian Wang wrote:
> My previous comment on v1 on prefering clobbering with VS = Initial
> handling aside...
> 
> On 8/8/25 20:36, Darius Rad wrote:
> > On Wed, Aug 06, 2025 at 07:03:28AM -0700, Drew Fustini wrote:
> > [...]
> >> diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
> >> index 3987f5f76a9deb0824e53a72df4c3bf90ac2bee1..b702c00351617165a4d8897c7df68eadcd2d562e 100644
> >> --- a/Documentation/arch/riscv/vector.rst
> >> +++ b/Documentation/arch/riscv/vector.rst
> >> @@ -134,7 +134,25 @@ processes in form of sysctl knob:
> >>  3.  Vector Register State Across System Calls
> >>  ---------------------------------------------
> >>  
> >> -As indicated by version 1.0 of the V extension [1], vector registers are
> >> -clobbered by system calls.
> >> +Linux adopts the syscall ABI proposed by version 1.0 of the V extension [1],
> >> +where vector registers are clobbered by system calls. Specifically:
> >> +
> >> +    Executing a system call causes all caller-saved vector registers
> >> +    (v0-v31, vl, vtype) and vstart to become unspecied.
> >> +
> > Perhaps:
> >
> > Clobbering the vector registers may prevent leaking information to user
> 
> No... Not clobbering does not "leak" anything. If you find that it leaks
> information, please report - that's a bug.
> 

That's why I wrote "may".  If such a bug existed, either now or in the
future, clobbering here would limit the scope of it.  But this may not be
important enough to mention.

> > space and aid in debugging, but can significantly increase system call
> > latency for some implementations.  [...]
> >
> >> +However, clobbering the vector registers can significantly increase system call
> >> +latency for some implementations. To mitigate this performance impact, a sysctl
> >> +knob is provided that controls whether vector state is always discarded in the
> >> +syscall path:
> >> +
> >> +* /proc/sys/abi/riscv_v_vstate_discard
> >> +
> >> +    Valid values are:
> >> +
> >> +    * 0: Vector state is not always clobbered in all syscalls
> >> +    * 1: Mandatory clobbering of vector state in all syscalls
> >> +
> >> +    Reading this file returns the current discard behavior. The initial state is
> >> +    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> >>  
> >>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
> >> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> >> index 36061f4732b7496a9c68a9a10f9959849dc2a95c..7bb8a8513135cbc105bd94d273012486a886f724 100644
> >> --- a/arch/riscv/Kconfig
> >> +++ b/arch/riscv/Kconfig
> >> @@ -656,6 +656,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
> >>  
> >>  	  If you don't know what to do here, say Y.
> >>  
> >> +config RISCV_ISA_V_VSTATE_DISCARD
> >> +	bool "Enable Vector state discard by default"
> >> +	depends on RISCV_ISA_V
> >> +	default n
> >> +	help
> > Perhaps add the following paragraph:
> >
> > Discarding vector state is more robust, but has negative performance
> > implications in certain implementations.
> 
> "Robust" is too vague... I don't think this word is helpful for anyone
> trying to understand what this does.
> 
> Vivian "dramforever" Wang
> 
> >
> >> +	  Say Y here if you want to always discard vector state in syscalls.
> >> +	  Otherwise, userspace has to enable it via the sysctl interface.
> >> +
> >> +	  If you don't know what to do here, say N.
> >> +
> >>  config RISCV_ISA_V_UCOPY_THRESHOLD
> >>  	int "Threshold size for vectorized user copies"
> >>  	depends on RISCV_ISA_V
> 
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Florian Weimer]

* Drew Fustini:

> From: Drew Fustini <dfustini@tenstorrent.com>
>
> Clobbering the vector registers can significantly increase system call
> latency for some implementations. To mitigate this performance impact, a
> sysctl knob is provided that controls whether the vector state is
> discarded in the syscall path:
>
> /proc/sys/abi/riscv_v_vstate_discard
>
> Valid values are:
>
> 0: Vector state is not always clobbered in all syscalls
> 1: Mandatory clobbering of vector state in all syscalls
>
> The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.

Can this be put into the system call number instead, or make it specific
to some system calls in other ways?

I think C libraries can use this optimization for their system calls
(after adjusting the assembler clobbers) because the vector state is
caller-saved in the standard calling convention.  But there is backwards
compatibility impact for turning this on for the entire process.

Thanks,
Florian

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Sat, Aug 09, 2025 at 10:40:46AM +0200, Florian Weimer wrote:
> * Drew Fustini:
> 
> > From: Drew Fustini <dfustini@tenstorrent.com>
> >
> > Clobbering the vector registers can significantly increase system call
> > latency for some implementations. To mitigate this performance impact, a
> > sysctl knob is provided that controls whether the vector state is
> > discarded in the syscall path:
> >
> > /proc/sys/abi/riscv_v_vstate_discard
> >
> > Valid values are:
> >
> > 0: Vector state is not always clobbered in all syscalls
> > 1: Mandatory clobbering of vector state in all syscalls
> >
> > The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> 
> Can this be put into the system call number instead, or make it specific
> to some system calls in other ways?

Do you mean the control the initial state of the sysctl, or not having a
sysctl for discard behavior at all?

> I think C libraries can use this optimization for their system calls
> (after adjusting the assembler clobbers) because the vector state is
> caller-saved in the standard calling convention.  But there is backwards
> compatibility impact for turning this on for the entire process.

The focus I have right now is allowing users to avoid the delay in
syscall entry on implementations where clobbering is slow. Palmer had
mentioned in my v1 [1] that he has 'a patch out for GCC that enables a
system-wide vector ABI, but I don't have time to test/benchmark it so 
it's kind of hard to justify'. It seems like creating a new ABI where
the vector registers are preserved across syscalls could be useful, but
I think it would be best to handle that possiblity later on.

Thanks,
Drew

[1] https://lore.kernel.org/linux-riscv/mhng-E49DDC7D-A330-4626-A122-4146AADDBB33@Palmers-Mini.rwc.dabbelt.com/

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Florian Weimer]

* Drew Fustini:

> On Sat, Aug 09, 2025 at 10:40:46AM +0200, Florian Weimer wrote:
>> * Drew Fustini:
>> 
>> > From: Drew Fustini <dfustini@tenstorrent.com>
>> >
>> > Clobbering the vector registers can significantly increase system call
>> > latency for some implementations. To mitigate this performance impact, a
>> > sysctl knob is provided that controls whether the vector state is
>> > discarded in the syscall path:
>> >
>> > /proc/sys/abi/riscv_v_vstate_discard
>> >
>> > Valid values are:
>> >
>> > 0: Vector state is not always clobbered in all syscalls
>> > 1: Mandatory clobbering of vector state in all syscalls
>> >
>> > The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
>> 
>> Can this be put into the system call number instead, or make it specific
>> to some system calls in other ways?
>
> Do you mean the control the initial state of the sysctl, or not having a
> sysctl for discard behavior at all?

It's seems rather strange to have a sysctl for such an ABI change
because it really has to be a per-process property.

>> I think C libraries can use this optimization for their system calls
>> (after adjusting the assembler clobbers) because the vector state is
>> caller-saved in the standard calling convention.  But there is backwards
>> compatibility impact for turning this on for the entire process.
>
> The focus I have right now is allowing users to avoid the delay in
> syscall entry on implementations where clobbering is slow. Palmer had
> mentioned in my v1 [1] that he has 'a patch out for GCC that enables a
> system-wide vector ABI, but I don't have time to test/benchmark it so 
> it's kind of hard to justify'. It seems like creating a new ABI where
> the vector registers are preserved across syscalls could be useful, but
> I think it would be best to handle that possiblity later on.

I'm confused.  Current glibc assumes that vector registers are preserved
across system calls because the assembler clobbers do not mention them.
Similar inline assembly probably has ended up in other projects, too.
It works by accident if glibc is compiled for a non-vector target, or if
it so happens that GCC never keeps vector registers alive across system
calls.

Thanks,
Florian

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Sun, Aug 10, 2025 at 09:45:45AM +0200, Florian Weimer wrote:
> * Drew Fustini:
> 
> > On Sat, Aug 09, 2025 at 10:40:46AM +0200, Florian Weimer wrote:
> >> * Drew Fustini:
> >> 
> >> > From: Drew Fustini <dfustini@tenstorrent.com>
> >> >
> >> > Clobbering the vector registers can significantly increase system call
> >> > latency for some implementations. To mitigate this performance impact, a
> >> > sysctl knob is provided that controls whether the vector state is
> >> > discarded in the syscall path:
> >> >
> >> > /proc/sys/abi/riscv_v_vstate_discard
> >> >
> >> > Valid values are:
> >> >
> >> > 0: Vector state is not always clobbered in all syscalls
> >> > 1: Mandatory clobbering of vector state in all syscalls
> >> >
> >> > The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> >> 
> >> Can this be put into the system call number instead, or make it specific
> >> to some system calls in other ways?
> >
> > Do you mean the control the initial state of the sysctl, or not having a
> > sysctl for discard behavior at all?
> 
> It's seems rather strange to have a sysctl for such an ABI change
> because it really has to be a per-process property.

The reason for sysctl is that I want a means to let a system to opt out 
of clobbering vector state on the syscall entry path. This is because it
adds significant overhead for some implementations. For example, it
results in a 25% longer syscall duration on the X280 core.

I would be in favor of reverting the mandatory clobbering behavior, but
Palmer says that it is useful for test suites. Since revert isn't an
option, I want a system wide policy control like this sysctl. It does
seem like there could be some advantages to per-process control but I
think that delves into ABI changes which I feel is a separate issue from
a system wide knob for "always clobber"/"do not always clobber".

> 
> >> I think C libraries can use this optimization for their system calls
> >> (after adjusting the assembler clobbers) because the vector state is
> >> caller-saved in the standard calling convention.  But there is backwards
> >> compatibility impact for turning this on for the entire process.
> >
> > The focus I have right now is allowing users to avoid the delay in
> > syscall entry on implementations where clobbering is slow. Palmer had
> > mentioned in my v1 [1] that he has 'a patch out for GCC that enables a
> > system-wide vector ABI, but I don't have time to test/benchmark it so 
> > it's kind of hard to justify'. It seems like creating a new ABI where
> > the vector registers are preserved across syscalls could be useful, but
> > I think it would be best to handle that possiblity later on.
> 
> I'm confused.  Current glibc assumes that vector registers are preserved
> across system calls because the assembler clobbers do not mention them.
> Similar inline assembly probably has ended up in other projects, too.
> It works by accident if glibc is compiled for a non-vector target, or if
> it so happens that GCC never keeps vector registers alive across system
> calls.

I wasn't trying to make any ABI changes with this sysctl patch. The
riscv kernel documentation states vector state is not preserved across
syscalls. I am not trying to change that policy.

Around the same time that Palmer added that statement to the vector
documentation, Bjorn added the code that always clobbers the vector
registers on syscall entry. This was done in order to ensure programs
were not relying on vector state being preserved. 

At the time 2 years ago, Palmer and Bjorn talked about how this could
be revisted if it turns out the clobbering process ended up being slow
on real hardware. This patch is my attempt to allow platforms with slow
vstate clobbering to opt out this strict mandatory clobbering on syscall
entry.

Thanks,
Drew

Re: [PATCH v2] riscv: Add sysctl to control discard of vstate during syscall

[Florian Weimer]

* Drew Fustini:

>> I'm confused.  Current glibc assumes that vector registers are preserved
>> across system calls because the assembler clobbers do not mention them.
>> Similar inline assembly probably has ended up in other projects, too.
>> It works by accident if glibc is compiled for a non-vector target, or if
>> it so happens that GCC never keeps vector registers alive across system
>> calls.
>
> I wasn't trying to make any ABI changes with this sysctl patch. The
> riscv kernel documentation states vector state is not preserved across
> syscalls. I am not trying to change that policy.

Oh, then glibc is just buggy in this area.  I had no idea.  If the
vector state is already clobbered, than the sysctl for experimentation
is fine.

Thanks,
Florian