[PATCH] riscv: Add sysctl to control discard of vstate during syscall

[PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

From: Drew Fustini <dfustini@tenstorrent.com>

Clobbering the vector registers can significantly increase system call
latency for some implementations. To mitigate this performance impact, a
policy mechanism is provided to administrators, distro maintainers, and
developers to control vector state discard in the form of a sysctl knob:

/proc/sys/abi/riscv_v_vstate_discard

Valid values are:

0: Do not discard vector state during syscall
1: Discard vector state during syscall

The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.

Fixes: 9657e9b7d253 ("riscv: Discard vector state on syscalls")
Signed-off-by: Drew Fustini <dfustini@tenstorrent.com>
---
 Documentation/arch/riscv/vector.rst | 15 +++++++++++++++
 arch/riscv/Kconfig                  | 10 ++++++++++
 arch/riscv/include/asm/vector.h     |  4 ++++
 arch/riscv/kernel/vector.c          | 16 +++++++++++++++-
 4 files changed, 44 insertions(+), 1 deletion(-)

I've tested the impact of riscv_v_vstate_discard() on the SiFive X280
cores [1] in the Tenstorrent Blackhole SoC [2]. The results from the
Blackhole P100 [3] card show that discarding the vector registers
increases null syscall latency by 25%.

The null syscall program [4] executes vsetvli and then calls getppid()
in a loop. The average duration of getppid() is 198 ns when registers
are clobbered in riscv_v_vstate_discard(). The average duration drops
to 149 ns when riscv_v_vstate_discard() skips clobbering the registers
as result of riscv_v_vstate_discard being set to 0.

$ sudo sysctl abi.riscv_v_vstate_discard=1
abi.riscv_v_vstate_discard = 1

$ ./null_syscall --vsetvli
vsetvli complete
 iterations: 1000000000
   duration: 198 seconds
avg latency: 198.73 ns

$ sudo sysctl abi.riscv_v_vstate_discard=0
abi.riscv_v_vstate_discard = 0

$ ./null_syscall --vsetvli
vsetvli complete
 iterations: 1000000000
   duration: 149 seconds
avg latency: 149.89 ns

I'm testing on the tt-blackhole-v6.16-rc1_vstate_discard [5] branch that
has 13 patches, including this one, on top of v6.16-rc1. Most are simple
yaml patches for dt bindings along with dts files and a bespoke network
driver. I don't think the other patches are relevant to this discussion.

This patch applies clean on its own to riscv/for-next and next-20250718.

[1] https://www.sifive.com/cores/intelligence-x200-series
[2] https://tenstorrent.com/en/hardware/blackhole
[3] https://github.com/tenstorrent/tt-bh-linux
[4] https://gist.github.com/tt-fustini/ab9b217756912ce75522b3cce11d0d58
[5] https://github.com/tenstorrent/linux/tree/tt-blackhole-v6.16-rc1_vstate_discard

diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
index 3987f5f76a9d..1edbce436015 100644
--- a/Documentation/arch/riscv/vector.rst
+++ b/Documentation/arch/riscv/vector.rst
@@ -137,4 +137,19 @@ processes in form of sysctl knob:
 As indicated by version 1.0 of the V extension [1], vector registers are
 clobbered by system calls.
 
+Clobbering the vector registers can significantly increase system call latency
+for some implementations. To mitigate the performance impact, a policy mechanism
+is provided to the administrators, distro maintainers, and developers to control
+the vstate discard in the form of a sysctl knob:
+
+* /proc/sys/abi/riscv_v_vstate_discard
+
+    Valid values are:
+
+    * 0: Do not discard vector state during syscall
+    * 1: Discard vector state during syscall
+
+    Reading this file returns the current discard behavior. The initial state is
+    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
+
 1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 0aeee50da016..c0039f21d1f0 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -655,6 +655,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
 
 	  If you don't know what to do here, say Y.
 
+config RISCV_ISA_V_VSTATE_DISCARD
+	bool "Enable Vector state discard by default"
+	depends on RISCV_ISA_V
+	default n
+	help
+	  Say Y here if you want to enable Vector state discard on syscall.
+	  Otherwise, userspace has to enable it via the sysctl interface.
+
+	  If you don't know what to do here, say N.
+
 config RISCV_ISA_V_UCOPY_THRESHOLD
 	int "Threshold size for vectorized user copies"
 	depends on RISCV_ISA_V
diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
index 45c9b426fcc5..77991013216b 100644
--- a/arch/riscv/include/asm/vector.h
+++ b/arch/riscv/include/asm/vector.h
@@ -40,6 +40,7 @@
 	_res;								\
 })
 
+extern bool riscv_v_vstate_discard_ctl;
 extern unsigned long riscv_v_vsize;
 int riscv_v_setup_vsize(void);
 bool insn_is_vector(u32 insn_buf);
@@ -270,6 +271,9 @@ static inline void __riscv_v_vstate_discard(void)
 {
 	unsigned long vl, vtype_inval = 1UL << (BITS_PER_LONG - 1);
 
+	if (READ_ONCE(riscv_v_vstate_discard_ctl) == 0)
+		return;
+
 	riscv_v_enable();
 	if (has_xtheadvector())
 		asm volatile (THEAD_VSETVLI_T4X0E8M8D1 : : : "t4");
diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c
index 184f780c932d..7a4c209ad337 100644
--- a/arch/riscv/kernel/vector.c
+++ b/arch/riscv/kernel/vector.c
@@ -26,6 +26,7 @@ static struct kmem_cache *riscv_v_user_cachep;
 static struct kmem_cache *riscv_v_kernel_cachep;
 #endif
 
+bool riscv_v_vstate_discard_ctl = IS_ENABLED(CONFIG_RISCV_ISA_V_VSTATE_DISCARD);
 unsigned long riscv_v_vsize __read_mostly;
 EXPORT_SYMBOL_GPL(riscv_v_vsize);
 
@@ -307,11 +308,24 @@ static const struct ctl_table riscv_v_default_vstate_table[] = {
 	},
 };
 
+static const struct ctl_table riscv_v_vstate_discard_table[] = {
+	{
+		.procname       = "riscv_v_vstate_discard",
+		.data           = &riscv_v_vstate_discard_ctl,
+		.maxlen         = sizeof(riscv_v_vstate_discard_ctl),
+		.mode           = 0644,
+		.proc_handler   = proc_dobool,
+	},
+};
+
 static int __init riscv_v_sysctl_init(void)
 {
-	if (has_vector() || has_xtheadvector())
+	if (has_vector() || has_xtheadvector()) {
 		if (!register_sysctl("abi", riscv_v_default_vstate_table))
 			return -EINVAL;
+		if (!register_sysctl("abi", riscv_v_vstate_discard_table))
+			return -EINVAL;
+	}
 	return 0;
 }
 
-- 
2.34.1


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Darius Rad]

On Fri, Jul 18, 2025 at 08:39:13PM -0700, Drew Fustini wrote:
> From: Drew Fustini <dfustini@tenstorrent.com>
> 
> Clobbering the vector registers can significantly increase system call
> latency for some implementations. To mitigate this performance impact, a
> policy mechanism is provided to administrators, distro maintainers, and
> developers to control vector state discard in the form of a sysctl knob:
> 
> /proc/sys/abi/riscv_v_vstate_discard
> 
> Valid values are:
> 
> 0: Do not discard vector state during syscall
> 1: Discard vector state during syscall
> 

Is the intention for this this mean "don't guarantee vector state is
clobbered" or "preserve vector state"?  I suspect it is the former, but the
wording seems unclear.  Additionally, if that's indeed the case, maybe the
documentation should more clearly articulate the tradeoff (performance vs.
security/robustness).

> The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> 
> Fixes: 9657e9b7d253 ("riscv: Discard vector state on syscalls")
> Signed-off-by: Drew Fustini <dfustini@tenstorrent.com>
> ---
>  Documentation/arch/riscv/vector.rst | 15 +++++++++++++++
>  arch/riscv/Kconfig                  | 10 ++++++++++
>  arch/riscv/include/asm/vector.h     |  4 ++++
>  arch/riscv/kernel/vector.c          | 16 +++++++++++++++-
>  4 files changed, 44 insertions(+), 1 deletion(-)
> 
> I've tested the impact of riscv_v_vstate_discard() on the SiFive X280
> cores [1] in the Tenstorrent Blackhole SoC [2]. The results from the
> Blackhole P100 [3] card show that discarding the vector registers
> increases null syscall latency by 25%.
> 
> The null syscall program [4] executes vsetvli and then calls getppid()
> in a loop. The average duration of getppid() is 198 ns when registers
> are clobbered in riscv_v_vstate_discard(). The average duration drops
> to 149 ns when riscv_v_vstate_discard() skips clobbering the registers
> as result of riscv_v_vstate_discard being set to 0.
> 
> $ sudo sysctl abi.riscv_v_vstate_discard=1
> abi.riscv_v_vstate_discard = 1
> 
> $ ./null_syscall --vsetvli
> vsetvli complete
>  iterations: 1000000000
>    duration: 198 seconds
> avg latency: 198.73 ns
> 
> $ sudo sysctl abi.riscv_v_vstate_discard=0
> abi.riscv_v_vstate_discard = 0
> 
> $ ./null_syscall --vsetvli
> vsetvli complete
>  iterations: 1000000000
>    duration: 149 seconds
> avg latency: 149.89 ns
> 
> I'm testing on the tt-blackhole-v6.16-rc1_vstate_discard [5] branch that
> has 13 patches, including this one, on top of v6.16-rc1. Most are simple
> yaml patches for dt bindings along with dts files and a bespoke network
> driver. I don't think the other patches are relevant to this discussion.
> 
> This patch applies clean on its own to riscv/for-next and next-20250718.
> 
> [1] https://www.sifive.com/cores/intelligence-x200-series
> [2] https://tenstorrent.com/en/hardware/blackhole
> [3] https://github.com/tenstorrent/tt-bh-linux
> [4] https://gist.github.com/tt-fustini/ab9b217756912ce75522b3cce11d0d58
> [5] https://github.com/tenstorrent/linux/tree/tt-blackhole-v6.16-rc1_vstate_discard
> 
> diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
> index 3987f5f76a9d..1edbce436015 100644
> --- a/Documentation/arch/riscv/vector.rst
> +++ b/Documentation/arch/riscv/vector.rst
> @@ -137,4 +137,19 @@ processes in form of sysctl knob:
>  As indicated by version 1.0 of the V extension [1], vector registers are
>  clobbered by system calls.
>  
> +Clobbering the vector registers can significantly increase system call latency
> +for some implementations. To mitigate the performance impact, a policy mechanism
> +is provided to the administrators, distro maintainers, and developers to control
> +the vstate discard in the form of a sysctl knob:
> +
> +* /proc/sys/abi/riscv_v_vstate_discard
> +
> +    Valid values are:
> +
> +    * 0: Do not discard vector state during syscall
> +    * 1: Discard vector state during syscall
> +
> +    Reading this file returns the current discard behavior. The initial state is
> +    controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> +
>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index 0aeee50da016..c0039f21d1f0 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -655,6 +655,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
>  
>  	  If you don't know what to do here, say Y.
>  
> +config RISCV_ISA_V_VSTATE_DISCARD
> +	bool "Enable Vector state discard by default"
> +	depends on RISCV_ISA_V
> +	default n
> +	help
> +	  Say Y here if you want to enable Vector state discard on syscall.
> +	  Otherwise, userspace has to enable it via the sysctl interface.
> +
> +	  If you don't know what to do here, say N.
> +
>  config RISCV_ISA_V_UCOPY_THRESHOLD
>  	int "Threshold size for vectorized user copies"
>  	depends on RISCV_ISA_V
> diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
> index 45c9b426fcc5..77991013216b 100644
> --- a/arch/riscv/include/asm/vector.h
> +++ b/arch/riscv/include/asm/vector.h
> @@ -40,6 +40,7 @@
>  	_res;								\
>  })
>  
> +extern bool riscv_v_vstate_discard_ctl;
>  extern unsigned long riscv_v_vsize;
>  int riscv_v_setup_vsize(void);
>  bool insn_is_vector(u32 insn_buf);
> @@ -270,6 +271,9 @@ static inline void __riscv_v_vstate_discard(void)
>  {
>  	unsigned long vl, vtype_inval = 1UL << (BITS_PER_LONG - 1);
>  
> +	if (READ_ONCE(riscv_v_vstate_discard_ctl) == 0)
> +		return;
> +
>  	riscv_v_enable();
>  	if (has_xtheadvector())
>  		asm volatile (THEAD_VSETVLI_T4X0E8M8D1 : : : "t4");
> diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c
> index 184f780c932d..7a4c209ad337 100644
> --- a/arch/riscv/kernel/vector.c
> +++ b/arch/riscv/kernel/vector.c
> @@ -26,6 +26,7 @@ static struct kmem_cache *riscv_v_user_cachep;
>  static struct kmem_cache *riscv_v_kernel_cachep;
>  #endif
>  
> +bool riscv_v_vstate_discard_ctl = IS_ENABLED(CONFIG_RISCV_ISA_V_VSTATE_DISCARD);
>  unsigned long riscv_v_vsize __read_mostly;
>  EXPORT_SYMBOL_GPL(riscv_v_vsize);
>  
> @@ -307,11 +308,24 @@ static const struct ctl_table riscv_v_default_vstate_table[] = {
>  	},
>  };
>  
> +static const struct ctl_table riscv_v_vstate_discard_table[] = {
> +	{
> +		.procname       = "riscv_v_vstate_discard",
> +		.data           = &riscv_v_vstate_discard_ctl,
> +		.maxlen         = sizeof(riscv_v_vstate_discard_ctl),
> +		.mode           = 0644,
> +		.proc_handler   = proc_dobool,
> +	},
> +};
> +
>  static int __init riscv_v_sysctl_init(void)
>  {
> -	if (has_vector() || has_xtheadvector())
> +	if (has_vector() || has_xtheadvector()) {
>  		if (!register_sysctl("abi", riscv_v_default_vstate_table))
>  			return -EINVAL;
> +		if (!register_sysctl("abi", riscv_v_vstate_discard_table))
> +			return -EINVAL;
> +	}
>  	return 0;
>  }
>  
> -- 
> 2.34.1
> 
> 
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linu

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Radim Krčmář]

2025-07-18T20:39:13-07:00, Drew Fustini <fustini@kernel.org>:
> From: Drew Fustini <dfustini@tenstorrent.com>
>
> Clobbering the vector registers can significantly increase system call
> latency for some implementations. To mitigate this performance impact, a
> policy mechanism is provided to administrators, distro maintainers, and
> developers to control vector state discard in the form of a sysctl knob:
>
> /proc/sys/abi/riscv_v_vstate_discard
>
> Valid values are:
>
> 0: Do not discard vector state during syscall
> 1: Discard vector state during syscall
>
> The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.

I think it is a bit more complicated to do this nicely...
Programs don't have to save/restore vector registers around syscalls
when compiled for riscv_v_vstate_discard=0, so running under
riscv_v_vstate_discard=1 would break them.

Shouldn't we have a way to prevent riscv_v_vstate_discard=0 executable
from running with riscv_v_vstate_discard=1?

> Fixes: 9657e9b7d253 ("riscv: Discard vector state on syscalls")

Programs compiled for riscv_v_vstate_discard=1 are compatible with 0, so
I think it would be simplest to revert that patch, and pretended it
never happened... (The issues will eventually go away.)

Shouldn't the RISC-V Linux syscall ABI be defined somewhere?
How come we could have broken it with 9657e9b7d253?

Thanks.

---
I don't think it makes much sense to clobber vector registers on a
syscall -- a kernel might not even touch vector registers, so they are
efforlessly preserved in that case.
If kernel needs to use vector registers in the syscall, then the kernel
needs to prevent any register leaks to userspace anyway by restoring
some state into them -- and why not restore the original one?

I think that main point of clobbering would be to optimize
context-switches after the userspace is not using vector registers
anymore, but it's terribly inefficient if the ratio of syscalls to
context switches is high.
Linux can also try to detect the situation, and turn to lazy vector
context-switch, with sstatus.VS=off, instead of eagerly restoring
clobbered state.
(A good indicator might be that the userspace hasn't dirtied the vectors
 since the last context-switch -- kernel didn't need to save the state,
 so it will restore lazily.)

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Radim Krčmář]

2025-07-21T14:35:38+02:00, Radim Krčmář <rkrcmar@ventanamicro.com>:
> Shouldn't the RISC-V Linux syscall ABI be defined somewhere?

To clarify this point.  My issue is with the following part in
Documentation/arch/riscv/vector.rst:

>>  As indicated by version 1.0 of the V extension [1], vector registers are
>>  clobbered by system calls.
>>  [...]
>>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc

The ISA does not say that vector registers are clobbered by system
calls.  All the ISA says is:

  "This Appendix is only a placeholder to help explain the conventions
   used in the code examples, and is not considered frozen or
   part of the ratification process.  The official RISC-V psABI document
   is being expanded to specify the vector calling conventions."

while the RISC-V psABI says:

  "The calling convention for system calls does not fall within the
   scope of this document. Please refer to the documentation of the
   RISC-V execution environment interface (e.g OS kernel ABI, SBI)."

We made a circular dependency, misinterpreted the ISA, and probably
implemented a suboptimal syscall ABI -- preserving vector registers
seems strictly better.

> How come we could have broken it with 9657e9b7d253?

We changed the ABI once, so maybe we can change it back?

Thanks.

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Mon, Jul 21, 2025 at 08:13:48AM -0400, Darius Rad wrote:
> On Fri, Jul 18, 2025 at 08:39:13PM -0700, Drew Fustini wrote:
> > From: Drew Fustini <dfustini@tenstorrent.com>
> > 
> > Clobbering the vector registers can significantly increase system call
> > latency for some implementations. To mitigate this performance impact, a
> > policy mechanism is provided to administrators, distro maintainers, and
> > developers to control vector state discard in the form of a sysctl knob:
> > 
> > /proc/sys/abi/riscv_v_vstate_discard
> > 
> > Valid values are:
> > 
> > 0: Do not discard vector state during syscall
> > 1: Discard vector state during syscall
> > 
> 
> Is the intention for this this mean "don't guarantee vector state is
> clobbered" or "preserve vector state"?  I suspect it is the former, but the
> wording seems unclear.  Additionally, if that's indeed the case, maybe the
> documentation should more clearly articulate the tradeoff (performance vs.
> security/robustness).

Thanks for your comment. I agree it could be worded better and the trade
off should be explained. I believe the motivation behind clobbering in
all syscalls was to make sure that userspace does not rely on the vector
being preserved through a syscall. The goal of this strictness was to
catch any programs that were misbehaving.

Thus, it might be better if I described it as:

0: Vector state is not strictly clobbered in all syscalls
1: Mandatory clobbering of vector state in all syscalls

Thanks,
Drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Mon, Jul 21, 2025 at 02:35:38PM +0200, Radim Krčmář wrote:
> 2025-07-18T20:39:13-07:00, Drew Fustini <fustini@kernel.org>:
> > From: Drew Fustini <dfustini@tenstorrent.com>
> >
> > Clobbering the vector registers can significantly increase system call
> > latency for some implementations. To mitigate this performance impact, a
> > policy mechanism is provided to administrators, distro maintainers, and
> > developers to control vector state discard in the form of a sysctl knob:
> >
> > /proc/sys/abi/riscv_v_vstate_discard
> >
> > Valid values are:
> >
> > 0: Do not discard vector state during syscall
> > 1: Discard vector state during syscall
> >
> > The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> 
> I think it is a bit more complicated to do this nicely...
> Programs don't have to save/restore vector registers around syscalls
> when compiled for riscv_v_vstate_discard=0, so running under
> riscv_v_vstate_discard=1 would break them.

Thanks for your comments. You raise a good point that this sysctl can
lead to the case where a program might be compiled to not save/restore
vector registers around syscalls. That same program would not work
correctly if the sysadmin changes riscv_v_vstate_discard to 1.

> Shouldn't we have a way to prevent riscv_v_vstate_discard=0 executable
> from running with riscv_v_vstate_discard=1?

Yes, this does make me concerned that a program could crash as a result
of this sysctl which would be confusing for the user as they may not
even be aware of this sysctl. I'll have to think more about how such a
protection could work.

> 
> > Fixes: 9657e9b7d253 ("riscv: Discard vector state on syscalls")
> 
> Programs compiled for riscv_v_vstate_discard=1 are compatible with 0, so
> I think it would be simplest to revert that patch, and pretended it
> never happened... (The issues will eventually go away.)

I agree that reverting the existing discard behavior would be the
simplest solution to the peformance issue observed on some
implementations. However, I believe there is also the desire to have a
way to enforce strict clobbering across syscalls to catch any incorrect
behavior while testing. I was hoping a syscall could allow both use
cases to be handled, but you raise good points about compatibility.

> Shouldn't the RISC-V Linux syscall ABI be defined somewhere?
> How come we could have broken it with 9657e9b7d253?

I may have been wrong to use a Fixes tag for 9657e9b7d253. I was trying
to highlight the original discussion that I was trying to address with
this sysctl patch.

> 
> Thanks.
> 
> ---
> I don't think it makes much sense to clobber vector registers on a
> syscall -- a kernel might not even touch vector registers, so they are
> efforlessly preserved in that case.
> If kernel needs to use vector registers in the syscall, then the kernel
> needs to prevent any register leaks to userspace anyway by restoring
> some state into them -- and why not restore the original one?
> 
> I think that main point of clobbering would be to optimize
> context-switches after the userspace is not using vector registers
> anymore, but it's terribly inefficient if the ratio of syscalls to
> context switches is high.
> Linux can also try to detect the situation, and turn to lazy vector
> context-switch, with sstatus.VS=off, instead of eagerly restoring
> clobbered state.
> (A good indicator might be that the userspace hasn't dirtied the vectors
>  since the last context-switch -- kernel didn't need to save the state,
>  so it will restore lazily.)

I think this is an interesting discussion to have. I was hoping this
patch would get people discussing if mandatory vector state cloberring
is really something that should be do in syscalls.

Thanks,
Drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Mon, Jul 21, 2025 at 04:54:25PM +0200, Radim Krčmář wrote:
> 2025-07-21T14:35:38+02:00, Radim Krčmář <rkrcmar@ventanamicro.com>:
> > Shouldn't the RISC-V Linux syscall ABI be defined somewhere?
> 
> To clarify this point.  My issue is with the following part in
> Documentation/arch/riscv/vector.rst:
> 
> >>  As indicated by version 1.0 of the V extension [1], vector registers are
> >>  clobbered by system calls.
> >>  [...]
> >>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
> 
> The ISA does not say that vector registers are clobbered by system
> calls.  All the ISA says is:
> 
>   "This Appendix is only a placeholder to help explain the conventions
>    used in the code examples, and is not considered frozen or
>    part of the ratification process.  The official RISC-V psABI document
>    is being expanded to specify the vector calling conventions."
> 
> while the RISC-V psABI says:
> 
>   "The calling convention for system calls does not fall within the
>    scope of this document. Please refer to the documentation of the
>    RISC-V execution environment interface (e.g OS kernel ABI, SBI)."
> 
> We made a circular dependency, misinterpreted the ISA, and probably
> implemented a suboptimal syscall ABI -- preserving vector registers
> seems strictly better.

Thanks for providing these references. It does seem like this is
something that an OS can decide and is not mandated by the ISA or psABI.

> > How come we could have broken it with 9657e9b7d253?
> 
> We changed the ABI once, so maybe we can change it back?

Reverting 9657e9b7d253 would solve the performance issue for some
implementations that I've highlighted in this patch. However, I am
interested to hear from others that feel the current mandatory
clobbering behavior is ideal for testing (and maybe security?).

Thanks,
Drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Mon, Jul 21, 2025 at 01:59:39PM -0700, Drew Fustini wrote:
> On Mon, Jul 21, 2025 at 08:13:48AM -0400, Darius Rad wrote:
> > On Fri, Jul 18, 2025 at 08:39:13PM -0700, Drew Fustini wrote:
> > > From: Drew Fustini <dfustini@tenstorrent.com>
> > > 
> > > Clobbering the vector registers can significantly increase system call
> > > latency for some implementations. To mitigate this performance impact, a
> > > policy mechanism is provided to administrators, distro maintainers, and
> > > developers to control vector state discard in the form of a sysctl knob:
> > > 
> > > /proc/sys/abi/riscv_v_vstate_discard
> > > 
> > > Valid values are:
> > > 
> > > 0: Do not discard vector state during syscall
> > > 1: Discard vector state during syscall
> > > 
> > 
> > Is the intention for this this mean "don't guarantee vector state is
> > clobbered" or "preserve vector state"?  I suspect it is the former, but the
> > wording seems unclear.  Additionally, if that's indeed the case, maybe the
> > documentation should more clearly articulate the tradeoff (performance vs.
> > security/robustness).
> 
> Thanks for your comment. I agree it could be worded better and the trade
> off should be explained. I believe the motivation behind clobbering in
> all syscalls was to make sure that userspace does not rely on the vector
> being preserved through a syscall. The goal of this strictness was to
> catch any programs that were misbehaving.
> 
> Thus, it might be better if I described it as:
> 
> 0: Vector state is not strictly clobbered in all syscalls
> 1: Mandatory clobbering of vector state in all syscalls

Darius - sorry, I seem to have failed to put you in the To: field of my
reply.

Thanks,
Drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Vivian Wang]

On 7/19/25 11:39, Drew Fustini wrote:

> From: Drew Fustini <dfustini@tenstorrent.com>
>
> Clobbering the vector registers can significantly increase system call
> latency for some implementations. To mitigate this performance impact, a
> policy mechanism is provided to administrators, distro maintainers, and
> developers to control vector state discard in the form of a sysctl knob:

So I had an idea: Is it possible to avoid repeatedly discarding the
state on every syscall by setting VS to Initial after discarding, and
avoiding discarding when VS is Initial? So:

if (VS == Clean || VS == Dirty) {
    clobber;
    VS = Initial;
}

This would avoid this problem with syscall-heavy user programs while
adding minimum overhead for everything else.

Note that I haven't checked closely on how VS = Initial is currently
used in Linux. This might require changes to vector init make the really
first-time initial state and the after-syscall initial state the same.
Or maybe I misunderstood entirely.

Vivian "dramforever" Wang

> /proc/sys/abi/riscv_v_vstate_discard
>
> Valid values are:
>
> 0: Do not discard vector state during syscall
> 1: Discard vector state during syscall
>
> The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
>
> Fixes: 9657e9b7d253 ("riscv: Discard vector state on syscalls")
> Signed-off-by: Drew Fustini <dfustini@tenstorrent.com>
> ---
>  Documentation/arch/riscv/vector.rst | 15 +++++++++++++++
>  arch/riscv/Kconfig                  | 10 ++++++++++
>  arch/riscv/include/asm/vector.h     |  4 ++++
>  arch/riscv/kernel/vector.c          | 16 +++++++++++++++-
>  4 files changed, 44 insertions(+), 1 deletion(-)
>
> [...]


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Radim Krčmář]

2025-07-24T05:55:54+08:00, Vivian Wang <wangruikang@iscas.ac.cn>:
> On 7/19/25 11:39, Drew Fustini wrote:
>> From: Drew Fustini <dfustini@tenstorrent.com>
>> Clobbering the vector registers can significantly increase system call
>> latency for some implementations. To mitigate this performance impact, a
>> policy mechanism is provided to administrators, distro maintainers, and
>> developers to control vector state discard in the form of a sysctl knob:
>
> So I had an idea: Is it possible to avoid repeatedly discarding the
> state on every syscall by setting VS to Initial after discarding, and
> avoiding discarding when VS is Initial? So:
>
> if (VS == Clean || VS == Dirty) {
>     clobber;
>     VS = Initial;
> }
>
> This would avoid this problem with syscall-heavy user programs while
> adding minimum overhead for everything else.

I think your proposal improves the existing code, but if a userspace is
using vectors, it's likely also restoring them after a syscall, so the
state would immediately get dirty, and the next syscall would again
needlessly clobber vector registers.

Preserving the vector state still seems better for userspaces that use
both vectors and syscalls.

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Vivian Wang]

On 7/25/25 18:18, Radim Krčmář wrote:
> 2025-07-24T05:55:54+08:00, Vivian Wang <wangruikang@iscas.ac.cn>:
>> On 7/19/25 11:39, Drew Fustini wrote:
>>> From: Drew Fustini <dfustini@tenstorrent.com>
>>> Clobbering the vector registers can significantly increase system call
>>> latency for some implementations. To mitigate this performance impact, a
>>> policy mechanism is provided to administrators, distro maintainers, and
>>> developers to control vector state discard in the form of a sysctl knob:
>> So I had an idea: Is it possible to avoid repeatedly discarding the
>> state on every syscall by setting VS to Initial after discarding, and
>> avoiding discarding when VS is Initial? So:
>>
>> if (VS == Clean || VS == Dirty) {
>>     clobber;
>>     VS = Initial;
>> }
>>
>> This would avoid this problem with syscall-heavy user programs while
>> adding minimum overhead for everything else.
> I think your proposal improves the existing code, but if a userspace is
> using vectors, it's likely also restoring them after a syscall, so the
> state would immediately get dirty, and the next syscall would again
> needlessly clobber vector registers.

Without any data to back it up, I would say that my understanding is
that this should be a rare case, only happening if e.g. someone is
adding printf debugging to their vector code. Otherwise, vector loops
should not have syscalls in them.

A more reasonable worry would be programs using RVV everywhere in all
sorts of common operations. In that case, alternating syscalls and
vectors would make the discarding wasteful.

> Preserving the vector state still seems better for userspaces that use
> both vectors and syscalls.

If we can expect e.g. userspace programs to primarily repeatedly use RVV
with no syscalls between loops, *or* primarily repeatedly use syscalls
with rare occurrences of RVV between syscalls. This way, the primarily
syscall programs can benefit from slightly switching, since there's no
need to save and restore state for those most of the time. In effect,
syscalls serves as a hint that RVV is over. The primarily RVV programs
should not be switching as much - if they are, that's a sign of CPU
resources being oversubscribed. 

Having said all of that, I am actually slightly more interested in why
vmv.v.vi is *so slow* on SiFive X280. I wonder if there would be a more
microarchitectural favorable ways to just put a bunch of ones in some
vector registers? Would 0 be better?

Vivian "dramforever" Wang


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Radim Krčmář]

2025-07-25T23:01:03+08:00, Vivian Wang <wangruikang@iscas.ac.cn>:
> On 7/25/25 18:18, Radim Krčmář wrote:
>> 2025-07-24T05:55:54+08:00, Vivian Wang <wangruikang@iscas.ac.cn>:
>>> On 7/19/25 11:39, Drew Fustini wrote:
>>>> From: Drew Fustini <dfustini@tenstorrent.com>
>>>> Clobbering the vector registers can significantly increase system call
>>>> latency for some implementations. To mitigate this performance impact, a
>>>> policy mechanism is provided to administrators, distro maintainers, and
>>>> developers to control vector state discard in the form of a sysctl knob:
>>> So I had an idea: Is it possible to avoid repeatedly discarding the
>>> state on every syscall by setting VS to Initial after discarding, and
>>> avoiding discarding when VS is Initial? So:
>>>
>>> if (VS == Clean || VS == Dirty) {
>>>     clobber;
>>>     VS = Initial;
>>> }
>>>
>>> This would avoid this problem with syscall-heavy user programs while
>>> adding minimum overhead for everything else.
>> I think your proposal improves the existing code, but if a userspace is
>> using vectors, it's likely also restoring them after a syscall, so the
>> state would immediately get dirty, and the next syscall would again
>> needlessly clobber vector registers.
>
> Without any data to back it up, I would say that my understanding is
> that this should be a rare case, only happening if e.g. someone is
> adding printf debugging to their vector code. Otherwise, vector loops
> should not have syscalls in them.
>
> A more reasonable worry would be programs using RVV everywhere in all
> sorts of common operations. In that case, alternating syscalls and
> vectors would make the discarding wasteful.

Good point.  Yeah, auto-vectorization might be hindered.

In the worst case, users could just notice that it's slowing programs
down, and disable it without looking for the cause.

>> Preserving the vector state still seems better for userspaces that use
>> both vectors and syscalls.
>
> If we can expect e.g. userspace programs to primarily repeatedly use RVV
> with no syscalls between loops, *or* primarily repeatedly use syscalls
> with rare occurrences of RVV between syscalls. This way, the primarily
> syscall programs can benefit from slightly switching, since there's no
> need to save and restore state for those most of the time. In effect,
> syscalls serves as a hint that RVV is over.

This would need deeper analysis, and we will probably never be correct
with a system-wide policy regardless -- a room for prctl?

I think there might be a lot of programs that have a repeating pattern
of compute -> syscall (e.g. to write results), and clobbering is losing
performance if a program does more than a single loop per switch.

>                                             The primarily RVV programs
> should not be switching as much - if they are, that's a sign of CPU
> resources being oversubscribed.

Yes, but clobbering only gives benefits on a switch, so we don't want to
clobber if there are more syscall than switches.

Well, there is a way: a syscall could just set VS=Initial, and if
userspace doesn't dirty vector registers, a restore would set the
registers to whatever the initial state is.
No vector registers touched on syscall, or save.

This works as we don't have to do anything when "clobbering" -- the
registers are unspecified after a syscall.
The downside is that users might (incorrectly) depend on the unspecified
value without dirtying, so the unspecified value could change at an
arbitrary point, which would provide some interesting debugging cases.

(And it's still suboptimal if software actually wants to preserve
 vectors across syscalls.)

> Having said all of that, I am actually slightly more interested in why
> vmv.v.vi is *so slow* on SiFive X280. I wonder if there would be a more
> microarchitectural favorable ways to just put a bunch of ones in some
> vector registers? Would 0 be better?

No idea, and there are a lot of options to try, but it would be quite
sad if we had to have special case for each implementation.

Thanks.

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Fri, Jul 25, 2025 at 08:47:04PM +0200, Radim Krčmář wrote:
> 2025-07-25T23:01:03+08:00, Vivian Wang <wangruikang@iscas.ac.cn>:
> > On 7/25/25 18:18, Radim Krčmář wrote:
> >> 2025-07-24T05:55:54+08:00, Vivian Wang <wangruikang@iscas.ac.cn>:
> >>> On 7/19/25 11:39, Drew Fustini wrote:
> >>>> From: Drew Fustini <dfustini@tenstorrent.com>
> >>>> Clobbering the vector registers can significantly increase system call
> >>>> latency for some implementations. To mitigate this performance impact, a
> >>>> policy mechanism is provided to administrators, distro maintainers, and
> >>>> developers to control vector state discard in the form of a sysctl knob:
> >>> So I had an idea: Is it possible to avoid repeatedly discarding the
> >>> state on every syscall by setting VS to Initial after discarding, and
> >>> avoiding discarding when VS is Initial? So:
> >>>
> >>> if (VS == Clean || VS == Dirty) {
> >>>     clobber;
> >>>     VS = Initial;
> >>> }
> >>>
> >>> This would avoid this problem with syscall-heavy user programs while
> >>> adding minimum overhead for everything else.
> >> I think your proposal improves the existing code, but if a userspace is
> >> using vectors, it's likely also restoring them after a syscall, so the
> >> state would immediately get dirty, and the next syscall would again
> >> needlessly clobber vector registers.
> >
> > Without any data to back it up, I would say that my understanding is
> > that this should be a rare case, only happening if e.g. someone is
> > adding printf debugging to their vector code. Otherwise, vector loops
> > should not have syscalls in them.
> >
> > A more reasonable worry would be programs using RVV everywhere in all
> > sorts of common operations. In that case, alternating syscalls and
> > vectors would make the discarding wasteful.
> 
> Good point.  Yeah, auto-vectorization might be hindered.

Yes, I think that userspace vector usage will become more common over
time even for "ordinary" programs as compilers and libraries improve.
For example, it may be the case that the majority of userspace binaries
will use vector once the ifunc memcpy patches go in.

> In the worst case, users could just notice that it's slowing programs
> down, and disable it without looking for the cause.

I think that a default policy of not clobbering in syscalls would be the
best trade off. I gave CONFIG_RISCV_ISA_V_VSTATE_DISCARD a default of n
in this patch, and I imagined that people like Palmer, who wanted it for
test suites, could change the default or use the sysctl.

> 
> >> Preserving the vector state still seems better for userspaces that use
> >> both vectors and syscalls.
> >
> > If we can expect e.g. userspace programs to primarily repeatedly use RVV
> > with no syscalls between loops, *or* primarily repeatedly use syscalls
> > with rare occurrences of RVV between syscalls. This way, the primarily
> > syscall programs can benefit from slightly switching, since there's no
> > need to save and restore state for those most of the time. In effect,
> > syscalls serves as a hint that RVV is over.
> 
> This would need deeper analysis, and we will probably never be correct
> with a system-wide policy regardless -- a room for prctl?
> 
> I think there might be a lot of programs that have a repeating pattern
> of compute -> syscall (e.g. to write results), and clobbering is losing
> performance if a program does more than a single loop per switch.

It's interesting that you mention prctl as it does seem like that could
play a role here. If people think that one syscall clobbering behavior
for the whole system is too limited, then maybe prctl could be a better
solution. I believe it should default to not clobbering. It could be
enabled for test suites in CI that want the strict clobbering, or for
programs that are known to work better with clobbering enabled.

Thanks,
Drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Mon, Jul 21, 2025 at 02:16:46PM -0700, Drew Fustini wrote:
> On Mon, Jul 21, 2025 at 02:35:38PM +0200, Radim Krčmář wrote:
> > 2025-07-18T20:39:13-07:00, Drew Fustini <fustini@kernel.org>:
> > > From: Drew Fustini <dfustini@tenstorrent.com>
> > >
> > > Clobbering the vector registers can significantly increase system call
> > > latency for some implementations. To mitigate this performance impact, a
> > > policy mechanism is provided to administrators, distro maintainers, and
> > > developers to control vector state discard in the form of a sysctl knob:
> > >
> > > /proc/sys/abi/riscv_v_vstate_discard
> > >
> > > Valid values are:
> > >
> > > 0: Do not discard vector state during syscall
> > > 1: Discard vector state during syscall
> > >
> > > The initial state is controlled by CONFIG_RISCV_ISA_V_VSTATE_DISCARD.
> > 
> > I think it is a bit more complicated to do this nicely...
> > Programs don't have to save/restore vector registers around syscalls
> > when compiled for riscv_v_vstate_discard=0, so running under
> > riscv_v_vstate_discard=1 would break them.
> 
> Thanks for your comments. You raise a good point that this sysctl can
> lead to the case where a program might be compiled to not save/restore
> vector registers around syscalls. That same program would not work
> correctly if the sysadmin changes riscv_v_vstate_discard to 1.
> 
> > Shouldn't we have a way to prevent riscv_v_vstate_discard=0 executable
> > from running with riscv_v_vstate_discard=1?
> 
> Yes, this does make me concerned that a program could crash as a result
> of this sysctl which would be confusing for the user as they may not
> even be aware of this sysctl. I'll have to think more about how such a
> protection could work.

After further consideration, I think that it would be appropriate for
programs that don't save/restore vector registers around syscalls to
break when riscv_v_vstate_discard=1. I believe the primary motivation
of the original change 2 years ago was to catch programs that rely on
the vector state being retained across syscalls. Thus, I think that this
behavior would be consistent with that.

Bjorn told me he is away at the moment but hopefully he can provide his
perspective when he is back.

Thanks,
Drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Palmer Dabbelt]

On Mon, 21 Jul 2025 07:54:25 PDT (-0700), rkrcmar@ventanamicro.com wrote:
> 2025-07-21T14:35:38+02:00, Radim Krčmář <rkrcmar@ventanamicro.com>:
>> Shouldn't the RISC-V Linux syscall ABI be defined somewhere?
>
> To clarify this point.  My issue is with the following part in
> Documentation/arch/riscv/vector.rst:
>
>>>  As indicated by version 1.0 of the V extension [1], vector registers are
>>>  clobbered by system calls.
>>>  [...]
>>>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
>
> The ISA does not say that vector registers are clobbered by system
> calls.  All the ISA says is:
>
>   "This Appendix is only a placeholder to help explain the conventions
>    used in the code examples, and is not considered frozen or
>    part of the ratification process.  The official RISC-V psABI document
>    is being expanded to specify the vector calling conventions."

It also says

    Executing a system call causes all caller-saved vector registers 
    (v0-v31, vl, vtype) and vstart to become unspecied.

in the ISA manual, a few sentences later in that page.  So that's what 
we were trying to get at with the documentation pointer, but maybe it's 
better to have something more explicit like

    diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
    index 3987f5f76a9d..e8591660a7bb 100644
    --- a/Documentation/arch/riscv/vector.rst
    +++ b/Documentation/arch/riscv/vector.rst
    @@ -134,7 +134,10 @@ processes in form of sysctl knob:
     3.  Vector Register State Across System Calls
     ---------------------------------------------
    
    -As indicated by version 1.0 of the V extension [1], vector registers are
    -clobbered by system calls.
    +Linux adopts the syscall ABI proposed  by version 1.0 of the V extension [1],
    +where vector registers are clobbered by system calls.  Specifically
    +
    +    Executing a system call causes all caller-saved vector registers
    +    (v0-v31, vl, vtype) and vstart to become unspecied.
    
     1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc

> while the RISC-V psABI says:
>
>   "The calling convention for system calls does not fall within the
>    scope of this document. Please refer to the documentation of the
>    RISC-V execution environment interface (e.g OS kernel ABI, SBI)."
>
> We made a circular dependency, misinterpreted the ISA, and probably
> implemented a suboptimal syscall ABI -- preserving vector registers
> seems strictly better.

We'd really need userspace to have an ABI that preserves vector 
registers for it to be useful in the kernel.  As it stands there's 
pretty much nothing that's going to have useful vector state over a 
syscall, as they're almost always hidden behind some C function and 
those clobber the vector state.  I have a patch out for GCC that enables 
a system-wide vector ABI, but I don't have time to test/benchmark it so 
it's kind of hard to justify.

That said:

My first guess here would be that trashing the V register 
state is still faster on the machines that triggered this patch, it's 
just that the way we're trashing it is slow.  We're doing some wacky 
things in there (VILL, LMUL, clearing to -1), so it's not surprising 
that some implementations are slow on these routines.

This came up during the original patch and we decided to just go with 
this way (which is recommended by the ISA) until someone could 
demonstrate it's slow, so sounds like it's time to go revisit those.  

So I'd start with something like

    diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
    index b61786d43c20..1fba33e62d2b 100644
    --- a/arch/riscv/include/asm/vector.h
    +++ b/arch/riscv/include/asm/vector.h
    @@ -287,7 +287,6 @@ static inline void __riscv_v_vstate_discard(void)
                    "vmv.v.i        v8, -1\n\t"
                    "vmv.v.i        v16, -1\n\t"
                    "vmv.v.i        v24, -1\n\t"
    -               "vsetvl         %0, x0, %1\n\t"
                    ".option pop\n\t"
                    : "=&r" (vl) : "r" (vtype_inval));

to try and see if we're tripping over bad implementation behavior, in 
which case we can just hide this all in the kernel.  Then we can split 
out these performance issues from other things like lazy save/restore 
and a V-preserving uABI, as it stands this is all sort of getting mixed 
up.

>> How come we could have broken it with 9657e9b7d253?
>
> We changed the ABI once, so maybe we can change it back?

We didn't change the ABI, the documentation always said "vector registers are
clobbered by system calls".

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Radim Krčmář]

2025-07-30T18:05:59-07:00, Palmer Dabbelt <palmer@dabbelt.com>:
> On Mon, 21 Jul 2025 07:54:25 PDT (-0700), rkrcmar@ventanamicro.com wrote:
>> 2025-07-21T14:35:38+02:00, Radim Krčmář <rkrcmar@ventanamicro.com>:
>>> Shouldn't the RISC-V Linux syscall ABI be defined somewhere?
>>
>> To clarify this point.  My issue is with the following part in
>> Documentation/arch/riscv/vector.rst:
>>
>>>>  As indicated by version 1.0 of the V extension [1], vector registers are
>>>>  clobbered by system calls.
>>>>  [...]
>>>>  1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
>>
>> The ISA does not say that vector registers are clobbered by system
>> calls.  All the ISA says is:
>>
>>   "This Appendix is only a placeholder to help explain the conventions
>>    used in the code examples, and is not considered frozen or
>>    part of the ratification process.  The official RISC-V psABI document
>>    is being expanded to specify the vector calling conventions."
>
> It also says
>
>     Executing a system call causes all caller-saved vector registers 
>     (v0-v31, vl, vtype) and vstart to become unspecied.
>
> in the ISA manual, a few sentences later in that page.

It also says

  Most OSes will choose to either leave these registers intact or reset
  them to their initial state to avoid leaking information across
  process boundaries.

Both options make sense, but we're not doing either.

>                                                         So that's what 
> we were trying to get at with the documentation pointer, but maybe it's 
> better to have something more explicit like
>
>     diff --git a/Documentation/arch/riscv/vector.rst b/Documentation/arch/riscv/vector.rst
>     index 3987f5f76a9d..e8591660a7bb 100644
>     --- a/Documentation/arch/riscv/vector.rst
>     +++ b/Documentation/arch/riscv/vector.rst
>     @@ -134,7 +134,10 @@ processes in form of sysctl knob:
>      3.  Vector Register State Across System Calls
>      ---------------------------------------------
>     
>     -As indicated by version 1.0 of the V extension [1], vector registers are
>     -clobbered by system calls.
>     +Linux adopts the syscall ABI proposed  by version 1.0 of the V extension [1],

The whole section is just a non-normative convention for its own code
examples, so I wouldn't say the V extension proposed it for anyone.

We can just say what Linux does without referencing anything, because
nothing tells Linux what to do.

>     +where vector registers are clobbered by system calls.  Specifically
>     +
>     +    Executing a system call causes all caller-saved vector registers
>     +    (v0-v31, vl, vtype) and vstart to become unspecied.

We still need to define which registers are caller-saved.

No vector registers are preserved in the current syscall ABI, so I'd
just omit "caller-saved", to define that all vector registers become
unspecified.

>      1: https://github.com/riscv/riscv-v-spec/blob/master/calling-convention.adoc
>
>> while the RISC-V psABI says:
>>
>>   "The calling convention for system calls does not fall within the
>>    scope of this document. Please refer to the documentation of the
>>    RISC-V execution environment interface (e.g OS kernel ABI, SBI)."
>>
>> We made a circular dependency, misinterpreted the ISA, and probably
>> implemented a suboptimal syscall ABI -- preserving vector registers
>> seems strictly better.
>
> We'd really need userspace to have an ABI that preserves vector 
> registers for it to be useful in the kernel.

I don't really like over-catering to the standard psABI -- it's a relic
of simpler times, and I hope programs will eventually be freed of it.

>                                               As it stands there's 
> pretty much nothing that's going to have useful vector state over a 
> syscall, as they're almost always hidden behind some C function and 
> those clobber the vector state.

Userspace doesn't even have to carry vector state over syscall -- just
using vector, doing syscall, using other vector, doing syscall is
hindered by the current design, because each syscall has to touch vector
registers for not good reason.

>                                  I have a patch out for GCC that enables 
> a system-wide vector ABI, but I don't have time to test/benchmark it so 
> it's kind of hard to justify.

You mean enabling the alternative psABI for vectors?

https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/712449f8efcf6b3acd9e2a2a7ddfe89486317877/riscv-cc.adoc#calling-convention-variant

> That said:
>
> My first guess here would be that trashing the V register 
> state is still faster on the machines that triggered this patch, it's 
> just that the way we're trashing it is slow.  We're doing some wacky 
> things in there (VILL, LMUL, clearing to -1), so it's not surprising 
> that some implementations are slow on these routines.

I am afraid we might end up with a commmandline, DT, or
mvendorid+marchid+mimpid hint to pick the best method at runtime, or
boot time benchmarking in case it's not even known.

> This came up during the original patch and we decided to just go with 
> this way (which is recommended by the ISA) until someone could 
> demonstrate it's slow, so sounds like it's time to go revisit those.  
>
> So I'd start with something like
>
>     diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
>     index b61786d43c20..1fba33e62d2b 100644
>     --- a/arch/riscv/include/asm/vector.h
>     +++ b/arch/riscv/include/asm/vector.h
>     @@ -287,7 +287,6 @@ static inline void __riscv_v_vstate_discard(void)
>                     "vmv.v.i        v8, -1\n\t"
>                     "vmv.v.i        v16, -1\n\t"
>                     "vmv.v.i        v24, -1\n\t"
>     -               "vsetvl         %0, x0, %1\n\t"
>                     ".option pop\n\t"
>                     : "=&r" (vl) : "r" (vtype_inval));
>
> to try and see if we're tripping over bad implementation behavior, in 
> which case we can just hide this all in the kernel.  Then we can split 
> out these performance issues from other things like lazy save/restore 
> and a V-preserving uABI, as it stands this is all sort of getting mixed 
> up.

Yeah, the discussion got a bit out of hand.

I don't see much point in doing minor changes to the current design, as
it isn't anywhere near the Pareto front.

If we want to touch vectors on syscalls, I think it makes sense to start
with Vivian's proposal -- eagerly initializing vectors in syscalls
provides at least some advantage when eventually doing a context switch.

(If the performance is still bad, then we can initialize lazily on vector
 restore, which should be optimal for everything except programs that
 want to preserve vectors across syscalls.)

>>> How come we could have broken it with 9657e9b7d253?
>>
>> We changed the ABI once, so maybe we can change it back?
>
> We didn't change the ABI, the documentation always said "vector registers are
> clobbered by system calls".

My bad, I didn't see the patch went in with the initial version, thanks.

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Wed, Jul 30, 2025 at 06:05:59PM -0700, Palmer Dabbelt wrote:
> My first guess here would be that trashing the V register state is still
> faster on the machines that triggered this patch, it's just that the way
> we're trashing it is slow.  We're doing some wacky things in there (VILL,
> LMUL, clearing to -1), so it's not surprising that some implementations are
> slow on these routines.
> 
> This came up during the original patch and we decided to just go with this
> way (which is recommended by the ISA) until someone could demonstrate it's
> slow, so sounds like it's time to go revisit those.
> 
> So I'd start with something like
> 
>    diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
>    index b61786d43c20..1fba33e62d2b 100644
>    --- a/arch/riscv/include/asm/vector.h
>    +++ b/arch/riscv/include/asm/vector.h
>    @@ -287,7 +287,6 @@ static inline void __riscv_v_vstate_discard(void)
>                    "vmv.v.i        v8, -1\n\t"
>                    "vmv.v.i        v16, -1\n\t"
>                    "vmv.v.i        v24, -1\n\t"
>    -               "vsetvl         %0, x0, %1\n\t"
>                    ".option pop\n\t"
>                    : "=&r" (vl) : "r" (vtype_inval));
> 
> to try and see if we're tripping over bad implementation behavior, in which
> case we can just hide this all in the kernel.  Then we can split out these
> performance issues from other things like lazy save/restore and a
> V-preserving uABI, as it stands this is all sort of getting mixed up.

Thank you for your insights and the suggestion of removing vsetvl.

Using our v6.16-rc1 branch [1], the avg duration of getppid() is 198 ns
with the existing upstream behavior in __riscv_v_vstate_discard():

debian@tt-blackhole:~$ ./null_syscall --vsetvli
vsetvli complete
 iterations: 1000000000
   duration: 198 seconds
avg latency: 198.10 ns

I removed 'vsetvl' as you suggested but the average duration only
decreased a very small amount to 197.5 ns, so it seems that the other
instructions are what is taking a lot of time on the X280 cores:

debian@tt-blackhole:~$ ./null_syscall --vsetvli
vsetvli complete
 iterations: 1000000000
   duration: 197 seconds
avg latency: 197.53 ns

This is compared to a duration of 150 ns when using this patch with
abi.riscv_v_vstate_discard=0 which skips all the clobbering assembly.

Do you have any other suggestions for the __riscv_v_vstate_discard()
inline assembly that might be worth me testing on the X280 cores?

Thanks,
Drew

[1] https://github.com/tenstorrent/linux/tree/tt-blackhole-v6.16-rc1

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH] riscv: Add sysctl to control discard of vstate during syscall

[Drew Fustini]

On Fri, Aug 01, 2025 at 02:41:51PM -0700, Drew Fustini wrote:
> On Wed, Jul 30, 2025 at 06:05:59PM -0700, Palmer Dabbelt wrote:
> > My first guess here would be that trashing the V register state is still
> > faster on the machines that triggered this patch, it's just that the way
> > we're trashing it is slow.  We're doing some wacky things in there (VILL,
> > LMUL, clearing to -1), so it's not surprising that some implementations are
> > slow on these routines.
> > 
> > This came up during the original patch and we decided to just go with this
> > way (which is recommended by the ISA) until someone could demonstrate it's
> > slow, so sounds like it's time to go revisit those.
> > 
> > So I'd start with something like
> > 
> >    diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
> >    index b61786d43c20..1fba33e62d2b 100644
> >    --- a/arch/riscv/include/asm/vector.h
> >    +++ b/arch/riscv/include/asm/vector.h
> >    @@ -287,7 +287,6 @@ static inline void __riscv_v_vstate_discard(void)
> >                    "vmv.v.i        v8, -1\n\t"
> >                    "vmv.v.i        v16, -1\n\t"
> >                    "vmv.v.i        v24, -1\n\t"
> >    -               "vsetvl         %0, x0, %1\n\t"
> >                    ".option pop\n\t"
> >                    : "=&r" (vl) : "r" (vtype_inval));
> > 
> > to try and see if we're tripping over bad implementation behavior, in which
> > case we can just hide this all in the kernel.  Then we can split out these
> > performance issues from other things like lazy save/restore and a
> > V-preserving uABI, as it stands this is all sort of getting mixed up.
> 
> Thank you for your insights and the suggestion of removing vsetvl.
> 
> Using our v6.16-rc1 branch [1], the avg duration of getppid() is 198 ns
> with the existing upstream behavior in __riscv_v_vstate_discard():
> 
> debian@tt-blackhole:~$ ./null_syscall --vsetvli
> vsetvli complete
>  iterations: 1000000000
>    duration: 198 seconds
> avg latency: 198.10 ns
> 
> I removed 'vsetvl' as you suggested but the average duration only
> decreased a very small amount to 197.5 ns, so it seems that the other
> instructions are what is taking a lot of time on the X280 cores:
> 
> debian@tt-blackhole:~$ ./null_syscall --vsetvli
> vsetvli complete
>  iterations: 1000000000
>    duration: 197 seconds
> avg latency: 197.53 ns
> 
> This is compared to a duration of 150 ns when using this patch with
> abi.riscv_v_vstate_discard=0 which skips all the clobbering assembly.
> 
> Do you have any other suggestions for the __riscv_v_vstate_discard()
> inline assembly that might be worth me testing on the X280 cores?

I have tried leaving vsetvl but removing vmv.v.i instructions instead.
This made a difference on the X280 and reduced duration from 198 ns to
161 ns. This compared to an average duration of 148 ns when doing no
clobbering at all.

However, removing the vmv.v.i from the discard assembly doesn't help
much on our own out-of-order core due to still having to update the
vector state in status. Thus I'm still keen to have some way to entirely
opt out of __riscv_v_vstate_discard() on the do_trap_ecall_u() path.

Thanks,
Drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv