[PATCH 0/2] riscv: Allow vlenb to be probed from DT

[PATCH 0/2] riscv: Allow vlenb to be probed from DT

[Charlie Jenkins]

The kernel currently requires all harts to have the same value in the
vlenb csr that is present when a hart supports vector. In order to read
this csr, the kernel needs to boot the hart. Adding vlenb to the DT will
allow the kernel to detect the inconsistency early and not waste time
trying to boot harts that it doesn't support.

Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>

---
The two patches in this series were previously part of a series "riscv:
Support vendor extensions and xtheadvector" but has been factored out
due to a lack of reviews on the thead specific parts so that series will
be updated separately.

---
Charlie Jenkins (1):
      riscv: vector: Use vlenb from DT

Conor Dooley (1):
      dt-bindings: riscv: cpus: add a vlen register length property

 Documentation/devicetree/bindings/riscv/cpus.yaml |  6 +++
 arch/riscv/include/asm/cpufeature.h               |  2 +
 arch/riscv/kernel/cpufeature.c                    | 47 +++++++++++++++++++++++
 arch/riscv/kernel/vector.c                        | 12 +++++-
 4 files changed, 66 insertions(+), 1 deletion(-)
---
base-commit: a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
change-id: 20240515-add_vlenb_to_dt-307bb406ecc5
-- 
- Charlie

[PATCH 1/2] dt-bindings: riscv: cpus: add a vlen register length property

[Charlie Jenkins]

From: Conor Dooley <conor.dooley@microchip.com>

Add a property analogous to the vlenb CSR so that software can detect
the vector length of each CPU prior to it being brought online.
Currently software has to assume that the vector length read from the
boot CPU applies to all possible CPUs. On T-Head CPUs implementing
pre-ratification vector, reading the th.vlenb CSR may produce an illegal
instruction trap, so this property is required on such systems.

Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
---
 Documentation/devicetree/bindings/riscv/cpus.yaml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/devicetree/bindings/riscv/cpus.yaml b/Documentation/devicetree/bindings/riscv/cpus.yaml
index d87dd50f1a4b..edcb6a7d9319 100644
--- a/Documentation/devicetree/bindings/riscv/cpus.yaml
+++ b/Documentation/devicetree/bindings/riscv/cpus.yaml
@@ -94,6 +94,12 @@ properties:
     description:
       The blocksize in bytes for the Zicboz cache operations.
 
+  riscv,vlenb:
+    $ref: /schemas/types.yaml#/definitions/uint32
+    description:
+      VLEN/8, the vector register length in bytes. This property is required in
+      systems where the vector register length is not identical on all harts.
+
   # RISC-V has multiple properties for cache op block sizes as the sizes
   # differ between individual CBO extensions
   cache-op-block-size: false

-- 
2.44.0

[PATCH 2/2] riscv: vector: Use vlenb from DT

[Charlie Jenkins]

If vlenb is provided in the device tree, prefer that over reading the
vlenb csr.

Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
---
 arch/riscv/include/asm/cpufeature.h |  2 ++
 arch/riscv/kernel/cpufeature.c      | 47 +++++++++++++++++++++++++++++++++++++
 arch/riscv/kernel/vector.c          | 12 +++++++++-
 3 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h
index 347805446151..0c4f08577015 100644
--- a/arch/riscv/include/asm/cpufeature.h
+++ b/arch/riscv/include/asm/cpufeature.h
@@ -31,6 +31,8 @@ DECLARE_PER_CPU(struct riscv_cpuinfo, riscv_cpuinfo);
 /* Per-cpu ISA extensions. */
 extern struct riscv_isainfo hart_isa[NR_CPUS];
 
+extern u32 riscv_vlenb_of;
+
 void riscv_user_isa_enable(void);
 
 #if defined(CONFIG_RISCV_MISALIGNED)
diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c
index 3ed2359eae35..6c143ea9592b 100644
--- a/arch/riscv/kernel/cpufeature.c
+++ b/arch/riscv/kernel/cpufeature.c
@@ -35,6 +35,8 @@ static DECLARE_BITMAP(riscv_isa, RISCV_ISA_EXT_MAX) __read_mostly;
 /* Per-cpu ISA extensions. */
 struct riscv_isainfo hart_isa[NR_CPUS];
 
+u32 riscv_vlenb_of;
+
 /**
  * riscv_isa_extension_base() - Get base extension word
  *
@@ -648,6 +650,46 @@ static int __init riscv_isa_fallback_setup(char *__unused)
 early_param("riscv_isa_fallback", riscv_isa_fallback_setup);
 #endif
 
+static int has_riscv_homogeneous_vlenb(void)
+{
+	int cpu;
+	u32 prev_vlenb = 0;
+	u32 vlenb;
+
+	/* Ignore vlenb if vector is not enabled in the kernel */
+	if (!IS_ENABLED(CONFIG_RISCV_ISA_V))
+		return 0;
+
+	for_each_possible_cpu(cpu) {
+		struct device_node *cpu_node;
+
+		cpu_node = of_cpu_device_node_get(cpu);
+		if (!cpu_node) {
+			pr_warn("Unable to find cpu node\n");
+			return -ENOENT;
+		}
+
+		if (of_property_read_u32(cpu_node, "riscv,vlenb", &vlenb)) {
+			of_node_put(cpu_node);
+
+			if (prev_vlenb)
+				return -ENOENT;
+			continue;
+		}
+
+		if (prev_vlenb && vlenb != prev_vlenb) {
+			of_node_put(cpu_node);
+			return -ENOENT;
+		}
+
+		prev_vlenb = vlenb;
+		of_node_put(cpu_node);
+	}
+
+	riscv_vlenb_of = vlenb;
+	return 0;
+}
+
 void __init riscv_fill_hwcap(void)
 {
 	char print_str[NUM_ALPHA_EXTS + 1];
@@ -671,6 +713,11 @@ void __init riscv_fill_hwcap(void)
 			pr_info("Falling back to deprecated \"riscv,isa\"\n");
 			riscv_fill_hwcap_from_isa_string(isa2hwcap);
 		}
+
+		if (elf_hwcap & COMPAT_HWCAP_ISA_V && has_riscv_homogeneous_vlenb() < 0) {
+			pr_warn("Unsupported heterogeneous vlen detected, vector extension disabled.\n");
+			elf_hwcap &= ~COMPAT_HWCAP_ISA_V;
+		}
 	}
 
 	/*
diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c
index 6727d1d3b8f2..e04586cdb7f0 100644
--- a/arch/riscv/kernel/vector.c
+++ b/arch/riscv/kernel/vector.c
@@ -33,7 +33,17 @@ int riscv_v_setup_vsize(void)
 {
 	unsigned long this_vsize;
 
-	/* There are 32 vector registers with vlenb length. */
+	/*
+	 * There are 32 vector registers with vlenb length.
+	 *
+	 * If the riscv,vlenb property was provided by the firmware, use that
+	 * instead of probing the CSRs.
+	 */
+	if (riscv_vlenb_of) {
+		this_vsize = riscv_vlenb_of * 32;
+		return 0;
+	}
+
 	riscv_v_enable();
 	this_vsize = csr_read(CSR_VLENB) * 32;
 	riscv_v_disable();

-- 
2.44.0

Re: [PATCH 0/2] riscv: Allow vlenb to be probed from DT

[Jessica Clarke]

On 15 May 2024, at 22:50, Charlie Jenkins <charlie@rivosinc.com> wrote:
> 
> The kernel currently requires all harts to have the same value in the
> vlenb csr that is present when a hart supports vector. In order to read
> this csr, the kernel needs to boot the hart. Adding vlenb to the DT will
> allow the kernel to detect the inconsistency early and not waste time
> trying to boot harts that it doesn't support.

That doesn’t seem sufficient justification to me. If it can be read
from the hardware, why should we have to put it in the FDT? The whole
point of the FDT is to communicate the hardware configuration that
isn’t otherwise discoverable.

As for T-HEAD stuff, if they need it they can have a custom property.
Though naively I’d assume there’s a way to avoid it still...

Jess

Re: [PATCH 0/2] riscv: Allow vlenb to be probed from DT

[Charlie Jenkins]

On Wed, May 15, 2024 at 11:25:16PM +0100, Jessica Clarke wrote:
> On 15 May 2024, at 22:50, Charlie Jenkins <charlie@rivosinc.com> wrote:
> > 
> > The kernel currently requires all harts to have the same value in the
> > vlenb csr that is present when a hart supports vector. In order to read
> > this csr, the kernel needs to boot the hart. Adding vlenb to the DT will
> > allow the kernel to detect the inconsistency early and not waste time
> > trying to boot harts that it doesn't support.
> 
> That doesn’t seem sufficient justification to me. If it can be read
> from the hardware, why should we have to put it in the FDT? The whole
> point of the FDT is to communicate the hardware configuration that
> isn’t otherwise discoverable.

Yes you are correct in that vlenb is discoverable on any conforming
chip. However, the motivation here is for making decisions about how to
boot a hart before it is booted. By placing it in the device tree, we
are able to disable vector before the chip is booted instead of trying
to boot the chip with vector enabled only to disable it later. In both
cases when there is different vlenb on different harts, all harts still
boot and the outcome is that vector is disabled. The difference is that
with the DT entry, no vector setup code needs to be ran on a booting
hart when the outcome will be that vector is not enabled.

> 
> As for T-HEAD stuff, if they need it they can have a custom property.
> Though naively I’d assume there’s a way to avoid it still...

T-Head does not expose vlenb on all of their chips so I do not know of
any other way of getting the vlenb without having it be provided in a
DT. That was the motivation for this patch in the first place, but
making this available to all vendors allows optimizations to happen
during boot.

- Charlie

> 
> Jess
> 

Re: [PATCH 0/2] riscv: Allow vlenb to be probed from DT

[Jessica Clarke]

On 16 May 2024, at 00:08, Charlie Jenkins <charlie@rivosinc.com> wrote:
> 
> On Wed, May 15, 2024 at 11:25:16PM +0100, Jessica Clarke wrote:
>> On 15 May 2024, at 22:50, Charlie Jenkins <charlie@rivosinc.com> wrote:
>>> 
>>> The kernel currently requires all harts to have the same value in the
>>> vlenb csr that is present when a hart supports vector. In order to read
>>> this csr, the kernel needs to boot the hart. Adding vlenb to the DT will
>>> allow the kernel to detect the inconsistency early and not waste time
>>> trying to boot harts that it doesn't support.
>> 
>> That doesn’t seem sufficient justification to me. If it can be read
>> from the hardware, why should we have to put it in the FDT? The whole
>> point of the FDT is to communicate the hardware configuration that
>> isn’t otherwise discoverable.
> 
> Yes you are correct in that vlenb is discoverable on any conforming
> chip. However, the motivation here is for making decisions about how to
> boot a hart before it is booted. By placing it in the device tree, we
> are able to disable vector before the chip is booted instead of trying
> to boot the chip with vector enabled only to disable it later. In both
> cases when there is different vlenb on different harts, all harts still
> boot and the outcome is that vector is disabled. The difference is that
> with the DT entry, no vector setup code needs to be ran on a booting
> hart when the outcome will be that vector is not enabled.

Why does vlen get this special treatment? You could make exactly the
same argument for the number of asid bits. The precedent in the kernel,
whether RISC-V or other architectures, is to not do this. You can
detect it, so you should, especially since optimising for an
exceptional, unexpected error case is not worthwhile.

>> As for T-HEAD stuff, if they need it they can have a custom property.
>> Though naively I’d assume there’s a way to avoid it still...
> 
> T-Head does not expose vlenb on all of their chips so I do not know of
> any other way of getting the vlenb without having it be provided in a
> DT. That was the motivation for this patch in the first place, but
> making this available to all vendors allows optimizations to happen
> during boot.

How does userspace read it then? But if T-HEAD need it, that means it
should be a thead,vlen, not a riscv,vlen.

Jess

Re: [PATCH 0/2] riscv: Allow vlenb to be probed from DT

[Charlie Jenkins]

On Thu, May 16, 2024 at 01:58:29AM +0100, Jessica Clarke wrote:
> On 16 May 2024, at 00:08, Charlie Jenkins <charlie@rivosinc.com> wrote:
> > 
> > On Wed, May 15, 2024 at 11:25:16PM +0100, Jessica Clarke wrote:
> >> On 15 May 2024, at 22:50, Charlie Jenkins <charlie@rivosinc.com> wrote:
> >>> 
> >>> The kernel currently requires all harts to have the same value in the
> >>> vlenb csr that is present when a hart supports vector. In order to read
> >>> this csr, the kernel needs to boot the hart. Adding vlenb to the DT will
> >>> allow the kernel to detect the inconsistency early and not waste time
> >>> trying to boot harts that it doesn't support.
> >> 
> >> That doesn’t seem sufficient justification to me. If it can be read
> >> from the hardware, why should we have to put it in the FDT? The whole
> >> point of the FDT is to communicate the hardware configuration that
> >> isn’t otherwise discoverable.
> > 
> > Yes you are correct in that vlenb is discoverable on any conforming
> > chip. However, the motivation here is for making decisions about how to
> > boot a hart before it is booted. By placing it in the device tree, we
> > are able to disable vector before the chip is booted instead of trying
> > to boot the chip with vector enabled only to disable it later. In both
> > cases when there is different vlenb on different harts, all harts still
> > boot and the outcome is that vector is disabled. The difference is that
> > with the DT entry, no vector setup code needs to be ran on a booting
> > hart when the outcome will be that vector is not enabled.
> 
> Why does vlen get this special treatment? You could make exactly the
> same argument for the number of asid bits. The precedent in the kernel,
> whether RISC-V or other architectures, is to not do this. You can
> detect it, so you should, especially since optimising for an
> exceptional, unexpected error case is not worthwhile.
> 
> >> As for T-HEAD stuff, if they need it they can have a custom property.
> >> Though naively I’d assume there’s a way to avoid it still...
> > 
> > T-Head does not expose vlenb on all of their chips so I do not know of
> > any other way of getting the vlenb without having it be provided in a
> > DT. That was the motivation for this patch in the first place, but
> > making this available to all vendors allows optimizations to happen
> > during boot.
> 
> How does userspace read it then? But if T-HEAD need it, that means it
> should be a thead,vlen, not a riscv,vlen.
> 
> Jess
> 

I'll let Palmer decide if it is reasonable to have vlenb allowed to be
placed in the device tree to support cores like ones made by thead which
don't support vlenb. Otherwise I will replace it with a thead-specific
binding. 

- Charlie