[PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Sean Anderson]

Cache geometry is exposed through the Cache Size ID register. There is
one register for each cache, and they are selected through the Cache
Size Selection register. If FEAT_CCIDX is implemented, the layout of
CCSIDR changes to allow a larger number of sets and ways.

Signed-off-by: Sean Anderson <sean.anderson@linux.dev>
---

 arch/arm64/include/asm/cache.h |  3 +++
 arch/arm64/kernel/cacheinfo.c  | 28 ++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)

diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h
index 99cd6546e72e..569330689a2f 100644
--- a/arch/arm64/include/asm/cache.h
+++ b/arch/arm64/include/asm/cache.h
@@ -8,6 +8,9 @@
 #define L1_CACHE_SHIFT		(6)
 #define L1_CACHE_BYTES		(1 << L1_CACHE_SHIFT)
 
+#define CCSIDR_CCIDX_NumSets		GENMASK_ULL(55, 32)
+#define CCSIDR_CCIDX_Associativity	GENMASK_ULL(23, 3)
+
 #define CLIDR_LOUU_SHIFT	27
 #define CLIDR_LOC_SHIFT		24
 #define CLIDR_LOUIS_SHIFT	21
diff --git a/arch/arm64/kernel/cacheinfo.c b/arch/arm64/kernel/cacheinfo.c
index 309942b06c5b..a0180d3f1631 100644
--- a/arch/arm64/kernel/cacheinfo.c
+++ b/arch/arm64/kernel/cacheinfo.c
@@ -34,8 +34,36 @@ static inline enum cache_type get_cache_type(int level)
 static void ci_leaf_init(struct cacheinfo *this_leaf,
 			 enum cache_type type, unsigned int level)
 {
+	u64 val;
+
 	this_leaf->level = level;
 	this_leaf->type = type;
+	if (type == CACHE_TYPE_NOCACHE)
+		return;
+
+	val = FIELD_PREP(CSSELR_EL1_Level, level - 1);
+	if (type == CACHE_TYPE_INST)
+		val |= CSSELR_EL1_InD;
+	write_sysreg(val, csselr_el1);
+
+	val = read_sysreg(ccsidr_el1);
+	this_leaf->coherency_line_size =
+		BIT(FIELD_GET(CCSIDR_EL1_LineSize, val) + 4);
+	if (FIELD_GET(ID_MMFR4_EL1_CCIDX,
+		      read_sanitised_ftr_reg(SYS_ID_AA64MMFR4_EL1))) {
+		this_leaf->number_of_sets =
+			FIELD_GET(CCSIDR_CCIDX_NumSets, val) + 1;
+		this_leaf->ways_of_associativity =
+			FIELD_GET(CCSIDR_CCIDX_Associativity, val) + 1;
+	} else {
+		this_leaf->number_of_sets =
+			FIELD_GET(CCSIDR_EL1_NumSets, val) + 1;
+		this_leaf->ways_of_associativity =
+			FIELD_GET(CCSIDR_EL1_Associativity, val) + 1;
+	}
+	this_leaf->size = this_leaf->coherency_line_size *
+			  this_leaf->number_of_sets *
+			  this_leaf->ways_of_associativity;
 }
 
 static void detect_cache_level(unsigned int *level_p, unsigned int *leaves_p)
-- 
2.35.1.1320.gc452695387.dirty

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Sudeep Holla]

On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
> Cache geometry is exposed through the Cache Size ID register. There is
> one register for each cache, and they are selected through the Cache
> Size Selection register. If FEAT_CCIDX is implemented, the layout of
> CCSIDR changes to allow a larger number of sets and ways.
> 

Please refer
Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")

-- 
Regards,
Sudeep

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Sean Anderson]

On 5/10/25 03:04, Sudeep Holla wrote:
> On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
>> Cache geometry is exposed through the Cache Size ID register. There is
>> one register for each cache, and they are selected through the Cache
>> Size Selection register. If FEAT_CCIDX is implemented, the layout of
>> CCSIDR changes to allow a larger number of sets and ways.
>> 
> 
> Please refer
> Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
> 

| The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
| in conjunction with set/way cache maintenance and are not guaranteed to
| represent the actual microarchitectural features of a design.
| 
| The architecture explicitly states:
| 
| | You cannot make any inference about the actual sizes of caches based
| | on these parameters.

However, on many cores (A53, A72, and surely others that I haven't
checked) these *do* expose the actual microarchitectural features of the
design. Maybe a whitelist would be suitable.

| Furthermore, CCSIDR_EL1.{WT,WB,RA,WA} have been removed retrospectively
| from ARMv8 and are now considered to be UNKNOWN.
| 
| Since the kernel doesn't make use of set/way cache maintenance and it is
| not possible for userspace to execute these instructions, we have no
| need for the CCSIDR information in the kernel.

Actually, these parameters are directly visible (and useful) to
userspace in the form of the cache size. Rather than make userspace
perform benchmarks, we can expose this information in a standard way.
There is of course [id]cache-size, but these properties are absent more
often than not:

$ git grep arm,cortex- 'arch/arm64/**.dtsi' | wc -l
1248
$ git grep d-cache-size 'arch/arm64/**.dtsi' | wc -l
320

--Sean

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Sudeep Holla]

On Mon, May 12, 2025 at 11:28:36AM -0400, Sean Anderson wrote:
> On 5/10/25 03:04, Sudeep Holla wrote:
> > On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
> >> Cache geometry is exposed through the Cache Size ID register. There is
> >> one register for each cache, and they are selected through the Cache
> >> Size Selection register. If FEAT_CCIDX is implemented, the layout of
> >> CCSIDR changes to allow a larger number of sets and ways.
> >> 
> > 
> > Please refer
> > Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
> > 
> 
> | The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
> | in conjunction with set/way cache maintenance and are not guaranteed to
> | represent the actual microarchitectural features of a design.
> | 
> | The architecture explicitly states:
> | 
> | | You cannot make any inference about the actual sizes of caches based
> | | on these parameters.
> 
> However, on many cores (A53, A72, and surely others that I haven't
> checked) these *do* expose the actual microarchitectural features of the
> design. Maybe a whitelist would be suitable.
> 
> | Furthermore, CCSIDR_EL1.{WT,WB,RA,WA} have been removed retrospectively
> | from ARMv8 and are now considered to be UNKNOWN.
> | 
> | Since the kernel doesn't make use of set/way cache maintenance and it is
> | not possible for userspace to execute these instructions, we have no
> | need for the CCSIDR information in the kernel.
> 
> Actually, these parameters are directly visible (and useful) to
> userspace in the form of the cache size. Rather than make userspace
> perform benchmarks, we can expose this information in a standard way.

Yes that is already present, which is DT or ACPI.

> There is of course [id]cache-size, but these properties are absent more
> often than not:
> 
> $ git grep arm,cortex- 'arch/arm64/**.dtsi' | wc -l
> 1248
> $ git grep d-cache-size 'arch/arm64/**.dtsi' | wc -l
> 320
>

Just to be clear, I am fine with exposing to the userspace, but just
not reading those registers as stated in the commit message I shared
earlier.

Why can't it be done via DT/ACPI ?

-- 
Regards,
Sudeep

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Sean Anderson]

On 5/12/25 11:34, Sudeep Holla wrote:
> On Mon, May 12, 2025 at 11:28:36AM -0400, Sean Anderson wrote:
>> On 5/10/25 03:04, Sudeep Holla wrote:
>> > On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
>> >> Cache geometry is exposed through the Cache Size ID register. There is
>> >> one register for each cache, and they are selected through the Cache
>> >> Size Selection register. If FEAT_CCIDX is implemented, the layout of
>> >> CCSIDR changes to allow a larger number of sets and ways.
>> >> 
>> > 
>> > Please refer
>> > Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
>> > 
>> 
>> | The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
>> | in conjunction with set/way cache maintenance and are not guaranteed to
>> | represent the actual microarchitectural features of a design.
>> | 
>> | The architecture explicitly states:
>> | 
>> | | You cannot make any inference about the actual sizes of caches based
>> | | on these parameters.
>> 
>> However, on many cores (A53, A72, and surely others that I haven't
>> checked) these *do* expose the actual microarchitectural features of the
>> design. Maybe a whitelist would be suitable.
>> 
>> | Furthermore, CCSIDR_EL1.{WT,WB,RA,WA} have been removed retrospectively
>> | from ARMv8 and are now considered to be UNKNOWN.
>> | 
>> | Since the kernel doesn't make use of set/way cache maintenance and it is
>> | not possible for userspace to execute these instructions, we have no
>> | need for the CCSIDR information in the kernel.
>> 
>> Actually, these parameters are directly visible (and useful) to
>> userspace in the form of the cache size. Rather than make userspace
>> perform benchmarks, we can expose this information in a standard way.
> 
> Yes that is already present, which is DT or ACPI.
> 
>> There is of course [id]cache-size, but these properties are absent more
>> often than not:
>> 
>> $ git grep arm,cortex- 'arch/arm64/**.dtsi' | wc -l
>> 1248
>> $ git grep d-cache-size 'arch/arm64/**.dtsi' | wc -l
>> 320
>>
> 
> Just to be clear, I am fine with exposing to the userspace, but just
> not reading those registers as stated in the commit message I shared
> earlier.
> 
> Why can't it be done via DT/ACPI ?
> 

Well, do you want to go through 200 SoCs and look up cache info in the
documentation? I think it would be more expedient to look at the TRMs
for two dozen cores and determine if the above registers expose the cache
info correctly.

--Sean

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Mark Rutland]

On Mon, May 12, 2025 at 11:28:36AM -0400, Sean Anderson wrote:
> On 5/10/25 03:04, Sudeep Holla wrote:
> > On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
> >> Cache geometry is exposed through the Cache Size ID register. There is
> >> one register for each cache, and they are selected through the Cache
> >> Size Selection register. If FEAT_CCIDX is implemented, the layout of
> >> CCSIDR changes to allow a larger number of sets and ways.
> >> 
> > 
> > Please refer
> > Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
> > 
> 
> | The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
> | in conjunction with set/way cache maintenance and are not guaranteed to
> | represent the actual microarchitectural features of a design.
> | 
> | The architecture explicitly states:
> | 
> | | You cannot make any inference about the actual sizes of caches based
> | | on these parameters.
> 
> However, on many cores (A53, A72, and surely others that I haven't
> checked) these *do* expose the actual microarchitectural features of the
> design. Maybe a whitelist would be suitable.

Then we have to maintain a whitelist forever, and running an old/distro
kernel on new HW won't give you useful values unless you provide
equivalent values in DT, in which case the kernel doesn't need to read
the registers anyway.

The architecture explcitly tells us not to use the values in this way,
and it's possible to place the values into DT when you know they're
meaningful.

I do not think we should resurrect the usage of these registers.

Mark.

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Sean Anderson]

On 5/12/25 11:36, Mark Rutland wrote:
> On Mon, May 12, 2025 at 11:28:36AM -0400, Sean Anderson wrote:
>> On 5/10/25 03:04, Sudeep Holla wrote:
>> > On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
>> >> Cache geometry is exposed through the Cache Size ID register. There is
>> >> one register for each cache, and they are selected through the Cache
>> >> Size Selection register. If FEAT_CCIDX is implemented, the layout of
>> >> CCSIDR changes to allow a larger number of sets and ways.
>> >> 
>> > 
>> > Please refer
>> > Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
>> > 
>> 
>> | The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
>> | in conjunction with set/way cache maintenance and are not guaranteed to
>> | represent the actual microarchitectural features of a design.
>> | 
>> | The architecture explicitly states:
>> | 
>> | | You cannot make any inference about the actual sizes of caches based
>> | | on these parameters.
>> 
>> However, on many cores (A53, A72, and surely others that I haven't
>> checked) these *do* expose the actual microarchitectural features of the
>> design. Maybe a whitelist would be suitable.
> 
> Then we have to maintain a whitelist forever,

There's no maintenance involved. The silicon is already fabbed, so it's
not like it's going to change any time soon.

> and running an old/distro
> kernel on new HW won't give you useful values unless you provide
> equivalent values in DT, in which case the kernel doesn't need to read
> the registers anyway.

Conversely (and far more likely IMO), running an old/distro devicetree
on a new kernel won't give you usefult values. Bootloaders tend not be
be updated very often (if ever), whereas kernels can (ideally) be
updated without changing userspace.

> The architecture explcitly tells us not to use the values in this way,
> and it's possible to place the values into DT when you know they're
> meaningful.

Well, maybe we can just use these registers for the hundreds of existing
devicetrees that lack values.

--Sean

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Will Deacon]

On Mon, May 12, 2025 at 11:56:28AM -0400, Sean Anderson wrote:
> On 5/12/25 11:36, Mark Rutland wrote:
> > On Mon, May 12, 2025 at 11:28:36AM -0400, Sean Anderson wrote:
> >> On 5/10/25 03:04, Sudeep Holla wrote:
> >> > On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
> >> >> Cache geometry is exposed through the Cache Size ID register. There is
> >> >> one register for each cache, and they are selected through the Cache
> >> >> Size Selection register. If FEAT_CCIDX is implemented, the layout of
> >> >> CCSIDR changes to allow a larger number of sets and ways.
> >> >> 
> >> > 
> >> > Please refer
> >> > Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
> >> > 
> >> 
> >> | The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
> >> | in conjunction with set/way cache maintenance and are not guaranteed to
> >> | represent the actual microarchitectural features of a design.
> >> | 
> >> | The architecture explicitly states:
> >> | 
> >> | | You cannot make any inference about the actual sizes of caches based
> >> | | on these parameters.
> >> 
> >> However, on many cores (A53, A72, and surely others that I haven't
> >> checked) these *do* expose the actual microarchitectural features of the
> >> design. Maybe a whitelist would be suitable.
> > 
> > Then we have to maintain a whitelist forever,
> 
> There's no maintenance involved. The silicon is already fabbed, so it's
> not like it's going to change any time soon.

The list is going to change though and it introduces divergent behaviour
that I'd much rather avoid. The mechanism is there for firmware to
provide the information and it's hardly onerous compared with other
(critical) things that it's tasked to provide such as interrupt routing
and GPIOs.

> > and running an old/distro
> > kernel on new HW won't give you useful values unless you provide
> > equivalent values in DT, in which case the kernel doesn't need to read
> > the registers anyway.
> 
> Conversely (and far more likely IMO), running an old/distro devicetree
> on a new kernel won't give you usefult values. Bootloaders tend not be
> be updated very often (if ever), whereas kernels can (ideally) be
> updated without changing userspace.

Updating the device-tree shouldn't require updating the bootloader.

> > The architecture explcitly tells us not to use the values in this way,
> > and it's possible to place the values into DT when you know they're
> > meaningful.
> 
> Well, maybe we can just use these registers for the hundreds of existing
> devicetrees that lack values.

The fact that the device-tree files tend to omit this information is
quite telling as to how useful it actually is. What would you like to
use it for?

Short of having an immediate functional or performance benefit by
exposing this stuff, I wonder if we could add a kselftest for it
instead?

Will

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Sean Anderson]

On 5/14/25 08:38, Will Deacon wrote:
> On Mon, May 12, 2025 at 11:56:28AM -0400, Sean Anderson wrote:
>> On 5/12/25 11:36, Mark Rutland wrote:
>> > On Mon, May 12, 2025 at 11:28:36AM -0400, Sean Anderson wrote:
>> >> On 5/10/25 03:04, Sudeep Holla wrote:
>> >> > On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
>> >> >> Cache geometry is exposed through the Cache Size ID register. There is
>> >> >> one register for each cache, and they are selected through the Cache
>> >> >> Size Selection register. If FEAT_CCIDX is implemented, the layout of
>> >> >> CCSIDR changes to allow a larger number of sets and ways.
>> >> >> 
>> >> > 
>> >> > Please refer
>> >> > Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
>> >> > 
>> >> 
>> >> | The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
>> >> | in conjunction with set/way cache maintenance and are not guaranteed to
>> >> | represent the actual microarchitectural features of a design.
>> >> | 
>> >> | The architecture explicitly states:
>> >> | 
>> >> | | You cannot make any inference about the actual sizes of caches based
>> >> | | on these parameters.
>> >> 
>> >> However, on many cores (A53, A72, and surely others that I haven't
>> >> checked) these *do* expose the actual microarchitectural features of the
>> >> design. Maybe a whitelist would be suitable.
>> > 
>> > Then we have to maintain a whitelist forever,
>> 
>> There's no maintenance involved. The silicon is already fabbed, so it's
>> not like it's going to change any time soon.
> 
> The list is going to change though and it introduces divergent behaviour
> that I'd much rather avoid. The mechanism is there for firmware to
> provide the information and it's hardly onerous compared with other
> (critical) things that it's tasked to provide such as interrupt routing
> and GPIOs.

The mechanism is also there for us to discover the cache sizes without
requiring any devicetree involvement.

>> > and running an old/distro
>> > kernel on new HW won't give you useful values unless you provide
>> > equivalent values in DT, in which case the kernel doesn't need to read
>> > the registers anyway.
>> 
>> Conversely (and far more likely IMO), running an old/distro devicetree
>> on a new kernel won't give you usefult values. Bootloaders tend not be
>> be updated very often (if ever), whereas kernels can (ideally) be
>> updated without changing userspace.
> 
> Updating the device-tree shouldn't require updating the bootloader.

Very often the release cycle for the devicetree is tied to the bootloader.
So they may not be updated very often.

>> > The architecture explcitly tells us not to use the values in this way,
>> > and it's possible to place the values into DT when you know they're
>> > meaningful.
>> 
>> Well, maybe we can just use these registers for the hundreds of existing
>> devicetrees that lack values.
> 
> The fact that the device-tree files tend to omit this information is
> quite telling as to how useful it actually is. What would you like to
> use it for?

Say you have a program that works on batches of data. You may want to
adjust the size of the batch to fit in the L1 (or L2) cache. One way to
do this is to benchmark various batch sizes and select an appropriate
size. But it would be more convenient to the user to pick a batch size
automatically without having to run a benchmark, just by reading from
sysfs.

> Short of having an immediate functional or performance benefit by
> exposing this stuff, I wonder if we could add a kselftest for it
> instead?

I'm not sure how well that will improve adoption. Do people even run
kselftest during board bringup?

--Sean

Re: [PATCH] arm64: cacheinfo: Report cache sets, ways, and line size

[Rob Herring]

On Mon, May 12, 2025 at 11:27 AM Sean Anderson <sean.anderson@linux.dev> wrote:
>
> On 5/12/25 11:36, Mark Rutland wrote:
> > On Mon, May 12, 2025 at 11:28:36AM -0400, Sean Anderson wrote:
> >> On 5/10/25 03:04, Sudeep Holla wrote:
> >> > On Fri, May 09, 2025 at 07:37:35PM -0400, Sean Anderson wrote:
> >> >> Cache geometry is exposed through the Cache Size ID register. There is
> >> >> one register for each cache, and they are selected through the Cache
> >> >> Size Selection register. If FEAT_CCIDX is implemented, the layout of
> >> >> CCSIDR changes to allow a larger number of sets and ways.
> >> >>
> >> >
> >> > Please refer
> >> > Commit a8d4636f96ad ("arm64: cacheinfo: Remove CCSIDR-based cache information probing")
> >> >
> >>
> >> | The CCSIDR_EL1.{NumSets,Associativity,LineSize} fields are only for use
> >> | in conjunction with set/way cache maintenance and are not guaranteed to
> >> | represent the actual microarchitectural features of a design.
> >> |
> >> | The architecture explicitly states:
> >> |
> >> | | You cannot make any inference about the actual sizes of caches based
> >> | | on these parameters.
> >>
> >> However, on many cores (A53, A72, and surely others that I haven't
> >> checked) these *do* expose the actual microarchitectural features of the
> >> design. Maybe a whitelist would be suitable.
> >
> > Then we have to maintain a whitelist forever,
>
> There's no maintenance involved. The silicon is already fabbed, so it's
> not like it's going to change any time soon.
>
> > and running an old/distro
> > kernel on new HW won't give you useful values unless you provide
> > equivalent values in DT, in which case the kernel doesn't need to read
> > the registers anyway.
>
> Conversely (and far more likely IMO), running an old/distro devicetree
> on a new kernel won't give you usefult values. Bootloaders tend not be
> be updated very often (if ever), whereas kernels can (ideally) be
> updated without changing userspace.
>
> > The architecture explcitly tells us not to use the values in this way,
> > and it's possible to place the values into DT when you know they're
> > meaningful.
>
> Well, maybe we can just use these registers for the hundreds of existing
> devicetrees that lack values.

If the lack of values is a problem, add a schema to require them. Just
adding them as required globally isn't going to work, but nothing
prevents someone from having additional schemas for platforms they
care about or we could figure out a way to opt-in to specific schemas.

Rob