[PATCH 0/2] arch/x86: Set L2 Cache ID on AMD processors

[PATCH 0/2] arch/x86: Set L2 Cache ID on AMD processors

[K Prateek Nayak]

commit 66558b730f253 ("sched: Add cluster scheduler level for x86")
defined cluster on x86 as the set of threads sharing the same L2 cache.
cluster_id on x86, maps to the l2c_id which currently only Intel
processors set.

This series sets the l2c_id on AMD processors with X86_FEATURE_TOPOEXT,
using the extended APIC ID and the "Cache Properties (L2)" CPUID
(0x8000001D EAX). On AMD processors without X86_FEATURE_TOPOEXT, current
behavior will continue.

Following are the changes in value reported by
"/sys/devices/system/cpu/cpuX/topology/cluster_id" on a 2P Milan system
(2 x 64C/128T) where L2 is per-core level and SMT sibling of CPU (X) is
CPU ((X + 128) % 256).

- tip:x86/core

  $ for i in {0..255}; do\
      echo -n "CPU$i cluster_id: ";\
      cat /sys/devices/system/cpu/cpu$i/topology/cluster_id;\
    done;

    CPU0 cluster_id: 65535
    CPU1 cluster_id: 65535
    CPU2 cluster_id: 65535
    CPU3 cluster_id: 65535
    CPU4 cluster_id: 65535
    ...
    CPU254 cluster_id: 65535
    CPU255 cluster_id: 65535

- tip:x86/core + this series

  $ for i in {0..255}; do\
      echo -n "CPU$i cluster_id: ";\
      cat /sys/devices/system/cpu/cpu$i/topology/cluster_id;\
    done;

    CPU0 cluster_id: 0
    CPU1 cluster_id: 1
    CPU2 cluster_id: 2
    CPU3 cluster_id: 3
    CPU4 cluster_id: 4
    CPU5 cluster_id: 5
    CPU6 cluster_id: 6
    CPU7 cluster_id: 7
    CPU8 cluster_id: 8
    ...
    CPU126 cluster_id: 126
    CPU127 cluster_id: 127
    CPU128 cluster_id: 0
    CPU129 cluster_id: 1
    CPU130 cluster_id: 2
    CPU131 cluster_id: 3
    CPU132 cluster_id: 4
    CPU133 cluster_id: 5
    CPU134 cluster_id: 6
    CPU135 cluster_id: 7
    CPU136 cluster_id: 8
    ...
    CPU254 cluster_id: 126
    CPU255 cluster_id: 127

Note: Hygon, theoretically, should be able to set the l2c_id using the
same cacheinfo_amd_init_l2c_id() function being added in Patch 1. Since
I do not have access to a Hygon machine to verify my theory, ccing Hygon
maintainer Pu Wen <puwen@hygon.cn> for l2c_id enablement on Hygon.

The series also adds documentation for clusters on x86 platforms and
applies cleanly on top of tip:x86/core at commit ce3ba2af9695
("x86: Suppress KMSAN reports in arch_within_stack_frames()")

---
K Prateek Nayak (2):
  arch/x86: Set L2 Cache ID on AMD and Hygon processors
  x86/Documentation: Add documentation about cluster

 Documentation/x86/topology.rst   | 31 +++++++++++++++++++++
 arch/x86/include/asm/cacheinfo.h |  1 +
 arch/x86/kernel/cpu/amd.c        |  1 +
 arch/x86/kernel/cpu/cacheinfo.c  | 47 ++++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/hygon.c      |  1 +
 5 files changed, 81 insertions(+)

-- 
2.34.1

[PATCH 1/2] arch/x86: Set L2 Cache ID on AMD processors

[K Prateek Nayak]

On AMD processors supporting X86_FEATURE_TOPOEXT set the l2c_id using the
Extended APIC ID and the Cache Properties CPUID.

Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 arch/x86/include/asm/cacheinfo.h |  1 +
 arch/x86/kernel/cpu/amd.c        |  1 +
 arch/x86/kernel/cpu/cacheinfo.c  | 36 ++++++++++++++++++++++++++++++++
 3 files changed, 38 insertions(+)

diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cacheinfo.h
index ce9685fc78d8..5e472fc364c8 100644
--- a/arch/x86/include/asm/cacheinfo.h
+++ b/arch/x86/include/asm/cacheinfo.h
@@ -7,6 +7,7 @@ extern unsigned int memory_caching_control;
 #define CACHE_MTRR 0x01
 #define CACHE_PAT  0x02
 
+void cacheinfo_amd_init_l2c_id(struct cpuinfo_x86 *c, int cpu);
 void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu);
 void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, int cpu);
 
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index f769d6d08b43..e68d31231666 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -358,6 +358,7 @@ static void amd_get_topology(struct cpuinfo_x86 *c)
 		if (!err)
 			c->x86_coreid_bits = get_count_order(c->x86_max_cores);
 
+		cacheinfo_amd_init_l2c_id(c, cpu);
 		cacheinfo_amd_init_llc_id(c, cpu);
 
 	} else if (cpu_has(c, X86_FEATURE_NODEID_MSR)) {
diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinfo.c
index f4e5aa27eec6..0baf2d9b1595 100644
--- a/arch/x86/kernel/cpu/cacheinfo.c
+++ b/arch/x86/kernel/cpu/cacheinfo.c
@@ -659,6 +659,42 @@ static int find_num_cache_leaves(struct cpuinfo_x86 *c)
 	return i;
 }
 
+void cacheinfo_amd_init_l2c_id(struct cpuinfo_x86 *c, int cpu)
+{
+	u32 eax, ebx, ecx, edx, num_sharing_cache;
+	int i = 0, bits;
+
+	/* Check if L2 cache identifiers exists. */
+	if (!cpuid_ecx(0x80000006))
+		return;
+
+	while (true) {
+		u32 level;
+
+		cpuid_count(0x8000001d, i, &eax, &ebx, &ecx, &edx);
+		if (!eax)
+			return;
+
+		/*
+		 * Check if the current leaf is for L2 cache using
+		 * eax[7:5] used to describe the cache level.
+		 */
+		level = (eax >> 5) & 0x7;
+		if (level == 2)
+			break;
+
+		++i;
+	}
+
+	/*
+	 * L2 ID is calculated from the number of threads
+	 * sharing the L2 cache.
+	 */
+	num_sharing_cache = ((eax >> 14) & 0xfff) + 1;
+	bits = get_count_order(num_sharing_cache);
+	per_cpu(cpu_l2c_id, cpu) = c->apicid >> bits;
+}
+
 void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu)
 {
 	/*
-- 
2.34.1

[PATCH 2/2] x86/Documentation: Add documentation about cluster

[K Prateek Nayak]

x86 processors map cluster to the L2 cache. Add documentation stating
the same, and provide more information on the values and API related to
CPU clusters exposed by the kernel.

Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
---
 Documentation/x86/topology.rst | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/Documentation/x86/topology.rst b/Documentation/x86/topology.rst
index 7f58010ea86a..35991d8cdef1 100644
--- a/Documentation/x86/topology.rst
+++ b/Documentation/x86/topology.rst
@@ -33,6 +33,7 @@ historical nature and should be cleaned up.
 The topology of a system is described in the units of:
 
     - packages
+    - cluster
     - cores
     - threads
 
@@ -90,6 +91,27 @@ Package-related topology information in the kernel:
         Cache. In general, it is a number identifying an LLC uniquely on the
         system.
 
+
+Clusters
+========
+A cluster consists of 1 or more threads. It does not matter whether the threads
+are SMT- or CMT-type threads. All the threads of a cluster share the same L2
+cache.
+
+Cluster-related topology information in the kernel:
+
+  - cluster_id:
+
+    A per-CPU variable containing:
+
+      - On Intel, the common upper bits of APIC ID of the list of CPUs sharing
+        the L2 Cache with lower bits set to 0.
+
+      - On AMD, with Topology Extension, the common upper bits of the Extended
+        APIC ID of the list of CPUs sharing the L2 Cache, left shifted to
+        remove trailing 0s.
+
+
 Cores
 =====
 A core consists of 1 or more threads. It does not matter whether the threads
@@ -125,6 +147,11 @@ Thread-related topology information in the kernel:
 
     The number of online threads is also printed in /proc/cpuinfo "siblings."
 
+  - topology_cluster_cpumask():
+
+    The cpumask contains all online threads in the cluster to which a thread
+    belongs.
+
   - topology_sibling_cpumask():
 
     The cpumask contains all online threads in the core to which a thread
@@ -138,6 +165,10 @@ Thread-related topology information in the kernel:
 
     The physical package ID to which a thread belongs.
 
+  - topology_cluster_id();
+
+    The ID of the cluster to which a thread belongs.
+
   - topology_core_id();
 
     The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
-- 
2.34.1

Re: [PATCH 2/2] x86/Documentation: Add documentation about cluster

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 2374 bytes --]

On Mon, Apr 10, 2023 at 10:05:27PM +0530, K Prateek Nayak wrote:
> diff --git a/Documentation/x86/topology.rst b/Documentation/x86/topology.rst
> index 7f58010ea86a..35991d8cdef1 100644
> --- a/Documentation/x86/topology.rst
> +++ b/Documentation/x86/topology.rst
> @@ -33,6 +33,7 @@ historical nature and should be cleaned up.
>  The topology of a system is described in the units of:
>  
>      - packages
> +    - cluster
>      - cores
>      - threads
>  
> @@ -90,6 +91,27 @@ Package-related topology information in the kernel:
>          Cache. In general, it is a number identifying an LLC uniquely on the
>          system.
>  
> +
> +Clusters
> +========
> +A cluster consists of 1 or more threads. It does not matter whether the threads
> +are SMT- or CMT-type threads. All the threads of a cluster share the same L2
> +cache.
> +
> +Cluster-related topology information in the kernel:
> +
> +  - cluster_id:
> +
> +    A per-CPU variable containing:
> +
> +      - On Intel, the common upper bits of APIC ID of the list of CPUs sharing
> +        the L2 Cache with lower bits set to 0.
> +
> +      - On AMD, with Topology Extension, the common upper bits of the Extended
> +        APIC ID of the list of CPUs sharing the L2 Cache, left shifted to
> +        remove trailing 0s.
> +
> +
>  Cores
>  =====
>  A core consists of 1 or more threads. It does not matter whether the threads
> @@ -125,6 +147,11 @@ Thread-related topology information in the kernel:
>  
>      The number of online threads is also printed in /proc/cpuinfo "siblings."
>  
> +  - topology_cluster_cpumask():
> +
> +    The cpumask contains all online threads in the cluster to which a thread
> +    belongs.
> +
>    - topology_sibling_cpumask():
>  
>      The cpumask contains all online threads in the core to which a thread
> @@ -138,6 +165,10 @@ Thread-related topology information in the kernel:
>  
>      The physical package ID to which a thread belongs.
>  
> +  - topology_cluster_id();
> +
> +    The ID of the cluster to which a thread belongs.
> +
>    - topology_core_id();
>  
>      The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo

The doc LGTM, thanks!

Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH 2/2] x86/Documentation: Add documentation about cluster

[Peter Zijlstra]

On Mon, Apr 10, 2023 at 10:05:27PM +0530, K Prateek Nayak wrote:
> x86 processors map cluster to the L2 cache. Add documentation stating
> the same, and provide more information on the values and API related to
> CPU clusters exposed by the kernel.
> 
> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
> ---
>  Documentation/x86/topology.rst | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/Documentation/x86/topology.rst b/Documentation/x86/topology.rst
> index 7f58010ea86a..35991d8cdef1 100644
> --- a/Documentation/x86/topology.rst
> +++ b/Documentation/x86/topology.rst
> @@ -33,6 +33,7 @@ historical nature and should be cleaned up.
>  The topology of a system is described in the units of:
>  
>      - packages
> +    - cluster
>      - cores
>      - threads
>  
> @@ -90,6 +91,27 @@ Package-related topology information in the kernel:
>          Cache. In general, it is a number identifying an LLC uniquely on the
>          system.
>  
> +
> +Clusters
> +========
> +A cluster consists of 1 or more threads. It does not matter whether the threads
> +are SMT- or CMT-type threads. All the threads of a cluster share the same L2
> +cache.

I'm not quite sure that's a correct discription of what a cluster is.

Yes, SMT will fundamentally share core-level caches (and should we not
always have SMT share all cache topoligies?)

But there is also x86 where L2 is shared between multiple cores -- while
the above seems to suggest L2 is single core only.

Re: [PATCH 2/2] x86/Documentation: Add documentation about cluster

[K Prateek Nayak]

Hello Peter,

Thank you for reviewing the patches.

On 4/11/2023 1:55 PM, Peter Zijlstra wrote:
> On Mon, Apr 10, 2023 at 10:05:27PM +0530, K Prateek Nayak wrote:
>> x86 processors map cluster to the L2 cache. Add documentation stating
>> the same, and provide more information on the values and API related to
>> CPU clusters exposed by the kernel.
>>
>> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
>> ---
>>  Documentation/x86/topology.rst | 31 +++++++++++++++++++++++++++++++
>>  1 file changed, 31 insertions(+)
>>
>> diff --git a/Documentation/x86/topology.rst b/Documentation/x86/topology.rst
>> index 7f58010ea86a..35991d8cdef1 100644
>> --- a/Documentation/x86/topology.rst
>> +++ b/Documentation/x86/topology.rst
>> @@ -33,6 +33,7 @@ historical nature and should be cleaned up.
>>  The topology of a system is described in the units of:
>>  
>>      - packages
>> +    - cluster
>>      - cores
>>      - threads
>>  
>> @@ -90,6 +91,27 @@ Package-related topology information in the kernel:
>>          Cache. In general, it is a number identifying an LLC uniquely on the
>>          system.
>>  
>> +
>> +Clusters
>> +========
>> +A cluster consists of 1 or more threads. It does not matter whether the threads
>> +are SMT- or CMT-type threads. All the threads of a cluster share the same L2
>> +cache.
> 
> I'm not quite sure that's a correct discription of what a cluster is.
> 
> Yes, SMT will fundamentally share core-level caches (and should we not
> always have SMT share all cache topoligies?)

I can reword the cluster description as follows:

"A cluster consists of threads of one or more cores sharing the same
L2 cache."

> 
> But there is also x86 where L2 is shared between multiple cores -- while
> the above seems to suggest L2 is single core only.

I hope the above rewording solves this confusion. Let me know otherwise.
--
Thanks and Regards,
Prateek

Re: [PATCH 2/2] x86/Documentation: Add documentation about cluster

[K Prateek Nayak]

Hello Bagas,

On 4/11/2023 9:28 AM, Bagas Sanjaya wrote:
> On Mon, Apr 10, 2023 at 10:05:27PM +0530, K Prateek Nayak wrote:
>> diff --git a/Documentation/x86/topology.rst b/Documentation/x86/topology.rst
>> index 7f58010ea86a..35991d8cdef1 100644
>> --- a/Documentation/x86/topology.rst
>> +++ b/Documentation/x86/topology.rst
>> @@ -33,6 +33,7 @@ historical nature and should be cleaned up.
>>  The topology of a system is described in the units of:
>>  
>>      - packages
>> +    - cluster
>>      - cores
>>      - threads
>>  
>> @@ -90,6 +91,27 @@ Package-related topology information in the kernel:
>>          Cache. In general, it is a number identifying an LLC uniquely on the
>>          system.
>>  
>> +
>> +Clusters
>> +========
>> +A cluster consists of 1 or more threads. It does not matter whether the threads
>> +are SMT- or CMT-type threads. All the threads of a cluster share the same L2
>> +cache.
>> +
>> +Cluster-related topology information in the kernel:
>> +
>> +  - cluster_id:
>> +
>> +    A per-CPU variable containing:
>> +
>> +      - On Intel, the common upper bits of APIC ID of the list of CPUs sharing
>> +        the L2 Cache with lower bits set to 0.
>> +
>> +      - On AMD, with Topology Extension, the common upper bits of the Extended
>> +        APIC ID of the list of CPUs sharing the L2 Cache, left shifted to
>> +        remove trailing 0s.
>> +
>> +
>>  Cores
>>  =====
>>  A core consists of 1 or more threads. It does not matter whether the threads
>> @@ -125,6 +147,11 @@ Thread-related topology information in the kernel:
>>  
>>      The number of online threads is also printed in /proc/cpuinfo "siblings."
>>  
>> +  - topology_cluster_cpumask():
>> +
>> +    The cpumask contains all online threads in the cluster to which a thread
>> +    belongs.
>> +
>>    - topology_sibling_cpumask():
>>  
>>      The cpumask contains all online threads in the core to which a thread
>> @@ -138,6 +165,10 @@ Thread-related topology information in the kernel:
>>  
>>      The physical package ID to which a thread belongs.
>>  
>> +  - topology_cluster_id();
>> +
>> +    The ID of the cluster to which a thread belongs.
>> +
>>    - topology_core_id();
>>  
>>      The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
> 
> The doc LGTM, thanks!
> 
> Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>

Thank you for reviewing the patch :)

--
Thanks and Regards,
Prateek

Re: [PATCH 0/2] arch/x86: Set L2 Cache ID on AMD processors

[Borislav Petkov]

On Tue, Apr 11, 2023 at 12:10:38PM +0000, Wen Pu wrote:
> I have tested the function cacheinfo_amd_init_l2c_id() on Hygon machine,
> it had the same result as yours above. So Hygon also should set the
> l2c_id. Could you please set it for Hygon, or should I send a separate
> patch?

Separate patch please.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 0/2] arch/x86: Set L2 Cache ID on AMD processors

[Oleksandr Natalenko]

Hello.

On pondělí 10. dubna 2023 18:35:25 CEST K Prateek Nayak wrote:
> commit 66558b730f253 ("sched: Add cluster scheduler level for x86")
> defined cluster on x86 as the set of threads sharing the same L2 cache.
> cluster_id on x86, maps to the l2c_id which currently only Intel
> processors set.
> 
> This series sets the l2c_id on AMD processors with X86_FEATURE_TOPOEXT,
> using the extended APIC ID and the "Cache Properties (L2)" CPUID
> (0x8000001D EAX). On AMD processors without X86_FEATURE_TOPOEXT, current
> behavior will continue.
> 
> Following are the changes in value reported by
> "/sys/devices/system/cpu/cpuX/topology/cluster_id" on a 2P Milan system
> (2 x 64C/128T) where L2 is per-core level and SMT sibling of CPU (X) is
> CPU ((X + 128) % 256).
> 
> - tip:x86/core
> 
>   $ for i in {0..255}; do\
>       echo -n "CPU$i cluster_id: ";\
>       cat /sys/devices/system/cpu/cpu$i/topology/cluster_id;\
>     done;
> 
>     CPU0 cluster_id: 65535
>     CPU1 cluster_id: 65535
>     CPU2 cluster_id: 65535
>     CPU3 cluster_id: 65535
>     CPU4 cluster_id: 65535
>     ...
>     CPU254 cluster_id: 65535
>     CPU255 cluster_id: 65535
> 
> - tip:x86/core + this series
> 
>   $ for i in {0..255}; do\
>       echo -n "CPU$i cluster_id: ";\
>       cat /sys/devices/system/cpu/cpu$i/topology/cluster_id;\
>     done;
> 
>     CPU0 cluster_id: 0
>     CPU1 cluster_id: 1
>     CPU2 cluster_id: 2
>     CPU3 cluster_id: 3
>     CPU4 cluster_id: 4
>     CPU5 cluster_id: 5
>     CPU6 cluster_id: 6
>     CPU7 cluster_id: 7
>     CPU8 cluster_id: 8
>     ...
>     CPU126 cluster_id: 126
>     CPU127 cluster_id: 127
>     CPU128 cluster_id: 0
>     CPU129 cluster_id: 1
>     CPU130 cluster_id: 2
>     CPU131 cluster_id: 3
>     CPU132 cluster_id: 4
>     CPU133 cluster_id: 5
>     CPU134 cluster_id: 6
>     CPU135 cluster_id: 7
>     CPU136 cluster_id: 8
>     ...
>     CPU254 cluster_id: 126
>     CPU255 cluster_id: 127
> 
> Note: Hygon, theoretically, should be able to set the l2c_id using the
> same cacheinfo_amd_init_l2c_id() function being added in Patch 1. Since
> I do not have access to a Hygon machine to verify my theory, ccing Hygon
> maintainer Pu Wen <puwen@hygon.cn> for l2c_id enablement on Hygon.
> 
> The series also adds documentation for clusters on x86 platforms and
> applies cleanly on top of tip:x86/core at commit ce3ba2af9695
> ("x86: Suppress KMSAN reports in arch_within_stack_frames()")
> 
> ---
> K Prateek Nayak (2):
>   arch/x86: Set L2 Cache ID on AMD and Hygon processors
>   x86/Documentation: Add documentation about cluster
> 
>  Documentation/x86/topology.rst   | 31 +++++++++++++++++++++
>  arch/x86/include/asm/cacheinfo.h |  1 +
>  arch/x86/kernel/cpu/amd.c        |  1 +
>  arch/x86/kernel/cpu/cacheinfo.c  | 47 ++++++++++++++++++++++++++++++++
>  arch/x86/kernel/cpu/hygon.c      |  1 +
>  5 files changed, 81 insertions(+)

Having the following CPU:

```
Architecture:            x86_64
CPU op-mode(s):        32-bit, 64-bit
Address sizes:         48 bits physical, 48 bits virtual
Byte Order:            Little Endian
CPU(s):                  32
On-line CPU(s) list:   0-31
Vendor ID:               AuthenticAMD
Model name:            AMD Ryzen 9 5950X 16-Core Processor
CPU family:          25
Model:               33
Thread(s) per core:  2
Core(s) per socket:  16
Socket(s):           1
Stepping:            2
Frequency boost:     enabled
CPU(s) scaling MHz:  37%
CPU max MHz:         5084,0000
CPU min MHz:         550,0000
BogoMIPS:            6789,07
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_
tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c r
drand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l
lc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb
sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip
_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid ove
rflow_recov succor smca fsrm
Virtualization features:
Virtualization:        AMD-V
Caches (sum of all):
L1d:                   512 KiB (16 instances)
L1i:                   512 KiB (16 instances)
L2:                    8 MiB (16 instances)
L3:                    64 MiB (2 instances)
NUMA:
NUMA node(s):          1
NUMA node0 CPU(s):     0-31
Vulnerabilities:
Itlb multihit:         Not affected
L1tf:                  Not affected
Mds:                   Not affected
Meltdown:              Not affected
Mmio stale data:       Not affected
Retbleed:              Not affected
Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Srbds:                 Not affected
Tsx async abort:       Not affected
```

Without the series:

```
/sys/devices/system/cpu/cpu0/topology/cluster_id:65535
/sys/devices/system/cpu/cpu1/topology/cluster_id:65535
/sys/devices/system/cpu/cpu2/topology/cluster_id:65535
/sys/devices/system/cpu/cpu3/topology/cluster_id:65535
/sys/devices/system/cpu/cpu4/topology/cluster_id:65535
/sys/devices/system/cpu/cpu5/topology/cluster_id:65535
/sys/devices/system/cpu/cpu6/topology/cluster_id:65535
/sys/devices/system/cpu/cpu7/topology/cluster_id:65535
/sys/devices/system/cpu/cpu8/topology/cluster_id:65535
/sys/devices/system/cpu/cpu9/topology/cluster_id:65535
/sys/devices/system/cpu/cpu10/topology/cluster_id:65535
/sys/devices/system/cpu/cpu11/topology/cluster_id:65535
/sys/devices/system/cpu/cpu12/topology/cluster_id:65535
/sys/devices/system/cpu/cpu13/topology/cluster_id:65535
/sys/devices/system/cpu/cpu14/topology/cluster_id:65535
/sys/devices/system/cpu/cpu15/topology/cluster_id:65535
/sys/devices/system/cpu/cpu16/topology/cluster_id:65535
/sys/devices/system/cpu/cpu17/topology/cluster_id:65535
/sys/devices/system/cpu/cpu18/topology/cluster_id:65535
/sys/devices/system/cpu/cpu19/topology/cluster_id:65535
/sys/devices/system/cpu/cpu20/topology/cluster_id:65535
/sys/devices/system/cpu/cpu21/topology/cluster_id:65535
/sys/devices/system/cpu/cpu22/topology/cluster_id:65535
/sys/devices/system/cpu/cpu23/topology/cluster_id:65535
/sys/devices/system/cpu/cpu24/topology/cluster_id:65535
/sys/devices/system/cpu/cpu25/topology/cluster_id:65535
/sys/devices/system/cpu/cpu26/topology/cluster_id:65535
/sys/devices/system/cpu/cpu27/topology/cluster_id:65535
/sys/devices/system/cpu/cpu28/topology/cluster_id:65535
/sys/devices/system/cpu/cpu29/topology/cluster_id:65535
/sys/devices/system/cpu/cpu30/topology/cluster_id:65535
/sys/devices/system/cpu/cpu31/topology/cluster_id:65535
```

With the series:

```
/sys/devices/system/cpu/cpu0/topology/cluster_id:0
/sys/devices/system/cpu/cpu1/topology/cluster_id:1
/sys/devices/system/cpu/cpu2/topology/cluster_id:2
/sys/devices/system/cpu/cpu3/topology/cluster_id:3
/sys/devices/system/cpu/cpu4/topology/cluster_id:4
/sys/devices/system/cpu/cpu5/topology/cluster_id:5
/sys/devices/system/cpu/cpu6/topology/cluster_id:6
/sys/devices/system/cpu/cpu7/topology/cluster_id:7
/sys/devices/system/cpu/cpu8/topology/cluster_id:8
/sys/devices/system/cpu/cpu9/topology/cluster_id:9
/sys/devices/system/cpu/cpu10/topology/cluster_id:10
/sys/devices/system/cpu/cpu11/topology/cluster_id:11
/sys/devices/system/cpu/cpu12/topology/cluster_id:12
/sys/devices/system/cpu/cpu13/topology/cluster_id:13
/sys/devices/system/cpu/cpu14/topology/cluster_id:14
/sys/devices/system/cpu/cpu15/topology/cluster_id:15
/sys/devices/system/cpu/cpu16/topology/cluster_id:0
/sys/devices/system/cpu/cpu17/topology/cluster_id:1
/sys/devices/system/cpu/cpu18/topology/cluster_id:2
/sys/devices/system/cpu/cpu19/topology/cluster_id:3
/sys/devices/system/cpu/cpu20/topology/cluster_id:4
/sys/devices/system/cpu/cpu21/topology/cluster_id:5
/sys/devices/system/cpu/cpu22/topology/cluster_id:6
/sys/devices/system/cpu/cpu23/topology/cluster_id:7
/sys/devices/system/cpu/cpu24/topology/cluster_id:8
/sys/devices/system/cpu/cpu25/topology/cluster_id:9
/sys/devices/system/cpu/cpu26/topology/cluster_id:10
/sys/devices/system/cpu/cpu27/topology/cluster_id:11
/sys/devices/system/cpu/cpu28/topology/cluster_id:12
/sys/devices/system/cpu/cpu29/topology/cluster_id:13
/sys/devices/system/cpu/cpu30/topology/cluster_id:14
/sys/devices/system/cpu/cpu31/topology/cluster_id:15
```

Hence,

Tested-by: Oleksandr Natalenko <oleksandr@natalenko.name>

Thanks.

-- 
Oleksandr Natalenko (post-factum)

Re: [PATCH 0/2] arch/x86: Set L2 Cache ID on AMD processors

[K Prateek Nayak]

Hello Oleksandr,

Thank you for testing the series.

On 4/13/2023 6:47 PM, Oleksandr Natalenko wrote:
> Hello.
> 
> On pondělí 10. dubna 2023 18:35:25 CEST K Prateek Nayak wrote:
>> commit 66558b730f253 ("sched: Add cluster scheduler level for x86")
>> defined cluster on x86 as the set of threads sharing the same L2 cache.
>> cluster_id on x86, maps to the l2c_id which currently only Intel
>> processors set.
>>
>> This series sets the l2c_id on AMD processors with X86_FEATURE_TOPOEXT,
>> using the extended APIC ID and the "Cache Properties (L2)" CPUID
>> (0x8000001D EAX). On AMD processors without X86_FEATURE_TOPOEXT, current
>> behavior will continue.
>>
>> Following are the changes in value reported by
>> "/sys/devices/system/cpu/cpuX/topology/cluster_id" on a 2P Milan system
>> (2 x 64C/128T) where L2 is per-core level and SMT sibling of CPU (X) is
>> CPU ((X + 128) % 256).
>>
>> - tip:x86/core
>>
>>   $ for i in {0..255}; do\
>>       echo -n "CPU$i cluster_id: ";\
>>       cat /sys/devices/system/cpu/cpu$i/topology/cluster_id;\
>>     done;
>>
>>     CPU0 cluster_id: 65535
>>     CPU1 cluster_id: 65535
>>     CPU2 cluster_id: 65535
>>     CPU3 cluster_id: 65535
>>     CPU4 cluster_id: 65535
>>     ...
>>     CPU254 cluster_id: 65535
>>     CPU255 cluster_id: 65535
>>
>> - tip:x86/core + this series
>>
>>   $ for i in {0..255}; do\
>>       echo -n "CPU$i cluster_id: ";\
>>       cat /sys/devices/system/cpu/cpu$i/topology/cluster_id;\
>>     done;
>>
>>     CPU0 cluster_id: 0
>>     CPU1 cluster_id: 1
>>     CPU2 cluster_id: 2
>>     CPU3 cluster_id: 3
>>     CPU4 cluster_id: 4
>>     CPU5 cluster_id: 5
>>     CPU6 cluster_id: 6
>>     CPU7 cluster_id: 7
>>     CPU8 cluster_id: 8
>>     ...
>>     CPU126 cluster_id: 126
>>     CPU127 cluster_id: 127
>>     CPU128 cluster_id: 0
>>     CPU129 cluster_id: 1
>>     CPU130 cluster_id: 2
>>     CPU131 cluster_id: 3
>>     CPU132 cluster_id: 4
>>     CPU133 cluster_id: 5
>>     CPU134 cluster_id: 6
>>     CPU135 cluster_id: 7
>>     CPU136 cluster_id: 8
>>     ...
>>     CPU254 cluster_id: 126
>>     CPU255 cluster_id: 127
>>
>> Note: Hygon, theoretically, should be able to set the l2c_id using the
>> same cacheinfo_amd_init_l2c_id() function being added in Patch 1. Since
>> I do not have access to a Hygon machine to verify my theory, ccing Hygon
>> maintainer Pu Wen <puwen@hygon.cn> for l2c_id enablement on Hygon.
>>
>> The series also adds documentation for clusters on x86 platforms and
>> applies cleanly on top of tip:x86/core at commit ce3ba2af9695
>> ("x86: Suppress KMSAN reports in arch_within_stack_frames()")
>>
>> ---
>> K Prateek Nayak (2):
>>   arch/x86: Set L2 Cache ID on AMD and Hygon processors
>>   x86/Documentation: Add documentation about cluster
>>
>>  Documentation/x86/topology.rst   | 31 +++++++++++++++++++++
>>  arch/x86/include/asm/cacheinfo.h |  1 +
>>  arch/x86/kernel/cpu/amd.c        |  1 +
>>  arch/x86/kernel/cpu/cacheinfo.c  | 47 ++++++++++++++++++++++++++++++++
>>  arch/x86/kernel/cpu/hygon.c      |  1 +
>>  5 files changed, 81 insertions(+)
> 
> Having the following CPU:
> 
> ```
> Architecture:            x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Address sizes:         48 bits physical, 48 bits virtual
> Byte Order:            Little Endian
> CPU(s):                  32
> On-line CPU(s) list:   0-31
> Vendor ID:               AuthenticAMD
> Model name:            AMD Ryzen 9 5950X 16-Core Processor
> CPU family:          25
> Model:               33
> Thread(s) per core:  2
> Core(s) per socket:  16
> Socket(s):           1
> Stepping:            2
> Frequency boost:     enabled
> CPU(s) scaling MHz:  37%
> CPU max MHz:         5084,0000
> CPU min MHz:         550,0000
> BogoMIPS:            6789,07
> Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_
> tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c r
> drand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l
> lc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb
> sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip
> _save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid ove
> rflow_recov succor smca fsrm
> Virtualization features:
> Virtualization:        AMD-V
> Caches (sum of all):
> L1d:                   512 KiB (16 instances)
> L1i:                   512 KiB (16 instances)
> L2:                    8 MiB (16 instances)
> L3:                    64 MiB (2 instances)
> NUMA:
> NUMA node(s):          1
> NUMA node0 CPU(s):     0-31
> Vulnerabilities:
> Itlb multihit:         Not affected
> L1tf:                  Not affected
> Mds:                   Not affected
> Meltdown:              Not affected
> Mmio stale data:       Not affected
> Retbleed:              Not affected
> Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
> Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
> Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
> Srbds:                 Not affected
> Tsx async abort:       Not affected
> ```
> 
> Without the series:
> 
> ```
> /sys/devices/system/cpu/cpu0/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu1/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu2/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu3/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu4/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu5/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu6/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu7/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu8/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu9/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu10/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu11/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu12/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu13/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu14/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu15/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu16/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu17/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu18/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu19/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu20/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu21/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu22/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu23/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu24/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu25/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu26/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu27/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu28/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu29/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu30/topology/cluster_id:65535
> /sys/devices/system/cpu/cpu31/topology/cluster_id:65535
> ```
> 
> With the series:
> 
> ```
> /sys/devices/system/cpu/cpu0/topology/cluster_id:0
> /sys/devices/system/cpu/cpu1/topology/cluster_id:1
> /sys/devices/system/cpu/cpu2/topology/cluster_id:2
> /sys/devices/system/cpu/cpu3/topology/cluster_id:3
> /sys/devices/system/cpu/cpu4/topology/cluster_id:4
> /sys/devices/system/cpu/cpu5/topology/cluster_id:5
> /sys/devices/system/cpu/cpu6/topology/cluster_id:6
> /sys/devices/system/cpu/cpu7/topology/cluster_id:7
> /sys/devices/system/cpu/cpu8/topology/cluster_id:8
> /sys/devices/system/cpu/cpu9/topology/cluster_id:9
> /sys/devices/system/cpu/cpu10/topology/cluster_id:10
> /sys/devices/system/cpu/cpu11/topology/cluster_id:11
> /sys/devices/system/cpu/cpu12/topology/cluster_id:12
> /sys/devices/system/cpu/cpu13/topology/cluster_id:13
> /sys/devices/system/cpu/cpu14/topology/cluster_id:14
> /sys/devices/system/cpu/cpu15/topology/cluster_id:15
> /sys/devices/system/cpu/cpu16/topology/cluster_id:0
> /sys/devices/system/cpu/cpu17/topology/cluster_id:1
> /sys/devices/system/cpu/cpu18/topology/cluster_id:2
> /sys/devices/system/cpu/cpu19/topology/cluster_id:3
> /sys/devices/system/cpu/cpu20/topology/cluster_id:4
> /sys/devices/system/cpu/cpu21/topology/cluster_id:5
> /sys/devices/system/cpu/cpu22/topology/cluster_id:6
> /sys/devices/system/cpu/cpu23/topology/cluster_id:7
> /sys/devices/system/cpu/cpu24/topology/cluster_id:8
> /sys/devices/system/cpu/cpu25/topology/cluster_id:9
> /sys/devices/system/cpu/cpu26/topology/cluster_id:10
> /sys/devices/system/cpu/cpu27/topology/cluster_id:11
> /sys/devices/system/cpu/cpu28/topology/cluster_id:12
> /sys/devices/system/cpu/cpu29/topology/cluster_id:13
> /sys/devices/system/cpu/cpu30/topology/cluster_id:14
> /sys/devices/system/cpu/cpu31/topology/cluster_id:15
> ```
> 
> Hence,
> 
> Tested-by: Oleksandr Natalenko <oleksandr@natalenko.name>

I've retained your tag in Patch 1 of v2
(https://lore.kernel.org/lkml/20230413172918.1500-2-kprateek.nayak@amd.com/)
since there are no functional changes for AMD in v2.

> 
> Thanks.
> 

--
Thanks and Regards,
Prateek