[PATCH v2 0/3] drm/amdkfd: Add support for discontiguous and CPU-less NUMA topologies

[PATCH v2 0/3] drm/amdkfd: Add support for discontiguous and CPU-less NUMA topologies

[Dhruv B]

This series fixes VCRAT generation in AMDKFD for systems that do not
provide a CRAT table and where the driver generates a Virtual CRAT
(VCRAT) for the CPU topology.

The current implementation uses for_each_online_node() to iterate over
NUMA nodes and assigns proximity domain IDs sequentially (0, 1, 2, ...)
regardless of the actual NUMA node IDs. It also assumes that every
online NUMA node contains both CPUs and memory, unconditionally
emitting Compute Unit and Memory CRAT subtypes for each node.

These assumptions break on systems with:

  Discontiguous (sparse) NUMA node IDs
  CPU-less NUMA nodes
  Memory-less NUMA nodes

For example:

  available: 3 nodes (0,2-3)

  node 0: CPUs present
  node 2: CPU-less
  node 3: CPU-less

In this case, the existing code iterates over nodes 0, 2 and 3 and
assigns proximity domains 0, 1 and 2 respectively, instead of 0, 2
and 3. The mismatch between NUMA node IDs and proximity domain IDs
results in an inconsistent KFD topology and causes topology parsing
to fail with:

  amdgpu: Virtual CRAT table created for GPU
  amdgpu: Error parsing VCRAT
  kfd: amdgpu: Error adding device to topology
  kfd: amdgpu: Error initializing KFD node

To fix the proximity domain mismatch, the VCRAT generation loop is
changed to iterate over the complete NUMA node ID range up to the
last online node, incrementing num_domains for every node ID in the
range. This preserves a direct 1:1 mapping between NUMA node ID and
proximity domain ID, while skipping resource entry creation for
offline or unpopulated node IDs.

To fix the unconditional subtype emission, explicit guards are added
so that Compute Unit entries are only written for nodes that have
valid CPU cores (kfd_numa_node_to_apic_id() != -1), and Memory
entries are only written for nodes that actually contain memory
(node_state(N_MEMORY)). This prevents invalid or empty CRAT subtype
entries from being emitted for CPU-less, memory-less, or empty NUMA
nodes.

On a system with contiguous NUMA nodes, numactl reports:

  $ numactl -H
  available: 4 nodes (0-3)
  node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
           18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
           34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
           50 51 52 53 54 55
  node 0 size: 881496 MB
  node 1 cpus: 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70
           71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
           87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
           102 103 104 105 106 107 108 109 110 111
  node 1 size: 885640 MB
  node 2 cpus: 112 113 114 115 116 117 118 119 120 121 122
           123 124 125 126 127 128 129 130 131 132 133 134
           135 136 137 138 139 140 141 142 143 144 145 146
           147 148 149 150 151 152 153 154 155 156 157 158
           159 160 161 162 163 164 165 166 167
  node 2 size: 885640 MB
  node 3 cpus: 168 169 170 171 172 173 174 175 176 177 178
           179 180 181 182 183 184 185 186 187 188 189 190
           191 192 193 194 195 196 197 198 199 200 201 202
           203 204 205 206 207 208 209 210 211 212 213 214
           215 216 217 218 219 220 221 222 223
  node 3 size: 885394 MB

The KFD topology sysfs reflects all 4 CPU nodes plus 2 GPUs:

  /sys/class/kfd/kfd/topology/nodes: 0  1  2  3  4  5

  0: cpu_cores_count 56  mem_banks_count 1  io_links_count 0
  1: cpu_cores_count 56  mem_banks_count 1  io_links_count 0
  2: cpu_cores_count 56  mem_banks_count 1  io_links_count 2
  3: cpu_cores_count 56  mem_banks_count 1  io_links_count 0
  4: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)
  5: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)

On the same system with sparse NUMA nodes, numactl reports:

  $ numactl -H
  available: 3 nodes (0,2-3)
  node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
           18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
           34 35 36 37 38 39
  node 0 size: 519305 MB
  node 2 cpus:
  node 2 size: 0 MB
  node 3 cpus:
  node 3 size: 0 MB

After this series the KFD topology sysfs correctly reflects all node
IDs including the absent node 1 as a placeholder:

  /sys/class/kfd/kfd/topology/nodes: 0  1  2  3  4  5

  0: cpu_cores_count 40  mem_banks_count 1  io_links_count 0
  1: cpu_cores_count 0   mem_banks_count 0  io_links_count 0 (Absent node, placeholder domain)
  2: cpu_cores_count 0   mem_banks_count 0  io_links_count 2 (CPU-less, GPUs attached here)
  3: cpu_cores_count 0   mem_banks_count 0  io_links_count 0 (CPU-less)
  4: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)
  5: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)

The GPU I/O links correctly reference proximity domain 2, which
corresponds directly to NUMA node 2 where the GPUs are physically
attached. This is confirmed by the io_links sysfs entries:

  node 4 io_link: node_from 4  node_to 2  type 2  max_bandwidth 10000
  node 5 io_link: node_from 5  node_to 2  type 2  max_bandwidth 10000

Both GPUs report node_to 2, matching the proximity domain of NUMA
node 2 (the CPU-less node to which the GPUs are connected).

This series consists of the following patches:

Patch 1 removes unused code in the CRAT initialization path as a
preparatory cleanup.

Patch 2 fixes the proximity domain mismatch by iterating through
the complete NUMA node ID range and preserving a direct NUMA-node-
to-proximity-domain mapping.

Patch 3 restricts Compute Unit and Memory CRAT subtype entries to
NUMA nodes that actually have the corresponding resources.
---
v1 -> v2
- Addressed Felix's comment and fixed the incorrect proximity domain mapping 
  caused due to crat_table->num_domains++ being at the start of the loop.

v1 - https://lore.kernel.org/all/cover.1791208447.git.dhruv.b@linux.ibm.com/
---
Dhruv B (3):
  drm/amdkfd: Remove unused code in CRAT initialization
  drm/amdkfd: Add support for discontiguous NUMA node topology
  drm/amdkfd: Restrict CU and memory entries to resource nodes

 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 60 ++++++++++++-----------
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  2 +-
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 11 +----
 3 files changed, 35 insertions(+), 38 deletions(-)

-- 
2.54.0 (Apple Git-157)

[PATCH v2 1/3] drm/amdkfd: Remove unused code in CRAT initialization

[Dhruv B]

The 'size' argument in kfd_create_crat_image_virtual() is not used
by callers to retrieve the allocated CRAT image size. The callers
rely solely on the returned image pointer and inspect the parsed CRAT
table header directly for entry counts and lengths.

Additionally, in kfd_topology_init(), the 'kdev' pointer assignment
via list_first_entry() after parsing the CPU CRAT table is not being
used and can be removed.

Clean up the function signature of kfd_create_crat_image_virtual() by
dropping the unused 'size' parameter, manage the buffer size locally,
and remove the unused variable assignments in kfd_topology_init().

Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Co-developed-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 10 ++++------
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  2 +-
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 11 ++---------
 3 files changed, 7 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 6e0df685503d..99b0c7f825fe 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -2411,7 +2411,6 @@ static int kfd_create_vcrat_image_gpu(void *pcrat_image,
  *
  *	@crat_image: VCRAT image created because ACPI does not have a
  *		     CRAT for this device
- *	@size: [OUT] size of virtual crat_image
  *	@flags:	COMPUTE_UNIT_CPU - Create VCRAT for CPU device
  *		COMPUTE_UNIT_GPU - Create VCRAT for GPU
  *		(COMPUTE_UNIT_CPU | COMPUTE_UNIT_GPU) - Create VCRAT for APU
@@ -2421,7 +2420,7 @@ static int kfd_create_vcrat_image_gpu(void *pcrat_image,
  *
  *	Return 0 if successful else return -ve value
  */
-int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
+int kfd_create_crat_image_virtual(void **crat_image,
 				  int flags, struct kfd_node *kdev,
 				  uint32_t proximity_domain)
 {
@@ -2449,9 +2448,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
 		pcrat_image = kvmalloc(dyn_size, GFP_KERNEL);
 		if (!pcrat_image)
 			return -ENOMEM;
-		*size = dyn_size;
 		pr_debug("CRAT size is %ld", dyn_size);
-		ret = kfd_create_vcrat_image_cpu(pcrat_image, size);
+		ret = kfd_create_vcrat_image_cpu(pcrat_image, &dyn_size);
 		break;
 	case COMPUTE_UNIT_GPU:
 		if (!kdev)
@@ -2459,8 +2457,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
 		pcrat_image = kvmalloc(VCRAT_SIZE_FOR_GPU, GFP_KERNEL);
 		if (!pcrat_image)
 			return -ENOMEM;
-		*size = VCRAT_SIZE_FOR_GPU;
-		ret = kfd_create_vcrat_image_gpu(pcrat_image, size, kdev,
+		dyn_size = VCRAT_SIZE_FOR_GPU;
+		ret = kfd_create_vcrat_image_gpu(pcrat_image, &dyn_size, kdev,
 						 proximity_domain);
 		break;
 	case (COMPUTE_UNIT_CPU | COMPUTE_UNIT_GPU):
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
index a8ca7ecb6d27..418a3939eda3 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
@@ -313,7 +313,7 @@ int kfd_get_gpu_cache_info(struct kfd_node *kdev, struct kfd_gpu_cache_info **pc
 void kfd_destroy_crat_image(void *crat_image);
 int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
 			 uint32_t proximity_domain);
-int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
+int kfd_create_crat_image_virtual(void **crat_image,
 				  int flags, struct kfd_node *kdev,
 				  uint32_t proximity_domain);
 
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index 35b3abe57b80..b38d12f3da84 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -996,7 +996,6 @@ static void kfd_add_non_crat_information(struct kfd_topology_device *kdev)
 int kfd_topology_init(void)
 {
 	void *crat_image = NULL;
-	size_t image_size = 0;
 	int ret;
 	struct list_head temp_topology_device_list;
 	int cpu_only_node = 0;
@@ -1023,8 +1022,7 @@ int kfd_topology_init(void)
 	 */
 	proximity_domain = 0;
 
-	ret = kfd_create_crat_image_virtual(&crat_image, &image_size,
-					    COMPUTE_UNIT_CPU, NULL,
+	ret = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_CPU, NULL,
 					    proximity_domain);
 	cpu_only_node = 1;
 	if (ret) {
@@ -1040,9 +1038,6 @@ int kfd_topology_init(void)
 		goto err;
 	}
 
-	kdev = list_first_entry(&temp_topology_device_list,
-				struct kfd_topology_device, list);
-
 	down_write(&topology_lock);
 	kfd_topology_update_device_list(&temp_topology_device_list,
 					&topology_device_list);
@@ -1871,11 +1866,9 @@ static int kfd_topology_add_device_locked(struct kfd_node *gpu,
 	int proximity_domain = ++topology_crat_proximity_domain;
 	struct list_head temp_topology_device_list;
 	void *crat_image = NULL;
-	size_t image_size = 0;
 	int res;
 
-	res = kfd_create_crat_image_virtual(&crat_image, &image_size,
-					    COMPUTE_UNIT_GPU, gpu,
+	res = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_GPU, gpu,
 					    proximity_domain);
 	if (res) {
 		dev_err(gpu->adev->dev, "Error creating VCRAT\n");
-- 
2.54.0 (Apple Git-157)

[PATCH v2 2/3] drm/amdkfd: Add support for discontiguous NUMA node topology

[Dhruv B]

On systems with discontiguous NUMA nodes, the GPU driver fails to load
with the following errors:

[    5.609767] amdgpu 0038:12:00.0: Error parsing VCRAT
[    5.609781] kfd kfd: Error adding device to topology
[    5.609848] kfd kfd: Error initializing KFD node

In the current KFD code, proximity domains are created contiguously
when generating the VCRAT. With discontiguous NUMA node IDs, this
creates a mismatch between the proximity domain ID and the NUMA node ID.
For example, if NUMA nodes 0, 2 and 3 are online, the current code
assigns proximity domains 0, 1 and 2 to these nodes respectively,
instead of domains 0, 2 and 3.

The KFD topology uses the NUMA node and proximity domain information to
create topology devices representing the CPU and memory resources
associated with each NUMA node. This topology is exposed to userspace
through sysfs, where the NUMA node and proximity domain information are
used to identify the topology devices. The mismatch between the NUMA
node ID and proximity domain therefore results in an inconsistent KFD
topology and causes topology parsing to fail.

Make the proximity domain ID correspond directly to the NUMA node ID by
iterating through the complete NUMA node ID range up to the last online
node. Account for every node ID when assigning proximity domains, while
skipping resource entry creation for offline or unpopulated nodes.
This preserves the NUMA-node-to-proximity-domain mapping and allows the
KFD topology and its sysfs representation to correctly reflect
discontiguous NUMA topologies.

Co-developed-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 99b0c7f825fe..7dad8f30ff2a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1899,7 +1899,7 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 	acpi_status status;
 	struct crat_subtype_generic *sub_type_hdr;
 	int avail_size = *size;
-	int numa_node_id;
+	int numa_node_id, last_node;
 #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
 	uint32_t entries = 0;
 #endif
@@ -1935,10 +1935,13 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 	crat_table->num_domains = 0;
 
 	sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
+	last_node = find_last_bit(node_online_map.bits, MAX_NUMNODES);
 
-	for_each_online_node(numa_node_id) {
-		if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
+	for (numa_node_id = 0; numa_node_id <= last_node; numa_node_id++) {
+		if (!node_online(numa_node_id)) {
+			crat_table->num_domains++;
 			continue;
+		}
 
 		/* Fill in Subtype: Compute Unit */
 		ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
@@ -1982,7 +1985,6 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 #else
 		pr_info("IO link not available for non x86 platforms\n");
 #endif
-
 		crat_table->num_domains++;
 	}
 
-- 
2.54.0 (Apple Git-157)

[PATCH v2 3/3] drm/amdkfd: Restrict CU and memory entries to resource nodes

[Dhruv B]

In heterogeneous or non-uniform NUMA systems, a NUMA node may be
CPU-less, memory-less, or both CPU-less and memory-less.

Previously, kfd_create_vcrat_image_cpu() assumed that every online NUMA
node contained both compute units and memory, attempting to create both
CU and memory CRAT entries unconditionally for every online node.

Add explicit guards to:
1. Only create Compute Unit CRAT entries if the NUMA node has valid
   CPU cores (kfd_numa_node_to_apic_id() != -1).
2. Only create Memory CRAT entries if the NUMA node actually contains
   memory (node_state(numa_node_id, N_MEMORY)).

This avoids emitting invalid or empty CRAT subtype entries for CPU-less,
memory-less, or both CPU-less and memory-less NUMA nodes in virtual CRAT
generation.

Co-developed-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 40 +++++++++++++++------------
 1 file changed, 22 insertions(+), 18 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 7dad8f30ff2a..329931b0edc6 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1944,28 +1944,32 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 		}
 
 		/* Fill in Subtype: Compute Unit */
-		ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
-			crat_table->num_domains,
-			(struct crat_subtype_computeunit *)sub_type_hdr);
-		if (ret < 0)
-			return ret;
-		crat_table->length += sub_type_hdr->length;
-		crat_table->total_entries++;
+		if (kfd_numa_node_to_apic_id(numa_node_id) != -1) {
+			ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
+						  crat_table->num_domains,
+						  (struct crat_subtype_computeunit *)sub_type_hdr);
+			if (ret < 0)
+				return ret;
+			crat_table->length += sub_type_hdr->length;
+			crat_table->total_entries++;
 
-		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
-			sub_type_hdr->length);
+			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
+				sub_type_hdr->length);
+		}
 
 		/* Fill in Subtype: Memory */
-		ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
-			crat_table->num_domains,
-			(struct crat_subtype_memory *)sub_type_hdr);
-		if (ret < 0)
-			return ret;
-		crat_table->length += sub_type_hdr->length;
-		crat_table->total_entries++;
+		if (node_state(numa_node_id, N_MEMORY)) {
+			ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
+							crat_table->num_domains,
+							(struct crat_subtype_memory *)sub_type_hdr);
+			if (ret < 0)
+				return ret;
+			crat_table->length += sub_type_hdr->length;
+			crat_table->total_entries++;
 
-		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
-			sub_type_hdr->length);
+			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
+				sub_type_hdr->length);
+		}
 
 		/* Fill in Subtype: IO Link */
 #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
-- 
2.54.0 (Apple Git-157)

Re: [PATCH v2 3/3] drm/amdkfd: Restrict CU and memory entries to resource nodes

[Felix Kuehling]

On 2026-10-06 08:51, Dhruv B wrote:
> In heterogeneous or non-uniform NUMA systems, a NUMA node may be
> CPU-less, memory-less, or both CPU-less and memory-less.
>
> Previously, kfd_create_vcrat_image_cpu() assumed that every online NUMA
> node contained both compute units and memory, attempting to create both
> CU and memory CRAT entries unconditionally for every online node.
>
> Add explicit guards to:
> 1. Only create Compute Unit CRAT entries if the NUMA node has valid
>     CPU cores (kfd_numa_node_to_apic_id() != -1).
> 2. Only create Memory CRAT entries if the NUMA node actually contains
>     memory (node_state(numa_node_id, N_MEMORY)).
>
> This avoids emitting invalid or empty CRAT subtype entries for CPU-less,
> memory-less, or both CPU-less and memory-less NUMA nodes in virtual CRAT
> generation.

@David Yat Sin, I think we currently support CPU-less and memory-less 
NUMA domains already. I know we have CPU-less NUMA domains on MI300A in 
NPS4/TPX mode. And I remember we've fixed bugs on NUMA CPU systems with 
only one DIMM populated, which would result in a memory-less NUMA domain.

Therefore, I'm not sure whether this patch is really needed and I'd be 
concerned about introducing a regression. Is ROCr able to handle the 
changes being proposed here? David, could you take a look at this and 
maybe even test it, or find someone who could help test it on the system 
configurations I described?

@Dhruv B, how much testing have you done on this patch series. Given 
that the bug in patch 2 slipped through in v1, I'm worried that it's not 
very thorough.

That said, the patch series looks good to me now, in theory.

Regards,
   Felix


>
> Co-developed-by: Donet Tom <donettom@linux.ibm.com>
> Signed-off-by: Donet Tom <donettom@linux.ibm.com>
> Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 40 +++++++++++++++------------
>   1 file changed, 22 insertions(+), 18 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> index 7dad8f30ff2a..329931b0edc6 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> @@ -1944,28 +1944,32 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
>   		}
>   
>   		/* Fill in Subtype: Compute Unit */
> -		ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
> -			crat_table->num_domains,
> -			(struct crat_subtype_computeunit *)sub_type_hdr);
> -		if (ret < 0)
> -			return ret;
> -		crat_table->length += sub_type_hdr->length;
> -		crat_table->total_entries++;
> +		if (kfd_numa_node_to_apic_id(numa_node_id) != -1) {
> +			ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
> +						  crat_table->num_domains,
> +						  (struct crat_subtype_computeunit *)sub_type_hdr);
> +			if (ret < 0)
> +				return ret;
> +			crat_table->length += sub_type_hdr->length;
> +			crat_table->total_entries++;
>   
> -		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
> -			sub_type_hdr->length);
> +			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
> +				sub_type_hdr->length);
> +		}
>   
>   		/* Fill in Subtype: Memory */
> -		ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
> -			crat_table->num_domains,
> -			(struct crat_subtype_memory *)sub_type_hdr);
> -		if (ret < 0)
> -			return ret;
> -		crat_table->length += sub_type_hdr->length;
> -		crat_table->total_entries++;
> +		if (node_state(numa_node_id, N_MEMORY)) {
> +			ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
> +							crat_table->num_domains,
> +							(struct crat_subtype_memory *)sub_type_hdr);
> +			if (ret < 0)
> +				return ret;
> +			crat_table->length += sub_type_hdr->length;
> +			crat_table->total_entries++;
>   
> -		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
> -			sub_type_hdr->length);
> +			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
> +				sub_type_hdr->length);
> +		}
>   
>   		/* Fill in Subtype: IO Link */
>   #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)

Re: [PATCH v2 3/3] drm/amdkfd: Restrict CU and memory entries to resource nodes

[Dhruv Bhogaonkar]

On 07/10/26 01:30, Felix Kuehling wrote:
>
> On 2026-10-06 08:51, Dhruv B wrote:
>> In heterogeneous or non-uniform NUMA systems, a NUMA node may be
>> CPU-less, memory-less, or both CPU-less and memory-less.
>>
>> Previously, kfd_create_vcrat_image_cpu() assumed that every online NUMA
>> node contained both compute units and memory, attempting to create both
>> CU and memory CRAT entries unconditionally for every online node.
>>
>> Add explicit guards to:
>> 1. Only create Compute Unit CRAT entries if the NUMA node has valid
>>     CPU cores (kfd_numa_node_to_apic_id() != -1).
>> 2. Only create Memory CRAT entries if the NUMA node actually contains
>>     memory (node_state(numa_node_id, N_MEMORY)).
>>
>> This avoids emitting invalid or empty CRAT subtype entries for CPU-less,
>> memory-less, or both CPU-less and memory-less NUMA nodes in virtual CRAT
>> generation.
>
> @David Yat Sin, I think we currently support CPU-less and memory-less 
> NUMA domains already. I know we have CPU-less NUMA domains on MI300A 
> in NPS4/TPX mode. And I remember we've fixed bugs on NUMA CPU systems 
> with only one DIMM populated, which would result in a memory-less NUMA 
> domain.
>
> Therefore, I'm not sure whether this patch is really needed and I'd be 
> concerned about introducing a regression. Is ROCr able to handle the 
> changes being proposed here? David, could you take a look at this and 
> maybe even test it, or find someone who could help test it on the 
> system configurations I described?

Thank you for the feedback, Felix. Apologies for the bug in v1 —
that was an oversight on our part.

 From the current code, the kfd_numa_node_to_apic_id() == -1
continue causes the entire loop body to be skipped
for a CPU-less NUMA node:

   for_each_online_node(numa_node_id) {
           if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
                   continue;

           ret = kfd_fill_cu_for_cpu(...);
           ...
           ret = kfd_fill_mem_info_for_cpu(...);
   }

When the loop body is skipped, neither a CU entry nor a memory entry is
generated for that node, and no KFD topology device appears to be
registered for it from this code path. From our reading, it appears that
CPU-less nodes are effectively absent from the generated VCRAT topology
rather than being represented with an empty or placeholder entry.

>
> @Dhruv B, how much testing have you done on this patch series. Given 
> that the bug in patch 2 slipped through in v1, I'm worried that it's 
> not very thorough.

Regarding testing, on the userspace side we ran the RCCL unit tests.

On the kernel side, we verified that the IO links are created correctly
for both topologies by inspecting the KFD topology sysfs entries after
driver load.

Here are the RCCL unit test results for both configurations, along with
the corresponding numactl and KFD topology sysfs output observed after
driver load.

---
Test environment — contiguous NUMA (4 nodes, 0-3)
--------------------------------------------------

numactl reports 4 contiguous nodes (0-3), all with CPUs and memory:

   $ numactl -H
   available: 4 nodes (0-3)
   node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
                22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
                41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
   node 0 size: 881496 MB
   node 0 free: 875909 MB
   node 1 cpus: 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
                75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93
                94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109
                110 111
   node 1 size: 885640 MB
   node 1 free: 883815 MB
   node 2 cpus: 112 113 114 115 116 117 118 119 120 121 122 123 124 125
                126 127 128 129 130 131 132 133 134 135 136 137 138 139
                140 141 142 143 144 145 146 147 148 149 150 151 152 153
                154 155 156 157 158 159 160 161 162 163 164 165 166 167
   node 2 size: 885640 MB
   node 2 free: 874248 MB
   node 3 cpus: 168 169 170 171 172 173 174 175 176 177 178 179 180 181
                182 183 184 185 186 187 188 189 190 191 192 193 194 195
                196 197 198 199 200 201 202 203 204 205 206 207 208 209
                210 211 212 213 214 215 216 217 218 219 220 221 222 223
   node 3 size: 885394 MB
   node 3 free: 884033 MB

The KFD topology sysfs reflects all 4 CPU nodes (0-3) plus 2 GPUs
(nodes 4 and 5), each CPU node with correct core and memory counts:

   /sys/class/kfd/kfd/topology/nodes: 0  1  2 3  4  5

   0: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
   1: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
   2: cpu_cores_count 56  mem_banks_count 1 io_links_count 2
   3: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
   4: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
   5: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)

RCCL unit tests — 948 cases, all PASS:

---
Test environment — discontiguous NUMA (3 nodes: 0, 2-3)
--------------------------------------------------------

numactl reports 3 nodes with a gap at node 1. Nodes 2 and 3 are
CPU-less and memory-less:

   $ numactl -H
   available: 3 nodes (0,2-3)
   node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
                22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
   node 0 size: 519305 MB
   node 0 free: 504963 MB
   node 2 cpus: (none)
   node 2 size: 0 MB
   node 2 free: 0 MB
   node 3 cpus: (none)
   node 3 size: 0 MB
   node 3 free: 0 MB

After this patch series the KFD topology sysfs correctly preserves the
NUMA node IDs as proximity domains. Node 1 appears as an empty
placeholder, nodes 2 and 3 have no cores or memory banks but carry the
GPU IO links, and both GPUs correctly reference node 2 as their attached
CPU-less domain:

   /sys/class/kfd/kfd/topology/nodes: 0  1  2 3  4  5

   0: cpu_cores_count 40  mem_banks_count 1 io_links_count 0
   1: cpu_cores_count  0  mem_banks_count 0 io_links_count 0  (absent, 
placeholder)
   2: cpu_cores_count  0  mem_banks_count 0 io_links_count 2  (CPU-less, 
GPUs attached)
   3: cpu_cores_count  0  mem_banks_count 0 io_links_count 0  (CPU-less)
   4: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
   5: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)

   node 4 io_link: node_from 4  node_to 2 type 2  max_bandwidth 10000
   node 5 io_link: node_from 5  node_to 2 type 2  max_bandwidth 10000

RCCL unit tests — 948 cases, all PASS:

We fully understand your concern about regressions. We would
greatly appreciate it if someone with access to the systems you mentioned
could test this series, and we are very happy to make any adjustments
based on that feedback.

Regards,
Dhruv

>
> That said, the patch series looks good to me now, in theory.
>
> Regards,
>   Felix
>
>
>>
>> Co-developed-by: Donet Tom <donettom@linux.ibm.com>
>> Signed-off-by: Donet Tom <donettom@linux.ibm.com>
>> Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
>> ---
>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 40 +++++++++++++++------------
>>   1 file changed, 22 insertions(+), 18 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c 
>> b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>> index 7dad8f30ff2a..329931b0edc6 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>> @@ -1944,28 +1944,32 @@ static int kfd_create_vcrat_image_cpu(void 
>> *pcrat_image, size_t *size)
>>           }
>>             /* Fill in Subtype: Compute Unit */
>> -        ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
>> -            crat_table->num_domains,
>> -            (struct crat_subtype_computeunit *)sub_type_hdr);
>> -        if (ret < 0)
>> -            return ret;
>> -        crat_table->length += sub_type_hdr->length;
>> -        crat_table->total_entries++;
>> +        if (kfd_numa_node_to_apic_id(numa_node_id) != -1) {
>> +            ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
>> +                          crat_table->num_domains,
>> +                          (struct crat_subtype_computeunit 
>> *)sub_type_hdr);
>> +            if (ret < 0)
>> +                return ret;
>> +            crat_table->length += sub_type_hdr->length;
>> +            crat_table->total_entries++;
>>   -        sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
>> -            sub_type_hdr->length);
>> +            sub_type_hdr = (typeof(sub_type_hdr))((char 
>> *)sub_type_hdr +
>> +                sub_type_hdr->length);
>> +        }
>>             /* Fill in Subtype: Memory */
>> -        ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
>> -            crat_table->num_domains,
>> -            (struct crat_subtype_memory *)sub_type_hdr);
>> -        if (ret < 0)
>> -            return ret;
>> -        crat_table->length += sub_type_hdr->length;
>> -        crat_table->total_entries++;
>> +        if (node_state(numa_node_id, N_MEMORY)) {
>> +            ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
>> +                            crat_table->num_domains,
>> +                            (struct crat_subtype_memory 
>> *)sub_type_hdr);
>> +            if (ret < 0)
>> +                return ret;
>> +            crat_table->length += sub_type_hdr->length;
>> +            crat_table->total_entries++;
>>   -        sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
>> -            sub_type_hdr->length);
>> +            sub_type_hdr = (typeof(sub_type_hdr))((char 
>> *)sub_type_hdr +
>> +                sub_type_hdr->length);
>> +        }
>>             /* Fill in Subtype: IO Link */
>>   #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)

Re: [PATCH v2 3/3] drm/amdkfd: Restrict CU and memory entries to resource nodes

[Kuehling, Felix]

On 2026-10-07 07:14, Dhruv Bhogaonkar wrote:
> [Some people who received this message don't often get email from 
> dhruvb@linux.ibm.com. Learn why this is important at 
> https://aka.ms/LearnAboutSenderIdentification ]
>
> On 07/10/26 01:30, Felix Kuehling wrote:
>>
>> On 2026-10-06 08:51, Dhruv B wrote:
>>> In heterogeneous or non-uniform NUMA systems, a NUMA node may be
>>> CPU-less, memory-less, or both CPU-less and memory-less.
>>>
>>> Previously, kfd_create_vcrat_image_cpu() assumed that every online NUMA
>>> node contained both compute units and memory, attempting to create both
>>> CU and memory CRAT entries unconditionally for every online node.
>>>
>>> Add explicit guards to:
>>> 1. Only create Compute Unit CRAT entries if the NUMA node has valid
>>>     CPU cores (kfd_numa_node_to_apic_id() != -1).
>>> 2. Only create Memory CRAT entries if the NUMA node actually contains
>>>     memory (node_state(numa_node_id, N_MEMORY)).
>>>
>>> This avoids emitting invalid or empty CRAT subtype entries for 
>>> CPU-less,
>>> memory-less, or both CPU-less and memory-less NUMA nodes in virtual 
>>> CRAT
>>> generation.
>>
>> @David Yat Sin, I think we currently support CPU-less and memory-less
>> NUMA domains already. I know we have CPU-less NUMA domains on MI300A
>> in NPS4/TPX mode. And I remember we've fixed bugs on NUMA CPU systems
>> with only one DIMM populated, which would result in a memory-less NUMA
>> domain.
>>
>> Therefore, I'm not sure whether this patch is really needed and I'd be
>> concerned about introducing a regression. Is ROCr able to handle the
>> changes being proposed here? David, could you take a look at this and
>> maybe even test it, or find someone who could help test it on the
>> system configurations I described?
>
> Thank you for the feedback, Felix. Apologies for the bug in v1 —
> that was an oversight on our part.
>
> From the current code, the kfd_numa_node_to_apic_id() == -1
> continue causes the entire loop body to be skipped
> for a CPU-less NUMA node:
>
>   for_each_online_node(numa_node_id) {
>           if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
>                   continue;
>
>           ret = kfd_fill_cu_for_cpu(...);
>           ...
>           ret = kfd_fill_mem_info_for_cpu(...);
>   }
>
> When the loop body is skipped, neither a CU entry nor a memory entry is
> generated for that node, and no KFD topology device appears to be
> registered for it from this code path. From our reading, it appears that
> CPU-less nodes are effectively absent from the generated VCRAT topology
> rather than being represented with an empty or placeholder entry.

OK. I'm a bit surprised about that, because MI300A in NPS4 mode should 
have three CPU-less NUMA nodes per socket. Maybe we never depended on 
them being represented as CPU-less NUMA nodes in the KFD topology. Then 
this change should be an improvement because it removes some ambiguity 
between NUMA node numbers and KFD proximity domains.


>
>>
>> @Dhruv B, how much testing have you done on this patch series. Given
>> that the bug in patch 2 slipped through in v1, I'm worried that it's
>> not very thorough.
>
> Regarding testing, on the userspace side we ran the RCCL unit tests.
>
> On the kernel side, we verified that the IO links are created correctly
> for both topologies by inspecting the KFD topology sysfs entries after
> driver load.
>
> Here are the RCCL unit test results for both configurations, along with
> the corresponding numactl and KFD topology sysfs output observed after
> driver load.

Sounds good. We still need to test MI300A to make sure the changes don't 
confuse ROCr or amdsmi. Applications mostly shouldn't care because they 
don't use KFD topology node numbers directly. I'm told that most of the 
MI300A systems in production seem to use NPS1, so this shouldn't affect 
them.

Regards,
   Felix


>
> ---
> Test environment — contiguous NUMA (4 nodes, 0-3)
> --------------------------------------------------
>
> numactl reports 4 contiguous nodes (0-3), all with CPUs and memory:
>
>   $ numactl -H
>   available: 4 nodes (0-3)
>   node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
>                22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
>                41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
>   node 0 size: 881496 MB
>   node 0 free: 875909 MB
>   node 1 cpus: 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
>                75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93
>                94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109
>                110 111
>   node 1 size: 885640 MB
>   node 1 free: 883815 MB
>   node 2 cpus: 112 113 114 115 116 117 118 119 120 121 122 123 124 125
>                126 127 128 129 130 131 132 133 134 135 136 137 138 139
>                140 141 142 143 144 145 146 147 148 149 150 151 152 153
>                154 155 156 157 158 159 160 161 162 163 164 165 166 167
>   node 2 size: 885640 MB
>   node 2 free: 874248 MB
>   node 3 cpus: 168 169 170 171 172 173 174 175 176 177 178 179 180 181
>                182 183 184 185 186 187 188 189 190 191 192 193 194 195
>                196 197 198 199 200 201 202 203 204 205 206 207 208 209
>                210 211 212 213 214 215 216 217 218 219 220 221 222 223
>   node 3 size: 885394 MB
>   node 3 free: 884033 MB
>
> The KFD topology sysfs reflects all 4 CPU nodes (0-3) plus 2 GPUs
> (nodes 4 and 5), each CPU node with correct core and memory counts:
>
>   /sys/class/kfd/kfd/topology/nodes: 0  1  2 3  4  5
>
>   0: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
>   1: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
>   2: cpu_cores_count 56  mem_banks_count 1 io_links_count 2
>   3: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
>   4: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>   5: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>
> RCCL unit tests — 948 cases, all PASS:
>
> ---
> Test environment — discontiguous NUMA (3 nodes: 0, 2-3)
> --------------------------------------------------------
>
> numactl reports 3 nodes with a gap at node 1. Nodes 2 and 3 are
> CPU-less and memory-less:
>
>   $ numactl -H
>   available: 3 nodes (0,2-3)
>   node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
>                22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
>   node 0 size: 519305 MB
>   node 0 free: 504963 MB
>   node 2 cpus: (none)
>   node 2 size: 0 MB
>   node 2 free: 0 MB
>   node 3 cpus: (none)
>   node 3 size: 0 MB
>   node 3 free: 0 MB
>
> After this patch series the KFD topology sysfs correctly preserves the
> NUMA node IDs as proximity domains. Node 1 appears as an empty
> placeholder, nodes 2 and 3 have no cores or memory banks but carry the
> GPU IO links, and both GPUs correctly reference node 2 as their attached
> CPU-less domain:
>
>   /sys/class/kfd/kfd/topology/nodes: 0  1  2 3  4  5
>
>   0: cpu_cores_count 40  mem_banks_count 1 io_links_count 0
>   1: cpu_cores_count  0  mem_banks_count 0 io_links_count 0 (absent,
> placeholder)
>   2: cpu_cores_count  0  mem_banks_count 0 io_links_count 2 (CPU-less,
> GPUs attached)
>   3: cpu_cores_count  0  mem_banks_count 0 io_links_count 0 (CPU-less)
>   4: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>   5: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>
>   node 4 io_link: node_from 4  node_to 2 type 2  max_bandwidth 10000
>   node 5 io_link: node_from 5  node_to 2 type 2  max_bandwidth 10000
>
> RCCL unit tests — 948 cases, all PASS:
>
> We fully understand your concern about regressions. We would
> greatly appreciate it if someone with access to the systems you mentioned
> could test this series, and we are very happy to make any adjustments
> based on that feedback.
>
> Regards,
> Dhruv
>
>>
>> That said, the patch series looks good to me now, in theory.
>>
>> Regards,
>>   Felix
>>
>>
>>>
>>> Co-developed-by: Donet Tom <donettom@linux.ibm.com>
>>> Signed-off-by: Donet Tom <donettom@linux.ibm.com>
>>> Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
>>> ---
>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 40 
>>> +++++++++++++++------------
>>>   1 file changed, 22 insertions(+), 18 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>> b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>> index 7dad8f30ff2a..329931b0edc6 100644
>>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>> @@ -1944,28 +1944,32 @@ static int kfd_create_vcrat_image_cpu(void
>>> *pcrat_image, size_t *size)
>>>           }
>>>             /* Fill in Subtype: Compute Unit */
>>> -        ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
>>> -            crat_table->num_domains,
>>> -            (struct crat_subtype_computeunit *)sub_type_hdr);
>>> -        if (ret < 0)
>>> -            return ret;
>>> -        crat_table->length += sub_type_hdr->length;
>>> -        crat_table->total_entries++;
>>> +        if (kfd_numa_node_to_apic_id(numa_node_id) != -1) {
>>> +            ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
>>> +                          crat_table->num_domains,
>>> +                          (struct crat_subtype_computeunit
>>> *)sub_type_hdr);
>>> +            if (ret < 0)
>>> +                return ret;
>>> +            crat_table->length += sub_type_hdr->length;
>>> +            crat_table->total_entries++;
>>>   -        sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
>>> -            sub_type_hdr->length);
>>> +            sub_type_hdr = (typeof(sub_type_hdr))((char
>>> *)sub_type_hdr +
>>> +                sub_type_hdr->length);
>>> +        }
>>>             /* Fill in Subtype: Memory */
>>> -        ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
>>> -            crat_table->num_domains,
>>> -            (struct crat_subtype_memory *)sub_type_hdr);
>>> -        if (ret < 0)
>>> -            return ret;
>>> -        crat_table->length += sub_type_hdr->length;
>>> -        crat_table->total_entries++;
>>> +        if (node_state(numa_node_id, N_MEMORY)) {
>>> +            ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
>>> +                            crat_table->num_domains,
>>> +                            (struct crat_subtype_memory
>>> *)sub_type_hdr);
>>> +            if (ret < 0)
>>> +                return ret;
>>> +            crat_table->length += sub_type_hdr->length;
>>> +            crat_table->total_entries++;
>>>   -        sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
>>> -            sub_type_hdr->length);
>>> +            sub_type_hdr = (typeof(sub_type_hdr))((char
>>> *)sub_type_hdr +
>>> +                sub_type_hdr->length);
>>> +        }
>>>             /* Fill in Subtype: IO Link */
>>>   #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)

Re: [PATCH v2 3/3] drm/amdkfd: Restrict CU and memory entries to resource nodes

[Dhruv Bhogaonkar]

On 08/10/26 02:31, Kuehling, Felix wrote:
> On 2026-10-07 07:14, Dhruv Bhogaonkar wrote:
>> [Some people who received this message don't often get email from 
>> dhruvb@linux.ibm.com. Learn why this is important at 
>> https://aka.ms/LearnAboutSenderIdentification ]
>>
>> On 07/10/26 01:30, Felix Kuehling wrote:
>>>
>>> On 2026-10-06 08:51, Dhruv B wrote:
>>>> In heterogeneous or non-uniform NUMA systems, a NUMA node may be
>>>> CPU-less, memory-less, or both CPU-less and memory-less.
>>>>
>>>> Previously, kfd_create_vcrat_image_cpu() assumed that every online 
>>>> NUMA
>>>> node contained both compute units and memory, attempting to create 
>>>> both
>>>> CU and memory CRAT entries unconditionally for every online node.
>>>>
>>>> Add explicit guards to:
>>>> 1. Only create Compute Unit CRAT entries if the NUMA node has valid
>>>>     CPU cores (kfd_numa_node_to_apic_id() != -1).
>>>> 2. Only create Memory CRAT entries if the NUMA node actually contains
>>>>     memory (node_state(numa_node_id, N_MEMORY)).
>>>>
>>>> This avoids emitting invalid or empty CRAT subtype entries for 
>>>> CPU-less,
>>>> memory-less, or both CPU-less and memory-less NUMA nodes in virtual 
>>>> CRAT
>>>> generation.
>>>
>>> @David Yat Sin, I think we currently support CPU-less and memory-less
>>> NUMA domains already. I know we have CPU-less NUMA domains on MI300A
>>> in NPS4/TPX mode. And I remember we've fixed bugs on NUMA CPU systems
>>> with only one DIMM populated, which would result in a memory-less NUMA
>>> domain.
>>>
>>> Therefore, I'm not sure whether this patch is really needed and I'd be
>>> concerned about introducing a regression. Is ROCr able to handle the
>>> changes being proposed here? David, could you take a look at this and
>>> maybe even test it, or find someone who could help test it on the
>>> system configurations I described?
>>
>> Thank you for the feedback, Felix. Apologies for the bug in v1 —
>> that was an oversight on our part.
>>
>> From the current code, the kfd_numa_node_to_apic_id() == -1
>> continue causes the entire loop body to be skipped
>> for a CPU-less NUMA node:
>>
>>   for_each_online_node(numa_node_id) {
>>           if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
>>                   continue;
>>
>>           ret = kfd_fill_cu_for_cpu(...);
>>           ...
>>           ret = kfd_fill_mem_info_for_cpu(...);
>>   }
>>
>> When the loop body is skipped, neither a CU entry nor a memory entry is
>> generated for that node, and no KFD topology device appears to be
>> registered for it from this code path. From our reading, it appears that
>> CPU-less nodes are effectively absent from the generated VCRAT topology
>> rather than being represented with an empty or placeholder entry.
>
> OK. I'm a bit surprised about that, because MI300A in NPS4 mode should 
> have three CPU-less NUMA nodes per socket. Maybe we never depended on 
> them being represented as CPU-less NUMA nodes in the KFD topology. 
> Then this change should be an improvement because it removes some 
> ambiguity between NUMA node numbers and KFD proximity domains.
>
>
>>
>>>
>>> @Dhruv B, how much testing have you done on this patch series. Given
>>> that the bug in patch 2 slipped through in v1, I'm worried that it's
>>> not very thorough.
>>
>> Regarding testing, on the userspace side we ran the RCCL unit tests.
>>
>> On the kernel side, we verified that the IO links are created correctly
>> for both topologies by inspecting the KFD topology sysfs entries after
>> driver load.
>>
>> Here are the RCCL unit test results for both configurations, along with
>> the corresponding numactl and KFD topology sysfs output observed after
>> driver load.
>
> Sounds good. We still need to test MI300A to make sure the changes 
> don't confuse ROCr or amdsmi. Applications mostly shouldn't care 
> because they don't use KFD topology node numbers directly. I'm told 
> that most of the MI300A systems in production seem to use NPS1, so 
> this shouldn't affect them.


Thank you, Felix, for the feedback and clarification. We appreciate
you taking the time to review the series.

We will wait for the MI300A testing results and will be happy to make
any changes based on the findings.

Regards,
Dhruv B
>
>
> Regards,
>   Felix
>
>
>>
>> ---
>> Test environment — contiguous NUMA (4 nodes, 0-3)
>> --------------------------------------------------
>>
>> numactl reports 4 contiguous nodes (0-3), all with CPUs and memory:
>>
>>   $ numactl -H
>>   available: 4 nodes (0-3)
>>   node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
>>                22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
>>                41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
>>   node 0 size: 881496 MB
>>   node 0 free: 875909 MB
>>   node 1 cpus: 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
>>                75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93
>>                94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109
>>                110 111
>>   node 1 size: 885640 MB
>>   node 1 free: 883815 MB
>>   node 2 cpus: 112 113 114 115 116 117 118 119 120 121 122 123 124 125
>>                126 127 128 129 130 131 132 133 134 135 136 137 138 139
>>                140 141 142 143 144 145 146 147 148 149 150 151 152 153
>>                154 155 156 157 158 159 160 161 162 163 164 165 166 167
>>   node 2 size: 885640 MB
>>   node 2 free: 874248 MB
>>   node 3 cpus: 168 169 170 171 172 173 174 175 176 177 178 179 180 181
>>                182 183 184 185 186 187 188 189 190 191 192 193 194 195
>>                196 197 198 199 200 201 202 203 204 205 206 207 208 209
>>                210 211 212 213 214 215 216 217 218 219 220 221 222 223
>>   node 3 size: 885394 MB
>>   node 3 free: 884033 MB
>>
>> The KFD topology sysfs reflects all 4 CPU nodes (0-3) plus 2 GPUs
>> (nodes 4 and 5), each CPU node with correct core and memory counts:
>>
>>   /sys/class/kfd/kfd/topology/nodes: 0  1  2 3  4  5
>>
>>   0: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
>>   1: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
>>   2: cpu_cores_count 56  mem_banks_count 1 io_links_count 2
>>   3: cpu_cores_count 56  mem_banks_count 1 io_links_count 0
>>   4: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>>   5: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>>
>> RCCL unit tests — 948 cases, all PASS:
>>
>> ---
>> Test environment — discontiguous NUMA (3 nodes: 0, 2-3)
>> --------------------------------------------------------
>>
>> numactl reports 3 nodes with a gap at node 1. Nodes 2 and 3 are
>> CPU-less and memory-less:
>>
>>   $ numactl -H
>>   available: 3 nodes (0,2-3)
>>   node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
>>                22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
>>   node 0 size: 519305 MB
>>   node 0 free: 504963 MB
>>   node 2 cpus: (none)
>>   node 2 size: 0 MB
>>   node 2 free: 0 MB
>>   node 3 cpus: (none)
>>   node 3 size: 0 MB
>>   node 3 free: 0 MB
>>
>> After this patch series the KFD topology sysfs correctly preserves the
>> NUMA node IDs as proximity domains. Node 1 appears as an empty
>> placeholder, nodes 2 and 3 have no cores or memory banks but carry the
>> GPU IO links, and both GPUs correctly reference node 2 as their attached
>> CPU-less domain:
>>
>>   /sys/class/kfd/kfd/topology/nodes: 0  1  2 3  4  5
>>
>>   0: cpu_cores_count 40  mem_banks_count 1 io_links_count 0
>>   1: cpu_cores_count  0  mem_banks_count 0 io_links_count 0 (absent,
>> placeholder)
>>   2: cpu_cores_count  0  mem_banks_count 0 io_links_count 2 (CPU-less,
>> GPUs attached)
>>   3: cpu_cores_count  0  mem_banks_count 0 io_links_count 0 (CPU-less)
>>   4: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>>   5: cpu_cores_count  0  simd_count 416  io_links_count 1  (GPU)
>>
>>   node 4 io_link: node_from 4  node_to 2 type 2  max_bandwidth 10000
>>   node 5 io_link: node_from 5  node_to 2 type 2  max_bandwidth 10000
>>
>> RCCL unit tests — 948 cases, all PASS:
>>
>> We fully understand your concern about regressions. We would
>> greatly appreciate it if someone with access to the systems you 
>> mentioned
>> could test this series, and we are very happy to make any adjustments
>> based on that feedback.
>>
>> Regards,
>> Dhruv
>>
>>>
>>> That said, the patch series looks good to me now, in theory.
>>>
>>> Regards,
>>>   Felix
>>>
>>>
>>>>
>>>> Co-developed-by: Donet Tom <donettom@linux.ibm.com>
>>>> Signed-off-by: Donet Tom <donettom@linux.ibm.com>
>>>> Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
>>>> ---
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 40 
>>>> +++++++++++++++------------
>>>>   1 file changed, 22 insertions(+), 18 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>>> b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>>> index 7dad8f30ff2a..329931b0edc6 100644
>>>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>>>> @@ -1944,28 +1944,32 @@ static int kfd_create_vcrat_image_cpu(void
>>>> *pcrat_image, size_t *size)
>>>>           }
>>>>             /* Fill in Subtype: Compute Unit */
>>>> -        ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
>>>> -            crat_table->num_domains,
>>>> -            (struct crat_subtype_computeunit *)sub_type_hdr);
>>>> -        if (ret < 0)
>>>> -            return ret;
>>>> -        crat_table->length += sub_type_hdr->length;
>>>> -        crat_table->total_entries++;
>>>> +        if (kfd_numa_node_to_apic_id(numa_node_id) != -1) {
>>>> +            ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
>>>> +                          crat_table->num_domains,
>>>> +                          (struct crat_subtype_computeunit
>>>> *)sub_type_hdr);
>>>> +            if (ret < 0)
>>>> +                return ret;
>>>> +            crat_table->length += sub_type_hdr->length;
>>>> +            crat_table->total_entries++;
>>>>   -        sub_type_hdr = (typeof(sub_type_hdr))((char 
>>>> *)sub_type_hdr +
>>>> -            sub_type_hdr->length);
>>>> +            sub_type_hdr = (typeof(sub_type_hdr))((char
>>>> *)sub_type_hdr +
>>>> +                sub_type_hdr->length);
>>>> +        }
>>>>             /* Fill in Subtype: Memory */
>>>> -        ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
>>>> -            crat_table->num_domains,
>>>> -            (struct crat_subtype_memory *)sub_type_hdr);
>>>> -        if (ret < 0)
>>>> -            return ret;
>>>> -        crat_table->length += sub_type_hdr->length;
>>>> -        crat_table->total_entries++;
>>>> +        if (node_state(numa_node_id, N_MEMORY)) {
>>>> +            ret = kfd_fill_mem_info_for_cpu(numa_node_id, 
>>>> &avail_size,
>>>> +                            crat_table->num_domains,
>>>> +                            (struct crat_subtype_memory
>>>> *)sub_type_hdr);
>>>> +            if (ret < 0)
>>>> +                return ret;
>>>> +            crat_table->length += sub_type_hdr->length;
>>>> +            crat_table->total_entries++;
>>>>   -        sub_type_hdr = (typeof(sub_type_hdr))((char 
>>>> *)sub_type_hdr +
>>>> -            sub_type_hdr->length);
>>>> +            sub_type_hdr = (typeof(sub_type_hdr))((char
>>>> *)sub_type_hdr +
>>>> +                sub_type_hdr->length);
>>>> +        }
>>>>             /* Fill in Subtype: IO Link */
>>>>   #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)