[PATCH 0/3] drm/amdkfd: Add support for discontiguous and CPU-less NUMA topologies

[PATCH 0/3] drm/amdkfd: Add support for discontiguous and CPU-less NUMA topologies

[Dhruv B]

This series fixes VCRAT generation in AMDKFD for systems that do not
provide a CRAT table and where the driver generates a Virtual CRAT
(VCRAT) for the CPU topology.

The current implementation uses for_each_online_node() to iterate over
NUMA nodes and assigns proximity domain IDs sequentially (0, 1, 2, ...)
regardless of the actual NUMA node IDs. It also assumes that every online
NUMA node contains both CPUs and memory, unconditionally emitting
Compute Unit and Memory CRAT subtypes for each node.

These assumptions break on systems with:

  Discontiguous (sparse) NUMA node IDs
  CPU-less NUMA nodes
  Memory-less NUMA nodes

For example:

  available: 3 nodes (0,2-3)

  node 0: CPUs present
  node 2: CPU-less
  node 3: CPU-less

In this case, the existing code iterates over nodes 0, 2 and 3 and
assigns proximity domains 0, 1 and 2 respectively, instead of 0, 2
and 3. The mismatch between NUMA node IDs and proximity domain IDs
results in an inconsistent KFD topology and causes topology parsing to
fail with:

  amdgpu 0038:12:00.0: Error parsing VCRAT
  kfd kfd: Error adding device to topology
  kfd kfd: Error initializing KFD node

To fix the proximity domain mismatch, the VCRAT generation loop is
changed to iterate over the complete NUMA node ID range up to the last
online node, incrementing num_domains for every node ID in the range.
This preserves a direct 1:1 mapping between NUMA node ID and proximity
domain ID, while skipping resource entry creation for offline or
unpopulated node IDs.

To fix the unconditional subtype emission, explicit guards are added so
that Compute Unit entries are only written for nodes that have valid CPU
cores (kfd_numa_node_to_apic_id() != -1), and Memory entries are only
written for nodes that actually contain memory (node_state(N_MEMORY)).
This prevents invalid or empty CRAT subtype entries from being emitted
for CPU-less, memory-less, or empty NUMA nodes.

This series consists of the following patches:

Patch 1 removes unused code in the CRAT initialization path.

Patch 2 fixes the proximity domain mismatch by iterating through the
complete NUMA node ID range and preserving a direct NUMA-node-to-
proximity-domain mapping.

Patch 3 restricts Compute Unit and Memory CRAT subtype entries to NUMA
nodes that actually have the corresponding resources.

Dhruv B (3):
  drm/amdkfd: Remove unused code in CRAT initialization
  drm/amdkfd: Add support for discontiguous NUMA node topology
  drm/amdkfd: Restrict CU and memory entries to resource nodes

 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 61 ++++++++++++-----------
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  2 +-
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 11 +---
 3 files changed, 35 insertions(+), 39 deletions(-)

-- 
2.54.0 (Apple Git-157)

[PATCH 1/3] drm/amdkfd: Remove unused code in CRAT initialization

[Dhruv B]

The 'size' argument in kfd_create_crat_image_virtual() is not used
by callers to retrieve the allocated CRAT image size. The callers
rely solely on the returned image pointer and inspect the parsed CRAT
table header directly for entry counts and lengths.

Additionally, in kfd_topology_init(), the 'kdev' pointer assignment
via list_first_entry() after parsing the CPU CRAT table is not being
used and can be removed.

Clean up the function signature of kfd_create_crat_image_virtual() by
dropping the unused 'size' parameter, manage the buffer size locally,
and remove the unused variable assignments in kfd_topology_init().

Co-developed-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 10 ++++------
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  2 +-
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 11 ++---------
 3 files changed, 7 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 6e0df685503d..99b0c7f825fe 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -2411,7 +2411,6 @@ static int kfd_create_vcrat_image_gpu(void *pcrat_image,
  *
  *	@crat_image: VCRAT image created because ACPI does not have a
  *		     CRAT for this device
- *	@size: [OUT] size of virtual crat_image
  *	@flags:	COMPUTE_UNIT_CPU - Create VCRAT for CPU device
  *		COMPUTE_UNIT_GPU - Create VCRAT for GPU
  *		(COMPUTE_UNIT_CPU | COMPUTE_UNIT_GPU) - Create VCRAT for APU
@@ -2421,7 +2420,7 @@ static int kfd_create_vcrat_image_gpu(void *pcrat_image,
  *
  *	Return 0 if successful else return -ve value
  */
-int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
+int kfd_create_crat_image_virtual(void **crat_image,
 				  int flags, struct kfd_node *kdev,
 				  uint32_t proximity_domain)
 {
@@ -2449,9 +2448,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
 		pcrat_image = kvmalloc(dyn_size, GFP_KERNEL);
 		if (!pcrat_image)
 			return -ENOMEM;
-		*size = dyn_size;
 		pr_debug("CRAT size is %ld", dyn_size);
-		ret = kfd_create_vcrat_image_cpu(pcrat_image, size);
+		ret = kfd_create_vcrat_image_cpu(pcrat_image, &dyn_size);
 		break;
 	case COMPUTE_UNIT_GPU:
 		if (!kdev)
@@ -2459,8 +2457,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
 		pcrat_image = kvmalloc(VCRAT_SIZE_FOR_GPU, GFP_KERNEL);
 		if (!pcrat_image)
 			return -ENOMEM;
-		*size = VCRAT_SIZE_FOR_GPU;
-		ret = kfd_create_vcrat_image_gpu(pcrat_image, size, kdev,
+		dyn_size = VCRAT_SIZE_FOR_GPU;
+		ret = kfd_create_vcrat_image_gpu(pcrat_image, &dyn_size, kdev,
 						 proximity_domain);
 		break;
 	case (COMPUTE_UNIT_CPU | COMPUTE_UNIT_GPU):
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
index a8ca7ecb6d27..418a3939eda3 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
@@ -313,7 +313,7 @@ int kfd_get_gpu_cache_info(struct kfd_node *kdev, struct kfd_gpu_cache_info **pc
 void kfd_destroy_crat_image(void *crat_image);
 int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
 			 uint32_t proximity_domain);
-int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
+int kfd_create_crat_image_virtual(void **crat_image,
 				  int flags, struct kfd_node *kdev,
 				  uint32_t proximity_domain);
 
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index 35b3abe57b80..b38d12f3da84 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -996,7 +996,6 @@ static void kfd_add_non_crat_information(struct kfd_topology_device *kdev)
 int kfd_topology_init(void)
 {
 	void *crat_image = NULL;
-	size_t image_size = 0;
 	int ret;
 	struct list_head temp_topology_device_list;
 	int cpu_only_node = 0;
@@ -1023,8 +1022,7 @@ int kfd_topology_init(void)
 	 */
 	proximity_domain = 0;
 
-	ret = kfd_create_crat_image_virtual(&crat_image, &image_size,
-					    COMPUTE_UNIT_CPU, NULL,
+	ret = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_CPU, NULL,
 					    proximity_domain);
 	cpu_only_node = 1;
 	if (ret) {
@@ -1040,9 +1038,6 @@ int kfd_topology_init(void)
 		goto err;
 	}
 
-	kdev = list_first_entry(&temp_topology_device_list,
-				struct kfd_topology_device, list);
-
 	down_write(&topology_lock);
 	kfd_topology_update_device_list(&temp_topology_device_list,
 					&topology_device_list);
@@ -1871,11 +1866,9 @@ static int kfd_topology_add_device_locked(struct kfd_node *gpu,
 	int proximity_domain = ++topology_crat_proximity_domain;
 	struct list_head temp_topology_device_list;
 	void *crat_image = NULL;
-	size_t image_size = 0;
 	int res;
 
-	res = kfd_create_crat_image_virtual(&crat_image, &image_size,
-					    COMPUTE_UNIT_GPU, gpu,
+	res = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_GPU, gpu,
 					    proximity_domain);
 	if (res) {
 		dev_err(gpu->adev->dev, "Error creating VCRAT\n");
-- 
2.54.0 (Apple Git-157)

[PATCH 2/3] drm/amdkfd: Add support for discontiguous NUMA node topology

[Dhruv B]

On systems with discontiguous NUMA nodes, the GPU driver fails to load
with the following errors:

[    5.609767] amdgpu 0038:12:00.0: Error parsing VCRAT
[    5.609781] kfd kfd: Error adding device to topology
[    5.609848] kfd kfd: Error initializing KFD node

In the current KFD code, proximity domains are created contiguously
when generating the VCRAT. With discontiguous NUMA node IDs, this
creates a mismatch between the proximity domain ID and the NUMA node ID.
For example, if NUMA nodes 0, 2 and 3 are online, the current code
assigns proximity domains 0, 1 and 2 to these nodes respectively,
instead of domains 0, 2 and 3.

The KFD topology uses the NUMA node and proximity domain information to
create topology devices representing the CPU and memory resources
associated with each NUMA node. This topology is exposed to userspace
through sysfs, where the NUMA node and proximity domain information are
used to identify the topology devices. The mismatch between the NUMA
node ID and proximity domain therefore results in an inconsistent KFD
topology and causes topology parsing to fail.

Make the proximity domain ID correspond directly to the NUMA node ID by
iterating through the complete NUMA node ID range up to the last online
node. Account for every node ID when assigning proximity domains, while
skipping resource entry creation for offline or unpopulated nodes.
This preserves the NUMA-node-to-proximity-domain mapping and allows the
KFD topology and its sysfs representation to correctly reflect
discontiguous NUMA topologies.

Co-developed-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 99b0c7f825fe..d4739db1c14c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1899,7 +1899,7 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 	acpi_status status;
 	struct crat_subtype_generic *sub_type_hdr;
 	int avail_size = *size;
-	int numa_node_id;
+	int numa_node_id, last_node;
 #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
 	uint32_t entries = 0;
 #endif
@@ -1935,9 +1935,12 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 	crat_table->num_domains = 0;
 
 	sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
+	last_node = find_last_bit(node_online_map.bits, MAX_NUMNODES);
 
-	for_each_online_node(numa_node_id) {
-		if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
+	for (numa_node_id = 0; numa_node_id <= last_node; numa_node_id++) {
+		crat_table->num_domains++;
+
+		if (!node_online(numa_node_id))
 			continue;
 
 		/* Fill in Subtype: Compute Unit */
@@ -1982,8 +1985,6 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 #else
 		pr_info("IO link not available for non x86 platforms\n");
 #endif
-
-		crat_table->num_domains++;
 	}
 
 	/* TODO: Add cache Subtype for CPU.
-- 
2.54.0 (Apple Git-157)

[PATCH 3/3] drm/amdkfd: Restrict CU and memory entries to resource nodes

[Dhruv B]

In heterogeneous or non-uniform NUMA systems, a NUMA node may be
CPU-less, memory-less, or both CPU-less and memory-less.

Previously, kfd_create_vcrat_image_cpu() assumed that every online NUMA
node contained both compute units and memory, attempting to create both
CU and memory CRAT entries unconditionally for every online node.

Add explicit guards to:
1. Only create Compute Unit CRAT entries if the NUMA node has valid
   CPU cores (kfd_numa_node_to_apic_id() != -1).
2. Only create Memory CRAT entries if the NUMA node actually contains
   memory (node_state(numa_node_id, N_MEMORY)).

This avoids emitting invalid or empty CRAT subtype entries for CPU-less,
memory-less, or both CPU-less and memory-less NUMA nodes in virtual CRAT
generation.

Co-developed-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 40 +++++++++++++++------------
 1 file changed, 22 insertions(+), 18 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index d4739db1c14c..489ddf68b751 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1944,28 +1944,32 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 			continue;
 
 		/* Fill in Subtype: Compute Unit */
-		ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
-			crat_table->num_domains,
-			(struct crat_subtype_computeunit *)sub_type_hdr);
-		if (ret < 0)
-			return ret;
-		crat_table->length += sub_type_hdr->length;
-		crat_table->total_entries++;
+		if (kfd_numa_node_to_apic_id(numa_node_id) != -1) {
+			ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
+						  crat_table->num_domains,
+						  (struct crat_subtype_computeunit *)sub_type_hdr);
+			if (ret < 0)
+				return ret;
+			crat_table->length += sub_type_hdr->length;
+			crat_table->total_entries++;
 
-		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
-			sub_type_hdr->length);
+			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
+				sub_type_hdr->length);
+		}
 
 		/* Fill in Subtype: Memory */
-		ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
-			crat_table->num_domains,
-			(struct crat_subtype_memory *)sub_type_hdr);
-		if (ret < 0)
-			return ret;
-		crat_table->length += sub_type_hdr->length;
-		crat_table->total_entries++;
+		if (node_state(numa_node_id, N_MEMORY)) {
+			ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
+							crat_table->num_domains,
+							(struct crat_subtype_memory *)sub_type_hdr);
+			if (ret < 0)
+				return ret;
+			crat_table->length += sub_type_hdr->length;
+			crat_table->total_entries++;
 
-		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
-			sub_type_hdr->length);
+			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
+				sub_type_hdr->length);
+		}
 
 		/* Fill in Subtype: IO Link */
 #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
-- 
2.54.0 (Apple Git-157)

Re: [PATCH 2/3] drm/amdkfd: Add support for discontiguous NUMA node topology

[Felix Kuehling]

On 2026-10-05 10:00, Dhruv B wrote:
> On systems with discontiguous NUMA nodes, the GPU driver fails to load
> with the following errors:
>
> [    5.609767] amdgpu 0038:12:00.0: Error parsing VCRAT
> [    5.609781] kfd kfd: Error adding device to topology
> [    5.609848] kfd kfd: Error initializing KFD node
>
> In the current KFD code, proximity domains are created contiguously
> when generating the VCRAT. With discontiguous NUMA node IDs, this
> creates a mismatch between the proximity domain ID and the NUMA node ID.
> For example, if NUMA nodes 0, 2 and 3 are online, the current code
> assigns proximity domains 0, 1 and 2 to these nodes respectively,
> instead of domains 0, 2 and 3.
>
> The KFD topology uses the NUMA node and proximity domain information to
> create topology devices representing the CPU and memory resources
> associated with each NUMA node. This topology is exposed to userspace
> through sysfs, where the NUMA node and proximity domain information are
> used to identify the topology devices. The mismatch between the NUMA
> node ID and proximity domain therefore results in an inconsistent KFD
> topology and causes topology parsing to fail.
>
> Make the proximity domain ID correspond directly to the NUMA node ID by
> iterating through the complete NUMA node ID range up to the last online
> node. Account for every node ID when assigning proximity domains, while
> skipping resource entry creation for offline or unpopulated nodes.
> This preserves the NUMA-node-to-proximity-domain mapping and allows the
> KFD topology and its sysfs representation to correctly reflect
> discontiguous NUMA topologies.
>
> Co-developed-by: Donet Tom <donettom@linux.ibm.com>
> Signed-off-by: Donet Tom <donettom@linux.ibm.com>
> Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 11 ++++++-----
>   1 file changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> index 99b0c7f825fe..d4739db1c14c 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> @@ -1899,7 +1899,7 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
>   	acpi_status status;
>   	struct crat_subtype_generic *sub_type_hdr;
>   	int avail_size = *size;
> -	int numa_node_id;
> +	int numa_node_id, last_node;
>   #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
>   	uint32_t entries = 0;
>   #endif
> @@ -1935,9 +1935,12 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
>   	crat_table->num_domains = 0;
>   
>   	sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
> +	last_node = find_last_bit(node_online_map.bits, MAX_NUMNODES);
>   
> -	for_each_online_node(numa_node_id) {
> -		if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
> +	for (numa_node_id = 0; numa_node_id <= last_node; numa_node_id++) {
> +		crat_table->num_domains++;

Doing this at the beginning of the loop body, doesn't that result in an 
incorrect proximity_domain being used in kfd_fill_cu_for_cpu? E.g. the 
first CPU will get proximity_domain 1 instead of 0.

Regards,
   Felix


> +
> +		if (!node_online(numa_node_id))
>   			continue;
>   
>   		/* Fill in Subtype: Compute Unit */
> @@ -1982,8 +1985,6 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
>   #else
>   		pr_info("IO link not available for non x86 platforms\n");
>   #endif
> -
> -		crat_table->num_domains++;
>   	}
>   
>   	/* TODO: Add cache Subtype for CPU.

Re: [PATCH 1/3] drm/amdkfd: Remove unused code in CRAT initialization

[Felix Kuehling]

On 2026-10-05 10:00, Dhruv B wrote:
> The 'size' argument in kfd_create_crat_image_virtual() is not used
> by callers to retrieve the allocated CRAT image size. The callers
> rely solely on the returned image pointer and inspect the parsed CRAT
> table header directly for entry counts and lengths.
>
> Additionally, in kfd_topology_init(), the 'kdev' pointer assignment
> via list_first_entry() after parsing the CPU CRAT table is not being
> used and can be removed.
>
> Clean up the function signature of kfd_create_crat_image_virtual() by
> dropping the unused 'size' parameter, manage the buffer size locally,
> and remove the unused variable assignments in kfd_topology_init().
>
> Co-developed-by: Donet Tom <donettom@linux.ibm.com>
> Signed-off-by: Donet Tom <donettom@linux.ibm.com>
> Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>

This patch is

Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>

I'll apply it with the rest of the series once that's ready.

Regards,
   Felix


> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 10 ++++------
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  2 +-
>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 11 ++---------
>   3 files changed, 7 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> index 6e0df685503d..99b0c7f825fe 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> @@ -2411,7 +2411,6 @@ static int kfd_create_vcrat_image_gpu(void *pcrat_image,
>    *
>    *	@crat_image: VCRAT image created because ACPI does not have a
>    *		     CRAT for this device
> - *	@size: [OUT] size of virtual crat_image
>    *	@flags:	COMPUTE_UNIT_CPU - Create VCRAT for CPU device
>    *		COMPUTE_UNIT_GPU - Create VCRAT for GPU
>    *		(COMPUTE_UNIT_CPU | COMPUTE_UNIT_GPU) - Create VCRAT for APU
> @@ -2421,7 +2420,7 @@ static int kfd_create_vcrat_image_gpu(void *pcrat_image,
>    *
>    *	Return 0 if successful else return -ve value
>    */
> -int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
> +int kfd_create_crat_image_virtual(void **crat_image,
>   				  int flags, struct kfd_node *kdev,
>   				  uint32_t proximity_domain)
>   {
> @@ -2449,9 +2448,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
>   		pcrat_image = kvmalloc(dyn_size, GFP_KERNEL);
>   		if (!pcrat_image)
>   			return -ENOMEM;
> -		*size = dyn_size;
>   		pr_debug("CRAT size is %ld", dyn_size);
> -		ret = kfd_create_vcrat_image_cpu(pcrat_image, size);
> +		ret = kfd_create_vcrat_image_cpu(pcrat_image, &dyn_size);
>   		break;
>   	case COMPUTE_UNIT_GPU:
>   		if (!kdev)
> @@ -2459,8 +2457,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
>   		pcrat_image = kvmalloc(VCRAT_SIZE_FOR_GPU, GFP_KERNEL);
>   		if (!pcrat_image)
>   			return -ENOMEM;
> -		*size = VCRAT_SIZE_FOR_GPU;
> -		ret = kfd_create_vcrat_image_gpu(pcrat_image, size, kdev,
> +		dyn_size = VCRAT_SIZE_FOR_GPU;
> +		ret = kfd_create_vcrat_image_gpu(pcrat_image, &dyn_size, kdev,
>   						 proximity_domain);
>   		break;
>   	case (COMPUTE_UNIT_CPU | COMPUTE_UNIT_GPU):
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
> index a8ca7ecb6d27..418a3939eda3 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
> @@ -313,7 +313,7 @@ int kfd_get_gpu_cache_info(struct kfd_node *kdev, struct kfd_gpu_cache_info **pc
>   void kfd_destroy_crat_image(void *crat_image);
>   int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
>   			 uint32_t proximity_domain);
> -int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
> +int kfd_create_crat_image_virtual(void **crat_image,
>   				  int flags, struct kfd_node *kdev,
>   				  uint32_t proximity_domain);
>   
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
> index 35b3abe57b80..b38d12f3da84 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
> @@ -996,7 +996,6 @@ static void kfd_add_non_crat_information(struct kfd_topology_device *kdev)
>   int kfd_topology_init(void)
>   {
>   	void *crat_image = NULL;
> -	size_t image_size = 0;
>   	int ret;
>   	struct list_head temp_topology_device_list;
>   	int cpu_only_node = 0;
> @@ -1023,8 +1022,7 @@ int kfd_topology_init(void)
>   	 */
>   	proximity_domain = 0;
>   
> -	ret = kfd_create_crat_image_virtual(&crat_image, &image_size,
> -					    COMPUTE_UNIT_CPU, NULL,
> +	ret = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_CPU, NULL,
>   					    proximity_domain);
>   	cpu_only_node = 1;
>   	if (ret) {
> @@ -1040,9 +1038,6 @@ int kfd_topology_init(void)
>   		goto err;
>   	}
>   
> -	kdev = list_first_entry(&temp_topology_device_list,
> -				struct kfd_topology_device, list);
> -
>   	down_write(&topology_lock);
>   	kfd_topology_update_device_list(&temp_topology_device_list,
>   					&topology_device_list);
> @@ -1871,11 +1866,9 @@ static int kfd_topology_add_device_locked(struct kfd_node *gpu,
>   	int proximity_domain = ++topology_crat_proximity_domain;
>   	struct list_head temp_topology_device_list;
>   	void *crat_image = NULL;
> -	size_t image_size = 0;
>   	int res;
>   
> -	res = kfd_create_crat_image_virtual(&crat_image, &image_size,
> -					    COMPUTE_UNIT_GPU, gpu,
> +	res = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_GPU, gpu,
>   					    proximity_domain);
>   	if (res) {
>   		dev_err(gpu->adev->dev, "Error creating VCRAT\n");

Re: [PATCH 2/3] drm/amdkfd: Add support for discontiguous NUMA node topology

[Dhruv Bhogaonkar]

On 06/10/26 01:48, Felix Kuehling wrote:
> On 2026-10-05 10:00, Dhruv B wrote:
>> On systems with discontiguous NUMA nodes, the GPU driver fails to load
>> with the following errors:
>>
>> [    5.609767] amdgpu 0038:12:00.0: Error parsing VCRAT
>> [    5.609781] kfd kfd: Error adding device to topology
>> [    5.609848] kfd kfd: Error initializing KFD node
>>
>> In the current KFD code, proximity domains are created contiguously
>> when generating the VCRAT. With discontiguous NUMA node IDs, this
>> creates a mismatch between the proximity domain ID and the NUMA node ID.
>> For example, if NUMA nodes 0, 2 and 3 are online, the current code
>> assigns proximity domains 0, 1 and 2 to these nodes respectively,
>> instead of domains 0, 2 and 3.
>>
>> The KFD topology uses the NUMA node and proximity domain information to
>> create topology devices representing the CPU and memory resources
>> associated with each NUMA node. This topology is exposed to userspace
>> through sysfs, where the NUMA node and proximity domain information are
>> used to identify the topology devices. The mismatch between the NUMA
>> node ID and proximity domain therefore results in an inconsistent KFD
>> topology and causes topology parsing to fail.
>>
>> Make the proximity domain ID correspond directly to the NUMA node ID by
>> iterating through the complete NUMA node ID range up to the last online
>> node. Account for every node ID when assigning proximity domains, while
>> skipping resource entry creation for offline or unpopulated nodes.
>> This preserves the NUMA-node-to-proximity-domain mapping and allows the
>> KFD topology and its sysfs representation to correctly reflect
>> discontiguous NUMA topologies.
>>
>> Co-developed-by: Donet Tom <donettom@linux.ibm.com>
>> Signed-off-by: Donet Tom <donettom@linux.ibm.com>
>> Signed-off-by: Dhruv B <dhruv.b@linux.ibm.com>
>> ---
>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 11 ++++++-----
>>   1 file changed, 6 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c 
>> b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>> index 99b0c7f825fe..d4739db1c14c 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
>> @@ -1899,7 +1899,7 @@ static int kfd_create_vcrat_image_cpu(void 
>> *pcrat_image, size_t *size)
>>       acpi_status status;
>>       struct crat_subtype_generic *sub_type_hdr;
>>       int avail_size = *size;
>> -    int numa_node_id;
>> +    int numa_node_id, last_node;
>>   #if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
>>       uint32_t entries = 0;
>>   #endif
>> @@ -1935,9 +1935,12 @@ static int kfd_create_vcrat_image_cpu(void 
>> *pcrat_image, size_t *size)
>>       crat_table->num_domains = 0;
>>         sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
>> +    last_node = find_last_bit(node_online_map.bits, MAX_NUMNODES);
>>   -    for_each_online_node(numa_node_id) {
>> -        if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
>> +    for (numa_node_id = 0; numa_node_id <= last_node; numa_node_id++) {
>> +        crat_table->num_domains++;
>
> Doing this at the beginning of the loop body, doesn't that result in 
> an incorrect proximity_domain being used in kfd_fill_cu_for_cpu? E.g. 
> the first CPU will get proximity_domain 1 instead of 0.


Thanks for pointing this out, Felix. Apologies, we missed this when 
making the change.

I'll correct this and will post a new version with the corrected changes.

Thanks,
Dhruv B


>
>
> Regards,
>   Felix
>
>
>> +
>> +        if (!node_online(numa_node_id))
>>               continue;
>>             /* Fill in Subtype: Compute Unit */
>> @@ -1982,8 +1985,6 @@ static int kfd_create_vcrat_image_cpu(void 
>> *pcrat_image, size_t *size)
>>   #else
>>           pr_info("IO link not available for non x86 platforms\n");
>>   #endif
>> -
>> -        crat_table->num_domains++;
>>       }
>>         /* TODO: Add cache Subtype for CPU.