[PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

[PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

[Donet Tom]

This series fixes topology device creation and proximity domain mappings in
AMDKFD when a system does not provide a CRAT table and the driver generates a
Virtual CRAT (VCRAT).

The current implementation assumes that CPU NUMA node IDs are contiguous and
that every NUMA node contains CPUs. During VCRAT generation, CPU topology
entries and proximity domains are created only for NUMA nodes that have CPUs.
GPU proximity domains are then allocated immediately after the CPU proximity
domains, and the GPU I/O link (proximity_domain_to) is initialized using the
NUMA node ID to which the GPU is attached, implicitly assuming that NUMA node
IDs and proximity domains have a one-to-one mapping.

These assumptions break on systems with:

Sparse (non-contiguous) NUMA node IDs
CPU-less NUMA nodes
CPU-less and memory-less NUMA nodes

For example:

available: 3 nodes (0,2-3)

node 0: CPUs present
node 2: CPU-less
node 3: CPU-less

In this case, the driver creates a CPU topology device only for node 0 and
assigns a single CPU proximity domain (0). However, the GPU VCRAT still
references the NUMA node ID to which the GPU is attached (for example, node 3)
in the proximity_domain_to field. Since no corresponding CPU proximity domain
exists for node 3, the parser cannot find a matching proximity domain during
VCRAT parsing, causing topology initialization to fail.

The failure is observed as:

amdgpu: Virtual CRAT table created for GPU
amdgpu: Error parsing VCRAT
kfd: amdgpu: Error adding device to topology
kfd: amdgpu: Error initializing KFD node


Since every online NUMA node is a valid topology object and can contain
CPUs, memory, I/O links, or any combination of these, topology devices
and proximity domains should be created for every online NUMA node
rather than only for NUMA nodes that contain CPUs.

To address this, this series introduces a new VCRAT subtype that
records the mapping between the NUMA node ID and the generated VCRAT
proximity domain. When topology devices are created, this information
is stored in the corresponding topology device, allowing the driver to
translate a NUMA node ID into its associated proximity domain whenever
required.

Returning to the previous example, the system contains three online
NUMA nodes, so three CPU topology devices and three proximity domains
are created, even though only one NUMA node contains CPUs. The NUMA
node ID is stored in each topology device together with its generated
proximity domain.

Later, when the GPU VCRAT is generated, the driver only knows the NUMA
node ID to which the GPU is attached (for example, node 3). Instead of
assuming that the NUMA node ID is equal to the proximity domain, the
driver walks the existing topology devices to locate the corresponding
NUMA node and retrieves its generated proximity domain. In this
example, NUMA node 3 maps to proximity domain 2, so
proximity_domain_to is populated with the correct value.

Since the GPU I/O link now references a valid proximity domain, VCRAT
parsing completes successfully and topology initialization proceeds
without errors on systems with sparse NUMA node IDs, CPU-less NUMA
nodes, and CPU-less/memory-less NUMA nodes.

This series consists of the following patches:

Patch 1 removes an unused argument from
kfd_create_crat_image_virtual() as a preparatory cleanup.

Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
NUMA node ID and its corresponding proximity domain.

Patch 3 populates the NUMA affinity entries during VCRAT generation
for every online NUMA node.

Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
NUMA node ID and proximity domain in the corresponding topology
device.

Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
GPU's NUMA node ID to the corresponding CPU proximity domain before
programming proximity_domain_to.

Patch 6 creates proximity domains and VCRAT entries for all online
NUMA nodes, including CPU-less and memory-less nodes.

Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
devices, exposing the associated NUMA node through the topology sysfs
interface.

Please note that the changes in this series are on a best effort basis from our
end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the
HW & SW stack) to kindly help with the review and provide feedback / comments on
these patches

Donet Tom (7):
  drm/amdgpu: Remove unused argument from kfd_create_crat_image_virtual
  drm/amdgpu: Add VCRAT NUMA affinity entry
  drm/amdgpu: Populate NUMA affinity entries in VCRAT
  drm/amdgpu: Parse NUMA affinity entries from VCRAT
  drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
  drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
    and memory-less NUMA nodes
  drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry

 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 +++++++++++++++-------
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
 drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
 4 files changed, 168 insertions(+), 56 deletions(-)

-- 
2.54.0

[PATCH 1/7] drm/amdgpu: Remove unused argument from kfd_create_crat_image_virtual

[Donet Tom]

The size argument is neither passed to nor returned from
kfd_create_crat_image_virtual(). Since it is unused, remove it from the
function and update the callers accordingly.

Signed-off-by: Donet Tom <donettom@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 11 +++++------
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  4 ++--
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 12 ++++--------
 3 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 2a239f45fc24..3190a4d6acc6 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -2412,8 +2412,8 @@ static int kfd_create_vcrat_image_gpu(void *pcrat_image,
  *
  *	Return 0 if successful else return -ve value
  */
-int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
-				  int flags, struct kfd_node *kdev,
+int kfd_create_crat_image_virtual(void **crat_image, int flags,
+				  struct kfd_node *kdev,
 				  uint32_t proximity_domain)
 {
 	void *pcrat_image = NULL;
@@ -2440,9 +2440,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
 		pcrat_image = kvmalloc(dyn_size, GFP_KERNEL);
 		if (!pcrat_image)
 			return -ENOMEM;
-		*size = dyn_size;
 		pr_debug("CRAT size is %ld", dyn_size);
-		ret = kfd_create_vcrat_image_cpu(pcrat_image, size);
+		ret = kfd_create_vcrat_image_cpu(pcrat_image, &dyn_size);
 		break;
 	case COMPUTE_UNIT_GPU:
 		if (!kdev)
@@ -2450,8 +2449,8 @@ int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
 		pcrat_image = kvmalloc(VCRAT_SIZE_FOR_GPU, GFP_KERNEL);
 		if (!pcrat_image)
 			return -ENOMEM;
-		*size = VCRAT_SIZE_FOR_GPU;
-		ret = kfd_create_vcrat_image_gpu(pcrat_image, size, kdev,
+		dyn_size = VCRAT_SIZE_FOR_GPU;
+		ret = kfd_create_vcrat_image_gpu(pcrat_image, &dyn_size, kdev,
 						 proximity_domain);
 		break;
 	case (COMPUTE_UNIT_CPU | COMPUTE_UNIT_GPU):
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
index a8ca7ecb6d27..252fa94aaab9 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
@@ -313,8 +313,8 @@ int kfd_get_gpu_cache_info(struct kfd_node *kdev, struct kfd_gpu_cache_info **pc
 void kfd_destroy_crat_image(void *crat_image);
 int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
 			 uint32_t proximity_domain);
-int kfd_create_crat_image_virtual(void **crat_image, size_t *size,
-				  int flags, struct kfd_node *kdev,
+int kfd_create_crat_image_virtual(void **crat_image, int flags,
+				  struct kfd_node *kdev,
 				  uint32_t proximity_domain);
 
 #endif /* KFD_CRAT_H_INCLUDED */
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index 00517c3d0e6a..f4b7cb44daff 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -996,7 +996,6 @@ static void kfd_add_non_crat_information(struct kfd_topology_device *kdev)
 int kfd_topology_init(void)
 {
 	void *crat_image = NULL;
-	size_t image_size = 0;
 	int ret;
 	struct list_head temp_topology_device_list;
 	int cpu_only_node = 0;
@@ -1023,9 +1022,8 @@ int kfd_topology_init(void)
 	 */
 	proximity_domain = 0;
 
-	ret = kfd_create_crat_image_virtual(&crat_image, &image_size,
-					    COMPUTE_UNIT_CPU, NULL,
-					    proximity_domain);
+	ret = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_CPU,
+					    NULL, proximity_domain);
 	cpu_only_node = 1;
 	if (ret) {
 		pr_err("Error creating VCRAT table for CPU\n");
@@ -1871,12 +1869,10 @@ static int kfd_topology_add_device_locked(struct kfd_node *gpu,
 	int proximity_domain = ++topology_crat_proximity_domain;
 	struct list_head temp_topology_device_list;
 	void *crat_image = NULL;
-	size_t image_size = 0;
 	int res;
 
-	res = kfd_create_crat_image_virtual(&crat_image, &image_size,
-					    COMPUTE_UNIT_GPU, gpu,
-					    proximity_domain);
+	res = kfd_create_crat_image_virtual(&crat_image, COMPUTE_UNIT_GPU,
+					    gpu, proximity_domain);
 	if (res) {
 		dev_err(gpu->adev->dev, "Error creating VCRAT\n");
 		topology_crat_proximity_domain--;
-- 
2.54.0

[PATCH 2/7] drm/amdgpu: Add VCRAT NUMA affinity entry

[Donet Tom]

Add a new VCRAT subtype to describe NUMA affinity information.
The new entry stores the NUMA node ID and proximity domain.

Signed-off-by: Donet Tom <donettom@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
index 252fa94aaab9..934cbb54b0ba 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.h
@@ -75,7 +75,8 @@ struct crat_header {
 #define CRAT_SUBTYPE_TLB_AFFINITY		3
 #define CRAT_SUBTYPE_CCOMPUTE_AFFINITY		4
 #define CRAT_SUBTYPE_IOLINK_AFFINITY		5
-#define CRAT_SUBTYPE_MAX			6
+#define CRAT_SUBTYPE_NUMA_AFFINITY		6
+#define CRAT_SUBTYPE_MAX			7
 
 /*
  * Do not change the value of CRAT_SIBLINGMAP_SIZE from 32
@@ -280,6 +281,18 @@ struct crat_subtype_iolink {
 	uint8_t		weight_xgmi;
 };
 
+/*
+ * HSA NUMA sub-type header
+ */
+struct crat_subtype_numa {
+	uint8_t		type;
+	uint8_t		length;
+	uint16_t	reserved;
+	uint32_t	flags;
+	uint32_t	numa_node;
+	uint32_t	proximity_domain;
+};
+
 /*
  * HSA generic sub-type header
  */
-- 
2.54.0

[PATCH 3/7] drm/amdgpu: Populate NUMA affinity entries in VCRAT

[Donet Tom]

Populate the new VCRAT NUMA affinity subtype with the NUMA node ID and
the corresponding proximity domain of each NUMA node.

Signed-off-by: Donet Tom <donettom@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 32 +++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 3190a4d6acc6..5613eb69423d 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1877,6 +1877,26 @@ static int kfd_fill_iolink_info_for_cpu(int numa_node_id, int *avail_size,
 }
 #endif
 
+static int kfd_fill_numa_info_for_cpu(int *avail_size,
+			struct crat_subtype_numa *sub_type_hdr,
+			int numa_node, int proximity_domain)
+{
+	*avail_size -= sizeof(struct crat_subtype_numa);
+	if (*avail_size < 0)
+		return -ENOMEM;
+
+	memset(sub_type_hdr, 0, sizeof(struct crat_subtype_numa));
+
+	/* Fill in subtype header data */
+	sub_type_hdr->type = CRAT_SUBTYPE_NUMA_AFFINITY;
+	sub_type_hdr->length = sizeof(struct crat_subtype_numa);
+	sub_type_hdr->flags = CRAT_SUBTYPE_FLAGS_ENABLED;
+	sub_type_hdr->numa_node = numa_node;
+	sub_type_hdr->proximity_domain = proximity_domain;
+
+	return 0;
+}
+
 /* kfd_create_vcrat_image_cpu - Create Virtual CRAT for CPU
  *
  *	@pcrat_image: Fill in VCRAT for CPU
@@ -1974,6 +1994,17 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 		pr_info("IO link not available for non x86 platforms\n");
 #endif
 
+		ret = kfd_fill_numa_info_for_cpu(&avail_size,
+				(struct crat_subtype_numa *)sub_type_hdr,
+				numa_node_id, crat_table->num_domains);
+		if (ret < 0)
+			return ret;
+		crat_table->length += sub_type_hdr->length;
+		crat_table->total_entries++;
+
+		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
+				sub_type_hdr->length);
+
 		crat_table->num_domains++;
 	}
 
@@ -2436,6 +2467,7 @@ int kfd_create_crat_image_virtual(void **crat_image, int flags,
 		dyn_size = sizeof(struct crat_header) +
 			num_nodes * (sizeof(struct crat_subtype_computeunit) +
 			sizeof(struct crat_subtype_memory) +
+			sizeof(struct crat_subtype_numa) +
 			(num_nodes - 1) * sizeof(struct crat_subtype_iolink));
 		pcrat_image = kvmalloc(dyn_size, GFP_KERNEL);
 		if (!pcrat_image)
-- 
2.54.0

[PATCH 4/7] drm/amdgpu: Parse NUMA affinity entries from VCRAT

[Donet Tom]

Parse the NUMA affinity entries from the VCRAT and populate the
corresponding NUMA node information in the topology device.

Signed-off-by: Donet Tom <donettom@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 23 +++++++++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/kfd_topology.h |  1 +
 2 files changed, 24 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 5613eb69423d..079bce6d904d 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1299,6 +1299,23 @@ static int kfd_parse_subtype_iolink(struct crat_subtype_iolink *iolink,
 	return 0;
 }
 
+static int kfd_parse_subtype_numa(struct crat_subtype_numa *numa,
+				struct list_head *device_list)
+{
+	struct kfd_topology_device *dev;
+
+	pr_debug("Found NUMA entry in CRAT table with proximity_domain=%u numa node=%u\n",
+			numa->proximity_domain, numa->numa_node);
+	list_for_each_entry(dev, device_list, list) {
+		if (numa->proximity_domain == dev->proximity_domain) {
+			dev->numa_node = numa->numa_node;
+			break;
+		}
+	}
+
+	return 0;
+}
+
 /* kfd_parse_subtype - parse subtypes and attach it to correct topology device
  * present in the device_list
  *	@sub_type_hdr - subtype section of crat_image
@@ -1311,6 +1328,7 @@ static int kfd_parse_subtype(struct crat_subtype_generic *sub_type_hdr,
 	struct crat_subtype_memory *mem;
 	struct crat_subtype_cache *cache;
 	struct crat_subtype_iolink *iolink;
+	struct crat_subtype_numa *numa;
 	int ret = 0;
 
 	switch (sub_type_hdr->type) {
@@ -1342,6 +1360,10 @@ static int kfd_parse_subtype(struct crat_subtype_generic *sub_type_hdr,
 		iolink = (struct crat_subtype_iolink *)sub_type_hdr;
 		ret = kfd_parse_subtype_iolink(iolink, device_list);
 		break;
+	case CRAT_SUBTYPE_NUMA_AFFINITY:
+		numa = (struct crat_subtype_numa *)sub_type_hdr;
+		ret = kfd_parse_subtype_numa(numa, device_list);
+		break;
 	default:
 		pr_warn("Unknown subtype %d in CRAT\n",
 				sub_type_hdr->type);
@@ -1389,6 +1411,7 @@ int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
 		if (!top_dev)
 			break;
 		top_dev->proximity_domain = proximity_domain++;
+		top_dev->numa_node = -1;
 	}
 
 	if (!top_dev) {
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
index ad63ba67b577..f5e00f5df922 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
@@ -144,6 +144,7 @@ struct kfd_perf_properties {
 struct kfd_topology_device {
 	struct list_head		list;
 	uint32_t			gpu_id;
+	uint32_t			numa_node;
 	uint32_t			proximity_domain;
 	struct kfd_node_properties	node_props;
 	struct list_head		mem_props;
-- 
2.54.0

[PATCH 5/7] drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes

[Donet Tom]

The VCRAT CPU builder, kfd_create_vcrat_image_cpu(), assigns proximity
domains sequentially while iterating over for_each_online_node(). As a
result, the first online node is assigned proximity domain 0, the second
online node is assigned proximity domain 1, and so on.

The GPU VCRAT builder creates GPU proximity domains after the CPU
proximity domains and programs the GPU-to-CPU affinity
(proximity_domain_to) using the NUMA node ID of the node to which the
GPU is attached. This is only correct when NUMA node IDs are contiguous,
because the CPU proximity domains happen to match the NUMA node IDs.

On systems with discontiguous NUMA node IDs, the CPU proximity domains no
longer correspond to the NUMA node IDs. For example:

available: 3 nodes (0,2,4)

Node IDs:           0    2    4
CPU proximity:      0    1    2
GPU proximity:                3

In this example, a GPU attached to NUMA node 4 is assigned GPU proximity
domain 3. However, its proximity_domain_to field is programmed with the
NUMA node ID (4) instead of the corresponding CPU proximity domain (2).

During VCRAT parsing, no CPU proximity domain with value 4 exists, so the
GPU affinity cannot be resolved and topology initialization fails:

amdgpu: Virtual CRAT table created for GPU
amdgpu: Error parsing VCRAT
kfd: amdgpu: Error adding device to topology
kfd: amdgpu: Error initializing KFD node

Fix this by translating the GPU's NUMA node ID to the corresponding CPU
proximity domain assigned during GPU VCRAT creation before programming
proximity_domain_to. This ensures that GPU affinity entries always
reference a valid CPU proximity domain, allowing VCRAT parsing and KFD
topology initialization to succeed on systems with discontiguous NUMA node
IDs.

Signed-off-by: Donet Tom <donettom@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 10 +++++++---
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 22 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/kfd_topology.h |  2 ++
 3 files changed, 31 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 079bce6d904d..b38208965fc8 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1267,6 +1267,7 @@ static int kfd_parse_subtype_iolink(struct crat_subtype_iolink *iolink,
 					iolink->recommended_transfer_size;
 
 			dev->node_props.io_links_count++;
+			dev->numa_node = kfd_proximity_domain_to_numa_node(id_to);
 			list_add_tail(&props->list, &dev->io_link_props);
 			break;
 		}
@@ -1888,8 +1889,10 @@ static int kfd_fill_iolink_info_for_cpu(int numa_node_id, int *avail_size,
 		sub_type_hdr->flags = CRAT_SUBTYPE_FLAGS_ENABLED;
 
 		/* Fill in IO link data */
-		sub_type_hdr->proximity_domain_from = numa_node_id;
-		sub_type_hdr->proximity_domain_to = nid;
+		sub_type_hdr->proximity_domain_from =
+			kfd_numa_node_to_proximity_domain(numa_node_id);
+		sub_type_hdr->proximity_domain_to =
+			kfd_numa_node_to_proximity_domain(nid);
 		sub_type_hdr->io_interface_type = link_type;
 
 		(*num_entries)++;
@@ -2237,7 +2240,8 @@ static int kfd_fill_gpu_direct_io_link_to_cpu(int *avail_size,
 	if (kdev->adev->pdev->dev.numa_node == NUMA_NO_NODE)
 		sub_type_hdr->proximity_domain_to = 0;
 	else
-		sub_type_hdr->proximity_domain_to = kdev->adev->pdev->dev.numa_node;
+		sub_type_hdr->proximity_domain_to =
+		    kfd_numa_node_to_proximity_domain(kdev->adev->pdev->dev.numa_node);
 #else
 	sub_type_hdr->proximity_domain_to = 0;
 #endif
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index f4b7cb44daff..d3269f0ffec3 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -2463,3 +2463,25 @@ void kfd_update_svm_support_properties(struct amdgpu_device *adev)
 	}
 	up_write(&topology_lock);
 }
+
+int kfd_proximity_domain_to_numa_node(uint32_t proximity_domain)
+{
+	struct kfd_topology_device *dev;
+
+	list_for_each_entry(dev, &topology_device_list, list) {
+		if (dev->proximity_domain == proximity_domain)
+			return dev->numa_node;
+	}
+	return -1;
+}
+
+int kfd_numa_node_to_proximity_domain(int target_node)
+{
+	struct kfd_topology_device *dev;
+
+	list_for_each_entry(dev, &topology_device_list, list) {
+		if (dev->numa_node == target_node)
+			return dev->proximity_domain;
+	}
+	return -1;
+}
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
index f5e00f5df922..312802754096 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
@@ -201,6 +201,8 @@ struct dmi_mem_device {
 struct kfd_topology_device *kfd_create_topology_device(
 		struct list_head *device_list);
 void kfd_release_topology_device_list(struct list_head *device_list);
+int kfd_numa_node_to_proximity_domain(int target_node);
+int kfd_proximity_domain_to_numa_node(uint32_t proximity_domain);
 
 #if IS_ENABLED(CONFIG_HSA_AMD)
 void kfd_update_svm_support_properties(struct amdgpu_device *adev);
-- 
2.54.0

[PATCH 6/7] drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less and memory-less NUMA nodes

[Donet Tom]

The current implementation creates proximity domains and the associated
VCRAT entries only for NUMA nodes that contain CPUs. However, Linux
systems can have CPU-less, memory-less, or CPU-less and memory-less
NUMA nodes.

When proximity domains and VCRAT entries are not created for these
NUMA nodes, the generated VCRAT is incomplete and KFD topology
initialization fails while parsing it.

For example:

  available: 3 nodes (0,2-3)

  node 0 cpus: 0-39
  node 0 size: 519305 MB

  node 2 cpus:
  node 2 size: 519305 MB

  node 3 cpus:
  node 3 size:

With the current implementation, only node 0 is assigned a CPU
proximity domain (domain 0). A GPU attached to NUMA node 3 is assigned
GPU proximity domain 1, and its proximity_domain_to field references
the CPU proximity domain corresponding to node 3. Since no CPU
proximity domain exists for node 3, the referenced proximity domain is
missing, causing VCRAT parsing to fail.

Fix this by creating proximity domains and the corresponding VCRAT
entries for all online NUMA nodes, including CPU-less and memory-less
nodes. This ensures that every online NUMA node has a corresponding
proximity domain, allowing GPU affinity information to be resolved
correctly during VCRAT parsing.

Signed-off-by: Donet Tom <donettom@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 78 ++++++++++++++-------------
 1 file changed, 41 insertions(+), 37 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index b38208965fc8..df698109c17a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1974,51 +1974,55 @@ static int kfd_create_vcrat_image_cpu(void *pcrat_image, size_t *size)
 	sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
 
 	for_each_online_node(numa_node_id) {
-		if (kfd_numa_node_to_apic_id(numa_node_id) == -1)
-			continue;
-
-		/* Fill in Subtype: Compute Unit */
-		ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
-			crat_table->num_domains,
-			(struct crat_subtype_computeunit *)sub_type_hdr);
-		if (ret < 0)
-			return ret;
-		crat_table->length += sub_type_hdr->length;
-		crat_table->total_entries++;
+		int apic_id = kfd_numa_node_to_apic_id(numa_node_id);
+		int mem = node_state(numa_node_id, N_MEMORY);
+
+		if (apic_id != -1) {
+			/* Fill in Subtype: Compute Unit */
+			ret = kfd_fill_cu_for_cpu(numa_node_id, &avail_size,
+				crat_table->num_domains,
+				(struct crat_subtype_computeunit *)sub_type_hdr);
+			if (ret < 0)
+				return ret;
+			crat_table->length += sub_type_hdr->length;
+			crat_table->total_entries++;
 
-		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
-			sub_type_hdr->length);
+			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
+				sub_type_hdr->length);
 
-		/* Fill in Subtype: Memory */
-		ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
-			crat_table->num_domains,
-			(struct crat_subtype_memory *)sub_type_hdr);
-		if (ret < 0)
-			return ret;
-		crat_table->length += sub_type_hdr->length;
-		crat_table->total_entries++;
+			/* Fill in Subtype: IO Link */
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+			ret = kfd_fill_iolink_info_for_cpu(numa_node_id, &avail_size,
+					&entries,
+					(struct crat_subtype_iolink *)sub_type_hdr);
+			if (ret < 0)
+				return ret;
 
-		sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
-			sub_type_hdr->length);
+			if (entries) {
+				crat_table->length += (sub_type_hdr->length * entries);
+				crat_table->total_entries += entries;
 
-		/* Fill in Subtype: IO Link */
-#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
-		ret = kfd_fill_iolink_info_for_cpu(numa_node_id, &avail_size,
-				&entries,
-				(struct crat_subtype_iolink *)sub_type_hdr);
-		if (ret < 0)
-			return ret;
+				sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
+						sub_type_hdr->length * entries);
+			}
+#else
+			pr_info("IO link not available for non x86 platforms\n");
+#endif
+		}
 
-		if (entries) {
-			crat_table->length += (sub_type_hdr->length * entries);
-			crat_table->total_entries += entries;
+		if (mem) {
+			/* Fill in Subtype: Memory */
+			ret = kfd_fill_mem_info_for_cpu(numa_node_id, &avail_size,
+				crat_table->num_domains,
+				(struct crat_subtype_memory *)sub_type_hdr);
+			if (ret < 0)
+				return ret;
+			crat_table->length += sub_type_hdr->length;
+			crat_table->total_entries++;
 
 			sub_type_hdr = (typeof(sub_type_hdr))((char *)sub_type_hdr +
-					sub_type_hdr->length * entries);
+				sub_type_hdr->length);
 		}
-#else
-		pr_info("IO link not available for non x86 platforms\n");
-#endif
 
 		ret = kfd_fill_numa_info_for_cpu(&avail_size,
 				(struct crat_subtype_numa *)sub_type_hdr,
-- 
2.54.0

[PATCH 7/7] drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry

[Donet Tom]

Added a numa_node sysfs attribute for CPU and GPU topology devices.
This attribute exposed the NUMA node associated with the corresponding
topology device through the topology sysfs interface.

Signed-off-by: Donet Tom <donettom@linux.ibm.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 15 +++++++++++++++
 drivers/gpu/drm/amd/amdkfd/kfd_topology.h |  1 +
 2 files changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index d3269f0ffec3..51780bc8aabc 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -418,6 +418,14 @@ static ssize_t node_show(struct kobject *kobj, struct attribute *attr,
 		return sysfs_show_32bit_val(buffer, offs, dev->gpu_id);
 	}
 
+	if (strcmp(attr->name, "numa_node") == 0) {
+		dev = container_of(attr, struct kfd_topology_device,
+				attr_numa_node);
+		if (dev->gpu && kfd_devcgroup_check_permission(dev->gpu))
+			return -EPERM;
+		return sysfs_show_32bit_val(buffer, offs, dev->numa_node);
+	}
+
 	if (strcmp(attr->name, "name") == 0) {
 		dev = container_of(attr, struct kfd_topology_device,
 				attr_name);
@@ -630,6 +638,7 @@ static void kfd_remove_sysfs_node_entry(struct kfd_topology_device *dev)
 
 	if (dev->kobj_node) {
 		sysfs_remove_file(dev->kobj_node, &dev->attr_gpuid);
+		sysfs_remove_file(dev->kobj_node, &dev->attr_numa_node);
 		sysfs_remove_file(dev->kobj_node, &dev->attr_name);
 		sysfs_remove_file(dev->kobj_node, &dev->attr_props);
 		kobject_del(dev->kobj_node);
@@ -693,6 +702,9 @@ static int kfd_build_sysfs_node_entry(struct kfd_topology_device *dev,
 	dev->attr_gpuid.name = "gpu_id";
 	dev->attr_gpuid.mode = KFD_SYSFS_FILE_MODE;
 	sysfs_attr_init(&dev->attr_gpuid);
+	dev->attr_numa_node.name = "numa_node";
+	dev->attr_numa_node.mode = KFD_SYSFS_FILE_MODE;
+	sysfs_attr_init(&dev->attr_numa_node);
 	dev->attr_name.name = "name";
 	dev->attr_name.mode = KFD_SYSFS_FILE_MODE;
 	sysfs_attr_init(&dev->attr_name);
@@ -700,6 +712,9 @@ static int kfd_build_sysfs_node_entry(struct kfd_topology_device *dev,
 	dev->attr_props.mode = KFD_SYSFS_FILE_MODE;
 	sysfs_attr_init(&dev->attr_props);
 	ret = sysfs_create_file(dev->kobj_node, &dev->attr_gpuid);
+	if (ret < 0)
+		return ret;
+	ret = sysfs_create_file(dev->kobj_node, &dev->attr_numa_node);
 	if (ret < 0)
 		return ret;
 	ret = sysfs_create_file(dev->kobj_node, &dev->attr_name);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
index 312802754096..90777af8e665 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h
@@ -160,6 +160,7 @@ struct kfd_topology_device {
 	struct kobject			*kobj_p2plink;
 	struct kobject			*kobj_perf;
 	struct attribute		attr_gpuid;
+	struct attribute		attr_numa_node;
 	struct attribute		attr_name;
 	struct attribute		attr_props;
 	union {
-- 
2.54.0

Re: [PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

[Felix Kuehling]

On 2026-08-04 05:52, Donet Tom wrote:
> This series fixes topology device creation and proximity domain mappings in
> AMDKFD when a system does not provide a CRAT table and the driver generates a
> Virtual CRAT (VCRAT).
>
> The current implementation assumes that CPU NUMA node IDs are contiguous and
> that every NUMA node contains CPUs. During VCRAT generation, CPU topology
> entries and proximity domains are created only for NUMA nodes that have CPUs.
> GPU proximity domains are then allocated immediately after the CPU proximity
> domains, and the GPU I/O link (proximity_domain_to) is initialized using the
> NUMA node ID to which the GPU is attached, implicitly assuming that NUMA node
> IDs and proximity domains have a one-to-one mapping.
>
> These assumptions break on systems with:
>
> Sparse (non-contiguous) NUMA node IDs
> CPU-less NUMA nodes
> CPU-less and memory-less NUMA nodes
>
> For example:
>
> available: 3 nodes (0,2-3)
>
> node 0: CPUs present
> node 2: CPU-less
> node 3: CPU-less
>
> In this case, the driver creates a CPU topology device only for node 0 and
> assigns a single CPU proximity domain (0). However, the GPU VCRAT still
> references the NUMA node ID to which the GPU is attached (for example, node 3)
> in the proximity_domain_to field. Since no corresponding CPU proximity domain
> exists for node 3, the parser cannot find a matching proximity domain during
> VCRAT parsing, causing topology initialization to fail.

Hi Tom,

Thank you for the explanation and the patch series. I may have some gaps 
in my understanding that I would like to clarify. In my mind, I was 
using "proximity domain" and "NUMA node" interchangeably. You're 
demonstrating that they are not the same thing. Is that just a different 
way of labeling the same thing, or are NUMA nodes and proximity domains 
fundamentally different concepts.

Your code in patch 5 (kfd_proximity_domain_to_numa_node and 
kfd_numa_node_to_proximity_domain) seems to imply that there is, in 
fact, a 1:1 mapping, as it assumes that there is a unique translation in 
both directions. Am I missing something?

If they are just different numbering systems, do we really need to keep 
track of both proximity domains and NUMA nodes in the KFD topology? Or 
would it be sufficient to only track NUMA nodes, if that's what we 
really care about in the uAPI (KFD sysfs)?

Thanks,
   Felix


>
> The failure is observed as:
>
> amdgpu: Virtual CRAT table created for GPU
> amdgpu: Error parsing VCRAT
> kfd: amdgpu: Error adding device to topology
> kfd: amdgpu: Error initializing KFD node
>
>
> Since every online NUMA node is a valid topology object and can contain
> CPUs, memory, I/O links, or any combination of these, topology devices
> and proximity domains should be created for every online NUMA node
> rather than only for NUMA nodes that contain CPUs.
>
> To address this, this series introduces a new VCRAT subtype that
> records the mapping between the NUMA node ID and the generated VCRAT
> proximity domain. When topology devices are created, this information
> is stored in the corresponding topology device, allowing the driver to
> translate a NUMA node ID into its associated proximity domain whenever
> required.
>
> Returning to the previous example, the system contains three online
> NUMA nodes, so three CPU topology devices and three proximity domains
> are created, even though only one NUMA node contains CPUs. The NUMA
> node ID is stored in each topology device together with its generated
> proximity domain.
>
> Later, when the GPU VCRAT is generated, the driver only knows the NUMA
> node ID to which the GPU is attached (for example, node 3). Instead of
> assuming that the NUMA node ID is equal to the proximity domain, the
> driver walks the existing topology devices to locate the corresponding
> NUMA node and retrieves its generated proximity domain. In this
> example, NUMA node 3 maps to proximity domain 2, so
> proximity_domain_to is populated with the correct value.
>
> Since the GPU I/O link now references a valid proximity domain, VCRAT
> parsing completes successfully and topology initialization proceeds
> without errors on systems with sparse NUMA node IDs, CPU-less NUMA
> nodes, and CPU-less/memory-less NUMA nodes.
>
> This series consists of the following patches:
>
> Patch 1 removes an unused argument from
> kfd_create_crat_image_virtual() as a preparatory cleanup.
>
> Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
> NUMA node ID and its corresponding proximity domain.
>
> Patch 3 populates the NUMA affinity entries during VCRAT generation
> for every online NUMA node.
>
> Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
> NUMA node ID and proximity domain in the corresponding topology
> device.
>
> Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
> GPU's NUMA node ID to the corresponding CPU proximity domain before
> programming proximity_domain_to.
>
> Patch 6 creates proximity domains and VCRAT entries for all online
> NUMA nodes, including CPU-less and memory-less nodes.
>
> Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
> devices, exposing the associated NUMA node through the topology sysfs
> interface.
>
> Please note that the changes in this series are on a best effort basis from our
> end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the
> HW & SW stack) to kindly help with the review and provide feedback / comments on
> these patches
>
> Donet Tom (7):
>    drm/amdgpu: Remove unused argument from kfd_create_crat_image_virtual
>    drm/amdgpu: Add VCRAT NUMA affinity entry
>    drm/amdgpu: Populate NUMA affinity entries in VCRAT
>    drm/amdgpu: Parse NUMA affinity entries from VCRAT
>    drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
>    drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
>      and memory-less NUMA nodes
>    drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
>
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 +++++++++++++++-------
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
>   drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
>   4 files changed, 168 insertions(+), 56 deletions(-)
>

Re: [PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

[Donet Tom]

On 8/5/26 3:57 AM, Felix Kuehling wrote:
>
> On 2026-08-04 05:52, Donet Tom wrote:
>> This series fixes topology device creation and proximity domain 
>> mappings in
>> AMDKFD when a system does not provide a CRAT table and the driver 
>> generates a
>> Virtual CRAT (VCRAT).
>>
>> The current implementation assumes that CPU NUMA node IDs are 
>> contiguous and
>> that every NUMA node contains CPUs. During VCRAT generation, CPU 
>> topology
>> entries and proximity domains are created only for NUMA nodes that 
>> have CPUs.
>> GPU proximity domains are then allocated immediately after the CPU 
>> proximity
>> domains, and the GPU I/O link (proximity_domain_to) is initialized 
>> using the
>> NUMA node ID to which the GPU is attached, implicitly assuming that 
>> NUMA node
>> IDs and proximity domains have a one-to-one mapping.
>>
>> These assumptions break on systems with:
>>
>> Sparse (non-contiguous) NUMA node IDs
>> CPU-less NUMA nodes
>> CPU-less and memory-less NUMA nodes
>>
>> For example:
>>
>> available: 3 nodes (0,2-3)
>>
>> node 0: CPUs present
>> node 2: CPU-less
>> node 3: CPU-less
>>
>> In this case, the driver creates a CPU topology device only for node 
>> 0 and
>> assigns a single CPU proximity domain (0). However, the GPU VCRAT still
>> references the NUMA node ID to which the GPU is attached (for 
>> example, node 3)
>> in the proximity_domain_to field. Since no corresponding CPU 
>> proximity domain
>> exists for node 3, the parser cannot find a matching proximity domain 
>> during
>> VCRAT parsing, causing topology initialization to fail.
>
> Hi Tom, 


Hi Felix,


>
> Thank you for the explanation and the patch series. I may have some 
> gaps in my understanding that I would like to clarify. In my mind, I 
> was using "proximity domain" and "NUMA node" interchangeably. You're 
> demonstrating that they are not the same thing. Is that just a 
> different way of labeling the same thing, or are NUMA nodes and 
> proximity domains fundamentally different concepts.
>
> Your code in patch 5 (kfd_proximity_domain_to_numa_node and 
> kfd_numa_node_to_proximity_domain) seems to imply that there is, in 
> fact, a 1:1 mapping, as it assumes that there is a unique translation 
> in both directions. Am I missing something?



Thanks for the comment.

IIUC, NUMA node IDs and proximity domains are different numbering 
schemes. We have proximity domains for both CPU and GPU devices. CPU 
proximity domains start from 0, and GPU proximity domains start after 
the last CPU proximity domain.

If the NUMA node IDs are contiguous, the CPU proximity domains happen to 
match the NUMA node IDs. However, if the NUMA node IDs are 
discontiguous, the CPU proximity domains and NUMA node IDs no longer 
have a one-to-one mapping.

>
> If they are just different numbering systems, do we really need to 
> keep track of both proximity domains and NUMA nodes in the KFD 
> topology? Or would it be sufficient to only track NUMA nodes, if 
> that's what we really care about in the uAPI (KFD sysfs)?



Thanks for the suggestion. I also think we don't need to keep track of 
both the proximity domains and the NUMA node IDs in KFD. Do you think 
the approach below would be reasonable?

Just to make sure I understand correctly, for CPU devices the proximity 
domain will be the same as the NUMA node ID, and the GPU proximity 
domains will start after the last CPU proximity domain.

In that case, the CPU proximity domains and NUMA node IDs will always 
have a one-to-one mapping, and the GPU proximity domains will follow 
after them.

For example, if a system has three NUMA nodes and two GPUs:

NUMA node IDs:            0   2   3
CPU proximity domains:    0   2   3
GPU proximity domains:    4   5

Would it be okay to proceed with this approach?

I think this approach should also resolve the driver loading issue.


Thanks
Donet Tom


>
> Thanks,
>   Felix
>
>
>>
>> The failure is observed as:
>>
>> amdgpu: Virtual CRAT table created for GPU
>> amdgpu: Error parsing VCRAT
>> kfd: amdgpu: Error adding device to topology
>> kfd: amdgpu: Error initializing KFD node
>>
>>
>> Since every online NUMA node is a valid topology object and can contain
>> CPUs, memory, I/O links, or any combination of these, topology devices
>> and proximity domains should be created for every online NUMA node
>> rather than only for NUMA nodes that contain CPUs.
>>
>> To address this, this series introduces a new VCRAT subtype that
>> records the mapping between the NUMA node ID and the generated VCRAT
>> proximity domain. When topology devices are created, this information
>> is stored in the corresponding topology device, allowing the driver to
>> translate a NUMA node ID into its associated proximity domain whenever
>> required.
>>
>> Returning to the previous example, the system contains three online
>> NUMA nodes, so three CPU topology devices and three proximity domains
>> are created, even though only one NUMA node contains CPUs. The NUMA
>> node ID is stored in each topology device together with its generated
>> proximity domain.
>>
>> Later, when the GPU VCRAT is generated, the driver only knows the NUMA
>> node ID to which the GPU is attached (for example, node 3). Instead of
>> assuming that the NUMA node ID is equal to the proximity domain, the
>> driver walks the existing topology devices to locate the corresponding
>> NUMA node and retrieves its generated proximity domain. In this
>> example, NUMA node 3 maps to proximity domain 2, so
>> proximity_domain_to is populated with the correct value.
>>
>> Since the GPU I/O link now references a valid proximity domain, VCRAT
>> parsing completes successfully and topology initialization proceeds
>> without errors on systems with sparse NUMA node IDs, CPU-less NUMA
>> nodes, and CPU-less/memory-less NUMA nodes.
>>
>> This series consists of the following patches:
>>
>> Patch 1 removes an unused argument from
>> kfd_create_crat_image_virtual() as a preparatory cleanup.
>>
>> Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
>> NUMA node ID and its corresponding proximity domain.
>>
>> Patch 3 populates the NUMA affinity entries during VCRAT generation
>> for every online NUMA node.
>>
>> Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
>> NUMA node ID and proximity domain in the corresponding topology
>> device.
>>
>> Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
>> GPU's NUMA node ID to the corresponding CPU proximity domain before
>> programming proximity_domain_to.
>>
>> Patch 6 creates proximity domains and VCRAT entries for all online
>> NUMA nodes, including CPU-less and memory-less nodes.
>>
>> Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
>> devices, exposing the associated NUMA node through the topology sysfs
>> interface.
>>
>> Please note that the changes in this series are on a best effort 
>> basis from our
>> end. Therefore, requesting the amd-gfx community (who have deeper 
>> knowledge of the
>> HW & SW stack) to kindly help with the review and provide feedback / 
>> comments on
>> these patches
>>
>> Donet Tom (7):
>>    drm/amdgpu: Remove unused argument from kfd_create_crat_image_virtual
>>    drm/amdgpu: Add VCRAT NUMA affinity entry
>>    drm/amdgpu: Populate NUMA affinity entries in VCRAT
>>    drm/amdgpu: Parse NUMA affinity entries from VCRAT
>>    drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
>>    drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
>>      and memory-less NUMA nodes
>>    drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
>>
>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 +++++++++++++++-------
>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
>>   4 files changed, 168 insertions(+), 56 deletions(-)
>>

Re: [PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

[Kuehling, Felix]

On 2026-08-05 03:09, Donet Tom wrote:
>
> On 8/5/26 3:57 AM, Felix Kuehling wrote:
>>
>> On 2026-08-04 05:52, Donet Tom wrote:
>>> This series fixes topology device creation and proximity domain 
>>> mappings in
>>> AMDKFD when a system does not provide a CRAT table and the driver 
>>> generates a
>>> Virtual CRAT (VCRAT).
>>>
>>> The current implementation assumes that CPU NUMA node IDs are 
>>> contiguous and
>>> that every NUMA node contains CPUs. During VCRAT generation, CPU 
>>> topology
>>> entries and proximity domains are created only for NUMA nodes that 
>>> have CPUs.
>>> GPU proximity domains are then allocated immediately after the CPU 
>>> proximity
>>> domains, and the GPU I/O link (proximity_domain_to) is initialized 
>>> using the
>>> NUMA node ID to which the GPU is attached, implicitly assuming that 
>>> NUMA node
>>> IDs and proximity domains have a one-to-one mapping.
>>>
>>> These assumptions break on systems with:
>>>
>>> Sparse (non-contiguous) NUMA node IDs
>>> CPU-less NUMA nodes
>>> CPU-less and memory-less NUMA nodes
>>>
>>> For example:
>>>
>>> available: 3 nodes (0,2-3)
>>>
>>> node 0: CPUs present
>>> node 2: CPU-less
>>> node 3: CPU-less
>>>
>>> In this case, the driver creates a CPU topology device only for node 
>>> 0 and
>>> assigns a single CPU proximity domain (0). However, the GPU VCRAT still
>>> references the NUMA node ID to which the GPU is attached (for 
>>> example, node 3)
>>> in the proximity_domain_to field. Since no corresponding CPU 
>>> proximity domain
>>> exists for node 3, the parser cannot find a matching proximity 
>>> domain during
>>> VCRAT parsing, causing topology initialization to fail.
>>
>> Hi Tom, 
>
>
> Hi Felix,
>
>
>>
>> Thank you for the explanation and the patch series. I may have some 
>> gaps in my understanding that I would like to clarify. In my mind, I 
>> was using "proximity domain" and "NUMA node" interchangeably. You're 
>> demonstrating that they are not the same thing. Is that just a 
>> different way of labeling the same thing, or are NUMA nodes and 
>> proximity domains fundamentally different concepts.
>>
>> Your code in patch 5 (kfd_proximity_domain_to_numa_node and 
>> kfd_numa_node_to_proximity_domain) seems to imply that there is, in 
>> fact, a 1:1 mapping, as it assumes that there is a unique translation 
>> in both directions. Am I missing something?
>
>
>
> Thanks for the comment.
>
> IIUC, NUMA node IDs and proximity domains are different numbering 
> schemes. We have proximity domains for both CPU and GPU devices. CPU 
> proximity domains start from 0, and GPU proximity domains start after 
> the last CPU proximity domain.
>
> If the NUMA node IDs are contiguous, the CPU proximity domains happen 
> to match the NUMA node IDs. However, if the NUMA node IDs are 
> discontiguous, the CPU proximity domains and NUMA node IDs no longer 
> have a one-to-one mapping.
>
>>
>> If they are just different numbering systems, do we really need to 
>> keep track of both proximity domains and NUMA nodes in the KFD 
>> topology? Or would it be sufficient to only track NUMA nodes, if 
>> that's what we really care about in the uAPI (KFD sysfs)?
>
>
>
> Thanks for the suggestion. I also think we don't need to keep track of 
> both the proximity domains and the NUMA node IDs in KFD. Do you think 
> the approach below would be reasonable?
>
> Just to make sure I understand correctly, for CPU devices the 
> proximity domain will be the same as the NUMA node ID, and the GPU 
> proximity domains will start after the last CPU proximity domain.
>
> In that case, the CPU proximity domains and NUMA node IDs will always 
> have a one-to-one mapping, and the GPU proximity domains will follow 
> after them.
>
> For example, if a system has three NUMA nodes and two GPUs:
>
> NUMA node IDs:            0   2   3
> CPU proximity domains:    0   2   3
> GPU proximity domains:    4   5
>
> Would it be okay to proceed with this approach?

Yes, this looks good to me.

Regards,
   Felix


>
> I think this approach should also resolve the driver loading issue.
>
>
> Thanks
> Donet Tom
>
>
>>
>> Thanks,
>>   Felix
>>
>>
>>>
>>> The failure is observed as:
>>>
>>> amdgpu: Virtual CRAT table created for GPU
>>> amdgpu: Error parsing VCRAT
>>> kfd: amdgpu: Error adding device to topology
>>> kfd: amdgpu: Error initializing KFD node
>>>
>>>
>>> Since every online NUMA node is a valid topology object and can contain
>>> CPUs, memory, I/O links, or any combination of these, topology devices
>>> and proximity domains should be created for every online NUMA node
>>> rather than only for NUMA nodes that contain CPUs.
>>>
>>> To address this, this series introduces a new VCRAT subtype that
>>> records the mapping between the NUMA node ID and the generated VCRAT
>>> proximity domain. When topology devices are created, this information
>>> is stored in the corresponding topology device, allowing the driver to
>>> translate a NUMA node ID into its associated proximity domain whenever
>>> required.
>>>
>>> Returning to the previous example, the system contains three online
>>> NUMA nodes, so three CPU topology devices and three proximity domains
>>> are created, even though only one NUMA node contains CPUs. The NUMA
>>> node ID is stored in each topology device together with its generated
>>> proximity domain.
>>>
>>> Later, when the GPU VCRAT is generated, the driver only knows the NUMA
>>> node ID to which the GPU is attached (for example, node 3). Instead of
>>> assuming that the NUMA node ID is equal to the proximity domain, the
>>> driver walks the existing topology devices to locate the corresponding
>>> NUMA node and retrieves its generated proximity domain. In this
>>> example, NUMA node 3 maps to proximity domain 2, so
>>> proximity_domain_to is populated with the correct value.
>>>
>>> Since the GPU I/O link now references a valid proximity domain, VCRAT
>>> parsing completes successfully and topology initialization proceeds
>>> without errors on systems with sparse NUMA node IDs, CPU-less NUMA
>>> nodes, and CPU-less/memory-less NUMA nodes.
>>>
>>> This series consists of the following patches:
>>>
>>> Patch 1 removes an unused argument from
>>> kfd_create_crat_image_virtual() as a preparatory cleanup.
>>>
>>> Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
>>> NUMA node ID and its corresponding proximity domain.
>>>
>>> Patch 3 populates the NUMA affinity entries during VCRAT generation
>>> for every online NUMA node.
>>>
>>> Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
>>> NUMA node ID and proximity domain in the corresponding topology
>>> device.
>>>
>>> Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
>>> GPU's NUMA node ID to the corresponding CPU proximity domain before
>>> programming proximity_domain_to.
>>>
>>> Patch 6 creates proximity domains and VCRAT entries for all online
>>> NUMA nodes, including CPU-less and memory-less nodes.
>>>
>>> Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
>>> devices, exposing the associated NUMA node through the topology sysfs
>>> interface.
>>>
>>> Please note that the changes in this series are on a best effort 
>>> basis from our
>>> end. Therefore, requesting the amd-gfx community (who have deeper 
>>> knowledge of the
>>> HW & SW stack) to kindly help with the review and provide feedback / 
>>> comments on
>>> these patches
>>>
>>> Donet Tom (7):
>>>    drm/amdgpu: Remove unused argument from 
>>> kfd_create_crat_image_virtual
>>>    drm/amdgpu: Add VCRAT NUMA affinity entry
>>>    drm/amdgpu: Populate NUMA affinity entries in VCRAT
>>>    drm/amdgpu: Parse NUMA affinity entries from VCRAT
>>>    drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
>>>    drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
>>>      and memory-less NUMA nodes
>>>    drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
>>>
>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 
>>> +++++++++++++++-------
>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
>>>   4 files changed, 168 insertions(+), 56 deletions(-)
>>>

Re: [PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

[Donet Tom]

On 8/5/26 9:46 PM, Kuehling, Felix wrote:
>
> On 2026-08-05 03:09, Donet Tom wrote:
>>
>> On 8/5/26 3:57 AM, Felix Kuehling wrote:
>>>
>>> On 2026-08-04 05:52, Donet Tom wrote:
>>>> This series fixes topology device creation and proximity domain 
>>>> mappings in
>>>> AMDKFD when a system does not provide a CRAT table and the driver 
>>>> generates a
>>>> Virtual CRAT (VCRAT).
>>>>
>>>> The current implementation assumes that CPU NUMA node IDs are 
>>>> contiguous and
>>>> that every NUMA node contains CPUs. During VCRAT generation, CPU 
>>>> topology
>>>> entries and proximity domains are created only for NUMA nodes that 
>>>> have CPUs.
>>>> GPU proximity domains are then allocated immediately after the CPU 
>>>> proximity
>>>> domains, and the GPU I/O link (proximity_domain_to) is initialized 
>>>> using the
>>>> NUMA node ID to which the GPU is attached, implicitly assuming that 
>>>> NUMA node
>>>> IDs and proximity domains have a one-to-one mapping.
>>>>
>>>> These assumptions break on systems with:
>>>>
>>>> Sparse (non-contiguous) NUMA node IDs
>>>> CPU-less NUMA nodes
>>>> CPU-less and memory-less NUMA nodes
>>>>
>>>> For example:
>>>>
>>>> available: 3 nodes (0,2-3)
>>>>
>>>> node 0: CPUs present
>>>> node 2: CPU-less
>>>> node 3: CPU-less
>>>>
>>>> In this case, the driver creates a CPU topology device only for 
>>>> node 0 and
>>>> assigns a single CPU proximity domain (0). However, the GPU VCRAT 
>>>> still
>>>> references the NUMA node ID to which the GPU is attached (for 
>>>> example, node 3)
>>>> in the proximity_domain_to field. Since no corresponding CPU 
>>>> proximity domain
>>>> exists for node 3, the parser cannot find a matching proximity 
>>>> domain during
>>>> VCRAT parsing, causing topology initialization to fail.
>>>
>>> Hi Tom, 
>>
>>
>> Hi Felix,
>>
>>
>>>
>>> Thank you for the explanation and the patch series. I may have some 
>>> gaps in my understanding that I would like to clarify. In my mind, I 
>>> was using "proximity domain" and "NUMA node" interchangeably. You're 
>>> demonstrating that they are not the same thing. Is that just a 
>>> different way of labeling the same thing, or are NUMA nodes and 
>>> proximity domains fundamentally different concepts.
>>>
>>> Your code in patch 5 (kfd_proximity_domain_to_numa_node and 
>>> kfd_numa_node_to_proximity_domain) seems to imply that there is, in 
>>> fact, a 1:1 mapping, as it assumes that there is a unique 
>>> translation in both directions. Am I missing something?
>>
>>
>>
>> Thanks for the comment.
>>
>> IIUC, NUMA node IDs and proximity domains are different numbering 
>> schemes. We have proximity domains for both CPU and GPU devices. CPU 
>> proximity domains start from 0, and GPU proximity domains start after 
>> the last CPU proximity domain.
>>
>> If the NUMA node IDs are contiguous, the CPU proximity domains happen 
>> to match the NUMA node IDs. However, if the NUMA node IDs are 
>> discontiguous, the CPU proximity domains and NUMA node IDs no longer 
>> have a one-to-one mapping.
>>
>>>
>>> If they are just different numbering systems, do we really need to 
>>> keep track of both proximity domains and NUMA nodes in the KFD 
>>> topology? Or would it be sufficient to only track NUMA nodes, if 
>>> that's what we really care about in the uAPI (KFD sysfs)?
>>
>>
>>
>> Thanks for the suggestion. I also think we don't need to keep track 
>> of both the proximity domains and the NUMA node IDs in KFD. Do you 
>> think the approach below would be reasonable?
>>
>> Just to make sure I understand correctly, for CPU devices the 
>> proximity domain will be the same as the NUMA node ID, and the GPU 
>> proximity domains will start after the last CPU proximity domain.
>>
>> In that case, the CPU proximity domains and NUMA node IDs will always 
>> have a one-to-one mapping, and the GPU proximity domains will follow 
>> after them.
>>
>> For example, if a system has three NUMA nodes and two GPUs:
>>
>> NUMA node IDs:            0   2   3
>> CPU proximity domains:    0   2   3
>> GPU proximity domains:    4   5
>>
>> Would it be okay to proceed with this approach?
>
> Yes, this looks good to me.


Thanks, Felix. I'll implement this approach and post a new version.

Thanks,
Donet Tom


>
> Regards,
>   Felix
>
>
>>
>> I think this approach should also resolve the driver loading issue.
>>
>>
>> Thanks
>> Donet Tom
>>
>>
>>>
>>> Thanks,
>>>   Felix
>>>
>>>
>>>>
>>>> The failure is observed as:
>>>>
>>>> amdgpu: Virtual CRAT table created for GPU
>>>> amdgpu: Error parsing VCRAT
>>>> kfd: amdgpu: Error adding device to topology
>>>> kfd: amdgpu: Error initializing KFD node
>>>>
>>>>
>>>> Since every online NUMA node is a valid topology object and can 
>>>> contain
>>>> CPUs, memory, I/O links, or any combination of these, topology devices
>>>> and proximity domains should be created for every online NUMA node
>>>> rather than only for NUMA nodes that contain CPUs.
>>>>
>>>> To address this, this series introduces a new VCRAT subtype that
>>>> records the mapping between the NUMA node ID and the generated VCRAT
>>>> proximity domain. When topology devices are created, this information
>>>> is stored in the corresponding topology device, allowing the driver to
>>>> translate a NUMA node ID into its associated proximity domain whenever
>>>> required.
>>>>
>>>> Returning to the previous example, the system contains three online
>>>> NUMA nodes, so three CPU topology devices and three proximity domains
>>>> are created, even though only one NUMA node contains CPUs. The NUMA
>>>> node ID is stored in each topology device together with its generated
>>>> proximity domain.
>>>>
>>>> Later, when the GPU VCRAT is generated, the driver only knows the NUMA
>>>> node ID to which the GPU is attached (for example, node 3). Instead of
>>>> assuming that the NUMA node ID is equal to the proximity domain, the
>>>> driver walks the existing topology devices to locate the corresponding
>>>> NUMA node and retrieves its generated proximity domain. In this
>>>> example, NUMA node 3 maps to proximity domain 2, so
>>>> proximity_domain_to is populated with the correct value.
>>>>
>>>> Since the GPU I/O link now references a valid proximity domain, VCRAT
>>>> parsing completes successfully and topology initialization proceeds
>>>> without errors on systems with sparse NUMA node IDs, CPU-less NUMA
>>>> nodes, and CPU-less/memory-less NUMA nodes.
>>>>
>>>> This series consists of the following patches:
>>>>
>>>> Patch 1 removes an unused argument from
>>>> kfd_create_crat_image_virtual() as a preparatory cleanup.
>>>>
>>>> Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
>>>> NUMA node ID and its corresponding proximity domain.
>>>>
>>>> Patch 3 populates the NUMA affinity entries during VCRAT generation
>>>> for every online NUMA node.
>>>>
>>>> Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
>>>> NUMA node ID and proximity domain in the corresponding topology
>>>> device.
>>>>
>>>> Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
>>>> GPU's NUMA node ID to the corresponding CPU proximity domain before
>>>> programming proximity_domain_to.
>>>>
>>>> Patch 6 creates proximity domains and VCRAT entries for all online
>>>> NUMA nodes, including CPU-less and memory-less nodes.
>>>>
>>>> Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
>>>> devices, exposing the associated NUMA node through the topology sysfs
>>>> interface.
>>>>
>>>> Please note that the changes in this series are on a best effort 
>>>> basis from our
>>>> end. Therefore, requesting the amd-gfx community (who have deeper 
>>>> knowledge of the
>>>> HW & SW stack) to kindly help with the review and provide feedback 
>>>> / comments on
>>>> these patches
>>>>
>>>> Donet Tom (7):
>>>>    drm/amdgpu: Remove unused argument from 
>>>> kfd_create_crat_image_virtual
>>>>    drm/amdgpu: Add VCRAT NUMA affinity entry
>>>>    drm/amdgpu: Populate NUMA affinity entries in VCRAT
>>>>    drm/amdgpu: Parse NUMA affinity entries from VCRAT
>>>>    drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
>>>>    drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
>>>>      and memory-less NUMA nodes
>>>>    drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
>>>>
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 
>>>> +++++++++++++++-------
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
>>>>   4 files changed, 168 insertions(+), 56 deletions(-)
>>>>