Re: [PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

[Donet Tom <donettom@linux.ibm.com>]

On 8/5/26 9:46 PM, Kuehling, Felix wrote:
>
> On 2026-08-05 03:09, Donet Tom wrote:
>>
>> On 8/5/26 3:57 AM, Felix Kuehling wrote:
>>>
>>> On 2026-08-04 05:52, Donet Tom wrote:
>>>> This series fixes topology device creation and proximity domain 
>>>> mappings in
>>>> AMDKFD when a system does not provide a CRAT table and the driver 
>>>> generates a
>>>> Virtual CRAT (VCRAT).
>>>>
>>>> The current implementation assumes that CPU NUMA node IDs are 
>>>> contiguous and
>>>> that every NUMA node contains CPUs. During VCRAT generation, CPU 
>>>> topology
>>>> entries and proximity domains are created only for NUMA nodes that 
>>>> have CPUs.
>>>> GPU proximity domains are then allocated immediately after the CPU 
>>>> proximity
>>>> domains, and the GPU I/O link (proximity_domain_to) is initialized 
>>>> using the
>>>> NUMA node ID to which the GPU is attached, implicitly assuming that 
>>>> NUMA node
>>>> IDs and proximity domains have a one-to-one mapping.
>>>>
>>>> These assumptions break on systems with:
>>>>
>>>> Sparse (non-contiguous) NUMA node IDs
>>>> CPU-less NUMA nodes
>>>> CPU-less and memory-less NUMA nodes
>>>>
>>>> For example:
>>>>
>>>> available: 3 nodes (0,2-3)
>>>>
>>>> node 0: CPUs present
>>>> node 2: CPU-less
>>>> node 3: CPU-less
>>>>
>>>> In this case, the driver creates a CPU topology device only for 
>>>> node 0 and
>>>> assigns a single CPU proximity domain (0). However, the GPU VCRAT 
>>>> still
>>>> references the NUMA node ID to which the GPU is attached (for 
>>>> example, node 3)
>>>> in the proximity_domain_to field. Since no corresponding CPU 
>>>> proximity domain
>>>> exists for node 3, the parser cannot find a matching proximity 
>>>> domain during
>>>> VCRAT parsing, causing topology initialization to fail.
>>>
>>> Hi Tom, 
>>
>>
>> Hi Felix,
>>
>>
>>>
>>> Thank you for the explanation and the patch series. I may have some 
>>> gaps in my understanding that I would like to clarify. In my mind, I 
>>> was using "proximity domain" and "NUMA node" interchangeably. You're 
>>> demonstrating that they are not the same thing. Is that just a 
>>> different way of labeling the same thing, or are NUMA nodes and 
>>> proximity domains fundamentally different concepts.
>>>
>>> Your code in patch 5 (kfd_proximity_domain_to_numa_node and 
>>> kfd_numa_node_to_proximity_domain) seems to imply that there is, in 
>>> fact, a 1:1 mapping, as it assumes that there is a unique 
>>> translation in both directions. Am I missing something?
>>
>>
>>
>> Thanks for the comment.
>>
>> IIUC, NUMA node IDs and proximity domains are different numbering 
>> schemes. We have proximity domains for both CPU and GPU devices. CPU 
>> proximity domains start from 0, and GPU proximity domains start after 
>> the last CPU proximity domain.
>>
>> If the NUMA node IDs are contiguous, the CPU proximity domains happen 
>> to match the NUMA node IDs. However, if the NUMA node IDs are 
>> discontiguous, the CPU proximity domains and NUMA node IDs no longer 
>> have a one-to-one mapping.
>>
>>>
>>> If they are just different numbering systems, do we really need to 
>>> keep track of both proximity domains and NUMA nodes in the KFD 
>>> topology? Or would it be sufficient to only track NUMA nodes, if 
>>> that's what we really care about in the uAPI (KFD sysfs)?
>>
>>
>>
>> Thanks for the suggestion. I also think we don't need to keep track 
>> of both the proximity domains and the NUMA node IDs in KFD. Do you 
>> think the approach below would be reasonable?
>>
>> Just to make sure I understand correctly, for CPU devices the 
>> proximity domain will be the same as the NUMA node ID, and the GPU 
>> proximity domains will start after the last CPU proximity domain.
>>
>> In that case, the CPU proximity domains and NUMA node IDs will always 
>> have a one-to-one mapping, and the GPU proximity domains will follow 
>> after them.
>>
>> For example, if a system has three NUMA nodes and two GPUs:
>>
>> NUMA node IDs:            0   2   3
>> CPU proximity domains:    0   2   3
>> GPU proximity domains:    4   5
>>
>> Would it be okay to proceed with this approach?
>
> Yes, this looks good to me.


Thanks, Felix. I'll implement this approach and post a new version.

Thanks,
Donet Tom


>
> Regards,
>   Felix
>
>
>>
>> I think this approach should also resolve the driver loading issue.
>>
>>
>> Thanks
>> Donet Tom
>>
>>
>>>
>>> Thanks,
>>>   Felix
>>>
>>>
>>>>
>>>> The failure is observed as:
>>>>
>>>> amdgpu: Virtual CRAT table created for GPU
>>>> amdgpu: Error parsing VCRAT
>>>> kfd: amdgpu: Error adding device to topology
>>>> kfd: amdgpu: Error initializing KFD node
>>>>
>>>>
>>>> Since every online NUMA node is a valid topology object and can 
>>>> contain
>>>> CPUs, memory, I/O links, or any combination of these, topology devices
>>>> and proximity domains should be created for every online NUMA node
>>>> rather than only for NUMA nodes that contain CPUs.
>>>>
>>>> To address this, this series introduces a new VCRAT subtype that
>>>> records the mapping between the NUMA node ID and the generated VCRAT
>>>> proximity domain. When topology devices are created, this information
>>>> is stored in the corresponding topology device, allowing the driver to
>>>> translate a NUMA node ID into its associated proximity domain whenever
>>>> required.
>>>>
>>>> Returning to the previous example, the system contains three online
>>>> NUMA nodes, so three CPU topology devices and three proximity domains
>>>> are created, even though only one NUMA node contains CPUs. The NUMA
>>>> node ID is stored in each topology device together with its generated
>>>> proximity domain.
>>>>
>>>> Later, when the GPU VCRAT is generated, the driver only knows the NUMA
>>>> node ID to which the GPU is attached (for example, node 3). Instead of
>>>> assuming that the NUMA node ID is equal to the proximity domain, the
>>>> driver walks the existing topology devices to locate the corresponding
>>>> NUMA node and retrieves its generated proximity domain. In this
>>>> example, NUMA node 3 maps to proximity domain 2, so
>>>> proximity_domain_to is populated with the correct value.
>>>>
>>>> Since the GPU I/O link now references a valid proximity domain, VCRAT
>>>> parsing completes successfully and topology initialization proceeds
>>>> without errors on systems with sparse NUMA node IDs, CPU-less NUMA
>>>> nodes, and CPU-less/memory-less NUMA nodes.
>>>>
>>>> This series consists of the following patches:
>>>>
>>>> Patch 1 removes an unused argument from
>>>> kfd_create_crat_image_virtual() as a preparatory cleanup.
>>>>
>>>> Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
>>>> NUMA node ID and its corresponding proximity domain.
>>>>
>>>> Patch 3 populates the NUMA affinity entries during VCRAT generation
>>>> for every online NUMA node.
>>>>
>>>> Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
>>>> NUMA node ID and proximity domain in the corresponding topology
>>>> device.
>>>>
>>>> Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
>>>> GPU's NUMA node ID to the corresponding CPU proximity domain before
>>>> programming proximity_domain_to.
>>>>
>>>> Patch 6 creates proximity domains and VCRAT entries for all online
>>>> NUMA nodes, including CPU-less and memory-less nodes.
>>>>
>>>> Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
>>>> devices, exposing the associated NUMA node through the topology sysfs
>>>> interface.
>>>>
>>>> Please note that the changes in this series are on a best effort 
>>>> basis from our
>>>> end. Therefore, requesting the amd-gfx community (who have deeper 
>>>> knowledge of the
>>>> HW & SW stack) to kindly help with the review and provide feedback 
>>>> / comments on
>>>> these patches
>>>>
>>>> Donet Tom (7):
>>>>    drm/amdgpu: Remove unused argument from 
>>>> kfd_create_crat_image_virtual
>>>>    drm/amdgpu: Add VCRAT NUMA affinity entry
>>>>    drm/amdgpu: Populate NUMA affinity entries in VCRAT
>>>>    drm/amdgpu: Parse NUMA affinity entries from VCRAT
>>>>    drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
>>>>    drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
>>>>      and memory-less NUMA nodes
>>>>    drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
>>>>
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 
>>>> +++++++++++++++-------
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
>>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
>>>>   4 files changed, 168 insertions(+), 56 deletions(-)
>>>>