[PATCH v2 0/3] drm/amdkfd: Add support for discontiguous and CPU-less NUMA topologies

[Dhruv B <dhruv.b@linux.ibm.com>]

This series fixes VCRAT generation in AMDKFD for systems that do not
provide a CRAT table and where the driver generates a Virtual CRAT
(VCRAT) for the CPU topology.

The current implementation uses for_each_online_node() to iterate over
NUMA nodes and assigns proximity domain IDs sequentially (0, 1, 2, ...)
regardless of the actual NUMA node IDs. It also assumes that every
online NUMA node contains both CPUs and memory, unconditionally
emitting Compute Unit and Memory CRAT subtypes for each node.

These assumptions break on systems with:

  Discontiguous (sparse) NUMA node IDs
  CPU-less NUMA nodes
  Memory-less NUMA nodes

For example:

  available: 3 nodes (0,2-3)

  node 0: CPUs present
  node 2: CPU-less
  node 3: CPU-less

In this case, the existing code iterates over nodes 0, 2 and 3 and
assigns proximity domains 0, 1 and 2 respectively, instead of 0, 2
and 3. The mismatch between NUMA node IDs and proximity domain IDs
results in an inconsistent KFD topology and causes topology parsing
to fail with:

  amdgpu: Virtual CRAT table created for GPU
  amdgpu: Error parsing VCRAT
  kfd: amdgpu: Error adding device to topology
  kfd: amdgpu: Error initializing KFD node

To fix the proximity domain mismatch, the VCRAT generation loop is
changed to iterate over the complete NUMA node ID range up to the
last online node, incrementing num_domains for every node ID in the
range. This preserves a direct 1:1 mapping between NUMA node ID and
proximity domain ID, while skipping resource entry creation for
offline or unpopulated node IDs.

To fix the unconditional subtype emission, explicit guards are added
so that Compute Unit entries are only written for nodes that have
valid CPU cores (kfd_numa_node_to_apic_id() != -1), and Memory
entries are only written for nodes that actually contain memory
(node_state(N_MEMORY)). This prevents invalid or empty CRAT subtype
entries from being emitted for CPU-less, memory-less, or empty NUMA
nodes.

On a system with contiguous NUMA nodes, numactl reports:

  $ numactl -H
  available: 4 nodes (0-3)
  node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
           18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
           34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
           50 51 52 53 54 55
  node 0 size: 881496 MB
  node 1 cpus: 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70
           71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
           87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
           102 103 104 105 106 107 108 109 110 111
  node 1 size: 885640 MB
  node 2 cpus: 112 113 114 115 116 117 118 119 120 121 122
           123 124 125 126 127 128 129 130 131 132 133 134
           135 136 137 138 139 140 141 142 143 144 145 146
           147 148 149 150 151 152 153 154 155 156 157 158
           159 160 161 162 163 164 165 166 167
  node 2 size: 885640 MB
  node 3 cpus: 168 169 170 171 172 173 174 175 176 177 178
           179 180 181 182 183 184 185 186 187 188 189 190
           191 192 193 194 195 196 197 198 199 200 201 202
           203 204 205 206 207 208 209 210 211 212 213 214
           215 216 217 218 219 220 221 222 223
  node 3 size: 885394 MB

The KFD topology sysfs reflects all 4 CPU nodes plus 2 GPUs:

  /sys/class/kfd/kfd/topology/nodes: 0  1  2  3  4  5

  0: cpu_cores_count 56  mem_banks_count 1  io_links_count 0
  1: cpu_cores_count 56  mem_banks_count 1  io_links_count 0
  2: cpu_cores_count 56  mem_banks_count 1  io_links_count 2
  3: cpu_cores_count 56  mem_banks_count 1  io_links_count 0
  4: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)
  5: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)

On the same system with sparse NUMA nodes, numactl reports:

  $ numactl -H
  available: 3 nodes (0,2-3)
  node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
           18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
           34 35 36 37 38 39
  node 0 size: 519305 MB
  node 2 cpus:
  node 2 size: 0 MB
  node 3 cpus:
  node 3 size: 0 MB

After this series the KFD topology sysfs correctly reflects all node
IDs including the absent node 1 as a placeholder:

  /sys/class/kfd/kfd/topology/nodes: 0  1  2  3  4  5

  0: cpu_cores_count 40  mem_banks_count 1  io_links_count 0
  1: cpu_cores_count 0   mem_banks_count 0  io_links_count 0 (Absent node, placeholder domain)
  2: cpu_cores_count 0   mem_banks_count 0  io_links_count 2 (CPU-less, GPUs attached here)
  3: cpu_cores_count 0   mem_banks_count 0  io_links_count 0 (CPU-less)
  4: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)
  5: cpu_cores_count 0   simd_count 416     io_links_count 1 (GPU)

The GPU I/O links correctly reference proximity domain 2, which
corresponds directly to NUMA node 2 where the GPUs are physically
attached. This is confirmed by the io_links sysfs entries:

  node 4 io_link: node_from 4  node_to 2  type 2  max_bandwidth 10000
  node 5 io_link: node_from 5  node_to 2  type 2  max_bandwidth 10000

Both GPUs report node_to 2, matching the proximity domain of NUMA
node 2 (the CPU-less node to which the GPUs are connected).

This series consists of the following patches:

Patch 1 removes unused code in the CRAT initialization path as a
preparatory cleanup.

Patch 2 fixes the proximity domain mismatch by iterating through
the complete NUMA node ID range and preserving a direct NUMA-node-
to-proximity-domain mapping.

Patch 3 restricts Compute Unit and Memory CRAT subtype entries to
NUMA nodes that actually have the corresponding resources.
---
v1 -> v2
- Addressed Felix's comment and fixed the incorrect proximity domain mapping 
  caused due to crat_table->num_domains++ being at the start of the loop.

v1 - https://lore.kernel.org/all/cover.1791208447.git.dhruv.b@linux.ibm.com/
---
Dhruv B (3):
  drm/amdkfd: Remove unused code in CRAT initialization
  drm/amdkfd: Add support for discontiguous NUMA node topology
  drm/amdkfd: Restrict CU and memory entries to resource nodes

 drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 60 ++++++++++++-----------
 drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  2 +-
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 11 +----
 3 files changed, 35 insertions(+), 38 deletions(-)

-- 
2.54.0 (Apple Git-157)