All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH V2 0/3] Add UALink infrastructure series 2
@ 2026-08-31 18:33 Alex Deucher
  2026-08-31 18:33 ` [PATCH 1/3] drm/gem: Add callback for when handle count goes to 0 Alex Deucher
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Alex Deucher @ 2026-08-31 18:33 UTC (permalink / raw)
  To: amd-gfx, dri-devel, Felix.Kuehling; +Cc: Alex Deucher

This adds the core infrastructure for supporting UALink (Ultra Accelerator Link)
connected scale up pods. I've split this into two series, one to add the core
infrastructure, and one to expose the new IOCTL interface and add the
documentation to avoid spamming the larger audience with the implemenation
defails.  This is the second series.

The first series can be found here:
V2: https://lists.freedesktop.org/archives/amd-gfx/2026-August/151914.html
V1: https://lists.freedesktop.org/archives/amd-gfx/2026-August/151200.html

Both series can be accessed via this git branch:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/ualink?ref_type=heads

This implements UALoE (UALink over Ethernet). An overview of the complete
solution can be found here:
https://www.amd.com/en/products/rackscale-solutions/helios.html

Overview

Connected GPUs in a pod can directly access the remove memory on another
GPU over UALink.  Unlike RMDA, there is no copy involved; it is direct
loads/stores over the fabric.  Shared memory can only be accessed by
a remote GPU if the memory was exported and the importer has been authorized.
For the memory to be shared, it must be part of a unified physical
address space shared between nodes.  This address space is called NPA (Nework
Physical Address) space.  This address space is partitioned between
the GPUs so that each GPU has it's own segment of the address space in which
to export its memory.  Each GPU maintains a dedicated set of page tables
for their NPA space similar to GPUVM.

Exported memory is pinned.  This is similar to how dma-bufs in VRAM are pinned
for P2P access.  Remote TLB shootdowns from the exporter are used when the
exported memory is freed in order to remove access by remote GPUs

To access remote memory, the driver can map NPA addresses into its per
process GPUVM page tables just like local memory.  Applications use
opaque handles to represent remote memory.  GPUs in a pod communicate
with eachother directly to exchange NPA addresses between importers
and exporters.  If a node goes offline or is reset, their peers will
clean up any remaining refrences that are lost when that happens.

User interface

Export Memory

To export memory, a handle must be created for an allocation
that can be shared with another node in the pod.  To do this
the exporter calls the GEM UALink IOCTL with the GEM handle
to the buffer it wants to export.  The IOCTL returns a
unique handle which can be shared with the remote host.

Import Memory

To import the memory, the handle from the remote node must be converted
from a unique handle to a local GEM object which represents the local reference
to the NPA space on the importer.  If the memory has already been
imported, it just returns a new reference to the existing object.  If not,
the importer queries the exporter to get the NPA address.  Once it has that
the importer can create a dma-buf to represent the NPA space used by the
allocation and that is returned to the application.

Proposed Userspace
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L971
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036

v2:
- Update the Documentation based on review feedback

Alex Deucher (1):
  Documentation: add initial UALink Documentation

Mukul Joshi (2):
  drm/gem: Add callback for when handle count goes to 0
  drm/amdgpu: Add ioctl infra for exporting/importing UALink handles

 Documentation/gpu/amdgpu/index.rst         |  1 +
 Documentation/gpu/amdgpu/ualink.rst        | 75 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 +++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  2 +
 drivers/gpu/drm/drm_gem.c                  |  5 +-
 include/drm/drm_gem.h                      | 11 ++++
 include/uapi/drm/amdgpu_drm.h              | 30 +++++++++
 9 files changed, 178 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/gpu/amdgpu/ualink.rst

-- 
2.55.0


^ permalink raw reply	[flat|nested] 12+ messages in thread
* [PATCH V2 0/3] Add UALink infrastructure series 2
@ 2026-08-24 17:01 Alex Deucher
  0 siblings, 0 replies; 12+ messages in thread
From: Alex Deucher @ 2026-08-24 17:01 UTC (permalink / raw)
  To: amd-gfx, dri-devel, Felix.Kuehling; +Cc: Alex Deucher

This adds the core infrastructure for supporting UALink (Ultra Accelerator Link)
connected scale up pods. I've split this into two series, one to add the core
infrastructure, and one to expose the new IOCTL interface and add the
documentation to avoid spamming the larger audience with the implemenation
defails.  This is the second series.

The first series can be found here:
https://lists.freedesktop.org/archives/amd-gfx/2026-August/151200.html

Both series can be accessed via this git branch:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/ualink?ref_type=heads

This implements UALoE (UALink over Ethernet). An overview of the complete
solution can be found here:
https://www.amd.com/en/products/rackscale-solutions/helios.html

Overview

Connected GPUs in a pod can directly access the remove memory on another
GPU over UALink.  Unlike RMDA, there is no copy involved; it is direct
loads/stores over the fabric.  Shared memory can only be accessed by
a remote GPU if the memory was exported and the importer has been authorized.
For the memory to be shared, it must be part of a unified physical
address space shared between nodes.  This address space is called NPA (Nework
Physical Address) space.  This address space is partitioned between
the GPUs so that each GPU has it's own segment of the address space in which
to export its memory.  Each GPU maintains a dedicated set of page tables
for their NPA space similar to GPUVM.

Exported memory is pinned.  This is similar to how dma-bufs in VRAM are pinned
for P2P access.  Remote TLB shootdowns from the exporter are used when the
exported memory is freed in order to remove access by remote GPUs.

To access remote memory, the driver can map NPA addresses into its per
process GPUVM page tables just like local memory.  Applications use
opaque handles to represent remote memory.  GPUs in a pod communicate
with eachother directly to exchange NPA addresses between importers
and exporters.  If a node goes offline or is reset, their peers will
clean up any remaining refrences that are lost when that happens.

User interface

Export Memory

To export memory, a handle must be created for an allocation
that can be shared with another node in the pod.  To do this
the exporter calls the GEM UALink IOCTL with the GEM handle
to the buffer it wants to export.  The IOCTL returns a
unique handle which can be shared with the remote host.

Import Memory

To import the memory, the handle from the remote node must be converted
from a unique handle to a local GEM object which represents the local reference
to the NPA space on the importer.  If the memory has already been
imported, it just returns a new reference to the existing object.  If not,
the importer queries the exporter to get the NPA address.  Once it has that
the importer can create a dma-buf to represent the NPA space used by the
allocation and that is returned to the application.

Proposed Userspace
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L971
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036

V2: Update the documentation per Felix' feedback.  Clarify that exported memory
is pinned.

Alex Deucher (1):
  Documentation: add initial UALink Documentation

Mukul Joshi (2):
  drm/gem: Add callback for when handle count goes to 0
  drm/amdgpu: Add ioctl infra for exporting/importing UALink handles

 Documentation/gpu/amdgpu/index.rst         |  1 +
 Documentation/gpu/amdgpu/ualink.rst        | 75 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 +++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  2 +
 drivers/gpu/drm/drm_gem.c                  |  5 +-
 include/drm/drm_gem.h                      | 11 ++++
 include/uapi/drm/amdgpu_drm.h              | 30 +++++++++
 9 files changed, 178 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/gpu/amdgpu/ualink.rst

-- 
2.55.0


^ permalink raw reply	[flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-09-09  6:55 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-31 18:33 [PATCH V2 0/3] Add UALink infrastructure series 2 Alex Deucher
2026-08-31 18:33 ` [PATCH 1/3] drm/gem: Add callback for when handle count goes to 0 Alex Deucher
2026-09-05  5:36   ` Thorsten Leemhuis
2026-09-05  9:31     ` Manuel Ebner
2026-09-05 22:25       ` Klara Modin
2026-09-05 23:41         ` Gary Guo
2026-09-08  4:33         ` Thorsten Leemhuis
2026-09-08 16:03         ` Alex Deucher
2026-09-08 17:57           ` Klara Modin
2026-08-31 18:33 ` [PATCH 2/3] drm/amdgpu: Add ioctl infra for exporting/importing UALink handles Alex Deucher
2026-08-31 18:33 ` [PATCH 3/3] Documentation: add initial UALink Documentation Alex Deucher
  -- strict thread matches above, loose matches on Subject: below --
2026-08-24 17:01 [PATCH V2 0/3] Add UALink infrastructure series 2 Alex Deucher

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.