[PATCH 0/3] Add UALink infrastructure series 2

[PATCH 0/3] Add UALink infrastructure series 2

[Alex Deucher]

This adds the core infrastructure for supporting UALink (Ultra Accelerator Link)
connected scale up pods. I've split this into two series, one to add the core
infrastructure, and one to expose the new IOCTL interface and add the
documentation to avoid spamming the larger audience with the implemenation
defails.  This is the second series.

The first series can be found here:
https://lists.freedesktop.org/archives/amd-gfx/2026-August/151200.html

Both series can be accessed via this git branch:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/ualink?ref_type=heads

This implements UALoE (UALink over Ethernet). An overview of the complete
solution can be found here:
https://www.amd.com/en/products/rackscale-solutions/helios.html

Overview

Connected GPUs in a pod can directly access the remove memory on another
GPU over UALink.  Unlike RMDA, there is no copy involved; it is direct
loads/stores over the fabric.  Shared memory can only be accessed by
a remote GPU if the memory was exported and the importer has been authorized.
For the memory to be shared, it must be part of a unified physical
address space shared between nodes.  This address space is called NPA (Nework
Physical Address) space.  This address space is partitioned between
the GPUs so that each GPU has it's own segment of the address space in which
to export its memory.  Each GPU maintains a dedicated set of page tables
for their NPA space similar to GPUVM.

Exported memory is not pinned.  The exporter may need to move memory
or make it unavailable to the importers.  This is handled via remote
TLB shootdowns on the exporter in the case of TTM evictions or MMU notifiers and
remote presense checks with retry on the importer.

To access remote memory, the driver can map NPA addresses into its per
process GPUVM page tables just like local memory.  Applications use
opaque handles to represent remote memory.  GPUs in a pod communicate
with eachother directly to exchange NPA addresses between importers
and exporters.  If a node goes offline or is reset, their peers will
clean up any remaining refrences that are lost when that happens.

User interface

Export Memory

To export memory, a handle must be created for an allocation
that can be shared with another node in the pod.  To do this
the exporter calls the GEM UALink IOCTL with the GEM handle
to the buffer it wants to export.  The IOCTL returns a
unique handle which can be shared with the remote host.

Import Memory

To import the memory, the handle from the remote node must be converted
from a unique handle to a local GEM object which represents the local reference
to the NPA space on the importer.  If the memory has already been
imported, it just returns a new reference to the existing object.  If not,
the importer queries the exporter to get the NPA address.  Once it has that
the importer can create a dma-buf to represent the NPA space used by the
allocation and that is returned to the application.

Proposed Userspace
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L971
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036

Alex Deucher (1):
  Documentation: add initial UALink Documentation

Mukul Joshi (2):
  drm/gem: Add callback for when handle count goes to 0
  drm/amdgpu: Add ioctl infra for exporting/importing UALink handles

 Documentation/gpu/amdgpu/index.rst         |  1 +
 Documentation/gpu/amdgpu/ualink.rst        | 74 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 +++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  2 +
 drivers/gpu/drm/drm_gem.c                  |  5 +-
 include/drm/drm_gem.h                      | 11 ++++
 include/uapi/drm/amdgpu_drm.h              | 30 +++++++++
 9 files changed, 177 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/gpu/amdgpu/ualink.rst

-- 
2.55.0

[PATCH 1/3] drm/gem: Add callback for when handle count goes to 0

[Alex Deucher]

From: Mukul Joshi <mukul.joshi@amd.com>

Add an optional callback for driver-specific cleanup when the GEM
handle of an object is freed. This will be used by AMDGPU to enable
freeing of memory exported to other nodes in a UALink pod once all
user mode references are gone.

The callback is called outside the object_name_lock and before
releasing the reference count on the GEM object

Suggested-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c |  9 +++++++++
 drivers/gpu/drm/drm_gem.c               |  5 ++++-
 include/drm/drm_gem.h                   | 11 +++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index f754a4a3a1c22..0d579517c03ce 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -386,6 +386,14 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str
 	return drm_gem_ttm_mmap(obj, vma);
 }
 
+static void amdgpu_gem_object_handle_free(struct drm_gem_object *gobj)
+{
+	struct amdgpu_bo *aobj = gem_to_amdgpu_bo(gobj);
+
+	amdgpu_ualink_revoke_exported_memory(aobj);
+
+}
+
 const struct drm_gem_object_funcs amdgpu_gem_object_funcs = {
 	.free = amdgpu_gem_object_free,
 	.open = amdgpu_gem_object_open,
@@ -395,6 +403,7 @@ const struct drm_gem_object_funcs amdgpu_gem_object_funcs = {
 	.vunmap = drm_gem_ttm_vunmap,
 	.mmap = amdgpu_gem_object_mmap,
 	.vm_ops = &amdgpu_gem_vm_ops,
+	.handle_free = amdgpu_gem_object_handle_free
 };
 
 static bool amdgpu_gem_are_domains_valid(u32 domains)
diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
index e3ed684ddcf29..6a86bd2a0343e 100644
--- a/drivers/gpu/drm/drm_gem.c
+++ b/drivers/gpu/drm/drm_gem.c
@@ -354,8 +354,11 @@ void drm_gem_object_handle_put_unlocked(struct drm_gem_object *obj)
 	}
 	mutex_unlock(&dev->object_name_lock);
 
-	if (final)
+	if (final) {
+		if (obj->funcs->handle_free)
+			obj->funcs->handle_free(obj);
 		drm_gem_object_put(obj);
+	}
 }
 
 /*
diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h
index 8a704f6a65c15..95d8ae6f85df7 100644
--- a/include/drm/drm_gem.h
+++ b/include/drm/drm_gem.h
@@ -227,6 +227,17 @@ struct drm_gem_object_funcs {
 	 */
 	size_t (*rss)(struct drm_gem_object *obj);
 
+	/**
+	 * @handle_free:
+	 *
+	 * This callback is called when the GEM handle count goes down to 0.
+	 * It is currently used by AMDGPU driver to release their exported BO
+	 * handles.
+	 *
+	 * This callback is optional.
+	 */
+	void (*handle_free)(struct drm_gem_object *obj);
+
 	/**
 	 * @vm_ops:
 	 *
-- 
2.55.0

[PATCH 2/3] drm/amdgpu: Add ioctl infra for exporting/importing UALink handles

[Alex Deucher]

From: Mukul Joshi <mukul.joshi@amd.com>

Add the ioctl infrastructure to support exporting and importing BOs
to facilitate NPA based memory sharing across GPUs in a rack scale
setup.

Proposed userspace:
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L971
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036

v2: Move the ioctl wire-up to the end of the series.

Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Signed-off-by: Horatio Zhang <hongkun.zhang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  2 +
 include/uapi/drm/amdgpu_drm.h              | 30 +++++++++++++++
 4 files changed, 78 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 04b21e456fbc0..25430ff39a6c0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -54,6 +54,7 @@
 #include "amdgpu_userq.h"
 #include "amdgpu_userq_fence.h"
 #include "../amdxcp/amdgpu_xcp_drv.h"
+#include "amdgpu_ualink.h"
 
 /*
  * KMS wrapper.
@@ -3119,6 +3120,7 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = {
 	DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(AMDGPU_GEM_LIST_HANDLES, amdgpu_gem_list_handles_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(AMDGPU_PROC_OPTIONS, amdgpu_proc_options_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
+	DRM_IOCTL_DEF_DRV(AMDGPU_UALINK_HANDLE, amdgpu_gem_ualink_handle_ioctl, DRM_AUTH|DRM_RENDER_ALLOW)
 };
 
 static const struct drm_driver amdgpu_kms_driver = {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
index 402d8941fb045..63374c6b9ebd5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
@@ -3963,6 +3963,50 @@ int amdgpu_ualink_export_handle(struct drm_device *dev, struct drm_file *filp,
 	return r;
 }
 
+int amdgpu_gem_ualink_handle_ioctl(struct drm_device *dev, void *data,
+				   struct drm_file *filp)
+{
+	union drm_amdgpu_ualink_handle *args = data;
+	struct amdgpu_device *adev = drm_to_adev(dev);
+	struct amdgpu_ualink_handle handle = {};
+	u32 gem_handle;
+	int r, fd = -1;
+
+	if (adev->ualink.info->accel_state !=
+	    AMDGPU_UALINK_ACCEL_STATE_ACTIVE) {
+		dev_err(adev->dev,
+			"ualink device is not in active state in vpod\n");
+		return -EOPNOTSUPP;
+	}
+
+	/* The input and output members of the ioctl argument alias each other.
+	 * Latch every input field before invoking the handlers, and only write
+	 * the output fields afterwards.
+	 */
+	switch (args->in.op) {
+	case DRM_AMDGPU_UALINK_HANDLE_OP_EXPORT:
+		gem_handle = args->in.gem_handle;
+		r = amdgpu_ualink_export_handle(dev, filp, gem_handle, &handle);
+		if (!r) {
+			args->out.export_ualink_handle[0] = handle.handle_lo;
+			args->out.export_ualink_handle[1] = handle.handle_hi;
+		}
+		break;
+	case DRM_AMDGPU_UALINK_HANDLE_OP_IMPORT:
+		handle.handle_lo = args->in.import_ualink_handle[0];
+		handle.handle_hi = args->in.import_ualink_handle[1];
+		r = amdgpu_ualink_import_handle(dev, &handle, &fd);
+		if (!r)
+			args->out.import_dmabuf_handle = fd;
+		break;
+	default:
+		r = -EINVAL;
+		break;
+	}
+
+	return r;
+}
+
 int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 {
 	int i, r;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h
index 63710b484c6a5..9908d3a277172 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h
@@ -409,6 +409,8 @@ int amdgpu_ualink_export_handle(struct drm_device *dev, struct drm_file *filp,
 int amdgpu_ualink_import_handle(struct drm_device *dev,
 				const struct amdgpu_ualink_handle *ualink_handle,
 				int *fd_out);
+int amdgpu_gem_ualink_handle_ioctl(struct drm_device *dev, void *data,
+				   struct drm_file *filp);
 void amdgpu_ualink_revoke_exported_memory(struct amdgpu_bo *bo);
 
 int ualink_ip_hw_init(struct amdgpu_ip_block *ip_block);
diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index d6d5402a1e789..0e8115673d8d2 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -59,6 +59,7 @@ extern "C" {
 #define DRM_AMDGPU_USERQ_WAIT		0x18
 #define DRM_AMDGPU_GEM_LIST_HANDLES	0x19
 #define DRM_AMDGPU_PROC_OPTIONS		0x1A
+#define DRM_AMDGPU_UALINK_HANDLE	0x1B
 
 #define DRM_IOCTL_AMDGPU_GEM_CREATE	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
 #define DRM_IOCTL_AMDGPU_GEM_MMAP	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)
@@ -81,6 +82,7 @@ extern "C" {
 #define DRM_IOCTL_AMDGPU_USERQ_WAIT	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_USERQ_WAIT, struct drm_amdgpu_userq_wait)
 #define DRM_IOCTL_AMDGPU_GEM_LIST_HANDLES DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_LIST_HANDLES, struct drm_amdgpu_gem_list_handles)
 #define DRM_IOCTL_AMDGPU_PROC_OPTIONS	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_PROC_OPTIONS, struct drm_amdgpu_proc_options)
+#define DRM_IOCTL_AMDGPU_UALINK_HANDLE DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_UALINK_HANDLE, union drm_amdgpu_ualink_handle)
 
 /**
  * DOC: memory domains
@@ -1705,6 +1707,34 @@ struct drm_amdgpu_proc_options {
 	} kfd_sigbus_delay;
 };
 
+#define DRM_AMDGPU_UALINK_HANDLE_OP_EXPORT		0
+#define DRM_AMDGPU_UALINK_HANDLE_OP_IMPORT		1
+
+struct drm_amdgpu_ualink_handle_in {
+	/* Export or import */
+	__u32 op;
+	/* For future use, no flags defined so far */
+	__u32 flags;
+	union {
+		/* GEM handle of the BO to export */
+		__u32 gem_handle;
+		/* UALink handle to import */
+		__u64 import_ualink_handle[2];
+	};
+};
+
+union drm_amdgpu_ualink_handle_out {
+	/* Exported UALink handle */
+	__u64 export_ualink_handle[2];
+	/** DMABuf representing the imported UALink handle */
+	__u32 import_dmabuf_handle;
+};
+
+union drm_amdgpu_ualink_handle {
+	struct drm_amdgpu_ualink_handle_in in;
+	union drm_amdgpu_ualink_handle_out out;
+};
+
 #if defined(__cplusplus)
 }
 #endif
-- 
2.55.0

[PATCH 3/3] Documentation: add initial UALink Documentation

[Alex Deucher]

Document the details of UALink on the GPU.

Cc: Felix.Kuehling@amd.com
Cc: Joseph.Greathouse@amd.com
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 Documentation/gpu/amdgpu/index.rst  |  1 +
 Documentation/gpu/amdgpu/ualink.rst | 74 +++++++++++++++++++++++++++++
 2 files changed, 75 insertions(+)
 create mode 100644 Documentation/gpu/amdgpu/ualink.rst

diff --git a/Documentation/gpu/amdgpu/index.rst b/Documentation/gpu/amdgpu/index.rst
index b2ab182236efb..ba2ee73278672 100644
--- a/Documentation/gpu/amdgpu/index.rst
+++ b/Documentation/gpu/amdgpu/index.rst
@@ -23,4 +23,5 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures.
    debugfs
    process-isolation
    amdgpu-glossary
+   ualink
    ptl
diff --git a/Documentation/gpu/amdgpu/ualink.rst b/Documentation/gpu/amdgpu/ualink.rst
new file mode 100644
index 0000000000000..1e0a6c52b7b7a
--- /dev/null
+++ b/Documentation/gpu/amdgpu/ualink.rst
@@ -0,0 +1,74 @@
+==============
+UALink Support
+==============
+
+Overview
+========
+
+Connected GPUs in a pod can directly access the remove memory on another GPU
+over UALink.  Unlike RMDA, there is no copy involved; it is direct loads/stores
+over the fabric.  Shared memory can only be accessed by a remote GPU if the
+memory was exported and the importer has been authorized. For the memory to be
+shared, it must be part of a unified physical address space shared between
+nodes.  This address space is called NPA (Nework Physical Address) space.  This
+address space is partitioned between the GPUs so that each GPU has its own
+segment of the address space in which to export its memory.  Each GPU maintains
+a dedicated set of page tables for their NPA space similar to GPUVM.  Note that
+this mechanism only allows for GPU access to remote memory.  The remote memory
+is not CPU accessible.
+
+Exported memory is not pinned.  The exporter may need to move memory or make it
+unavailable to the importers.  This is handled via remote TLB shootdowns on the
+exporter in the case of TTM evictions or MMU notifiers and remote presense
+checks with retry on the importer.
+
+To access remote memory, the driver can map NPA addresses into its per process
+GPUVM page tables just like local memory.  Applications use opaque handles to
+represent remote memory.  GPUs in a pod communicate with eachother directly to
+exchange NPA addresses between importers and exporters.  If a node goes offline
+or is reset, their peers will clean up any remaining refrences that are lost
+when that happens.
+
+On the importer, the NPA space is like another physical address space. NPA
+addresses can be used as physical addresses for GPUVM to provide GPU virtual
+addresses to the memory for processes using the GPU.
+
+On the exporter, the NPA space provides a way to expose discontiguous local
+memory as a contiguous address range for remote GPUs.  This allows the exporter
+to locally manage the pages mapped into the NPA space.
+
+Remote NPAs are managed like another device specific TTM pool similar to
+doorbells or VRAM, however they cannot be CPU mapped.
+
+
+User Interface
+==============
+Two IOCTLs are provided to export and import remote memory.
+
+Export Memory
+-------------
+To export memory, a UALINK handle must be created for an allocation that can be
+shared with another node in the pod.  To do this the exporter calls the GEM
+UALink IOCTL with the GEM handle to the buffer it wants to export.  The IOCTL
+returns a unique 128 bit handle which can be shared with the remote host.
+Calling export on the same GEM handle always returns the same UALink handle.
+The UALink handle is destroyed when the GEM object is freed.
+
+Import Memory
+-------------
+To import remote memory, the UALink handle from the remote node must be
+converted from a UALink handle to a local GEM object which represents the local
+reference to the NPA space on the importer.  If the memory has already been
+imported, it just returns a new reference to the existing GEM object.  If not,
+the importer queries the exporter to get the NPA address.  Once it has that, the
+importer can create the GEM to represent the NPA space used by the allocation.
+The GEM object is then exported to the caller as a dma-buf. The dma-buf is
+leveraged for dynamic attachment which provides the ability to revoke access
+when necessary.
+
+
+Device to Device Communications
+===============================
+
+Devices communicate via a protocol implemented in firmware.  Mesages sent to a
+remote node generate an interrupt on that node for servicing.
-- 
2.55.0

Re: [PATCH 3/3] Documentation: add initial UALink Documentation

[Felix Kuehling]

On 2026-08-21 15:52, Alex Deucher wrote:
> Document the details of UALink on the GPU.
>
> Cc: Felix.Kuehling@amd.com
> Cc: Joseph.Greathouse@amd.com
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>   Documentation/gpu/amdgpu/index.rst  |  1 +
>   Documentation/gpu/amdgpu/ualink.rst | 74 +++++++++++++++++++++++++++++
>   2 files changed, 75 insertions(+)
>   create mode 100644 Documentation/gpu/amdgpu/ualink.rst
>
> diff --git a/Documentation/gpu/amdgpu/index.rst b/Documentation/gpu/amdgpu/index.rst
> index b2ab182236efb..ba2ee73278672 100644
> --- a/Documentation/gpu/amdgpu/index.rst
> +++ b/Documentation/gpu/amdgpu/index.rst
> @@ -23,4 +23,5 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures.
>      debugfs
>      process-isolation
>      amdgpu-glossary
> +   ualink
>      ptl
> diff --git a/Documentation/gpu/amdgpu/ualink.rst b/Documentation/gpu/amdgpu/ualink.rst
> new file mode 100644
> index 0000000000000..1e0a6c52b7b7a
> --- /dev/null
> +++ b/Documentation/gpu/amdgpu/ualink.rst
> @@ -0,0 +1,74 @@
> +==============
> +UALink Support
> +==============
> +
> +Overview
> +========
> +
> +Connected GPUs in a pod can directly access the remove memory on another GPU
> +over UALink.  Unlike RMDA, there is no copy involved; it is direct loads/stores
> +over the fabric.  Shared memory can only be accessed by a remote GPU if the
> +memory was exported and the importer has been authorized. For the memory to be
> +shared, it must be part of a unified physical address space shared between
> +nodes.  This address space is called NPA (Nework Physical Address) space.  This
> +address space is partitioned between the GPUs so that each GPU has its own
> +segment of the address space in which to export its memory.  Each GPU maintains
> +a dedicated set of page tables for their NPA space similar to GPUVM.  Note that
> +this mechanism only allows for GPU access to remote memory.  The remote memory
> +is not CPU accessible.
> +
> +Exported memory is not pinned.  The exporter may need to move memory or make it
> +unavailable to the importers.  This is handled via remote TLB shootdowns on the
> +exporter in the case of TTM evictions or MMU notifiers and remote presense
> +checks with retry on the importer.

This is no longer true. We ended up having to pin exported memory. This 
was also discussed with Christian already. In principle it's similar to 
how DMABufs in VRAM are pinned for RDMA access.

Remote TLB shootdown will only be used when exported memory is freed in 
order to remove access by remote GPUs.


> +
> +To access remote memory, the driver can map NPA addresses into its per process
> +GPUVM page tables just like local memory.  Applications use opaque handles to
> +represent remote memory.  GPUs in a pod communicate with eachother directly to
> +exchange NPA addresses between importers and exporters.  If a node goes offline
> +or is reset, their peers will clean up any remaining refrences that are lost
> +when that happens.

One important point here is, that NPA addresses are exchanged between 
the kernel mode drivers (through the scale-up fabric). NPA addresses are 
never given to user mode.


> +
> +On the importer, the NPA space is like another physical address space. NPA
> +addresses can be used as physical addresses for GPUVM to provide GPU virtual
> +addresses to the memory for processes using the GPU.
> +
> +On the exporter, the NPA space provides a way to expose discontiguous local
> +memory as a contiguous address range for remote GPUs.  This allows the exporter
> +to locally manage the pages mapped into the NPA space.
> +
> +Remote NPAs are managed like another device specific TTM pool similar to
> +doorbells or VRAM, however they cannot be CPU mapped.
> +
> +
> +User Interface
> +==============
> +Two IOCTLs are provided to export and import remote memory.
> +
> +Export Memory
> +-------------
> +To export memory, a UALINK handle must be created for an allocation that can be
> +shared with another node in the pod.  To do this the exporter calls the GEM
> +UALink IOCTL with the GEM handle to the buffer it wants to export.  The IOCTL
> +returns a unique 128 bit handle which can be shared with the remote host.
> +Calling export on the same GEM handle always returns the same UALink handle.
> +The UALink handle is destroyed when the GEM object is freed.

More precisely, the UALink handle is destroyed when the GEM handle count 
reaches 0.

Regards,
   Felix


> +
> +Import Memory
> +-------------
> +To import remote memory, the UALink handle from the remote node must be
> +converted from a UALink handle to a local GEM object which represents the local
> +reference to the NPA space on the importer.  If the memory has already been
> +imported, it just returns a new reference to the existing GEM object.  If not,
> +the importer queries the exporter to get the NPA address.  Once it has that, the
> +importer can create the GEM to represent the NPA space used by the allocation.
> +The GEM object is then exported to the caller as a dma-buf. The dma-buf is
> +leveraged for dynamic attachment which provides the ability to revoke access
> +when necessary.
> +
> +
> +Device to Device Communications
> +===============================
> +
> +Devices communicate via a protocol implemented in firmware.  Mesages sent to a
> +remote node generate an interrupt on that node for servicing.

Re: [PATCH 3/3] Documentation: add initial UALink Documentation

[Alex Deucher]

On Fri, Aug 21, 2026 at 5:06 PM Felix Kuehling <felix.kuehling@amd.com> wrote:
>
> On 2026-08-21 15:52, Alex Deucher wrote:
> > Document the details of UALink on the GPU.
> >
> > Cc: Felix.Kuehling@amd.com
> > Cc: Joseph.Greathouse@amd.com
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> > ---
> >   Documentation/gpu/amdgpu/index.rst  |  1 +
> >   Documentation/gpu/amdgpu/ualink.rst | 74 +++++++++++++++++++++++++++++
> >   2 files changed, 75 insertions(+)
> >   create mode 100644 Documentation/gpu/amdgpu/ualink.rst
> >
> > diff --git a/Documentation/gpu/amdgpu/index.rst b/Documentation/gpu/amdgpu/index.rst
> > index b2ab182236efb..ba2ee73278672 100644
> > --- a/Documentation/gpu/amdgpu/index.rst
> > +++ b/Documentation/gpu/amdgpu/index.rst
> > @@ -23,4 +23,5 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures.
> >      debugfs
> >      process-isolation
> >      amdgpu-glossary
> > +   ualink
> >      ptl
> > diff --git a/Documentation/gpu/amdgpu/ualink.rst b/Documentation/gpu/amdgpu/ualink.rst
> > new file mode 100644
> > index 0000000000000..1e0a6c52b7b7a
> > --- /dev/null
> > +++ b/Documentation/gpu/amdgpu/ualink.rst
> > @@ -0,0 +1,74 @@
> > +==============
> > +UALink Support
> > +==============
> > +
> > +Overview
> > +========
> > +
> > +Connected GPUs in a pod can directly access the remove memory on another GPU
> > +over UALink.  Unlike RMDA, there is no copy involved; it is direct loads/stores
> > +over the fabric.  Shared memory can only be accessed by a remote GPU if the
> > +memory was exported and the importer has been authorized. For the memory to be
> > +shared, it must be part of a unified physical address space shared between
> > +nodes.  This address space is called NPA (Nework Physical Address) space.  This
> > +address space is partitioned between the GPUs so that each GPU has its own
> > +segment of the address space in which to export its memory.  Each GPU maintains
> > +a dedicated set of page tables for their NPA space similar to GPUVM.  Note that
> > +this mechanism only allows for GPU access to remote memory.  The remote memory
> > +is not CPU accessible.
> > +
> > +Exported memory is not pinned.  The exporter may need to move memory or make it
> > +unavailable to the importers.  This is handled via remote TLB shootdowns on the
> > +exporter in the case of TTM evictions or MMU notifiers and remote presense
> > +checks with retry on the importer.
>
> This is no longer true. We ended up having to pin exported memory. This
> was also discussed with Christian already. In principle it's similar to
> how DMABufs in VRAM are pinned for RDMA access.
>
> Remote TLB shootdown will only be used when exported memory is freed in
> order to remove access by remote GPUs.
>
>
> > +
> > +To access remote memory, the driver can map NPA addresses into its per process
> > +GPUVM page tables just like local memory.  Applications use opaque handles to
> > +represent remote memory.  GPUs in a pod communicate with eachother directly to
> > +exchange NPA addresses between importers and exporters.  If a node goes offline
> > +or is reset, their peers will clean up any remaining refrences that are lost
> > +when that happens.
>
> One important point here is, that NPA addresses are exchanged between
> the kernel mode drivers (through the scale-up fabric). NPA addresses are
> never given to user mode.
>
>
> > +
> > +On the importer, the NPA space is like another physical address space. NPA
> > +addresses can be used as physical addresses for GPUVM to provide GPU virtual
> > +addresses to the memory for processes using the GPU.
> > +
> > +On the exporter, the NPA space provides a way to expose discontiguous local
> > +memory as a contiguous address range for remote GPUs.  This allows the exporter
> > +to locally manage the pages mapped into the NPA space.
> > +
> > +Remote NPAs are managed like another device specific TTM pool similar to
> > +doorbells or VRAM, however they cannot be CPU mapped.
> > +
> > +
> > +User Interface
> > +==============
> > +Two IOCTLs are provided to export and import remote memory.
> > +
> > +Export Memory
> > +-------------
> > +To export memory, a UALINK handle must be created for an allocation that can be
> > +shared with another node in the pod.  To do this the exporter calls the GEM
> > +UALink IOCTL with the GEM handle to the buffer it wants to export.  The IOCTL
> > +returns a unique 128 bit handle which can be shared with the remote host.
> > +Calling export on the same GEM handle always returns the same UALink handle.
> > +The UALink handle is destroyed when the GEM object is freed.
>
> More precisely, the UALink handle is destroyed when the GEM handle count
> reaches 0.

Thanks.  I've incorporated your comments.

Alex

>
> Regards,
>    Felix
>
>
> > +
> > +Import Memory
> > +-------------
> > +To import remote memory, the UALink handle from the remote node must be
> > +converted from a UALink handle to a local GEM object which represents the local
> > +reference to the NPA space on the importer.  If the memory has already been
> > +imported, it just returns a new reference to the existing GEM object.  If not,
> > +the importer queries the exporter to get the NPA address.  Once it has that, the
> > +importer can create the GEM to represent the NPA space used by the allocation.
> > +The GEM object is then exported to the caller as a dma-buf. The dma-buf is
> > +leveraged for dynamic attachment which provides the ability to revoke access
> > +when necessary.
> > +
> > +
> > +Device to Device Communications
> > +===============================
> > +
> > +Devices communicate via a protocol implemented in firmware.  Mesages sent to a
> > +remote node generate an interrupt on that node for servicing.

[PATCH 3/3] Documentation: add initial UALink Documentation

[Alex Deucher]

Document the details of UALink on the GPU.

v2: add updates from Felix

Cc: Felix.Kuehling@amd.com
Cc: Joseph.Greathouse@amd.com
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 Documentation/gpu/amdgpu/index.rst  |  1 +
 Documentation/gpu/amdgpu/ualink.rst | 75 +++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)
 create mode 100644 Documentation/gpu/amdgpu/ualink.rst

diff --git a/Documentation/gpu/amdgpu/index.rst b/Documentation/gpu/amdgpu/index.rst
index b2ab182236efb..ba2ee73278672 100644
--- a/Documentation/gpu/amdgpu/index.rst
+++ b/Documentation/gpu/amdgpu/index.rst
@@ -23,4 +23,5 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures.
    debugfs
    process-isolation
    amdgpu-glossary
+   ualink
    ptl
diff --git a/Documentation/gpu/amdgpu/ualink.rst b/Documentation/gpu/amdgpu/ualink.rst
new file mode 100644
index 0000000000000..e15c8621e5a1a
--- /dev/null
+++ b/Documentation/gpu/amdgpu/ualink.rst
@@ -0,0 +1,75 @@
+==============
+UALink Support
+==============
+
+Overview
+========
+
+Connected GPUs in a pod can directly access the remove memory on another GPU
+over UALink.  Unlike RMDA, there is no copy involved; it is direct loads/stores
+over the fabric.  Shared memory can only be accessed by a remote GPU if the
+memory was exported and the importer has been authorized. For the memory to be
+shared, it must be part of a unified physical address space shared between
+nodes.  This address space is called NPA (Nework Physical Address) space.  This
+address space is partitioned between the GPUs so that each GPU has its own
+segment of the address space in which to export its memory.  Each GPU maintains
+a dedicated set of page tables for their NPA space similar to GPUVM.  Note that
+this mechanism only allows for GPU access to remote memory.  The remote memory
+is not CPU accessible.
+
+Exported memory is pinned.  This is similar to how dma-bufs in VRAM are pinned
+for P2P access.  Remote TLB shootdowns from the exporter are used when the
+exported memory is freed in order to remove access by remote GPUs.
+
+To access remote memory, the driver can map NPA addresses into its per process
+GPUVM page tables just like local memory.  Applications use opaque handles to
+represent remote memory.  GPUs in a pod communicate with eachother directly to
+exchange NPA addresses between importers and exporters.  If a node goes offline
+or is reset, their peers will clean up any remaining refrences that are lost
+when that happens.  NPA addresses are exchanged directly between the kernel
+drivers using the scale up fabric.  NPA addresses are never exposed to user
+space.
+
+On the importer, the NPA space is like another physical address space. NPA
+addresses can be used as physical addresses for GPUVM to provide GPU virtual
+addresses to the memory for processes using the GPU.
+
+On the exporter, the NPA space provides a way to expose discontiguous local
+memory as a contiguous address range for remote GPUs.  This allows the exporter
+to locally manage the pages mapped into the NPA space.
+
+Remote NPAs are managed like another device specific TTM pool similar to
+doorbells or VRAM, however they cannot be CPU mapped.
+
+
+User Interface
+==============
+Two IOCTLs are provided to export and import remote memory.
+
+Export Memory
+-------------
+To export memory, a UALINK handle must be created for an allocation that can be
+shared with another node in the pod.  To do this the exporter calls the GEM
+UALink IOCTL with the GEM handle to the buffer it wants to export.  The IOCTL
+returns a unique 128 bit handle which can be shared with the remote host.
+Calling export on the same GEM handle always returns the same UALink handle.
+The UALink handle is destroyed when the GEM object reference count reaches 0.
+
+Import Memory
+-------------
+To import remote memory, the UALink handle from the remote node must be
+converted from a UALink handle to a local GEM object which represents the local
+reference to the NPA space on the importer.  If the memory has already been
+imported, it just returns a new reference to the existing GEM object.  If not,
+the importer queries the exporter to get the NPA address.  Once it has that, the
+importer can create the GEM to represent the NPA space used by the allocation.
+The GEM object is then exported to the caller as a dma-buf. The dma-buf is
+leveraged for dynamic attachment which provides the ability to revoke access
+when necessary.
+
+
+Device to Device Communications
+===============================
+
+Devices communicate via a protocol implemented in firmware.  Mesages sent to a
+remote node generate an interrupt on that node for servicing.
-- 
2.55.0

[PATCH 3/3] Documentation: add initial UALink Documentation

[Alex Deucher]

Document the details of UALink on the GPU.

v2: add updates from Felix

Cc: Felix.Kuehling@amd.com
Cc: Joseph.Greathouse@amd.com
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 Documentation/gpu/amdgpu/index.rst  |  1 +
 Documentation/gpu/amdgpu/ualink.rst | 75 +++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)
 create mode 100644 Documentation/gpu/amdgpu/ualink.rst

diff --git a/Documentation/gpu/amdgpu/index.rst b/Documentation/gpu/amdgpu/index.rst
index b2ab182236efb..ba2ee73278672 100644
--- a/Documentation/gpu/amdgpu/index.rst
+++ b/Documentation/gpu/amdgpu/index.rst
@@ -23,4 +23,5 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures.
    debugfs
    process-isolation
    amdgpu-glossary
+   ualink
    ptl
diff --git a/Documentation/gpu/amdgpu/ualink.rst b/Documentation/gpu/amdgpu/ualink.rst
new file mode 100644
index 0000000000000..e15c8621e5a1a
--- /dev/null
+++ b/Documentation/gpu/amdgpu/ualink.rst
@@ -0,0 +1,75 @@
+==============
+UALink Support
+==============
+
+Overview
+========
+
+Connected GPUs in a pod can directly access the remove memory on another GPU
+over UALink.  Unlike RMDA, there is no copy involved; it is direct loads/stores
+over the fabric.  Shared memory can only be accessed by a remote GPU if the
+memory was exported and the importer has been authorized. For the memory to be
+shared, it must be part of a unified physical address space shared between
+nodes.  This address space is called NPA (Nework Physical Address) space.  This
+address space is partitioned between the GPUs so that each GPU has its own
+segment of the address space in which to export its memory.  Each GPU maintains
+a dedicated set of page tables for their NPA space similar to GPUVM.  Note that
+this mechanism only allows for GPU access to remote memory.  The remote memory
+is not CPU accessible.
+
+Exported memory is pinned.  This is similar to how dma-bufs in VRAM are pinned
+for P2P access.  Remote TLB shootdowns from the exporter are used when the
+exported memory is freed in order to remove access by remote GPUs.
+
+To access remote memory, the driver can map NPA addresses into its per process
+GPUVM page tables just like local memory.  Applications use opaque handles to
+represent remote memory.  GPUs in a pod communicate with eachother directly to
+exchange NPA addresses between importers and exporters.  If a node goes offline
+or is reset, their peers will clean up any remaining refrences that are lost
+when that happens.  NPA addresses are exchanged directly between the kernel
+drivers using the scale up fabric.  NPA addresses are never exposed to user
+space.
+
+On the importer, the NPA space is like another physical address space. NPA
+addresses can be used as physical addresses for GPUVM to provide GPU virtual
+addresses to the memory for processes using the GPU.
+
+On the exporter, the NPA space provides a way to expose discontiguous local
+memory as a contiguous address range for remote GPUs.  This allows the exporter
+to locally manage the pages mapped into the NPA space.
+
+Remote NPAs are managed like another device specific TTM pool similar to
+doorbells or VRAM, however they cannot be CPU mapped.
+
+
+User Interface
+==============
+Two IOCTLs are provided to export and import remote memory.
+
+Export Memory
+-------------
+To export memory, a UALINK handle must be created for an allocation that can be
+shared with another node in the pod.  To do this the exporter calls the GEM
+UALink IOCTL with the GEM handle to the buffer it wants to export.  The IOCTL
+returns a unique 128 bit handle which can be shared with the remote host.
+Calling export on the same GEM handle always returns the same UALink handle.
+The UALink handle is destroyed when the GEM object reference count reaches 0.
+
+Import Memory
+-------------
+To import remote memory, the UALink handle from the remote node must be
+converted from a UALink handle to a local GEM object which represents the local
+reference to the NPA space on the importer.  If the memory has already been
+imported, it just returns a new reference to the existing GEM object.  If not,
+the importer queries the exporter to get the NPA address.  Once it has that, the
+importer can create the GEM to represent the NPA space used by the allocation.
+The GEM object is then exported to the caller as a dma-buf. The dma-buf is
+leveraged for dynamic attachment which provides the ability to revoke access
+when necessary.
+
+
+Device to Device Communications
+===============================
+
+Devices communicate via a protocol implemented in firmware.  Mesages sent to a
+remote node generate an interrupt on that node for servicing.
-- 
2.55.0

Re: [PATCH 0/3] Add UALink infrastructure series 2

[Dave Airlie]

> https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036
>
> Alex Deucher (1):
>   Documentation: add initial UALink Documentation
>
> Mukul Joshi (2):
>   drm/gem: Add callback for when handle count goes to 0
>   drm/amdgpu: Add ioctl infra for exporting/importing UALink handles
>
>  Documentation/gpu/amdgpu/index.rst         |  1 +
>  Documentation/gpu/amdgpu/ualink.rst        | 74 ++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  2 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |  9 +++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 +++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  2 +
>  drivers/gpu/drm/drm_gem.c                  |  5 +-
>  include/drm/drm_gem.h                      | 11 ++++

This gem patch is kinda sneaking in here, I'd have preferred to have
more discussion on the list around this sort of addition to the gem
code.

Dave.

RE: [PATCH 0/3] Add UALink infrastructure series 2

[Deucher, Alexander]

Public

> -----Original Message-----
> From: Dave Airlie <airlied@gmail.com>
> Sent: Sunday, September 27, 2026 5:17 PM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: amd-gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org; Kuehling,
> Felix <Felix.Kuehling@amd.com>
> Subject: Re: [PATCH 0/3] Add UALink infrastructure series 2
>
> > https://github.com/ROCm/rocm-
> systems/blob/35959f8e1260c7cee3e51a740e32
> > 0be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036
> >
> > Alex Deucher (1):
> >   Documentation: add initial UALink Documentation
> >
> > Mukul Joshi (2):
> >   drm/gem: Add callback for when handle count goes to 0
> >   drm/amdgpu: Add ioctl infra for exporting/importing UALink handles
> >
> >  Documentation/gpu/amdgpu/index.rst         |  1 +
> >  Documentation/gpu/amdgpu/ualink.rst        | 74
> ++++++++++++++++++++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  2 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |  9 +++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 +++++++++++++
> > drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  2 +
> >  drivers/gpu/drm/drm_gem.c                  |  5 +-
> >  include/drm/drm_gem.h                      | 11 ++++
>
> This gem patch is kinda sneaking in here, I'd have preferred to have more
> discussion on the list around this sort of addition to the gem code.
>

Sorry about that.  That was mainly why I sent these patches to the wider audience in the first place, but I ended up spending so much time writing up the overview of UALink, I completely forgot to call out the gem change as well.

Alex