Re: [PATCH 00/95] Add UALink instrastructure series 1

[Philip Yang <yangp@amd.com>]

[-- Attachment #1: Type: text/plain, Size: 12384 bytes --]

Two nitpicks below.

Regards,
Philip

On 2026-08-21 15:33, Alex Deucher wrote:
> This adds the core infrastructure for supporting UALink (Ultra Accelerator Link)
> connected scale up pods. I've split this into two series, one to add the core
> infrastructure, and one to expose the new IOCTL interface and add the
> documentation to avoid spamming the larger audience with the implemenation
> defails.  This is the first series.
>
> This implements UALoE (UALink over Ethernet). An overview of the complete
> solution can be found here:
> https://www.amd.com/en/products/rackscale-solutions/helios.html
>
> Overview
>
> Connected GPUs in a pod can directly access the remove memory on another
s/remove/remote/
> GPU over UALink.  Unlike RMDA, there is no copy involved; it is direct
> loads/stores over the fabric.  Shared memory can only be accessed by
> a remote GPU if the memory was exported and the importer has been authorized.
> For the memory to be shared, it must be part of a unified physical
> address space shared between nodes.  This address space is called NPA (Nework
> Physical Address) space.  This address space is partitioned between
> the GPUs so that each GPU has it's own segment of the address space in which
> to export its memory.  Each GPU maintains a dedicated set of page tables
> for their NPA space similar to GPUVM.
>
> Exported memory is not pinned.  The exporter may need to move memory
> or make it unavailable to the importers.  This is handled via remote
> TLB shootdowns on the exporter in the case of TTM evictions or MMU notifiers and
> remote presense checks with retry on the importer.
>
> To access remote memory, the driver can map NPA addresses into its per
> process GPUVM page tables just like local memory.  Applications use
> opaque handles to represent remote memory.  GPUs in a pod communicate
> with eachother directly to exchange NPA addresses between importers
s/eachother/each other/
> and exporters.  If a node goes offline or is reset, their peers will
> clean up any remaining refrences that are lost when that happens.
>
> User interface
>
> Export Memory
>
> To export memory, a handle must be created for an allocation
> that can be shared with another node in the pod.  To do this
> the exporter calls the GEM UALink IOCTL with the GEM handle
> to the buffer it wants to export.  The IOCTL returns a
> unique handle which can be shared with the remote host.
>
> Import Memory
>
> To import the memory, the handle from the remote node must be converted
> from a unique handle to a local GEM object which represents the local reference
> to the NPA space on the importer.  If the memory has already been
> imported, it just returns a new reference to the existing object.  If not,
> the importer queries the exporter to get the NPA address.  Once it has that
> the importer can create a dma-buf to represent the NPA space used by the
> allocation and that is returned to the application.
>
> Alex Sierra (1):
>    drm/amdgpu: log remote memory MTYPE for GC 12.1.0
>
> Felix Kuehling (17):
>    drm/amdgpu: Add sysfs API for UALink information
>    drm/amdgpu: Add sysfs API for UALink physical pod setup
>    drm/amdgpu: Add sysfs API for UALink virtual pod config
>    drm/amdgpu: Add sysfs API for UALink station configuration
>    drm/amdgpu: Implement PSP cmd UAL_GET_CONFIG
>    drm/amdgpu: Query initial UALink config from PSP
>    drm/amdgpu: Implement PSP cmd UAL_SET_PPOD_CONFIG
>    drm/amdgpu: Set physical pod configuration to PSP
>    drm/amdgpu: Implement PSP cmd UAL_SET_VPOD_CONFIG
>    drm/amdgpu: Set virtual pod configuration to PSP
>    drm/amdgpu: Implement PSP cmd UAL_SET_STATION_CONFIG
>    drm/amdgpu: Set UALink station config to PSP
>    drm/amdgpu: Implement PSP cmd UAL_SET_NPA_CONFIG
>    drm/amdgpu: Enable/disable NPA address translation using PSP
>    drm/amdgpu: Prevent double-free of drm_exec
>    drm/amdgpu: Pin page tables in NPA VMs
>    drm/amdgpu: Initialize NPA PT/PDs to noretry
>
> Gangliang Xie (1):
>    drm/amdgpu: add handler for nHT error
>
> Harish Kasiviswanathan (2):
>    drm/amdgpu: Add helper function to check psp xgmi ta
>    drm/amdgpu: Add ual_config_state to ual_get_config
>
> Jinzhou Su (1):
>    drm/amdgpu: Fix NULL pointer issue during ualink init
>
> Le Ma (1):
>    drm/amdgpu: extend PSP command polling sleep range
>
> Lijo Lazar (15):
>    drm/amdgpu: Add psp ualink command interfaces
>    drm/amdgpu: Fetch asp ualink interface version
>    drm/amdgpu: Separate out ualink init sequences
>    drm/amdgpu: Add ualink as separate ip block
>    drm/admgpu: Seggregate ualink nht messaging
>    drm/amdgpu: Assign accel state based on ASP config
>    drm/amdgpu: Drop duplicate vpod check functions
>    drm/amdgpu: Add support to send ASP completion
>    drm/amdgpu: Add handlers for ualink notifications
>    drm/amdgpu: Improve ualink state transitions
>    drm/amdgpu: Use uniform logic for inband/sideband
>    drm/amdgpu: Add name for ualink ip block
>    drm/amdgpu: Move ualink ip version related changes
>    drm/amdgpu: Add hw_fini for ualink
>    drm/amdgpu: Expose ualink info under each xcp
>
> Mukul Joshi (42):
>    drm/amdgpu: Add UALink manager core infrastructure
>    drm/amdgpu: Add a new NPA Address space
>    drm/amdgpu: Add address allocator for NPA addresses
>    drm/amdgpu: Initialize VM for NPA addr management
>    drm/amdgpu: Rework VMID reservation logic
>    drm/amdgpu: Reserve VMID for NPA VM
>    drm/amdgpu: Use reserved VMID for NPA VM
>    drm/amdgpu: Enable UALink Manager when pod becomes active
>    drm/amdgpu: Create a drm client for UALink NPA BOs
>    drm/amdgpu: Control NPA DMA-buf importing
>    drm/amdgpu: Add ualink handle to BOs
>    drm/amdgpu: Implement UALink handle export
>    drm/amdgpu: Add connection state management
>    drm/amdgpu: Implement UALink handle import ioctl
>    drm/amdgpu: Implement mechanism to revoke exported memory
>    drm/amdgpu: lock UALink import invalidation via drm_exec
>    drm/amdgpu: Cleanup exported UALink handles
>    drm/amdgpu: Cleanup imported UALink handles
>    drm/amdgpu: Handle connection reset
>    drm/amdgpu: Setup PTE mappings for NPA addresses
>    drm/amdgpu: Add handling for remote interrupts
>    drm/amdgpu: Send TLB shootdown on exported memory unmap
>    drm/amdgpu: Handle local GPUs in UALink import
>    drm/amdgpu: Add debugfs to drop UALink protocol messages
>    drm/amdgpu: Temporarily disable sending remote TLB shootdowns
>    drm/amdgpu: Temporarily Flush TLB on NPA mapping always
>    drm/amdgpu: fix NPA-RELEASE race in UALink exporter cleanup
>    drm/amdgpu: initialize UALink importer node list head
>    drm/amdgpu: Fix initialization flags for UALink XAs
>    drm/amdgpu: fix dma_buf leak in UALink exporter cleanup
>    drm/amdgpu: Fix uninitialized fence in UALink NPA unmap
>    drm/amdgpu: Use vm->last_update fence in UALink NPA unmap paths
>    drm/amdgpu: always use MTYPE_UC for remote memory on GFX 12.1
>    drm/amdkfd: program compute MQD coherent_aql_mtype on GFX 12.1
>    drm/amdgpu: Fix GART and SDMA entity leak on vPod reconfiguration
>    drm/amdgpu: Add UALink diagnostic logging for vpod commit/activation
>    drm/amdgpu: Handle UALink vPod reconfiguration while ACTIVE
>    drm/amdgpu: Cleanup UALink XA entries on manager stop
>    drm/amdgpu: Handle concurrent UALINK handle import race
>    drm/amdgpu: create UALink NPA import BO directly in the NPA domain
>    drm/amdgpu: add mtype_remote module parameter
>    drm/amdgpu: Honor mtype overrides for NPA remote memory
>
> Philip Yang (15):
>    drm/amdgpu: Fix UALink vPod double-activation
>    drm/amdgpu: Add UALink remote state structures and API declarations
>    drm/amdgpu: Add UALink NPA address layout helpers
>    drm/amdgpu: Add UALink NPA address computation for ring buffers
>    drm/amdgpu: Add UALink NPA VM mapping for ring buffers
>    drm/amdgpu: Add UALink SDMA scheduler entities
>    drm/amdgpu: Add UALink GART helpers for NPA address access
>    drm/amdgpu: Add UALink ring buffer allocation and firmware init
>    drm/amdgpu: Add UALink remote command packets and SDMA dispatch
>    drm/amdgpu: Add UALink firmware writeback address configuration
>    drm/amdgpu: Add UALink cross-GPU TLB shootdown and remote interrupt
>    drm/amdgpu: Add UALink software init, teardown, and reset
>    drm/amdgpu: Add UALink IH ring and enable interrupt
>    drm/amdgpu: UALink use LSDMA to send remote interrupt command
>    drm/amdgpu: Increase UALink soft ring size
>
>   drivers/gpu/drm/amd/amdgpu/Makefile           |    3 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu.h           |    5 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c   |   14 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |    2 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c |   21 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    5 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |    7 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h       |    1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c       |   33 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h       |   27 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h        |    1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ip.c        |    1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c       |   17 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h       |    7 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |   11 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    5 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c       |  280 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h       |   27 +
>   .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h    |    1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   17 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    4 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c    | 6174 +++++++++++++++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h    |  421 ++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |   44 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |    4 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c     |   29 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c       |    6 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.h       |    7 +
>   drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c        |    2 +-
>   drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c        |    2 +-
>   drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c        |    2 +-
>   drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c        |   51 +-
>   drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c         |    2 +-
>   drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c         |    2 +-
>   drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c         |    2 +-
>   drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c       |    2 +-
>   drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c        |    4 +
>   drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c        |    5 +
>   drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c        |   26 +-
>   drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c        |   98 +-
>   drivers/gpu/drm/amd/amdgpu/gmc_v12_1.h        |    1 +
>   drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c         |    6 +
>   drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c         |    6 +
>   drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c         |    6 +
>   drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c         |    6 +
>   drivers/gpu/drm/amd/amdgpu/ih_v7_0.c          |    8 +
>   drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h       |  162 +
>   drivers/gpu/drm/amd/amdgpu/psp_v15_0_8.c      |   48 +-
>   drivers/gpu/drm/amd/amdgpu/soc_v1_0.c         |    6 -
>   drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c      |  143 +
>   drivers/gpu/drm/amd/amdgpu/ualink_v1_0.h      |   30 +
>   .../drm/amd/amdkfd/kfd_mqd_manager_v12_1.c    |    7 +
>   drivers/gpu/drm/amd/amdkfd/kfd_svm.c          |    8 +-
>   drivers/gpu/drm/amd/include/amd_shared.h      |    2 +
>   .../ivsrcid/mpnht/irqsrcs_mpnht_15_0.h        |   30 +
>   drivers/gpu/drm/amd/include/v12_structs.h     |    2 +-
>   include/drm/ttm/ttm_resource.h                |    2 +-
>   include/uapi/drm/amdgpu_drm.h                 |    6 +
>   58 files changed, 7782 insertions(+), 69 deletions(-)
>   create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
>   create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h
>   create mode 100644 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c
>   create mode 100644 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.h
>   create mode 100644 drivers/gpu/drm/amd/include/ivsrcid/mpnht/irqsrcs_mpnht_15_0.h
>

[-- Attachment #2: Type: text/html, Size: 12636 bytes --]