[PATCH 0/3] Spurious OOM with P2P dma-buf imports on multi-GPU systems

[Matthew Auld <matthew.auld@intel.com>]

On multi-GPU systems with large amounts of VRAM, huge multi-GPU
workloads can see device allocations fail with OOM even though there is
plenty of free VRAM on every card and host memory is largely unused.

The Level Zero runtime creates one context spanning all GPUs and, by
default, makes every device allocation resident on each P2P-capable
peer. Each allocation on one card is therefore exported as a dma-buf
and imported by every other card, so the amount of imported memory per
device grows with both the number of GPUs and the size of their VRAM.
Disabling P2P in the runtime makes the failures go away.

Investigating this with a standalone reproducer and some instrumentation
uncovered three independent issues, each of which can turn the P2P
import fan-out into an allocation failure:

1. Imports consume fake mmap offset space. TTM allocates a
   drm_vma_offset node for ttm_bo_type_sg BOs, but xe never allows CPU
   faults on imports. The offset space is 1 TiB per DRM device on
   64-bit, so with enough peer imports it runs out and the import fails
   with -ENOSPC.

2. Imports are charged against the host-RAM-sized TT budget. Each
   device's TT manager is sized to total host RAM, and imported BOs
   placed in XE_PL_TT are charged against it even though their pages
   belong to the exporter (peer VRAM in the P2P case). Once the
   combined size of the peer imports exceeds host RAM, binding further
   imports fails, while actual host memory use stays low.

3. Exporting a deferred-backing VRAM BO allocates host pages. A BO
   created with DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING starts in system
   placement with a ttm_tt. Its first move into VRAM has no source data
   and goes through ttm_bo_move_null(), which keeps the stale ttm_tt.
   ttm_bo_setup_export() then populates that ttm_tt on export, so every
   exported VRAM BO also pins as many host pages as its size, which are
   never used. These pages are neither accounted against the TT budget
   nor shrinkable, so with large exports the host runs out of memory
   and the OOM killer can be invoked. The runtime only uses deferred
   backing once VRAM use is above a threshold, so it is exactly the
   large allocations that are affected.

With all three patches applied the reproducer (based off customer
workload) passes with the runtime defaults, and host memory use stays
flat throughout.

Patches are posted for posterity and discussion, and not necessarily to
be merged as-is in upstream. Each issue can be fixed properly upstream
separately.

---
Himal Prasad Ghimiray (1):
      drm/xe: Exclude imported dma-buf BOs from the TT budget

Matthew Auld (1):
      drm/xe: don't allocate fake mmap offsets for imported dma-bufs

Thomas Hellström (1):
      drm/ttm, drm/xe: Drop the ttm_tt on null moves to VRAM

 drivers/gpu/drm/ttm/ttm_bo_util.c    | 20 +++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.c           | 12 +++++++++---
 drivers/gpu/drm/xe/xe_device_types.h |  6 ++++++
 drivers/gpu/drm/xe/xe_dma_buf.c      |  9 +++++++++
 drivers/gpu/drm/xe/xe_query.c        |  3 ++-
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c  | 38 ++++++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.h  |  3 +++
 include/drm/ttm/ttm_bo.h             |  2 ++
 8 files changed, 85 insertions(+), 8 deletions(-)
---
base-commit: 5cd64b7b38f36620fbc117c53b4866b0590a3a93
change-id: 20261107-oom-fun-20d9362a40cf