All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC 0/7] drm/ttm, drm/xe: Per-BO host NUMA node placement
@ 2026-11-02 10:18 Himal Prasad Ghimiray
  2026-11-02 10:18 ` [RFC 1/7] drm/ttm/pool: Add ttm_pool_alloc_node() for caller-selected NUMA node Himal Prasad Ghimiray
                   ` (9 more replies)
  0 siblings, 10 replies; 19+ messages in thread
From: Himal Prasad Ghimiray @ 2026-11-02 10:18 UTC (permalink / raw)
  To: Christian König, Huang Rui, Thomas Hellström,
	Rodrigo Vivi
  Cc: Matthew Auld, Matthew Brost, dri-devel, intel-xe,
	Himal Prasad Ghimiray

The backing pages of an Xe system memory BO are allocated on the device
local node (dev_to_node()), or on the allocating CPU's node when the
device has none. That is fine when CPU and GPU access a buffer from the
same place, but not in general:

- On multi-socket systems, and where the GPU's PCIe/CXL link attaches
  to a different node than the threads using a buffer, the cost of
  accessing a BO depends on which node backs its pages. The device
  local node is not always the best choice.
- A runtime often knows where a BO should live: the node of the CPU
  threads producing or consuming it, the node closest to the link the
  GPU uses, or a node chosen to balance memory bandwidth. Today it has
  no way to express that for driver allocated system memory.
- mbind()/set_mempolicy() do not help: BO pages are allocated by the
  driver, not through the task's fault path, and BO CPU mappings are
  VM_PFNMAP.

This series lets userspace select the host NUMA node per BO at
creation time.

UAPI:
- New DRM_XE_GEM_CREATE_SET_PROPERTY_NUMA_NODE property on the existing
  GEM_CREATE set-property extension.
- The node is a preference (best effort, like MPOL_PREFERRED); the
  allocation falls back to other nodes under memory pressure.
- A negative value, or not setting the property, keeps the current
  default placement.
- Invalid or offline nodes, and BOs that can't be placed in system
  memory, are rejected with -EINVAL.
- The placement is preserved across shrinker backup / restore.
- A richer placement policy can be added later as a separate extension
  without changing this one.

Why a node aware TTM allocation API rather than one ttm_pool per node
in the driver:

The NUMA node is a property of each allocation, not of the pool. TTM
already tracks pages per node (per-node LRUs in the pool caches,
per-node shrinker, per-node cache limits) and __ttm_pool_alloc()
already takes a node internally. ttm_pool_alloc_node() and
ttm_tt_restore_node() only let a caller pass that node in, following
the alloc_pages_node() / kmalloc_node() pattern, with no new state.

One pool per node in the driver would work for Xe today, but:
- TTM core helpers (ttm_tt_restore(), ttm_tt_backup(), pool debugfs)
  all use the device pool. The driver would have to bypass them for
  populate, restore, backup and free, and re-check that whenever TTM
  adds or changes a helper. The restore issue fixed in this series is
  an example: the placement was silently lost after backup.
- It relies on non dma_alloc pools having no per-pool state. For
  dma_alloc pools, N pools mean N separate page caches that can't
  share pages.
- Every TTM driver that wants per-BO placement would have to duplicate
  the same per-node pool handling.

The existing ttm_pool_alloc() and ttm_tt_restore() become wrappers
passing NUMA_NO_NODE, so current users are unaffected.

Himal Prasad Ghimiray (7):
  drm/ttm/pool: Add ttm_pool_alloc_node() for caller-selected NUMA node
  drm/ttm: Add NUMA node aware restore helpers
  drm/ttm/pool: enforce per-node cache limits on free
  drm/xe/bo: Plumb a per-BO host NUMA node hint into tt populate
  drm/xe/bo: Preserve the host NUMA node across backup / restore
  drm/xe/uapi: Add GEM_CREATE NUMA node placement property
  drm/xe/uapi: Wire up GEM_CREATE NUMA node placement handler

 drivers/gpu/drm/ttm/ttm_pool.c          | 104 +++++++++++++++++++---
 drivers/gpu/drm/ttm/ttm_pool_internal.h |   4 +
 drivers/gpu/drm/ttm/ttm_tt.c            |  20 ++++-
 drivers/gpu/drm/xe/tests/xe_bo.c        |  12 +--
 drivers/gpu/drm/xe/tests/xe_dma_buf.c   |   4 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c   |   8 +-
 drivers/gpu/drm/xe/xe_bo.c              | 109 +++++++++++++++++++-----
 drivers/gpu/drm/xe/xe_bo.h              |   5 +-
 drivers/gpu/drm/xe/xe_bo_types.h        |   7 ++
 drivers/gpu/drm/xe/xe_dma_buf.c         |   3 +-
 include/drm/ttm/ttm_pool.h              |   2 +
 include/drm/ttm/ttm_tt.h                |   2 +
 include/uapi/drm/xe_drm.h               |   7 ++
 13 files changed, 238 insertions(+), 49 deletions(-)

-- 
2.43.0


^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2026-11-02 18:24 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-02 10:18 [RFC 0/7] drm/ttm, drm/xe: Per-BO host NUMA node placement Himal Prasad Ghimiray
2026-11-02 10:18 ` [RFC 1/7] drm/ttm/pool: Add ttm_pool_alloc_node() for caller-selected NUMA node Himal Prasad Ghimiray
2026-11-02 10:26   ` sashiko-bot
2026-11-02 10:50   ` Christian König
2026-11-02 12:46     ` Thomas Hellström
2026-11-02 10:18 ` [RFC 2/7] drm/ttm: Add NUMA node aware restore helpers Himal Prasad Ghimiray
2026-11-02 10:26   ` sashiko-bot
2026-11-02 10:18 ` [RFC 3/7] drm/ttm/pool: enforce per-node cache limits on free Himal Prasad Ghimiray
2026-11-02 10:18 ` [RFC 4/7] drm/xe/bo: Plumb a per-BO host NUMA node hint into tt populate Himal Prasad Ghimiray
2026-11-02 10:18 ` [RFC 5/7] drm/xe/bo: Preserve the host NUMA node across backup / restore Himal Prasad Ghimiray
2026-11-02 10:18 ` [RFC 6/7] drm/xe/uapi: Add GEM_CREATE NUMA node placement property Himal Prasad Ghimiray
2026-11-02 10:27   ` sashiko-bot
2026-11-02 10:18 ` [RFC 7/7] drm/xe/uapi: Wire up GEM_CREATE NUMA node placement handler Himal Prasad Ghimiray
2026-11-02 10:29   ` sashiko-bot
2026-11-02 11:12   ` Matthew Auld
2026-11-02 18:24     ` Himal prasad Ghimiray
2026-11-02 12:44 ` ✓ CI.KUnit: success for drm/ttm, drm/xe: Per-BO host NUMA node placement Patchwork
2026-11-02 13:22 ` ✓ Xe.CI.BAT: " Patchwork
2026-11-02 14:55 ` ✓ Xe.CI.FULL: " Patchwork

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.