Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [RFC PATCH 00/19] drm, drm/xe: Multi-device GPUSVM
@ 2025-03-12 21:03 Thomas Hellström
  2025-03-12 21:03 ` [RFC PATCH 01/19] drm/xe: Introduce CONFIG_DRM_XE_GPUSVM Thomas Hellström
                   ` (27 more replies)
  0 siblings, 28 replies; 34+ messages in thread
From: Thomas Hellström @ 2025-03-12 21:03 UTC (permalink / raw)
  To: intel-xe, dri-devel
  Cc: Thomas Hellström, himal.prasad.ghimiray, apopple, airlied,
	Simona Vetter, felix.kuehling, Matthew Brost,
	Christian König, dakr, Mrozek, Michal, Joonas Lahtinen

This RFC implements and requests comments for a way to handle SVM with multi-device,
typically with fast interconnects. It adds generic code and helpers in drm, and
device-specific code for xe.

For SVM, devices set up maps of device-private struct pages, using a struct dev_pagemap,
The CPU virtual address space (mm), can then be set up using special page-table entries
to point to such pages, but they can't be accessed directly by the CPU, but possibly
by other devices using a fast interconnect. This series aims to provide helpers to
identify pagemaps that take part in such a fast interconnect and to aid in migrating
between them.

This is initially done by augmenting the struct dev_pagemap with a struct drm_pagemap,
and having the struct drm_pagemap implement a "populate_mm" method, where a region of
the CPU virtual address space (mm) is populated with device_private pages from the
dev_pagemap associated with the drm_pagemap, migrating data from system memory or other
devices if necessary. The drm_pagemap_populate_mm() function is then typically called
from a fault handler, using the struct drm_pagemap pointer of choice. It could be
referencing a local drm_pagemap or a remote one. The migration is now completely done
by drm_pagemap callbacks, (typically using a copy-engine local to the dev_pagemap local
memory).

In addition there are helpers to build a drm_pagemap UAPI using file-descripors
representing struct drm_pagemaps, and a helper to register devices with a common
fast interconnect. The UAPI is intended to be private to the device, but if drivers
agree to identify struct drm_pagemaps by file descriptors one could in theory
do cross-driver multi-device SVM if a use-case were found.

The implementation for the Xe driver uses dynamic pagemaps which are created on first
use and removed 5s after the last reference is gone. Pagemaps are revoked on
device unbind, and data is then migrated to system.

Status:
This is a POC series. It has been tested with an IGT test soon to be published, with a
DG1 drm_pagemap and a BattleMage SVM client. There is separate work ongoing for the
gpu_madvise functionality.

The Xe implementation of the "populate_mm()" callback is
still rudimentary and doesn't migrate from foreign devices. It should be tuned to do
smarter choices.

Any feedback appreciated.

Patch overview:
Patch 1:
- Extends the way the Xe driver can compile out SVM support and pagemaps.
Patch 2:
- Fixes an existing potential UAF in the Xe SVM code.
Patch 3:
- Introduces the drm_pagemap.c file and moves drm_pagemap functionality to it.
Patch 4:
- Adds a populate_mm op to drm_pagemap.
Patch 5:
- Implement Xe's version of the populate_mm op.
Patch 6:
- Refcount struct drm_pagemap.
Patch 7:
- Cleanup patch.
Patch 8:
- Add a bo_remove callback for Xe, Used during device unbind.
Patch 9:
- Add a drm_pagemap utility to calculate a common owner structure
Patch 10:
- Adopt GPUSVM to a (sort of) dynamic owner.
Patch 11:
- Xe calculates the dev_private owner using the drm_pagemap utility.
Patch 12:
- Update the Xe page-table code to handle per range mixed system / device_private placement.
Patch 13:
- Modify GPUSVM to allow such placements.
Patch 14:
- Add a preferred pagemap to use by the Xe fault handler.
Patch 15:
- Add a utility that converts between drm_pagemaps and file-descriptors and back.
Patch 16:
- Fix Xe so that also devices without fault capability can publish drm_pagemaps.
Patch 17:
- Add the devmem_open UAPI, creating a drm_pagemap file descriptor from a
  (device, region) pair.
Patch 18:
- (Only for POC) Add an GPU madvise prefer_devmem IOCTL.
Patch 19:
- (Only for POC) Implement pcie p2p DMA as a fast interconnect and test.

Matthew Brost (1):
  drm/gpusvm, drm/pagemap: Move migration functionality to drm_pagemap

Thomas Hellström (18):
  drm/xe: Introduce CONFIG_DRM_XE_GPUSVM
  drm/xe/svm: Fix a potential bo UAF
  drm/pagemap: Add a populate_mm op
  drm/xe: Implement and use the drm_pagemap populate_mm op
  drm/pagemap, drm/xe: Add refcounting to struct drm_pagemap and manage
    lifetime
  drm/pagemap: Get rid of the struct
    drm_pagemap_zdd::device_private_page_owner field
  drm/xe/bo: Add a bo remove callback
  drm/pagemap_util: Add a utility to assign an owner to a set of
    interconnected gpus
  drm/gpusvm, drm/xe: Move the device private owner to the
    drm_gpusvm_ctx
  drm/xe: Use the drm_pagemap_util helper to get a svm pagemap owner
  drm/xe: Make the PT code handle placement per PTE rather than per vma
    / range
  drm/gpusvm: Allow mixed mappings
  drm/xe: Add a preferred dpagemap
  drm/pagemap/util: Add file descriptors pointing to struct drm_pagemap
  drm/xe/migrate: Allow xe_migrate_vram() also on non-pagefault capable
    devices
  drm/xe/uapi: Add the devmem_open ioctl
  drm/xe/uapi: HAX: Add the xe_madvise_prefer_devmem IOCTL
  drm/xe: HAX: Use pcie p2p dma to test fast interconnect

 Documentation/gpu/rfc/gpusvm.rst     |  12 +-
 drivers/gpu/drm/Makefile             |   7 +-
 drivers/gpu/drm/drm_gpusvm.c         | 782 +---------------------
 drivers/gpu/drm/drm_pagemap.c        | 940 +++++++++++++++++++++++++++
 drivers/gpu/drm/drm_pagemap_util.c   | 203 ++++++
 drivers/gpu/drm/xe/Kconfig           |  24 +-
 drivers/gpu/drm/xe/Makefile          |   2 +-
 drivers/gpu/drm/xe/xe_bo.c           |  65 +-
 drivers/gpu/drm/xe/xe_bo.h           |   2 +
 drivers/gpu/drm/xe/xe_bo_types.h     |   2 +-
 drivers/gpu/drm/xe/xe_device.c       |   8 +
 drivers/gpu/drm/xe/xe_device_types.h |  30 +-
 drivers/gpu/drm/xe/xe_migrate.c      |   8 +-
 drivers/gpu/drm/xe/xe_pt.c           | 112 ++--
 drivers/gpu/drm/xe/xe_query.c        |   2 +-
 drivers/gpu/drm/xe/xe_svm.c          | 716 +++++++++++++++++---
 drivers/gpu/drm/xe/xe_svm.h          | 158 ++++-
 drivers/gpu/drm/xe/xe_tile.c         |  20 +-
 drivers/gpu/drm/xe/xe_tile.h         |  33 +
 drivers/gpu/drm/xe/xe_vm.c           |   6 +-
 drivers/gpu/drm/xe/xe_vm_types.h     |   7 +
 include/drm/drm_gpusvm.h             | 102 +--
 include/drm/drm_pagemap.h            | 190 +++++-
 include/drm/drm_pagemap_util.h       |  59 ++
 include/uapi/drm/xe_drm.h            |  39 ++
 25 files changed, 2458 insertions(+), 1071 deletions(-)
 create mode 100644 drivers/gpu/drm/drm_pagemap.c
 create mode 100644 drivers/gpu/drm/drm_pagemap_util.c
 create mode 100644 include/drm/drm_pagemap_util.h

-- 
2.48.1


^ permalink raw reply	[flat|nested] 34+ messages in thread

end of thread, other threads:[~2025-03-17  9:21 UTC | newest]

Thread overview: 34+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-03-12 21:03 [RFC PATCH 00/19] drm, drm/xe: Multi-device GPUSVM Thomas Hellström
2025-03-12 21:03 ` [RFC PATCH 01/19] drm/xe: Introduce CONFIG_DRM_XE_GPUSVM Thomas Hellström
2025-03-12 21:03 ` [RFC PATCH 02/19] drm/xe/svm: Fix a potential bo UAF Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 03/19] drm/gpusvm, drm/pagemap: Move migration functionality to drm_pagemap Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 04/19] drm/pagemap: Add a populate_mm op Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 05/19] drm/xe: Implement and use the drm_pagemap " Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 06/19] drm/pagemap, drm/xe: Add refcounting to struct drm_pagemap and manage lifetime Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 07/19] drm/pagemap: Get rid of the struct drm_pagemap_zdd::device_private_page_owner field Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 08/19] drm/xe/bo: Add a bo remove callback Thomas Hellström
2025-03-14 13:05   ` Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 09/19] drm/pagemap_util: Add a utility to assign an owner to a set of interconnected gpus Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 10/19] drm/gpusvm, drm/xe: Move the device private owner to the drm_gpusvm_ctx Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 11/19] drm/xe: Use the drm_pagemap_util helper to get a svm pagemap owner Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 12/19] drm/xe: Make the PT code handle placement per PTE rather than per vma / range Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 13/19] drm/gpusvm: Allow mixed mappings Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 14/19] drm/xe: Add a preferred dpagemap Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 15/19] drm/pagemap/util: Add file descriptors pointing to struct drm_pagemap Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 16/19] drm/xe/migrate: Allow xe_migrate_vram() also on non-pagefault capable devices Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 17/19] drm/xe/uapi: Add the devmem_open ioctl Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 18/19] drm/xe/uapi: HAX: Add the xe_madvise_prefer_devmem IOCTL Thomas Hellström
2025-03-12 21:04 ` [RFC PATCH 19/19] drm/xe: HAX: Use pcie p2p dma to test fast interconnect Thomas Hellström
2025-03-12 21:10 ` ✓ CI.Patch_applied: success for drm, drm/xe: Multi-device GPUSVM Patchwork
2025-03-12 21:11 ` ✗ CI.checkpatch: warning " Patchwork
2025-03-12 21:12 ` ✓ CI.KUnit: success " Patchwork
2025-03-12 21:29 ` ✓ CI.Build: " Patchwork
2025-03-12 21:31 ` ✗ CI.Hooks: failure " Patchwork
2025-03-12 21:33 ` ✓ CI.checksparse: success " Patchwork
2025-03-12 22:06 ` ✗ Xe.CI.BAT: failure " Patchwork
2025-03-13 10:19 ` [RFC PATCH 00/19] " Christian König
2025-03-13 12:50   ` Thomas Hellström
2025-03-13 12:57     ` Christian König
2025-03-13 15:55       ` Thomas Hellström
2025-03-17  9:20       ` Thomas Hellström
2025-03-13 13:24 ` ✗ Xe.CI.Full: failure for " Patchwork

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox