Re: [PATCH v3 00/25] CPU binds and ULLS on migration queue

Summers, Stuart
2026-03-05T22:56:57Z

One question I have reading through some of the ULLS patches, why do we
need to do this in the kernel? This adds quite a bit of complexity here
that IMO might be a better fit for userspace migration, particularly
for the userspace that already handles ULLS like L0. What is the
benefit of doing this in the kernel vs adding a new API to allow some
kind of opt in for migration in the UMD?

The CPU binding generally even outside of the ULLS piece makes sense to
me, so I don't think this is blocking particularly. But it would be
nice to have a little more detail on the above before we move forward
here.

Thanks,
Stuart

On Fri, 2026-02-27 at 17:34 -0800, Matthew Brost wrote:
> We now have data demonstrating the need for CPU binds and ULLS on the
> migration queue, based on results generated from [1].
>
> On BMG, measurements show that when the GPU is continuously
> processing
> faults, copy jobs run approximately 30–40µs faster (depending on the
> test case) with ULLS compared to traditional GuC submission with SLPC
> enabled on the migration queue. Startup from a cold GPU shows an even
> larger speedup. Given the critical nature of fault performance, ULLS
> appears to be a worthwhile feature.
>
> In addition to driver telemetry, UMD compute benchmarks consistently
> show over 1GB/s improvement in pagefault benchmarks with ULLS
> enabled.
>
> ULLS will consume more power (not yet measured) due to a continuously
> running batch on the paging engine. However, compute UMDs already do
> this on engines exposed to users, so this seems like a worthwhile
> tradeoff. To mitigate power concerns, ULLS will exit after a period
> of
> time in which no faults have been processed.
>
> CPU binds are required for ULLS to function, as the migration queue
> needs exclusive access to the paging hardware engine. Thus, CPU binds
> are included here.
>
> Beyond being a requirement for ULLS, CPU binds should also reduce
> VM-bind latency, provide clearer multi-tile and TLB-invalidation
> layering, reduce pressure on GuC during fault storms as it is
> bypassed,
> and decouple kernel binds from unrelated copy/clear jobs—especially
> beneficial when faults are serviced in parallel. In a parallel-
> faulting
> test case, average bind time was reduced by approximately 15µs. In
> the
> worst case, 2MB copy time (~60–140µs) × (number of pagefault threads
> −
> 1) of latency would otherwise be added to a single fault. Reducing
> this
> latency increases overall throughput of the fault handler.
>
> This series can be merged in phases:
>
> Phase 1: CPU binds (patches 1–13)
> Phase 2: CPU-bind components and multi-tile relayers (patches 14–17)
> Phase 3: ULLS on the migration execution queue (patches 18–25)
>
> v2:
>  - Use delayed worker to exit ULLS mode in an effort to save on power
>  - Various other cleanups
> v3:
>  - CPU bind component, multi-tile relayer
>  - Split CPU bind patches in many small patches
>
> Matt
>
> [1] https://patchwork.freedesktop.org/series/149811/
>
> Matthew Brost (25):
>   drm/xe: Drop struct xe_migrate_pt_update argument from
> populate/clear
>     vfuns
>   drm/xe: Add xe_migrate_update_pgtables_cpu_execute helper
>   drm/xe: Decouple exec queue idle check from LRC
>   drm/xe: Add job count to GuC exec queue snapshot
>   drm/xe: Update xe_bo_put_deferred arguments to include writeback
> flag
>   drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
>   drm/xe: Update scheduler job layer to support PT jobs
>   drm/xe: Add helpers to access PT ops
>   drm/xe: Add struct xe_pt_job_ops
>   drm/xe: Update GuC submission backend to run PT jobs
>   drm/xe: Store level in struct xe_vm_pgtable_update
>   drm/xe: Don't use migrate exec queue for page fault binds
>   drm/xe: Enable CPU binds for jobs
>   drm/xe: Remove unused arguments from xe_migrate_pt_update_ops
>   drm/xe: Make bind queues operate cross-tile
>   drm/xe: Add CPU bind layer
>   drm/xe: Add device flag to enable PT mirroring across tiles
>   drm/xe: Add xe_hw_engine_write_ring_tail
>   drm/xe: Add ULLS support to LRC
>   drm/xe: Add ULLS migration job support to migration layer
>   drm/xe: Add MI_SEMAPHORE_WAIT instruction defs
>   drm/xe: Add ULLS migration job support to ring ops
>   drm/xe: Add ULLS migration job support to GuC submission
>   drm/xe: Enter ULLS for migration jobs upon page fault or SVM
> prefetch
>   drm/xe: Add modparam to enable / disable ULLS on migrate queue
>
>  drivers/gpu/drm/xe/Makefile                   |   1 +
>  .../gpu/drm/xe/instructions/xe_mi_commands.h  |   6 +
>  drivers/gpu/drm/xe/xe_bo.c                    |   8 +-
>  drivers/gpu/drm/xe/xe_bo.h                    |  11 +-
>  drivers/gpu/drm/xe/xe_bo_types.h              |   2 -
>  drivers/gpu/drm/xe/xe_cpu_bind.c              | 296 +++++++
>  drivers/gpu/drm/xe/xe_cpu_bind.h              | 118 +++
>  drivers/gpu/drm/xe/xe_debugfs.c               |   1 +
>  drivers/gpu/drm/xe/xe_defaults.h              |   1 +
>  drivers/gpu/drm/xe/xe_device.c                |  17 +-
>  drivers/gpu/drm/xe/xe_device_types.h          |  11 +
>  drivers/gpu/drm/xe/xe_drm_client.c            |   2 +-
>  drivers/gpu/drm/xe/xe_exec_queue.c            | 163 ++--
>  drivers/gpu/drm/xe/xe_exec_queue.h            |  18 +-
>  drivers/gpu/drm/xe/xe_exec_queue_types.h      |  21 +-
>  drivers/gpu/drm/xe/xe_guc_submit.c            |  82 +-
>  drivers/gpu/drm/xe/xe_guc_submit_types.h      |   2 +
>  drivers/gpu/drm/xe/xe_hw_engine.c             |  10 +
>  drivers/gpu/drm/xe/xe_hw_engine.h             |   1 +
>  drivers/gpu/drm/xe/xe_lrc.c                   |  51 ++
>  drivers/gpu/drm/xe/xe_lrc.h                   |   3 +
>  drivers/gpu/drm/xe/xe_lrc_types.h             |   4 +
>  drivers/gpu/drm/xe/xe_migrate.c               | 585 +++++--------
>  drivers/gpu/drm/xe/xe_migrate.h               |  93 +--
>  drivers/gpu/drm/xe/xe_module.c                |   4 +
>  drivers/gpu/drm/xe/xe_module.h                |   1 +
>  drivers/gpu/drm/xe/xe_pagefault.c             |   3 +
>  drivers/gpu/drm/xe/xe_pci.c                   |   2 +
>  drivers/gpu/drm/xe/xe_pci_types.h             |   1 +
>  drivers/gpu/drm/xe/xe_pt.c                    | 773 +++++++++++-----
> --
>  drivers/gpu/drm/xe/xe_pt.h                    |  12 +-
>  drivers/gpu/drm/xe/xe_pt_types.h              |  49 +-
>  drivers/gpu/drm/xe/xe_ring_ops.c              |  31 +
>  drivers/gpu/drm/xe/xe_sched_job.c             | 100 ++-
>  drivers/gpu/drm/xe/xe_sched_job_types.h       |  36 +-
>  drivers/gpu/drm/xe/xe_sync.c                  |  20 +-
>  drivers/gpu/drm/xe/xe_tlb_inval_job.c         |  28 +-
>  drivers/gpu/drm/xe/xe_tlb_inval_job.h         |   4 +-
>  drivers/gpu/drm/xe/xe_vm.c                    | 241 +++---
>  drivers/gpu/drm/xe/xe_vm.h                    |   3 +
>  drivers/gpu/drm/xe/xe_vm_types.h              |  22 +-
>  41 files changed, 1658 insertions(+), 1179 deletions(-)
>  create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.c
>  create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.h
>