Re: [RFC PATCH v2 1/1] drm/xe: keep VM-bound WC BOs resident during reclaim

[Matthew Brost <matthew.brost@intel.com>]

On Wed, Aug 12, 2026 at 04:28:47PM +0800, Neil Zhong wrote:
> Hi Matt,
> 
> > Can you give me instructions on how to recreate this on our end and your
> > machine, memory details? I have a bunch of various reproducers which I
> > have been using for shrinker work and the more the better.
> 
> Yes. The test system and workload are as follows.
> 
>   - Panther Lake iGPU using shared system memory
>   - 8 GiB installed memory; /proc/meminfo reports 7,723,392 KiB
>   - 4 KiB base pages
>   - four zram swap devices, 3,858,416 KiB in total
>   - Linux 6.18.15, x86_64, PREEMPT_DYNAMIC
>   - X11 fullscreen output at 3840x2160 and 60 Hz
>   - a 3840x2160, 60 fps, HEVC HDR video played continuously in a loop
> 
> The player uses hardware decoding. Each decoded frame is imported through
> DMA-BUF and processed by libplacebo/OpenGL for HDR tone mapping before
> presentation. It calls glFlush() for every rendered frame. During steady
> playback, the player accounts for about 1.85 GiB of logical Xe BO
> allocation, including about 672 MiB reported as shared.
> 
> The player is not currently public. However, an equivalent pipeline should

This will become a problem if we introduce a uAPI, as we need an
open-source consumer of that uAPI. Fortunately, it looks like we are
already moving toward pinning uAPIs for several other reasons as well.

> reproduce the condition if it keeps the decoded surfaces, HDR intermediate
> render targets and presentation surfaces alive, rather than creating a
> small synthetic BO set. 
> 
> No additional memory-pressure tool was used for the first reproduction.
> I start the video, let its working set reach steady state, and then capture
> ten minutes while playback continues. On the 8 GiB system, the priority-only
> run had MemAvailable between 2.89 and 3.15 GiB, while 97.8% of kswapd wakeups
> were for order-10 allocations. Thus this reproduces without forcing an

We have upstream fixes for the order-10 allocations to avoid triggering
reclaim for anything other than order-0 or order-9 allocations. I shared
that patch in a previous reply, and it is probably worth pulling in as a
mitigation, plus the core MM series.

> order-0 shortage.
> 
> I collected the trace with:
> 
>   sudo ./capture_xe_memory_churn.sh \
>           -t 600 \
>           -s 0.2 \
>           -o ./xe-memory-churn
> 
> `-t 600` records ten minutes. `-t 0` can instead be used to record until
> Ctrl-C. The 0.2 second option is only the /proc and TTM-pool sampling
> interval; ftrace events are recorded continuously.
> 
> The player also logs one line after each frame submission in this form:
> 
>   gl_sw_submit_frame timing: ... Flush=123.456 ms
> 
> The log prefix contains the wall-clock timestamp. An equivalent reproducer
> can record the time immediately after glFlush() returns and the measured
> duration. The trace script inserts a wall-clock epoch marker into a
> mono_raw ftrace stream so that the two timelines can be aligned.
> 
> The figures in my previous email came from the 6.18.15 kernel with [9]-[13]
> backported, the fragmentation check applied to direct reclaim as well, and
> the high-plus-min watermark experiment described there. The default-device
> watermark capture used vm.min_free_kbytes=131072. The follow-up used 50000;
> the workload and trace procedure were otherwise unchanged.
> 
> > Also a quick write up how you extracted these numbers from reproducer so
> > I can recreate on my end.
> 
> The capture script creates temporary entry and return kprobes for:
> 
>   ttm_tt_backup()
>   ttm_tt_restore()
>   ttm_pool_alloc()
>   ttm_pool_free()
>   ttm_pool_shrink()
> 
> It also traces the Xe shrinker, TTM restore and cache-attribute functions,
> kswapd and direct-reclaim events, compaction, and allocation
> fragmentation.
> 
> At ttm_tt_backup() entry, the probe records the ttm_tt pointer and
> num_pages. At return, it records the positive return value, which is the
> number of pages actually backed up. I sum those successful return values,
> not the requested page count, when reporting backup volume. A later
> ttm_tt_restore() is matched to the most recent successful backup using the
> same ttm_tt pointer. That provides per-object backup-to-restore latency and
> repeated-cycle counts.
> 
> The byte-volume calculation is:
> 
>   backup bytes = sum(successful backup return pages) * PAGE_SIZE
>   restore bytes = sum(num_pages for matched restores) * PAGE_SIZE
> 
> The reported backup-plus-restore volume is the sum of those two values.
> It is cumulative migration/copy traffic, not resident memory and not net
> memory freed. A shmem backup remains resident system memory unless those
> shmem pages are subsequently swapped out.
> 
> > This customer, in particular, utilizes priority bands to express this
> > heuristic (e.g., the compositor is the highest priority, any
> > non-privileged UI-related content is normal priority, and everything
> > else is low priority).
> 
> I think priority bands are useful for relative reclaim ordering, but they
> do not by themselves express the guarantee needed here.
> 
> Consider a system with one large GPU workload. If nearly all reclaimable
> BOs belong to that client and are placed in the high-priority band, the low
> bands will contain few or no candidates. When enough memory is requested,
> the shrinker must eventually enter the high band. In that case, high
> priority delays reclaim but does not prevent it. The priority-only test
> showed exactly this limitation: over ten minutes there were 6,743
> successful backups and 6,693 restores, with 157 Flush calls over 16.667 ms
> and a maximum Flush of 494.434 ms.
> 
> I am not suggesting that every BO of a high-priority client should be
> unreclaimable. Whether reclaim is acceptable depends on the workload, and
> the kernel cannot infer that semantic from WC, VM-bound state, client count
> or BO size alone. For example, the driver-visible behavior of these two
> workloads can look very similar:
> 
>   1. Foreground 4K60 HDR playback. Its active decoded surfaces, HDR render
>      targets and presentation surfaces have a 16.667 ms deadline. Backing
>      them up and restoring them causes an immediate and clearly visible
>      product failure. These BOs should avoid eviction and shrinking while
>      they are part of the active visual pipeline.
> 
>   2. Background image recognition or classification. Its BOs have no
>      presentation deadline. Reclaiming them under system memory pressure
>      is reasonable, even if the job later has to reconstruct its working
>      set.

In this case, do you have two VMs sharing BOs? If the foreground and
background tasks share a single VM, it does not matter which BOs are
shrunk or restored because, when a VM is validated during an exec IOCTL,
all BOs mapped within that VM are restored.

I assume that in this case you are using two VMs that share buffers as
needed via dma-buf. Is that correct? If not, pinning is not going to
help unless the pinned set includes everything.

> 
> Priority bands cannot distinguish those cases if both clients assign their

Assuming there are two VMs here, set your foreground queue's priority to
NORMAL and your background task queue's priority to LOW. (Alternatively,
if the foreground task has CAP_SYS_ADMIN privileges, you could set it to
HIGH, etc.) This should cause the background BOs to be shrunk rather
than the foreground BOs.
 
There is another problem in Xe when VMs share BOs. The way a VM is
locked and validated during exec IOCTLs can cause cross-VM lock
contention get stuck behing shrinking. This issue exists regardless of
whether a priority-based solution or pinning is used.
 
For example, assume there are two VMs, one for the foreground workload
and one for the background workload. We correctly evict the background
BOs when needed, and some set of dma-buf buffers is shared between the
two VMs. Below a flow that shows a contention problem:

1. a private BO from background is shrunk
2. background exec IOCTL
	2.1. grab all dma-resv locks (including some shared with
	     foreground)
	2.2. restore shunk BO
	2.3. submit GPU job
3. foreground exec IOCTL (in parallel with 2)
	3.1 grab all dma-resv locks (including some shared with
	    background)
	3.2. submit GPU job

In this example 3.1 can get stuck behind 2.2 (an unrelated restore),
thus 3 can miss a presentation deadline.

We likely need our Xe IOCTL and GPUVM code to be smart enough to perform
multiple lock-and-validate passes. For example, we could initially lock
only the eviction set, validate everything, and then repeat the process
until the eviction set is empty. Only after that would we lock and
validate everything else.

This is an existing problem that really needs to be fixed. I'll probably
take a look at addressing it.

> current working set a high relative priority. This is why I think the
> business semantic has to come from userspace. Priority bands can remain the
> general ordering mechanism, while a separate, explicit and bounded
> NO_SHRINK or latency-critical mark protects only the BOs in an active visual
> pipeline. The mark should be removed as soon as a surface leaves that
> working set.
> 
> > To be clear - this would be an addition to fixes discussed above, right?
> 
> Yes. I see explicit workload-semantic protection as an addition to the
> core MM fragmentation/shrinker fixes and the Xe/TTM priority bands, not a
> replacement for either. I agree that testing series 168651 together with
> the priority bands is still useful for general working-set preservation.
> It can reduce accidental reclaim, while an accounted NO_SHRINK contract
> handles the smaller set for which a missed presentation deadline is not an
> acceptable reclaim tradeoff.

I don't think anyone is opposed to pinning if we can get the right
permission control in place (most likely cgroups).

Matt

> 
> Thanks,
> Neil