Re: [PATCH v5 0/4] drm/xe/forcewake: add delayed-release optimization

["Bai, Zongyao" <zongyao.bai@intel.com>]

On 9/3/2026 11:59 AM, Umesh Nerlige Ramappa wrote:
> Hi Zongyao,
>
> On Thu, Aug 13, 2026 at 12:06:50AM +0000, Zongyao Bai wrote:
>> Engine-cycle queries used by tools such as VTune and PTI can arrive in
>> bursts. Releasing forcewake immediately after every query makes the next
>> query issue another wake request and wait for its ACK.
>>
>> Add an opt-in delayed-release path that keeps an idle forcewake domain
>> awake for a configurable interval, allowing a subsequent engine-cycle
>> query to reuse it. Other forcewake users retain the existing immediate
>> release behavior. The default hold interval is 100 us and can be changed
>> through configfs before binding the device.
>>
>> The series also synchronizes engine-cycle queries with unplug and 
>> flushes
>> pending delayed releases before suspend, shutdown, remove, and probe
>> unwind, ensuring timer callbacks cannot access MMIO after power-off or
>> teardown.
>>
>> On BMG, this reduced the Level Zero timestamp median from approximately
>> 48.5 us to 9.4 us.
>
> Can you please add some more info here on what this measurement is and 
> how this is affecting the VTune/PTI use cases? In other words, what 
> breaks if we don't have these changes?
>
> The purpose of engine_cycles query is just to capture CPU and GPU 
> timestamps as close to each other as possible and then the user would 
> use that to map the two timelines in whatever they are trying to 
> correlate, so I am interested in understanding the use case that this 
> is improving.
>
> Thanks,
> Umesh
Hi Umesh,

The main issue without this patch is the latency introduced when the XE 
forcewake domain goes to sleep immediately after a query completes.

In VTune/PTI workloads, APIs such as zeDeviceGetGlobalTimestamps() may 
be called at a very high frequency to correlate CPU and GPU timelines. 
It's not a one time query.
Since the forcewake domain is released right after the MMIO read, the 
next query often arrives after only a very short interval and must go 
through the full forcewake acquisition and MMIO read sequence again.
On bmg, we measured that this additional forcewake overhead can be 
around 48.5 µs, even with RC6 disabled.

This patch introduces a delayed forcewake release for the engine-cycle 
query path:DRM_XE_DEVICE_QUERY_ENGINE_CYCLES;
Instead of immediately releasing the forcewake domain after the read, we 
keep it active for a configurable period.
Currently 100 µs by default, chosen as a reasonable default balancing 
power and reuse; tunable via configfs.

With this change, after a zeDeviceGetGlobalTimestamps() call, the 
forcewake domain remains active for a short period.
If another query arrives within that window, it can access the registers 
directly without repeating the forcewake sequence.
The timer is then re-armed for subsequent accesses.

For the forcewake state transitions, please refer to the diagram in: 
drivers/gpu/drm/xe/xe_force_wake_types.h
And more details can be found in GSD-11225.

Thanks,
Zongyao
>>
>> v5:
>> - Split the original patch into four patches: unplug synchronization,
>>   delayed-release infrastructure, power-boundary flushing, and opt-in
>>   enablement/configuration.
>> - Preserve a newer hold interval when an older timer callback is 
>> running,
>>   and settle an outstanding sleep ACK before waking a domain again.
>>   (Sashiko, Matt)
>> - Synchronously cancel delayed-release timers during forcewake cleanup.
>>   (Sashiko, Matt)
>> - Flush delayed releases during shutdown, device removal, and
>>   post-registration probe unwind in addition to system/runtime suspend.
>>   (Matt)
>>
>> Zongyao Bai (4):
>>  drm/xe/forcewake: synchronize engine-cycle access with unplug
>>  drm/xe/forcewake: add delayed-release state machine
>>  drm/xe/forcewake: flush delayed release at power boundaries
>>  drm/xe/forcewake: enable configurable delayed forcewake release
>>
>> drivers/gpu/drm/xe/xe_configfs.c         |  72 ++++++++
>> drivers/gpu/drm/xe/xe_configfs.h         |   5 +
>> drivers/gpu/drm/xe/xe_defaults.h         |   1 +
>> drivers/gpu/drm/xe/xe_device.c           |  16 +-
>> drivers/gpu/drm/xe/xe_device_types.h     |   3 +
>> drivers/gpu/drm/xe/xe_force_wake.c       | 211 +++++++++++++++++++++--
>> drivers/gpu/drm/xe/xe_force_wake.h       |  46 ++++-
>> drivers/gpu/drm/xe/xe_force_wake_types.h |  68 +++++++-
>> drivers/gpu/drm/xe/xe_gt.c               |  12 +-
>> drivers/gpu/drm/xe/xe_query.c            |  36 +++-
>> 10 files changed, 437 insertions(+), 33 deletions(-)
>>
>> -- 
>> 2.43.0
>>