RT wakeup latency due to unbound workqueue execution (runtime mitigation?)

RT wakeup latency due to unbound workqueue execution (runtime mitigation?)

[vigashini kesavan]

Hi,

I am investigating occasional cyclictest wakeup latency spikes on a
PREEMPT_RT system (Linux 6.1.158-rt) and have traced it to the
execution of unbound workqueue items (flush_memcg_stats_dwork,
wb_workfn) on the measurement CPU.

These are observed under default Boot-time parameters, without any
modification to parameters such as isolcpus, nohz_full parameters or
workqueue cpumask. The intention is to analyze latency behaviour under
default boot-time configuration, relying only on runtime mechanisms
such as cgroup configuration or task level CPU affinity.
Manual pinning of kworker threads through task affinity was tried
without success.

Question:
Is there a supported or recommended runtime mechanism to prevent
unbound workqueue execution on a specific CPU. Or to defer this class
of work away from any CPU? (without any boot-time workqueue cpumasks)

Also, is there any option to attribute the unbound workqueue execution
to any kernel subsystems or runtime activities that could queue this
work. To better understand any potential userspace triggers for this.

Any guidance would be appreciated.

Thanks,
Vigashini Kesavan

Re: RT wakeup latency due to unbound workqueue execution (runtime mitigation?)

[Sebastian Andrzej Siewior]

On 2026-02-02 13:42:07 [+0200], vigashini kesavan wrote:
> Hi,
> 
> I am investigating occasional cyclictest wakeup latency spikes on a
> PREEMPT_RT system (Linux 6.1.158-rt) and have traced it to the
> execution of unbound workqueue items (flush_memcg_stats_dwork,
> wb_workfn) on the measurement CPU.
> 
> These are observed under default Boot-time parameters, without any
> modification to parameters such as isolcpus, nohz_full parameters or
> workqueue cpumask. The intention is to analyze latency behaviour under
> default boot-time configuration, relying only on runtime mechanisms
> such as cgroup configuration or task level CPU affinity.
> Manual pinning of kworker threads through task affinity was tried
> without success.

You can't pin an unbound worker to a CPU, this just does not work long
term. If you want this work to be invoked only a specific CPU you could
do replace the queue_delayed_work() invokcation with something like
	queue_delayed_work_on(1, system_wq, &stats_flush_dwork, FLUSH_TIME);
to have it invoked always on CPU1.

> Question:
> Is there a supported or recommended runtime mechanism to prevent
> unbound workqueue execution on a specific CPU. Or to defer this class
> of work away from any CPU? (without any boot-time workqueue cpumasks)

The hack above would be one way. But since you asked for recommended:
There is /sys/devices/virtual/workqueue/cpumask which contains the CPUs
for "unbound" workqueues. This has an effect on all of them.

> Also, is there any option to attribute the unbound workqueue execution
> to any kernel subsystems or runtime activities that could queue this
> work. To better understand any potential userspace triggers for this.

That folder
	/sys/devices/virtual/workqueue/

contains all the options you have. Some workqueues are dedicated and
have a name and allow changes. Other do not. The one you named, does
not use a specific workqueue but a generic one.

> Any guidance would be appreciated.
> 
> Thanks,
> Vigashini Kesavan

Sebastian