arm64 s2idle vs. workqueues

[Kevin Hilman <khilman@baylibre.com>]

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular
why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
AM62x), which doesn't (yet) have any DT defined idle-states, so is just
doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

  rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the
full 8 seconds, but one of them keeps waking due to the A53
arm_arch_timer firing, and processing misc. workqueue related activity
(example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
the freezer part of suspend to stop/freeze them.  However, I am a bit
surprised to see this non-frozen workqueue activity happening often
enough (few times per second) to prevent all 4 CPUs from being idle for
long periods at the same time, thus preventing a deeper cluster-idle
state.

Is there something else I'm missing that is needed to keep these
workqueues quiet for longer?  I had assumed that most of this workqueue
work would be deferred, and shouldn't need to wakeup a CPU just to run.

In case it's helpful I have published a trace.dat from trace-cmd which
captures power, sched, irq, timer and workqueue events.  With
kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
are all nicely idle for 8 sec while CPU2 is waking due to the timer and
workqueue activity.

Any pointers to how to improve this situation, or what else needs to be
tweaked here would be greatly appreciated,

Thanks,

Kevin


[1]
function, workqueue name
------------------------
- page_pool_release_retry(), "events"
- vmstat_shepherd(), "events"
- vmstat_update(), "mm_percpu_wq"
- crng_reseed(), "events_unbound"
- kfree_rcu_monitor(), "events"
- flush_memcg_stats_dwork(), "events_unbound"
- neigh_managed_work(), "events_power_efficient"
- async_run_entry_fn(), "async"
- deferred_probe_work_func(), "events"
- tcp_orphan_update(), timer expiry

[2] https://drive.google.com/file/d/1U51eTTeb4_13-CZWa2llHXTh9DfZ_4sF/view?usp=sharing