Re: arm64 s2idle vs. workqueues

[Kevin Hilman <khilman@baylibre.com>]

Christian Loehle <christian.loehle@arm.com> writes:

> On 10/10/24 11:23, Rafael J. Wysocki wrote:
>> Hi Kevin,
>> 
>> On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
>>>
>>> Hello,
>>>
>>> Looking for some pointers/tips on debugging s2idle, and in particular
>>> why it is not staying in an idle state as long as expected.
>>>
>>> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
>>> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
>>> doing a WFI when idle.
>>>
>>> I'm doing an 8-second s2idle with RTC wakeup by using:
>>>
>>>   rtcwake -m freeze -s8
>>>
>>> and what I see is that 3 of the CPUs stay in their idle state for the
>>> full 8 seconds, but one of them keeps waking due to the A53
>>> arm_arch_timer firing, and processing misc. workqueue related activity
>>> (example work listed below[1].)
>>>
>>> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
>>> the freezer part of suspend to stop/freeze them.  However, I am a bit
>>> surprised to see this non-frozen workqueue activity happening often
>>> enough (few times per second) to prevent all 4 CPUs from being idle for
>>> long periods at the same time, thus preventing a deeper cluster-idle
>>> state.
>>>
>>> Is there something else I'm missing that is needed to keep these
>>> workqueues quiet for longer?  I had assumed that most of this workqueue
>>> work would be deferred, and shouldn't need to wakeup a CPU just to run.
>>>
>>> In case it's helpful I have published a trace.dat from trace-cmd which
>>> captures power, sched, irq, timer and workqueue events.  With
>>> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
>>> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
>>> workqueue activity.
>>>
>>> Any pointers to how to improve this situation, or what else needs to be
>>> tweaked here would be greatly appreciated,
>> 
>> It looks like tick_freeze() is not called, which only happens in
>> enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
>> there are any idle states with an .enter_s2idle() callback.
>> 
>> So does the cpuidle driver for this platform provide .enter_s2idle()
>> callbacks for any of its idle states?
>
> AFAICT there shouldn't be a cpuidle driver that initialized and WFI is
> entered through arch code. The trace.dat indicates that, too.

That's correct.  

> @Kevin
> I assume you can add a state in the DT, disable it and everything works
> as expected?

Thanks for the suggestion.  I tried that quickly, but ran into a couple
problems:

1) If the state is added, but disabled, that leaves only a WFI state,
   and the current PSCI CPUidle driver fails out and defers to arch idle
   if there's only the 1 WFI state[1]

2) The PSCI implementation on this SoC does not implement OSI mode, and
   the PSCI CPUidle driver also doesn't setup the s2idle callbacks
   unless OSI is supported[2]

I'm going to hack a bit around these limitations just to see if I can
get fully idle, but long term, it looks like I have some TF-A work to do
for the PSCI implementation on this SoC.  But at least now I understand
the s2idle callbacks and the missing tick_freeze() path, so it makes
sense why my platform is not fully idle/suspended.  And the good news is
that the TF-A implementation for these TI SoCs is fully upstream, and I
can build & use my own version, so that will be the next step.

Thanks for the pointers!

Kevin

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n361
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n221