arm64 s2idle vs. workqueues

arm64 s2idle vs. workqueues

[Kevin Hilman]

Hello,

Looking for some pointers/tips on debugging s2idle, and in particular
why it is not staying in an idle state as long as expected.

I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
AM62x), which doesn't (yet) have any DT defined idle-states, so is just
doing a WFI when idle.

I'm doing an 8-second s2idle with RTC wakeup by using:

  rtcwake -m freeze -s8

and what I see is that 3 of the CPUs stay in their idle state for the
full 8 seconds, but one of them keeps waking due to the A53
arm_arch_timer firing, and processing misc. workqueue related activity
(example work listed below[1].)

I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
the freezer part of suspend to stop/freeze them.  However, I am a bit
surprised to see this non-frozen workqueue activity happening often
enough (few times per second) to prevent all 4 CPUs from being idle for
long periods at the same time, thus preventing a deeper cluster-idle
state.

Is there something else I'm missing that is needed to keep these
workqueues quiet for longer?  I had assumed that most of this workqueue
work would be deferred, and shouldn't need to wakeup a CPU just to run.

In case it's helpful I have published a trace.dat from trace-cmd which
captures power, sched, irq, timer and workqueue events.  With
kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
are all nicely idle for 8 sec while CPU2 is waking due to the timer and
workqueue activity.

Any pointers to how to improve this situation, or what else needs to be
tweaked here would be greatly appreciated,

Thanks,

Kevin


[1]
function, workqueue name
------------------------
- page_pool_release_retry(), "events"
- vmstat_shepherd(), "events"
- vmstat_update(), "mm_percpu_wq"
- crng_reseed(), "events_unbound"
- kfree_rcu_monitor(), "events"
- flush_memcg_stats_dwork(), "events_unbound"
- neigh_managed_work(), "events_power_efficient"
- async_run_entry_fn(), "async"
- deferred_probe_work_func(), "events"
- tcp_orphan_update(), timer expiry

[2] https://drive.google.com/file/d/1U51eTTeb4_13-CZWa2llHXTh9DfZ_4sF/view?usp=sharing

Re: arm64 s2idle vs. workqueues

[Saravana Kannan]

All I know is that shouldn't be happening. Not because I know the
code, but the Pixel Watch 2 uses s2idle and it stays fully suspended
perfectly well.

-Saravana

On Wed, Oct 9, 2024 at 5:19 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
> Hello,
>
> Looking for some pointers/tips on debugging s2idle, and in particular
> why it is not staying in an idle state as long as expected.
>
> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
> doing a WFI when idle.
>
> I'm doing an 8-second s2idle with RTC wakeup by using:
>
>   rtcwake -m freeze -s8
>
> and what I see is that 3 of the CPUs stay in their idle state for the
> full 8 seconds, but one of them keeps waking due to the A53
> arm_arch_timer firing, and processing misc. workqueue related activity
> (example work listed below[1].)
>
> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
> the freezer part of suspend to stop/freeze them.  However, I am a bit
> surprised to see this non-frozen workqueue activity happening often
> enough (few times per second) to prevent all 4 CPUs from being idle for
> long periods at the same time, thus preventing a deeper cluster-idle
> state.
>
> Is there something else I'm missing that is needed to keep these
> workqueues quiet for longer?  I had assumed that most of this workqueue
> work would be deferred, and shouldn't need to wakeup a CPU just to run.
>
> In case it's helpful I have published a trace.dat from trace-cmd which
> captures power, sched, irq, timer and workqueue events.  With
> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
> workqueue activity.
>
> Any pointers to how to improve this situation, or what else needs to be
> tweaked here would be greatly appreciated,
>
> Thanks,
>
> Kevin
>
>
> [1]
> function, workqueue name
> ------------------------
> - page_pool_release_retry(), "events"
> - vmstat_shepherd(), "events"
> - vmstat_update(), "mm_percpu_wq"
> - crng_reseed(), "events_unbound"
> - kfree_rcu_monitor(), "events"
> - flush_memcg_stats_dwork(), "events_unbound"
> - neigh_managed_work(), "events_power_efficient"
> - async_run_entry_fn(), "async"
> - deferred_probe_work_func(), "events"
> - tcp_orphan_update(), timer expiry
>
> [2] https://drive.google.com/file/d/1U51eTTeb4_13-CZWa2llHXTh9DfZ_4sF/view?usp=sharing

Re: arm64 s2idle vs. workqueues

[Rafael J. Wysocki]

Hi Kevin,

On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
>
> Hello,
>
> Looking for some pointers/tips on debugging s2idle, and in particular
> why it is not staying in an idle state as long as expected.
>
> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
> doing a WFI when idle.
>
> I'm doing an 8-second s2idle with RTC wakeup by using:
>
>   rtcwake -m freeze -s8
>
> and what I see is that 3 of the CPUs stay in their idle state for the
> full 8 seconds, but one of them keeps waking due to the A53
> arm_arch_timer firing, and processing misc. workqueue related activity
> (example work listed below[1].)
>
> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
> the freezer part of suspend to stop/freeze them.  However, I am a bit
> surprised to see this non-frozen workqueue activity happening often
> enough (few times per second) to prevent all 4 CPUs from being idle for
> long periods at the same time, thus preventing a deeper cluster-idle
> state.
>
> Is there something else I'm missing that is needed to keep these
> workqueues quiet for longer?  I had assumed that most of this workqueue
> work would be deferred, and shouldn't need to wakeup a CPU just to run.
>
> In case it's helpful I have published a trace.dat from trace-cmd which
> captures power, sched, irq, timer and workqueue events.  With
> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
> workqueue activity.
>
> Any pointers to how to improve this situation, or what else needs to be
> tweaked here would be greatly appreciated,

It looks like tick_freeze() is not called, which only happens in
enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
there are any idle states with an .enter_s2idle() callback.

So does the cpuidle driver for this platform provide .enter_s2idle()
callbacks for any of its idle states?

Cheers, Rafael

Re: arm64 s2idle vs. workqueues

[Christian Loehle]

On 10/10/24 11:23, Rafael J. Wysocki wrote:
> Hi Kevin,
> 
> On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
>>
>> Hello,
>>
>> Looking for some pointers/tips on debugging s2idle, and in particular
>> why it is not staying in an idle state as long as expected.
>>
>> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
>> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
>> doing a WFI when idle.
>>
>> I'm doing an 8-second s2idle with RTC wakeup by using:
>>
>>   rtcwake -m freeze -s8
>>
>> and what I see is that 3 of the CPUs stay in their idle state for the
>> full 8 seconds, but one of them keeps waking due to the A53
>> arm_arch_timer firing, and processing misc. workqueue related activity
>> (example work listed below[1].)
>>
>> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
>> the freezer part of suspend to stop/freeze them.  However, I am a bit
>> surprised to see this non-frozen workqueue activity happening often
>> enough (few times per second) to prevent all 4 CPUs from being idle for
>> long periods at the same time, thus preventing a deeper cluster-idle
>> state.
>>
>> Is there something else I'm missing that is needed to keep these
>> workqueues quiet for longer?  I had assumed that most of this workqueue
>> work would be deferred, and shouldn't need to wakeup a CPU just to run.
>>
>> In case it's helpful I have published a trace.dat from trace-cmd which
>> captures power, sched, irq, timer and workqueue events.  With
>> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
>> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
>> workqueue activity.
>>
>> Any pointers to how to improve this situation, or what else needs to be
>> tweaked here would be greatly appreciated,
> 
> It looks like tick_freeze() is not called, which only happens in
> enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
> there are any idle states with an .enter_s2idle() callback.
> 
> So does the cpuidle driver for this platform provide .enter_s2idle()
> callbacks for any of its idle states?

AFAICT there shouldn't be a cpuidle driver that initialized and WFI is
entered through arch code. The trace.dat indicates that, too.

@Kevin
I assume you can add a state in the DT, disable it and everything works
as expected?

Re: arm64 s2idle vs. workqueues

[Sudeep Holla]

On Thu, Oct 10, 2024 at 11:33:09AM +0100, Christian Loehle wrote:
> On 10/10/24 11:23, Rafael J. Wysocki wrote:
> > Hi Kevin,
> > 
> > On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
> >>
> >> Hello,
> >>
> >> Looking for some pointers/tips on debugging s2idle, and in particular
> >> why it is not staying in an idle state as long as expected.
> >>
> >> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
> >> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
> >> doing a WFI when idle.
> >>
> >> I'm doing an 8-second s2idle with RTC wakeup by using:
> >>
> >>   rtcwake -m freeze -s8
> >>
> >> and what I see is that 3 of the CPUs stay in their idle state for the
> >> full 8 seconds, but one of them keeps waking due to the A53
> >> arm_arch_timer firing, and processing misc. workqueue related activity
> >> (example work listed below[1].)
> >>
> >> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
> >> the freezer part of suspend to stop/freeze them.  However, I am a bit
> >> surprised to see this non-frozen workqueue activity happening often
> >> enough (few times per second) to prevent all 4 CPUs from being idle for
> >> long periods at the same time, thus preventing a deeper cluster-idle
> >> state.
> >>
> >> Is there something else I'm missing that is needed to keep these
> >> workqueues quiet for longer?  I had assumed that most of this workqueue
> >> work would be deferred, and shouldn't need to wakeup a CPU just to run.
> >>
> >> In case it's helpful I have published a trace.dat from trace-cmd which
> >> captures power, sched, irq, timer and workqueue events.  With
> >> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
> >> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
> >> workqueue activity.
> >>
> >> Any pointers to how to improve this situation, or what else needs to be
> >> tweaked here would be greatly appreciated,
> > 
> > It looks like tick_freeze() is not called, which only happens in
> > enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
> > there are any idle states with an .enter_s2idle() callback.
> > 
> > So does the cpuidle driver for this platform provide .enter_s2idle()
> > callbacks for any of its idle states?
> 
> AFAICT there shouldn't be a cpuidle driver that initialized and WFI is
> entered through arch code. The trace.dat indicates that, too.
> 
> @Kevin
> I assume you can add a state in the DT, disable it and everything works
> as expected?
>

Good point, I didn't realise that will be quickest way, I was thinking of
writing a small wfi only idle driver.

-- 
Regards,
Sudeep

Re: arm64 s2idle vs. workqueues

[Kevin Hilman]

Christian Loehle <christian.loehle@arm.com> writes:

> On 10/10/24 11:23, Rafael J. Wysocki wrote:
>> Hi Kevin,
>> 
>> On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
>>>
>>> Hello,
>>>
>>> Looking for some pointers/tips on debugging s2idle, and in particular
>>> why it is not staying in an idle state as long as expected.
>>>
>>> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
>>> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
>>> doing a WFI when idle.
>>>
>>> I'm doing an 8-second s2idle with RTC wakeup by using:
>>>
>>>   rtcwake -m freeze -s8
>>>
>>> and what I see is that 3 of the CPUs stay in their idle state for the
>>> full 8 seconds, but one of them keeps waking due to the A53
>>> arm_arch_timer firing, and processing misc. workqueue related activity
>>> (example work listed below[1].)
>>>
>>> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
>>> the freezer part of suspend to stop/freeze them.  However, I am a bit
>>> surprised to see this non-frozen workqueue activity happening often
>>> enough (few times per second) to prevent all 4 CPUs from being idle for
>>> long periods at the same time, thus preventing a deeper cluster-idle
>>> state.
>>>
>>> Is there something else I'm missing that is needed to keep these
>>> workqueues quiet for longer?  I had assumed that most of this workqueue
>>> work would be deferred, and shouldn't need to wakeup a CPU just to run.
>>>
>>> In case it's helpful I have published a trace.dat from trace-cmd which
>>> captures power, sched, irq, timer and workqueue events.  With
>>> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
>>> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
>>> workqueue activity.
>>>
>>> Any pointers to how to improve this situation, or what else needs to be
>>> tweaked here would be greatly appreciated,
>> 
>> It looks like tick_freeze() is not called, which only happens in
>> enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
>> there are any idle states with an .enter_s2idle() callback.
>> 
>> So does the cpuidle driver for this platform provide .enter_s2idle()
>> callbacks for any of its idle states?
>
> AFAICT there shouldn't be a cpuidle driver that initialized and WFI is
> entered through arch code. The trace.dat indicates that, too.

That's correct.  

> @Kevin
> I assume you can add a state in the DT, disable it and everything works
> as expected?

Thanks for the suggestion.  I tried that quickly, but ran into a couple
problems:

1) If the state is added, but disabled, that leaves only a WFI state,
   and the current PSCI CPUidle driver fails out and defers to arch idle
   if there's only the 1 WFI state[1]

2) The PSCI implementation on this SoC does not implement OSI mode, and
   the PSCI CPUidle driver also doesn't setup the s2idle callbacks
   unless OSI is supported[2]

I'm going to hack a bit around these limitations just to see if I can
get fully idle, but long term, it looks like I have some TF-A work to do
for the PSCI implementation on this SoC.  But at least now I understand
the s2idle callbacks and the missing tick_freeze() path, so it makes
sense why my platform is not fully idle/suspended.  And the good news is
that the TF-A implementation for these TI SoCs is fully upstream, and I
can build & use my own version, so that will be the next step.

Thanks for the pointers!

Kevin

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n361
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n221

Re: arm64 s2idle vs. workqueues

[Saravana Kannan]

If you have some lessons learned from this and present at the next
LPC, that'd be awesome! Or some LWN writeup, etc.

One of the issues you might hit... drivers doing stuff in hotplug
notifiers that are per CPU. We need a better solution for that.

-Saravana

On Thu, Oct 10, 2024 at 12:20 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
> Christian Loehle <christian.loehle@arm.com> writes:
>
> > On 10/10/24 11:23, Rafael J. Wysocki wrote:
> >> Hi Kevin,
> >>
> >> On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Looking for some pointers/tips on debugging s2idle, and in particular
> >>> why it is not staying in an idle state as long as expected.
> >>>
> >>> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
> >>> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
> >>> doing a WFI when idle.
> >>>
> >>> I'm doing an 8-second s2idle with RTC wakeup by using:
> >>>
> >>>   rtcwake -m freeze -s8
> >>>
> >>> and what I see is that 3 of the CPUs stay in their idle state for the
> >>> full 8 seconds, but one of them keeps waking due to the A53
> >>> arm_arch_timer firing, and processing misc. workqueue related activity
> >>> (example work listed below[1].)
> >>>
> >>> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
> >>> the freezer part of suspend to stop/freeze them.  However, I am a bit
> >>> surprised to see this non-frozen workqueue activity happening often
> >>> enough (few times per second) to prevent all 4 CPUs from being idle for
> >>> long periods at the same time, thus preventing a deeper cluster-idle
> >>> state.
> >>>
> >>> Is there something else I'm missing that is needed to keep these
> >>> workqueues quiet for longer?  I had assumed that most of this workqueue
> >>> work would be deferred, and shouldn't need to wakeup a CPU just to run.
> >>>
> >>> In case it's helpful I have published a trace.dat from trace-cmd which
> >>> captures power, sched, irq, timer and workqueue events.  With
> >>> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
> >>> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
> >>> workqueue activity.
> >>>
> >>> Any pointers to how to improve this situation, or what else needs to be
> >>> tweaked here would be greatly appreciated,
> >>
> >> It looks like tick_freeze() is not called, which only happens in
> >> enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
> >> there are any idle states with an .enter_s2idle() callback.
> >>
> >> So does the cpuidle driver for this platform provide .enter_s2idle()
> >> callbacks for any of its idle states?
> >
> > AFAICT there shouldn't be a cpuidle driver that initialized and WFI is
> > entered through arch code. The trace.dat indicates that, too.
>
> That's correct.
>
> > @Kevin
> > I assume you can add a state in the DT, disable it and everything works
> > as expected?
>
> Thanks for the suggestion.  I tried that quickly, but ran into a couple
> problems:
>
> 1) If the state is added, but disabled, that leaves only a WFI state,
>    and the current PSCI CPUidle driver fails out and defers to arch idle
>    if there's only the 1 WFI state[1]
>
> 2) The PSCI implementation on this SoC does not implement OSI mode, and
>    the PSCI CPUidle driver also doesn't setup the s2idle callbacks
>    unless OSI is supported[2]
>
> I'm going to hack a bit around these limitations just to see if I can
> get fully idle, but long term, it looks like I have some TF-A work to do
> for the PSCI implementation on this SoC.  But at least now I understand
> the s2idle callbacks and the missing tick_freeze() path, so it makes
> sense why my platform is not fully idle/suspended.  And the good news is
> that the TF-A implementation for these TI SoCs is fully upstream, and I
> can build & use my own version, so that will be the next step.
>
> Thanks for the pointers!
>
> Kevin
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n361
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/cpuidle/cpuidle-psci.c?h=v6.11#n221

Re: arm64 s2idle vs. workqueues

[Sudeep Holla]

On Thu, Oct 10, 2024 at 12:23:25PM +0200, Rafael J. Wysocki wrote:
> Hi Kevin,
> 
> On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
> >
> > Hello,
> >
> > Looking for some pointers/tips on debugging s2idle, and in particular
> > why it is not staying in an idle state as long as expected.
> >
> > I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
> > AM62x), which doesn't (yet) have any DT defined idle-states, so is just
> > doing a WFI when idle.
> >
> > I'm doing an 8-second s2idle with RTC wakeup by using:
> >
> >   rtcwake -m freeze -s8
> >
> > and what I see is that 3 of the CPUs stay in their idle state for the
> > full 8 seconds, but one of them keeps waking due to the A53
> > arm_arch_timer firing, and processing misc. workqueue related activity
> > (example work listed below[1].)
> >
> > I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
> > the freezer part of suspend to stop/freeze them.  However, I am a bit
> > surprised to see this non-frozen workqueue activity happening often
> > enough (few times per second) to prevent all 4 CPUs from being idle for
> > long periods at the same time, thus preventing a deeper cluster-idle
> > state.
> >
> > Is there something else I'm missing that is needed to keep these
> > workqueues quiet for longer?  I had assumed that most of this workqueue
> > work would be deferred, and shouldn't need to wakeup a CPU just to run.
> >
> > In case it's helpful I have published a trace.dat from trace-cmd which
> > captures power, sched, irq, timer and workqueue events.  With
> > kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
> > are all nicely idle for 8 sec while CPU2 is waking due to the timer and
> > workqueue activity.
> >
> > Any pointers to how to improve this situation, or what else needs to be
> > tweaked here would be greatly appreciated,
> 
> It looks like tick_freeze() is not called, which only happens in
> enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
> there are any idle states with an .enter_s2idle() callback.
> 
> So does the cpuidle driver for this platform provide .enter_s2idle()
> callbacks for any of its idle states?

IIUC, his setup/platform doesn't support any deeper idle states than the
shallowest WFI on all arm systems. I was about to ask if he has already
tried to build a cpuidle driver with just that WFI state or is he relying
on default arch_cpu_idle() which is one possible reason to take different
path and hence not executing the expected s2idle path.

-- 
Regards,
Sudeep

Re: arm64 s2idle vs. workqueues

[Kevin Hilman]

"Rafael J. Wysocki" <rafael@kernel.org> writes:

> Hi Kevin,
>
> On Thu, Oct 10, 2024 at 2:19 AM Kevin Hilman <khilman@baylibre.com> wrote:
>>
>> Hello,
>>
>> Looking for some pointers/tips on debugging s2idle, and in particular
>> why it is not staying in an idle state as long as expected.
>>
>> I'm attempting to use s2idle on a 4-core, single cluster ARM64 SoC (TI
>> AM62x), which doesn't (yet) have any DT defined idle-states, so is just
>> doing a WFI when idle.
>>
>> I'm doing an 8-second s2idle with RTC wakeup by using:
>>
>>   rtcwake -m freeze -s8
>>
>> and what I see is that 3 of the CPUs stay in their idle state for the
>> full 8 seconds, but one of them keeps waking due to the A53
>> arm_arch_timer firing, and processing misc. workqueue related activity
>> (example work listed below[1].)
>>
>> I realize that these workqueues are not WQ_FREEZABLE, so I don't expect
>> the freezer part of suspend to stop/freeze them.  However, I am a bit
>> surprised to see this non-frozen workqueue activity happening often
>> enough (few times per second) to prevent all 4 CPUs from being idle for
>> long periods at the same time, thus preventing a deeper cluster-idle
>> state.
>>
>> Is there something else I'm missing that is needed to keep these
>> workqueues quiet for longer?  I had assumed that most of this workqueue
>> work would be deferred, and shouldn't need to wakeup a CPU just to run.
>>
>> In case it's helpful I have published a trace.dat from trace-cmd which
>> captures power, sched, irq, timer and workqueue events.  With
>> kernelshark, it's pretty obvious to visualize what's happening: CPU0,1,3
>> are all nicely idle for 8 sec while CPU2 is waking due to the timer and
>> workqueue activity.
>>
>> Any pointers to how to improve this situation, or what else needs to be
>> tweaked here would be greatly appreciated,
>
> It looks like tick_freeze() is not called, which only happens in
> enter_s2idle_proper() that is called from cpuidle_enter_s2idle() if
> there are any idle states with an .enter_s2idle() callback.

Ah, thank you for the pointer, this is indeed what is missing.

> So does the cpuidle driver for this platform provide .enter_s2idle()
> callbacks for any of its idle states?

Unfortunately, no, so that explains my problem.

Thanks,

Kevin