Re: Fwd: ARM64: CPUIdle driver is not select any Idle state other then WFI

[Sudeep Holla <sudeep.holla@arm.com>]

On Wed, Dec 11, 2024 at 11:20:52AM +0530, Dhruva Gole wrote:
> Hi Vivek,
>
> On Oct 14, 2024 at 16:06:34 +0530, Vivek yadav wrote:
> > ---------- Forwarded message ---------
> > From: Vivek yadav <linux.ninja23@gmail.com>
> > Date: Fri, Oct 11, 2024 at 3:14 PM
> > Subject: ARM64: CPUIdle driver is not select any Idle state other then WFI
> > To: <linux-pm@vger.kernel.org>
>
> + Kevin, Vignesh and few colleagues at TI who have been working on this as
> well.
>
> >
> >
> > Hi @all,
> >
> > I am working on one custom SoC. Where I add one CPUIdle state for
> > ``arm,cortex-a55`` processor.
>
> Any further luck on this?
>
> I have also been working on something similar[1] but on an A53 core on
> TI-K3 AM62x processor.

Does upstream DTS have support for this platform to understand it better ?
Even reference to any complete DT file for the platform will help.

> >
> > idle-states {
> >       entry-method = "psci";
> >        cpu_ret_l: cpu-retention-l {
> >          compatible = "arm,idle-state";
> >          arm,psci-suspend-param = <0x00010001>;
> >          local-timer-stop;
> >          entry-latency-us = <55>;
> >           exit-latency-us = <140>;
> >           min-residency-us = <780>;
> >     };
> > };
> >
> > I am using ``Menu governor`` with the ``psci_idle driver`` in its original form.
> > After booting Linux I find out that the CPUIdle core is never going
> > inside the ``cpu-retention`` state.
> > To check time spent by CPU in any state. I am using the below command.
> >
> > ``cat /sys/devices/system/cpu/cpu*/cpuidle/state*/time``
>
> What I was seeing is in a multi core system (2 or more) all cores don't
> enter the idle-state simultaneously. There's something keeping atleast 1
> core always busy. However I could definitely see entry into TF-A from 1
> core at a time.

Does the platform have system timers ? What are the deeper idle states ?
If it is retention state with local timers on, I doubt if my suspicion of
CPU acting as broadcast timer in absence of a better/system timer.

> I then switched to a single core system to see if we were atall able to
> enter TF-A when only 1 core was available for linux, it turned out that
> with the "local-timer-stop" property that we have, this is never
> possible.
>

Yes my suspicion seems correct now but I can't confirm unless I understand
the platform completely.

> See this chunk in the kernel cpuidle driver:
> 	if (broadcast && tick_broadcast_enter()) {
>
> When I dug deeper into tick_broadcast_enter it always returns something
> non zero and hence in my case it was entering the if block and tried to
> find a deepest state. Then the deepest state would always return WFI and
> not the idle-state I had added.
>
> What we found out was on our kernel we end up using
>
> kernel/time/tick-broadcast-hrtimer.c
>
> This always seems to be keeping atleast 1 CPU busy and prevents idle.
> If we remove the local-timer-stop it was helping us, but we still need
> to dig into the full impact of what that entails and I am still
> interested in finding out how so many other users of similar idle-state
> implementation are able to do so without trouble.
>

Interesting. So if the platform is functional removing local-timer-stop,
I am bit confused. Either there is something else that is getting it out
from the idle state so, it should be fine and it could be just some
misconfiguration.

> Arm64 recommends to use arch_timer instead of external timers. Once we
> enter el3, timer interrupts to el1 is blocked and hence it's equivalent
> to local-timer-stop, so it does make sense to keep this property, but
> then how are others able to enter idle-states for all plugged CPUs at
> the same time?
>

Some systems have system timer that can take over as broadcast timer when
CPUs enter deeper idle states where the local timers are stopped.

--
Regards,
Sudeep