Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

[黄少波]

From: huangshaobo2075@phytium.com.cn
To: linux-pm@vger.kernel.org
Cc: rafael@kernel.org, lenb@kernel.org, deepthi@linux.vnet.ibm.com, khilman@kernel.org
Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

Hi,

I'm working on an ARM64 platform with a big.LITTLE CPU topology. While parsing the ACPI tables,
I noticed that the C-state latency and residency values differ between the big and LITTLE cores,
as expected.

However, I found that the current cpuidle framework only allows a single global `cpuidle_driver`,
and all CPUs share the same `cpuidle_driver->states[]` array. As a result, only the first core to
initialize (usually a LITTLE core) sets up the C-states, and the same values are applied to all cores,
including the big ones. This leads to incorrect idle behavior on asymmetric platforms.

I believe this behavior was introduced by commit 46bcfad7a819
("cpuidle: Single/Global registration of idle states").

I understand this design was introduced in 2011 to simplify cpuidle and reduce memory usage:
https://lkml.org/lkml/2011/4/25/83

However, on today's heterogeneous SoCs, this global model no longer suffices. For proper modeling,
we need support for per-cluster or per-core cpuidle drivers, or at least some mechanism to allow
different idle state parameters per CPU.

Has there been any discussion or work toward lifting this limitation?

Thanks,

Shaobo Huang


信息安全声明：本邮件包含信息归发件人所在组织所有,发件人所在组织对该邮件拥有所有权利。请接收者注意保密,未经发件人书面许可,不得向任何第三方组织和个人透露本邮件所含信息。
Information Security Notice: The information contained in this mail is solely property of the sender's organization.This mail communication is confidential.Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.

Re: Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

[Rafael J. Wysocki]

On Mon, Jun 16, 2025 at 9:14 AM 黄少波 <huangshaobo2075@phytium.com.cn> wrote:
>
> From: huangshaobo2075@phytium.com.cn
> To: linux-pm@vger.kernel.org
> Cc: rafael@kernel.org, lenb@kernel.org, deepthi@linux.vnet.ibm.com, khilman@kernel.org
> Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs
>
> Hi,
>
> I'm working on an ARM64 platform with a big.LITTLE CPU topology. While parsing the ACPI tables,
> I noticed that the C-state latency and residency values differ between the big and LITTLE cores,
> as expected.
>
> However, I found that the current cpuidle framework only allows a single global `cpuidle_driver`,
> and all CPUs share the same `cpuidle_driver->states[]` array.

Not really, see bl_idle_init() in particular.

However, on systems with ACPI on which _CST is used for idle state
description, there's only one cpuidle driver and one table of idle
states for all CPUs.

> As a result, only the first core to
> initialize (usually a LITTLE core) sets up the C-states, and the same values are applied to all cores,
> including the big ones. This leads to incorrect idle behavior on asymmetric platforms.
>
> I believe this behavior was introduced by commit 46bcfad7a819
> ("cpuidle: Single/Global registration of idle states").
>
> I understand this design was introduced in 2011 to simplify cpuidle and reduce memory usage:
> https://lkml.org/lkml/2011/4/25/83
>
> However, on today's heterogeneous SoCs, this global model no longer suffices. For proper modeling,
> we need support for per-cluster or per-core cpuidle drivers, or at least some mechanism to allow
> different idle state parameters per CPU.
>
> Has there been any discussion or work toward lifting this limitation?

No, there's not been any discussion on this so far, but why does the
platform firmware on this system use _CST for idle state description?
_LPI would be a better option AFAICS.

Re: Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

[黄少波]

> On Mon, Jun 16, 2025 at 9:14 AM 黄少波 <huangshaobo2075@phytium.com.cn> wrote:
> >
> > From: huangshaobo2075@phytium.com.cn
> > To: linux-pm@vger.kernel.org
> > Cc: rafael@kernel.org, lenb@kernel.org, deepthi@linux.vnet.ibm.com, khilman@kernel.org
> > Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs
> >
> > Hi,
> >
> > I'm working on an ARM64 platform with a big.LITTLE CPU topology. While parsing the ACPI tables,
> > I noticed that the C-state latency and residency values differ between the big and LITTLE cores,
> > as expected.
> >
> > However, I found that the current cpuidle framework only allows a single global `cpuidle_driver`,
> > and all CPUs share the same `cpuidle_driver->states[]` array.
> 
> Not really, see bl_idle_init() in particular.
> 
> However, on systems with ACPI on which _CST is used for idle state
> description, there's only one cpuidle driver and one table of idle
> states for all CPUs.
> 
> > As a result, only the first core to
> > initialize (usually a LITTLE core) sets up the C-states, and the same values are applied to all cores,
> > including the big ones. This leads to incorrect idle behavior on asymmetric platforms.
> >
> > I believe this behavior was introduced by commit 46bcfad7a819
> > ("cpuidle: Single/Global registration of idle states").
> >
> > I understand this design was introduced in 2011 to simplify cpuidle and reduce memory usage:
> > https://lkml.org/lkml/2011/4/25/83
> >
> > However, on today's heterogeneous SoCs, this global model no longer suffices. For proper modeling,
> > we need support for per-cluster or per-core cpuidle drivers, or at least some mechanism to allow
> > different idle state parameters per CPU.
> >
> > Has there been any discussion or work toward lifting this limitation?
> 
> No, there's not been any discussion on this so far, but why does the
> platform firmware on this system use _CST for idle state description?
> _LPI would be a better option AFAICS.

From: huangshaobo2075@phytium.com.cn
To: linux-pm@vger.kernel.org
Cc: rafael@kernel.org, lenb@kernel.org, deepthi@linux.vnet.ibm.com, khilman@kernel.org
Subject: [cpuidle][follow-up] _LPI path also assumes shared cpuidle_driver, breaks on big.LITTLE

Hi all,

Thanks Rafael for the quick feedback on my earlier question regarding asymmetric idle states on big.LITTLE SoCs.

As a follow-up, after reviewing the implementation further, I noticed that `acpi_processor_setup_lpi_states()` is only called from within
`acpi_processor_setup_cpuidle_states()`, which is itself only invoked once — during initialization of the first CPU.

Specifically:

- The kernel uses a shared global cpuidle_driver instance (`acpi_idle_driver`), which is only initialized once in `acpi_processor_power_init()`:

  static int acpi_processor_registered;
  ...
  if (!acpi_processor_registered) {
      acpi_processor_setup_cpuidle_states(pr);
      cpuidle_register_driver(&acpi_idle_driver);
  }

- Subsequent CPUs only register their cpuidle_device without updating the shared driver's states[] array.

- Even in `acpi_processor_power_state_has_changed()`, only `pr->id == 0` triggers `acpi_processor_setup_cpuidle_states()`, meaning the global
  state array is populated based only on CPU 0.

    int acpi_processor_power_state_has_changed(struct acpi_processor *pr) {
        ...
        if (pr->id == 0 && cpuidle_get_driver() == &acpi_idle_driver) {
            ...
            acpi_processor_setup_cpuidle_states(pr);
            ...
        }
    }

The ACPI _LPI specification allows firmware to describe per-CPU or per-cluster idle state characteristics (latency, residency, etc.). However,
the current implementation effectively ignores this flexibility and applies the first CPU’s states globally.

This appears to be a design limitation that prevents accurate modeling of asymmetric CPU topologies like big.LITTLE. As a result, idle states may
not reflect the correct characteristics for all cores, leading to suboptimal behavior.

Could you please confirm whether this behavior is an intentional carry-over from the legacy _CST-based implementation, or simply an area that hasn't
been fully extended to support _LPI's per-CPU flexibility?

We are actively using the ACPI-based _LPI method on asymmetric ARM64 platforms, so this behavior is directly impacting us. I’d be happy to help investigate
a fix if the current design is not intentional.

Additionally, I’ve reviewed the `bl_idle_init()` implementation in the big.LITTLE device-tree-based driver and confirmed it does not support ACPI.
Since our platform relies on _LPI via ACPI, this limitation is particularly relevant in our use case.

Thanks for your time and insights!

Best regards,
Shaobo Huang


信息安全声明：本邮件包含信息归发件人所在组织所有,发件人所在组织对该邮件拥有所有权利。请接收者注意保密,未经发件人书面许可,不得向任何第三方组织和个人透露本邮件所含信息。
Information Security Notice: The information contained in this mail is solely property of the sender's organization.This mail communication is confidential.Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.

Re: Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

[Pierre Gondois]

Hello Shaobo,

On 6/16/25 09:14, 黄少波 wrote:
> From: huangshaobo2075@phytium.com.cn
> To: linux-pm@vger.kernel.org
> Cc: rafael@kernel.org, lenb@kernel.org, deepthi@linux.vnet.ibm.com, khilman@kernel.org
> Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs
>
> Hi,
>
> I'm working on an ARM64 platform with a big.LITTLE CPU topology. While parsing the ACPI tables,
> I noticed that the C-state latency and residency values differ between the big and LITTLE cores,
> as expected.
>
> However, I found that the current cpuidle framework only allows a single global `cpuidle_driver`,
> and all CPUs share the same `cpuidle_driver->states[]` array. As a result, only the first core to
> initialize (usually a LITTLE core) sets up the C-states, and the same values are applied to all cores,
> including the big ones. This leads to incorrect idle behavior on asymmetric platforms.
>
> I believe this behavior was introduced by commit 46bcfad7a819
> ("cpuidle: Single/Global registration of idle states").
>
> I understand this design was introduced in 2011 to simplify cpuidle and reduce memory usage:
> https://lkml.org/lkml/2011/4/25/83
>
> However, on today's heterogeneous SoCs, this global model no longer suffices. For proper modeling,
> we need support for per-cluster or per-core cpuidle drivers, or at least some mechanism to allow
> different idle state parameters per CPU.
>
> Has there been any discussion or work toward lifting this limitation?
>
> Thanks,
>
> Shaobo Huang
>
>
> 信息安全声明：本邮件包含信息归发件人所在组织所有,发件人所在组织对该邮件拥有所有权利。请接收者注意保密,未经发件人书面许可,不得向任何第三方组织和个人透露本邮件所含信息。
> Information Security Notice: The information contained in this mail is solely property of the sender's organization.This mail communication is confidential.Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.

Just to confirm that I can reproduce this.

Sudeep will be back in a few days, in case he has more background or
another view on the topic,
but I agree asymmetric _LPI states should ideally be visible to the kernel.

I am having a look at the code aswell. In case you have an
implementation, I can provide additional testing.

Regards,

Pierre

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

[黄少波]

> Hello Shaobo,
> 
> On 6/16/25 09:14, 黄少波 wrote:
> > From: huangshaobo2075@phytium.com.cn
> > To: linux-pm@vger.kernel.org
> > Cc: rafael@kernel.org, lenb@kernel.org, deepthi@linux.vnet.ibm.com, khilman@kernel.org
> > Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs
> >
> > Hi,
> >
> > I'm working on an ARM64 platform with a big.LITTLE CPU topology. While parsing the ACPI tables,
> > I noticed that the C-state latency and residency values differ between the big and LITTLE cores,
> > as expected.
> >
> > However, I found that the current cpuidle framework only allows a single global `cpuidle_driver`,
> > and all CPUs share the same `cpuidle_driver->states[]` array. As a result, only the first core to
> > initialize (usually a LITTLE core) sets up the C-states, and the same values are applied to all cores,
> > including the big ones. This leads to incorrect idle behavior on asymmetric platforms.
> >
> > I believe this behavior was introduced by commit 46bcfad7a819
> > ("cpuidle: Single/Global registration of idle states").
> >
> > I understand this design was introduced in 2011 to simplify cpuidle and reduce memory usage:
> > https://lkml.org/lkml/2011/4/25/83
> >
> > However, on today's heterogeneous SoCs, this global model no longer suffices. For proper modeling,
> > we need support for per-cluster or per-core cpuidle drivers, or at least some mechanism to allow
> > different idle state parameters per CPU.
> >
> > Has there been any discussion or work toward lifting this limitation?
> >
> > Thanks,
> >
> > Shaobo Huang
> >
> >
> > 信息安全声明：本邮件包含信息归发件人所在组织所有,发件人所在组织对该邮件拥有所有权利。请接收者注意保密,未经发件人书面许可,不得向任何第三方组织和个人透露本邮件所含信息。
> > Information Security Notice: The information contained in this mail is solely property of the sender's organization.This mail communication is confidential.Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.
> 
> Just to confirm that I can reproduce this.
> 
> Sudeep will be back in a few days, in case he has more background or
> another view on the topic,
> but I agree asymmetric _LPI states should ideally be visible to the kernel.
> 
> I am having a look at the code aswell. In case you have an
> implementation, I can provide additional testing.
> 
> Regards,
> 
> Pierre
> 
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.


From: huangshaobo2075@phytium.com.cn
To: pierre.gondois@arm.com, linux-pm@vger.kernel.org
Cc: rafael@kernel.org, lenb@kernel.org, khilman@kernel.org, christian.loehle@arm.com, Sudeep.Holla@arm.com
Subject: [cpuidle] Handling asymmetric _LPI states in cpuidle

Hi Pierre,

Thanks for confirming the issue and looking into the code.

I haven’t started implementing yet, but I’ve been thinking about how to represent the asymmetry properly.

Initially, I considered grouping CPUs by cluster, so that cores in the same cluster would share a `cpuidle_driver`.
But I realized that this assumption doesn’t always hold, especially on systems where a cluster may contain heterogeneous
CPUs (e.g. big.LITTLE designs with shared power domains).

So now I’m leaning towards grouping CPUs that have the same set of _LPI states — regardless of their topology — and assigning
one cpuidle_driver per such group. This avoids using a single global driver, and also avoids the overhead of per-CPU drivers
when not necessary.

This is just an initial idea, and I’m still exploring possible approaches. If you have any suggestions or alternative ideas,
I’d be happy to discuss further or even collaborate on an implementation.

Best regards,  
Shaobo Huang

信息安全声明：本邮件包含信息归发件人所在组织所有,发件人所在组织对该邮件拥有所有权利。请接收者注意保密,未经发件人书面许可,不得向任何第三方组织和个人透露本邮件所含信息。
Information Security Notice: The information contained in this mail is solely property of the sender's organization.This mail communication is confidential.Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.

Re: Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

[Sudeep Holla]

On Mon, Jun 16, 2025 at 03:14:18PM +0800, 黄少波 wrote:
> 
> 信息安全声明：本邮件包含信息归发件人所在组织所有,发件人所在组织对该邮件拥有所有权利。请接收者注意保密,未经发件人书面许可,不得向任何第三方组织和个人透露本邮件所含信息。
> Information Security Notice: The information contained in this mail is
> solely property of the sender's organization.This mail communication is
> confidential.Recipients named above are obligated to maintain secrecy and
> are not permitted to disclose the contents of this communication to others.

This is not possible on the list, so I am just deleting all your emails.

-- 
Regards,
Sudeep

Re: Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs

[Sudeep Holla]

On Mon, Jun 16, 2025 at 01:29:15PM +0200, Rafael J. Wysocki wrote:
> On Mon, Jun 16, 2025 at 9:14 AM 黄少波 <huangshaobo2075@phytium.com.cn> wrote:
> >
> > From: huangshaobo2075@phytium.com.cn
> > To: linux-pm@vger.kernel.org
> > Cc: rafael@kernel.org, lenb@kernel.org, deepthi@linux.vnet.ibm.com, khilman@kernel.org
> > Subject: [cpuidle] Limitation: cannot model asymmetric C-state latencies on big.LITTLE SoCs
> >
> > Hi,
> >
> > I'm working on an ARM64 platform with a big.LITTLE CPU topology. While parsing the ACPI tables,
> > I noticed that the C-state latency and residency values differ between the big and LITTLE cores,
> > as expected.
> >
> > However, I found that the current cpuidle framework only allows a single global `cpuidle_driver`,
> > and all CPUs share the same `cpuidle_driver->states[]` array.
> 
> Not really, see bl_idle_init() in particular.
> 
> However, on systems with ACPI on which _CST is used for idle state
> description, there's only one cpuidle driver and one table of idle
> states for all CPUs.
> 

Indeed.

> > As a result, only the first core to
> > initialize (usually a LITTLE core) sets up the C-states, and the same values are applied to all cores,
> > including the big ones. This leads to incorrect idle behavior on asymmetric platforms.
> >
> > I believe this behavior was introduced by commit 46bcfad7a819
> > ("cpuidle: Single/Global registration of idle states").
> >
> > I understand this design was introduced in 2011 to simplify cpuidle and reduce memory usage:
> > https://lkml.org/lkml/2011/4/25/83
> >
> > However, on today's heterogeneous SoCs, this global model no longer suffices. For proper modeling,
> > we need support for per-cluster or per-core cpuidle drivers, or at least some mechanism to allow
> > different idle state parameters per CPU.
> >
> > Has there been any discussion or work toward lifting this limitation?
> 
> No, there's not been any discussion on this so far, but why does the
> platform firmware on this system use _CST for idle state description?
> _LPI would be a better option AFAICS.
> 

Absolutely _LPI is better and I believe _LPI is used in this case. However
at the time I added _LPI support, the expectation was to use ACPI on SMP
systems without such variations in the idle states. So even with _LPIs,
only one cpuidle driver and hence only one table of idle states for all CPUs
was added. We can enhance the support for HMP systems with different set of
idle states if required. All we need is to allocate the driver instead of
using acpi_idle_driver IIRC. The initialisation of the states is already done
per cpu.

-- 
Regards,
Sudeep