TEO as default governor ?

TEO as default governor ?

[Daniel Lezcano]

Hi,

I think we can agree the teo governor is better then the menu governor.

Would it make sense to make the teo governor the default governor ?


-- 
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: TEO as default governor ?

[Ulf Hansson]

On Tue, 11 Mar 2025 at 17:31, Daniel Lezcano <daniel.lezcano@linaro.org> wrote:
>
>
> Hi,
>
> I think we can agree the teo governor is better then the menu governor.
>
> Would it make sense to make the teo governor the default governor ?

In this regards, I just wanted to share that I have been working on
adding domain-idlestate-statistics to genpd to see how those are being
predicted/selected for a group of CPUs (aka CPU-cluster) - and
together with the cpuidle-psci-domain (PSCI OSI mode).

I will send out those patches for genpd shortly, it should help if
people want to run some tests at their side with these kind of
platforms.

Kind regards
Uffe

Re: TEO as default governor ?

[Christian Loehle]

On 3/11/25 16:31, Daniel Lezcano wrote:
> 
> Hi,
> 
> I think we can agree the teo governor is better then the menu governor.
> 
> Would it make sense to make the teo governor the default governor ?
> 
> 

Rafael's position seems to be quite conservative here.
Fact is menu is still the default on many systems.
Even worse, the really bad performance disadvantage when
using menu in an intercept-heavy workload has been fixed by Rafael :)
https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/

FWIW I proposed this a while ago:
https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/

Re: TEO as default governor ?

[Rafael J. Wysocki]

On Tue, Mar 11, 2025 at 5:47 PM Christian Loehle
<christian.loehle@arm.com> wrote:
>
> On 3/11/25 16:31, Daniel Lezcano wrote:
> >
> > Hi,
> >
> > I think we can agree the teo governor is better then the menu governor.
> >
> > Would it make sense to make the teo governor the default governor ?
> >
> >
>
> Rafael's position seems to be quite conservative here.
> Fact is menu is still the default on many systems.
> Even worse, the really bad performance disadvantage when
> using menu in an intercept-heavy workload has been fixed by Rafael :)
> https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/
>
> FWIW I proposed this a while ago:
> https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/

It will help if one can make a really convincing case for this change
(that is, show that menu with the most recent fixes included is really
significantly worse on their platform).

Re: TEO as default governor ?

[Daniel Lezcano]

On 11/03/2025 18:34, Rafael J. Wysocki wrote:
> On Tue, Mar 11, 2025 at 5:47 PM Christian Loehle
> <christian.loehle@arm.com> wrote:
>>
>> On 3/11/25 16:31, Daniel Lezcano wrote:
>>>
>>> Hi,
>>>
>>> I think we can agree the teo governor is better then the menu governor.
>>>
>>> Would it make sense to make the teo governor the default governor ?
>>>
>>>
>>
>> Rafael's position seems to be quite conservative here.
>> Fact is menu is still the default on many systems.
>> Even worse, the really bad performance disadvantage when
>> using menu in an intercept-heavy workload has been fixed by Rafael :)
>> https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/
>>
>> FWIW I proposed this a while ago:
>> https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/
> 
> It will help if one can make a really convincing case for this change
> (that is, show that menu with the most recent fixes included is really
> significantly worse on their platform).

For all the platforms I've been testing, the teo governor is always the 
best one.

Using the menu governor has also an impact on the user experience as it 
lags on mobile.

After studying the history of the menu governor few years ago, it 
appeared the menu governor was introduced before the SMP was widely 
used. The strength of the menu governor was the ability to find 
repeating intervals but with he multiplication of the cores, the IPIs 
were introduced which increased the entropy of the busy-idle cycles 
duration, thus making the duration much more random and altering the 
menu governor prediction accuracy.



-- 
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: TEO as default governor ?

[Rafael J. Wysocki]

On Tue, Mar 11, 2025 at 7:00 PM Daniel Lezcano
<daniel.lezcano@linaro.org> wrote:
>
> On 11/03/2025 18:34, Rafael J. Wysocki wrote:
> > On Tue, Mar 11, 2025 at 5:47 PM Christian Loehle
> > <christian.loehle@arm.com> wrote:
> >>
> >> On 3/11/25 16:31, Daniel Lezcano wrote:
> >>>
> >>> Hi,
> >>>
> >>> I think we can agree the teo governor is better then the menu governor.
> >>>
> >>> Would it make sense to make the teo governor the default governor ?
> >>>
> >>>
> >>
> >> Rafael's position seems to be quite conservative here.
> >> Fact is menu is still the default on many systems.
> >> Even worse, the really bad performance disadvantage when
> >> using menu in an intercept-heavy workload has been fixed by Rafael :)
> >> https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/
> >>
> >> FWIW I proposed this a while ago:
> >> https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/
> >
> > It will help if one can make a really convincing case for this change
> > (that is, show that menu with the most recent fixes included is really
> > significantly worse on their platform).
>
> For all the platforms I've been testing, the teo governor is always the
> best one.

Great!  Can you please share any numbers?

> Using the menu governor has also an impact on the user experience as it
> lags on mobile.

Well, I'm not quite sure what you mean here?

> After studying the history of the menu governor few years ago, it
> appeared the menu governor was introduced before the SMP was widely
> used. The strength of the menu governor was the ability to find
> repeating intervals but with he multiplication of the cores, the IPIs
> were introduced which increased the entropy of the busy-idle cycles
> duration, thus making the duration much more random and altering the
> menu governor prediction accuracy.

While this arguably is the case, menu has also been changed quite a
bit since its introduction.

What I'm looking for really is a set of numbers showing the difference
and clearly pointing out that teo should be preferred.

Re: TEO as default governor ?

[Daniel Lezcano]

On 11/03/2025 19:26, Rafael J. Wysocki wrote:
> On Tue, Mar 11, 2025 at 7:00 PM Daniel Lezcano
> <daniel.lezcano@linaro.org> wrote:
>>
>> On 11/03/2025 18:34, Rafael J. Wysocki wrote:
>>> On Tue, Mar 11, 2025 at 5:47 PM Christian Loehle
>>> <christian.loehle@arm.com> wrote:
>>>>
>>>> On 3/11/25 16:31, Daniel Lezcano wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I think we can agree the teo governor is better then the menu governor.
>>>>>
>>>>> Would it make sense to make the teo governor the default governor ?
>>>>>
>>>>>
>>>>
>>>> Rafael's position seems to be quite conservative here.
>>>> Fact is menu is still the default on many systems.
>>>> Even worse, the really bad performance disadvantage when
>>>> using menu in an intercept-heavy workload has been fixed by Rafael :)
>>>> https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/
>>>>
>>>> FWIW I proposed this a while ago:
>>>> https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/
>>>
>>> It will help if one can make a really convincing case for this change
>>> (that is, show that menu with the most recent fixes included is really
>>> significantly worse on their platform).
>>
>> For all the platforms I've been testing, the teo governor is always the
>> best one.
> 
> Great!  Can you please share any numbers?

There are some at a publication doing an evaluation of the irq 
prediction [1]

>> Using the menu governor has also an impact on the user experience as it
>> lags on mobile.
> 
> Well, I'm not quite sure what you mean here?

For example, the user feels the lag when touching the screen on a mobile 
or scrolling a document. Changing from menu to teo solves this issue.


>> After studying the history of the menu governor few years ago, it
>> appeared the menu governor was introduced before the SMP was widely
>> used. The strength of the menu governor was the ability to find
>> repeating intervals but with he multiplication of the cores, the IPIs
>> were introduced which increased the entropy of the busy-idle cycles
>> duration, thus making the duration much more random and altering the
>> menu governor prediction accuracy.
> 
> While this arguably is the case, menu has also been changed quite a
> bit since its introduction.
> 
> What I'm looking for really is a set of numbers showing the difference
> and clearly pointing out that teo should be preferred.

Ok, let me check if I can find a platform doing energy measurement.

I guess x86 is not a good target as the firmware overcomes the kernel 
decisions right ?


   -- D.


[1] 
https://hal.science/hal-04095844v1/file/O_S_level_interrupt_prediction_for_performance_and_energy_management_on_Android.pdf

-- 
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: TEO as default governor ?

[Rafael J. Wysocki]

On Tue, Mar 11, 2025 at 7:42 PM Daniel Lezcano
<daniel.lezcano@linaro.org> wrote:
>
> On 11/03/2025 19:26, Rafael J. Wysocki wrote:
> > On Tue, Mar 11, 2025 at 7:00 PM Daniel Lezcano
> > <daniel.lezcano@linaro.org> wrote:
> >>
> >> On 11/03/2025 18:34, Rafael J. Wysocki wrote:
> >>> On Tue, Mar 11, 2025 at 5:47 PM Christian Loehle
> >>> <christian.loehle@arm.com> wrote:
> >>>>
> >>>> On 3/11/25 16:31, Daniel Lezcano wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I think we can agree the teo governor is better then the menu governor.
> >>>>>
> >>>>> Would it make sense to make the teo governor the default governor ?
> >>>>>
> >>>>>
> >>>>
> >>>> Rafael's position seems to be quite conservative here.
> >>>> Fact is menu is still the default on many systems.
> >>>> Even worse, the really bad performance disadvantage when
> >>>> using menu in an intercept-heavy workload has been fixed by Rafael :)
> >>>> https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/
> >>>>
> >>>> FWIW I proposed this a while ago:
> >>>> https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/
> >>>
> >>> It will help if one can make a really convincing case for this change
> >>> (that is, show that menu with the most recent fixes included is really
> >>> significantly worse on their platform).
> >>
> >> For all the platforms I've been testing, the teo governor is always the
> >> best one.
> >
> > Great!  Can you please share any numbers?
>
> There are some at a publication doing an evaluation of the irq
> prediction [1]
>
> >> Using the menu governor has also an impact on the user experience as it
> >> lags on mobile.
> >
> > Well, I'm not quite sure what you mean here?
>
> For example, the user feels the lag when touching the screen on a mobile
> or scrolling a document. Changing from menu to teo solves this issue.
>
>
> >> After studying the history of the menu governor few years ago, it
> >> appeared the menu governor was introduced before the SMP was widely
> >> used. The strength of the menu governor was the ability to find
> >> repeating intervals but with he multiplication of the cores, the IPIs
> >> were introduced which increased the entropy of the busy-idle cycles
> >> duration, thus making the duration much more random and altering the
> >> menu governor prediction accuracy.
> >
> > While this arguably is the case, menu has also been changed quite a
> > bit since its introduction.
> >
> > What I'm looking for really is a set of numbers showing the difference
> > and clearly pointing out that teo should be preferred.
>
> Ok, let me check if I can find a platform doing energy measurement.
>
> I guess x86 is not a good target as the firmware overcomes the kernel
> decisions right ?

In many cases, yes, it does.

You basically need to disable the feature called "C1 demotion", please
see the recent patch series from Artem adding an interface for this:

https://lore.kernel.org/linux-pm/20250220151702.2153579-1-dedekind1@gmail.com/

Re: TEO as default governor ?

[Christian Loehle]

On 3/11/25 18:51, Rafael J. Wysocki wrote:
> On Tue, Mar 11, 2025 at 7:42 PM Daniel Lezcano
> <daniel.lezcano@linaro.org> wrote:
>>
>> On 11/03/2025 19:26, Rafael J. Wysocki wrote:
>>> On Tue, Mar 11, 2025 at 7:00 PM Daniel Lezcano
>>> <daniel.lezcano@linaro.org> wrote:
>>>>
>>>> On 11/03/2025 18:34, Rafael J. Wysocki wrote:
>>>>> On Tue, Mar 11, 2025 at 5:47 PM Christian Loehle
>>>>> <christian.loehle@arm.com> wrote:
>>>>>>
>>>>>> On 3/11/25 16:31, Daniel Lezcano wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I think we can agree the teo governor is better then the menu governor.
>>>>>>>
>>>>>>> Would it make sense to make the teo governor the default governor ?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Rafael's position seems to be quite conservative here.
>>>>>> Fact is menu is still the default on many systems.
>>>>>> Even worse, the really bad performance disadvantage when
>>>>>> using menu in an intercept-heavy workload has been fixed by Rafael :)
>>>>>> https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/
>>>>>>
>>>>>> FWIW I proposed this a while ago:
>>>>>> https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/
>>>>>
>>>>> It will help if one can make a really convincing case for this change
>>>>> (that is, show that menu with the most recent fixes included is really
>>>>> significantly worse on their platform).
>>>>
>>>> For all the platforms I've been testing, the teo governor is always the
>>>> best one.
>>>
>>> Great!  Can you please share any numbers?
>>
>> There are some at a publication doing an evaluation of the irq
>> prediction [1]
>>
>>>> Using the menu governor has also an impact on the user experience as it
>>>> lags on mobile.
>>>
>>> Well, I'm not quite sure what you mean here?
>>
>> For example, the user feels the lag when touching the screen on a mobile
>> or scrolling a document. Changing from menu to teo solves this issue.
>>
>>
>>>> After studying the history of the menu governor few years ago, it
>>>> appeared the menu governor was introduced before the SMP was widely
>>>> used. The strength of the menu governor was the ability to find
>>>> repeating intervals but with he multiplication of the cores, the IPIs
>>>> were introduced which increased the entropy of the busy-idle cycles
>>>> duration, thus making the duration much more random and altering the
>>>> menu governor prediction accuracy.
>>>
>>> While this arguably is the case, menu has also been changed quite a
>>> bit since its introduction.
>>>
>>> What I'm looking for really is a set of numbers showing the difference
>>> and clearly pointing out that teo should be preferred.
>>
>> Ok, let me check if I can find a platform doing energy measurement.
>>
>> I guess x86 is not a good target as the firmware overcomes the kernel
>> decisions right ?
> 
> In many cases, yes, it does.

So do we btw to some extent (arm that is).
The kernel currently has no ultimate and general knowledge if anything
beyond WFI a) has been entered (without observing it's side effects) and
b) when this was done (i.e. how long that request was delayed for).
Even for WFI many platforms have essentially different levels of deepness
the kernel has neither knowledge nor control over (in the general case). [0]
Additionally we have the issue Uffe mentioned of cluster-sleeps, which are
manageable in OSI-mode (like he proposes), but many (I'd say most) platforms
out there are still PC-mode.
The entire cpuidle landscape has become quite nondeterministic I'm afraid.

There should still be workloads where teo performs better, but ultimately
there's also systems+workloads combinations where menu would (now) be
preferred.
(Generally teo still tends to prefer shallower idle states, on mobile that
is quite nice because a) we often are intercept-heavy b) we have quite
efficient WFI and c) for longer idle periods we try to use system suspend
anyway.)

Those are my two cents, without the mentioned menu patches teo was far
ahead in many workloads, now I think it's much more balanced.
I'm happy to revisit this too.

[0]
https://developer.arm.com/documentation/101433/0102/Functional-description/Power-management-/Core-power-modes/Core-dynamic-retention-mode
This has been mentioned previously on LKML to illustrate how WFI != WFI.

Re: TEO as default governor ?

[Christian Loehle]

On 3/11/25 18:00, Daniel Lezcano wrote:
> On 11/03/2025 18:34, Rafael J. Wysocki wrote:
>> On Tue, Mar 11, 2025 at 5:47 PM Christian Loehle
>> <christian.loehle@arm.com> wrote:
>>>
>>> On 3/11/25 16:31, Daniel Lezcano wrote:
>>>>
>>>> Hi,
>>>>
>>>> I think we can agree the teo governor is better then the menu governor.
>>>>
>>>> Would it make sense to make the teo governor the default governor ?
>>>>
>>>>
>>>
>>> Rafael's position seems to be quite conservative here.
>>> Fact is menu is still the default on many systems.
>>> Even worse, the really bad performance disadvantage when
>>> using menu in an intercept-heavy workload has been fixed by Rafael :)
>>> https://lore.kernel.org/lkml/bc7f915b-8d9f-4e05-9939-8b7ecc078f85@arm.com/
>>>
>>> FWIW I proposed this a while ago:
>>> https://lore.kernel.org/lkml/20240905092645.2885200-3-christian.loehle@arm.com/
>>
>> It will help if one can make a really convincing case for this change
>> (that is, show that menu with the most recent fixes included is really
>> significantly worse on their platform).
> 
> For all the platforms I've been testing, the teo governor is always the best one.
> 
> Using the menu governor has also an impact on the user experience as it lags on mobile.
> 
> After studying the history of the menu governor few years ago, it appeared the menu governor was introduced before the SMP was widely used. The strength of the menu governor was the ability to find repeating intervals but with he multiplication of the cores, the IPIs were introduced which increased the entropy of the busy-idle cycles duration, thus making the duration much more random and altering the menu governor prediction accuracy.

Cross-posting Doug's reply here and +CC
https://lore.kernel.org/lkml/005801db9397$266ddac0$73499040$@telus.net/

Daniel if you're testing both and struggle to find a strong advantage
for teo:
teo currently has the intercept-mechanism, trying to find a shallower state
until we are at < 1/2 intercepts as wakeups.
Rafael's idea was to make this configurable, e.g. allowing only 20%
intercepts (more aggressive in selecting shallow states) or 80% (less
aggressive in selecting shallow states).
This would be a minimal change in code and the rest of teo is unaffected,
but gives users a choice, something menu won't offer.