[PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

	Hi all,

When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:

    cpu cpu0: EM: invalid perf. state: -22

This happens because the Operating Points Parameters tables do not list
voltages, as they are all identical.  Previously, it was assumed they
were optional, and unused, when none of the CPU nodes is tied to a
regulator using the "cpu-supply" property.  This assumption turned out
to be incorrect, causing the reported error message.

This RFC patch series fixes this by adding the missing voltages.

Note that the Energy Model calculates energy efficiency by dividing the
(estimated) CPU power consumption by CPU core clock frequency.  When all
voltages have the same value, the former is proportional to clock
frequency, and energy efficiency becomes a constant.  Hence all
operating points are considered to have the same efficiency, and the
Energy Model always picks the one with the highest clock rate (see also
[1]).

Alternatively, the Energy Model could be changed to silently ignore OPP
tables with missing frequencies.  IMHO this is not an unusual case.

Which approach should be taken?
Thanks for your comments!

[1] "PM: EM: Question Potential Issue with EM and OPP Table in cpufreq
     ondemand Governor"
    https://lore.kernel.org/all/a2ca883e-122e-43a1-b377-c43956b5b3be@arm.com

Geert Uytterhoeven (2):
  arm64: dts: renesas: r8a774c0: Re-add voltages to OPP table
  arm64: dts: renesas: r8a77990: Re-add voltages to OPP table

 arch/arm64/boot/dts/renesas/r8a774c0.dtsi | 3 +++
 arch/arm64/boot/dts/renesas/r8a77990.dtsi | 3 +++
 2 files changed, 6 insertions(+)

-- 
2.34.1

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

[PATCH/RFC 1/2] arm64: dts: renesas: r8a774c0: Re-add voltages to OPP table

[Geert Uytterhoeven]

When CONFIG_ENERGY_MODEL=y:

    cpu cpu0: EM: invalid perf. state: -22

When removing the (incorrect) voltages from the Operating Points
Parameters tables, it was assumed they were optional, and unused, when
none of the CPU nodes is tied to a regulator using the "cpu-supply"
property.  This assumption turned out to be incorrect, causing the
reported error message.

Fix this by re-adding the (correct) voltages.  Note that because all
voltages are identical, all operating points are considered to have the
same efficiency, and the energy model always picks the one with the
highest clock rate.

Reported-by: Renesas Test Team via Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Fixes: 554edc3e9239bb81 ("arm64: dts: renesas: r8a774c0: Remove bogus voltages from OPP table")
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 arch/arm64/boot/dts/renesas/r8a774c0.dtsi | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/boot/dts/renesas/r8a774c0.dtsi b/arch/arm64/boot/dts/renesas/r8a774c0.dtsi
index 7655d5e3a034166e..5d4db20033212218 100644
--- a/arch/arm64/boot/dts/renesas/r8a774c0.dtsi
+++ b/arch/arm64/boot/dts/renesas/r8a774c0.dtsi
@@ -49,14 +49,17 @@ cluster1_opp: opp-table-1 {
 		opp-shared;
 		opp-800000000 {
 			opp-hz = /bits/ 64 <800000000>;
+			opp-microvolt = <1030000>;
 			clock-latency-ns = <300000>;
 		};
 		opp-1000000000 {
 			opp-hz = /bits/ 64 <1000000000>;
+			opp-microvolt = <1030000>;
 			clock-latency-ns = <300000>;
 		};
 		opp-1200000000 {
 			opp-hz = /bits/ 64 <1200000000>;
+			opp-microvolt = <1030000>;
 			clock-latency-ns = <300000>;
 			opp-suspend;
 		};
-- 
2.34.1

[PATCH/RFC 2/2] arm64: dts: renesas: r8a77990: Re-add voltages to OPP table

[Geert Uytterhoeven]

When CONFIG_ENERGY_MODEL=y:

    cpu cpu0: EM: invalid perf. state: -22

When removing the (incorrect) voltages from the Operating Points
Parameters tables, it was assumed they were optional, and unused, when
none of the CPU nodes is tied to a regulator using the "cpu-supply"
property.  This assumption turned out to be incorrect, causing the
reported error message.

Fix this by re-adding the (correct) voltages.  Note that because all
voltages are identical, all operating points are considered to have the
same efficiency, and the energy model always picks the one with the
highest clock rate.

Reported-by: Renesas Test Team via Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Fixes: fb76b0fae3ca8803 ("arm64: dts: renesas: r8a77990: Remove bogus voltages from OPP table")
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 arch/arm64/boot/dts/renesas/r8a77990.dtsi | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/boot/dts/renesas/r8a77990.dtsi b/arch/arm64/boot/dts/renesas/r8a77990.dtsi
index 233af3081e84a407..26d08f2cbcb6417b 100644
--- a/arch/arm64/boot/dts/renesas/r8a77990.dtsi
+++ b/arch/arm64/boot/dts/renesas/r8a77990.dtsi
@@ -49,14 +49,17 @@ cluster1_opp: opp-table-1 {
 		opp-shared;
 		opp-800000000 {
 			opp-hz = /bits/ 64 <800000000>;
+			opp-microvolt = <1030000>;
 			clock-latency-ns = <300000>;
 		};
 		opp-1000000000 {
 			opp-hz = /bits/ 64 <1000000000>;
+			opp-microvolt = <1030000>;
 			clock-latency-ns = <300000>;
 		};
 		opp-1200000000 {
 			opp-hz = /bits/ 64 <1200000000>;
+			opp-microvolt = <1030000>;
 			clock-latency-ns = <300000>;
 			opp-suspend;
 		};
-- 
2.34.1

RE: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Biju Das]

Hi Geert,

> -----Original Message-----
> From: Geert Uytterhoeven <geert+renesas@glider.be>
> Subject: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables
> 
> 	Hi all,
> 
> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
> 
>     cpu cpu0: EM: invalid perf. state: -22
> 
> This happens because the Operating Points Parameters tables do not list voltages, as they are all
> identical.  Previously, it was assumed they were optional, and unused, when none of the CPU nodes is
> tied to a regulator using the "cpu-supply" property.  This assumption turned out to be incorrect,
> causing the reported error message.
> 
> This RFC patch series fixes this by adding the missing voltages.
> 
> Note that the Energy Model calculates energy efficiency by dividing the
> (estimated) CPU power consumption by CPU core clock frequency.  When all voltages have the same value,
> the former is proportional to clock frequency, and energy efficiency becomes a constant.  Hence all
> operating points are considered to have the same efficiency, and the Energy Model always picks the one
> with the highest clock rate (see also [1]).
> 
> Alternatively, the Energy Model could be changed to silently ignore OPP tables with missing
> frequencies.  IMHO this is not an unusual case.

I guess cooling uses OPP table for frequency down scaling at higher temp. So, we need
OPP tables for thermal governors??

Cheers,
Biju

> 
> Which approach should be taken?
> Thanks for your comments!
> 
> [1] "PM: EM: Question Potential Issue with EM and OPP Table in cpufreq
>      ondemand Governor"
>     https://lore.kernel.org/all/a2ca883e-122e-43a1-b377-c43956b5b3be@arm.com
> 
> Geert Uytterhoeven (2):
>   arm64: dts: renesas: r8a774c0: Re-add voltages to OPP table
>   arm64: dts: renesas: r8a77990: Re-add voltages to OPP table
> 
>  arch/arm64/boot/dts/renesas/r8a774c0.dtsi | 3 +++  arch/arm64/boot/dts/renesas/r8a77990.dtsi | 3 +++
>  2 files changed, 6 insertions(+)
> 
> --
> 2.34.1
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But when I'm talking to
> journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

Hi Biju,

On Tue, Oct 8, 2024 at 11:30 AM Biju Das <biju.das.jz@bp.renesas.com> wrote:
> > -----Original Message-----
> > From: Geert Uytterhoeven <geert+renesas@glider.be>
> > Subject: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables
> >
> > When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
> >
> >     cpu cpu0: EM: invalid perf. state: -22
> >
> > This happens because the Operating Points Parameters tables do not list voltages, as they are all
> > identical.  Previously, it was assumed they were optional, and unused, when none of the CPU nodes is
> > tied to a regulator using the "cpu-supply" property.  This assumption turned out to be incorrect,
> > causing the reported error message.
> >
> > This RFC patch series fixes this by adding the missing voltages.
> >
> > Note that the Energy Model calculates energy efficiency by dividing the
> > (estimated) CPU power consumption by CPU core clock frequency.  When all voltages have the same value,
> > the former is proportional to clock frequency, and energy efficiency becomes a constant.  Hence all
> > operating points are considered to have the same efficiency, and the Energy Model always picks the one
> > with the highest clock rate (see also [1]).
> >
> > Alternatively, the Energy Model could be changed to silently ignore OPP tables with missing
> > frequencies.  IMHO this is not an unusual case.
>
> I guess cooling uses OPP table for frequency down scaling at higher temp. So, we need
> OPP tables for thermal governors??

Yes, you still need OPP tables for thermal governance.
But the Energy Model cannot really use it when the voltages are identical.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
<geert+renesas@glider.be> wrote:
> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
>
>     cpu cpu0: EM: invalid perf. state: -22
>
> This happens because the Operating Points Parameters tables do not list
> voltages, as they are all identical.  Previously, it was assumed they
> were optional, and unused, when none of the CPU nodes is tied to a
> regulator using the "cpu-supply" property.  This assumption turned out
> to be incorrect, causing the reported error message.
>
> This RFC patch series fixes this by adding the missing voltages.
>
> Note that the Energy Model calculates energy efficiency by dividing the
> (estimated) CPU power consumption by CPU core clock frequency.  When all
> voltages have the same value, the former is proportional to clock
> frequency, and energy efficiency becomes a constant.  Hence all
> operating points are considered to have the same efficiency, and the
> Energy Model always picks the one with the highest clock rate (see also
> [1]).
>
> Alternatively, the Energy Model could be changed to silently ignore OPP
> tables with missing frequencies.  IMHO this is not an unusual case.
>
> Which approach should be taken?
> Thanks for your comments!

Any comments from the Energy Model and PM people?
Thanks!

> [1] "PM: EM: Question Potential Issue with EM and OPP Table in cpufreq
>      ondemand Governor"
>     https://lore.kernel.org/all/a2ca883e-122e-43a1-b377-c43956b5b3be@arm.com
>
> Geert Uytterhoeven (2):
>   arm64: dts: renesas: r8a774c0: Re-add voltages to OPP table
>   arm64: dts: renesas: r8a77990: Re-add voltages to OPP table

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Lukasz Luba]

Hi Geert,

On 10/22/24 14:36, Geert Uytterhoeven wrote:
> On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
> <geert+renesas@glider.be> wrote:
>> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
>>
>>      cpu cpu0: EM: invalid perf. state: -22
>>
>> This happens because the Operating Points Parameters tables do not list
>> voltages, as they are all identical.  Previously, it was assumed they
>> were optional, and unused, when none of the CPU nodes is tied to a
>> regulator using the "cpu-supply" property.  This assumption turned out
>> to be incorrect, causing the reported error message.
>>
>> This RFC patch series fixes this by adding the missing voltages.
>>
>> Note that the Energy Model calculates energy efficiency by dividing the
>> (estimated) CPU power consumption by CPU core clock frequency.  When all
>> voltages have the same value, the former is proportional to clock
>> frequency, and energy efficiency becomes a constant.  Hence all
>> operating points are considered to have the same efficiency, and the
>> Energy Model always picks the one with the highest clock rate (see also
>> [1]).
>>
>> Alternatively, the Energy Model could be changed to silently ignore OPP
>> tables with missing frequencies.  IMHO this is not an unusual case.
>>
>> Which approach should be taken?
>> Thanks for your comments!
> 
> Any comments from the Energy Model and PM people?

My apologies for delay.

So you had issue with bogus Voltage values and removed them.

There is another way to setup EM properly, via DT:
"opp-microwatt" [1].

That micro watt value won't confuse other subsystems, like
your regulator fwk. It will only be used by the EM fwk.

This would be an alternative to your voltage values.
Sounds better to you?

Do you know from /sys/kernel/debug/energy_model/
the current power values?

Regards,
Lukasz

[1] 
https://www.kernel.org/doc/html/v6.11/power/energy-model.html#registration-of-em-using-dt

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

Hi Lukasz,

On Fri, Oct 25, 2024 at 5:40 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> On 10/22/24 14:36, Geert Uytterhoeven wrote:
> > On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
> > <geert+renesas@glider.be> wrote:
> >> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
> >>
> >>      cpu cpu0: EM: invalid perf. state: -22
> >>
> >> This happens because the Operating Points Parameters tables do not list
> >> voltages, as they are all identical.  Previously, it was assumed they
> >> were optional, and unused, when none of the CPU nodes is tied to a
> >> regulator using the "cpu-supply" property.  This assumption turned out
> >> to be incorrect, causing the reported error message.
> >>
> >> This RFC patch series fixes this by adding the missing voltages.
> >>
> >> Note that the Energy Model calculates energy efficiency by dividing the
> >> (estimated) CPU power consumption by CPU core clock frequency.  When all
> >> voltages have the same value, the former is proportional to clock
> >> frequency, and energy efficiency becomes a constant.  Hence all
> >> operating points are considered to have the same efficiency, and the
> >> Energy Model always picks the one with the highest clock rate (see also
> >> [1]).
> >>
> >> Alternatively, the Energy Model could be changed to silently ignore OPP
> >> tables with missing frequencies.  IMHO this is not an unusual case.
> >>
> >> Which approach should be taken?
> >> Thanks for your comments!
> >
> > Any comments from the Energy Model and PM people?
>
> My apologies for delay.
>
> So you had issue with bogus Voltage values and removed them.
>
> There is another way to setup EM properly, via DT:
> "opp-microwatt" [1].
>
> That micro watt value won't confuse other subsystems, like
> your regulator fwk. It will only be used by the EM fwk.
>
> This would be an alternative to your voltage values.
> Sounds better to you?

For opp-microwatt, I do need to know the actual power consumption
of the core, right?

Full system power consumption while running the in-kernel
Dhrystones benchmark:

800 MHz: avg 4972,55 mW, stdef 20,474 mW
1000 MHz: avg 5025,93 mW, stdef 18,644 mW
1200 MHz: avg 5059,63 mW, stdef 15,425 mW

The system also has test points across a 0.005 Ohm sense resistor in
the DVFS power supply line, but no on-board measurement sensor (like
the MAX9611 on Salvator-X(S)), so I haven't measured anything
there yet.

> Do you know from /sys/kernel/debug/energy_model/
> the current power values?

With this series applied:

root@ebisu:~# grep -r . /sys/kernel/debug/energy_model/
/sys/kernel/debug/energy_model/cpu0/ps:1200000/inefficient:0
/sys/kernel/debug/energy_model/cpu0/ps:1200000/performance:1024
/sys/kernel/debug/energy_model/cpu0/ps:1200000/cost:3443
/sys/kernel/debug/energy_model/cpu0/ps:1200000/power:352643
/sys/kernel/debug/energy_model/cpu0/ps:1200000/frequency:1200000
/sys/kernel/debug/energy_model/cpu0/ps:1000000/inefficient:1
/sys/kernel/debug/energy_model/cpu0/ps:1000000/performance:853
/sys/kernel/debug/energy_model/cpu0/ps:1000000/cost:3445
/sys/kernel/debug/energy_model/cpu0/ps:1000000/power:293869
/sys/kernel/debug/energy_model/cpu0/ps:1000000/frequency:1000000
/sys/kernel/debug/energy_model/cpu0/ps:800000/inefficient:1
/sys/kernel/debug/energy_model/cpu0/ps:800000/performance:682
/sys/kernel/debug/energy_model/cpu0/ps:800000/cost:3447
/sys/kernel/debug/energy_model/cpu0/ps:800000/power:235095
/sys/kernel/debug/energy_model/cpu0/ps:800000/frequency:800000
/sys/kernel/debug/energy_model/cpu0/flags:0x3
/sys/kernel/debug/energy_model/cpu0/cpus:0-1

Thanks!

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Lukasz Luba]

On 10/28/24 11:34, Geert Uytterhoeven wrote:
> Hi Lukasz,
> 
> On Fri, Oct 25, 2024 at 5:40 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
>> On 10/22/24 14:36, Geert Uytterhoeven wrote:
>>> On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
>>> <geert+renesas@glider.be> wrote:
>>>> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
>>>>
>>>>       cpu cpu0: EM: invalid perf. state: -22
>>>>
>>>> This happens because the Operating Points Parameters tables do not list
>>>> voltages, as they are all identical.  Previously, it was assumed they
>>>> were optional, and unused, when none of the CPU nodes is tied to a
>>>> regulator using the "cpu-supply" property.  This assumption turned out
>>>> to be incorrect, causing the reported error message.
>>>>
>>>> This RFC patch series fixes this by adding the missing voltages.
>>>>
>>>> Note that the Energy Model calculates energy efficiency by dividing the
>>>> (estimated) CPU power consumption by CPU core clock frequency.  When all
>>>> voltages have the same value, the former is proportional to clock
>>>> frequency, and energy efficiency becomes a constant.  Hence all
>>>> operating points are considered to have the same efficiency, and the
>>>> Energy Model always picks the one with the highest clock rate (see also
>>>> [1]).
>>>>
>>>> Alternatively, the Energy Model could be changed to silently ignore OPP
>>>> tables with missing frequencies.  IMHO this is not an unusual case.
>>>>
>>>> Which approach should be taken?
>>>> Thanks for your comments!
>>>
>>> Any comments from the Energy Model and PM people?
>>
>> My apologies for delay.
>>
>> So you had issue with bogus Voltage values and removed them.
>>
>> There is another way to setup EM properly, via DT:
>> "opp-microwatt" [1].
>>
>> That micro watt value won't confuse other subsystems, like
>> your regulator fwk. It will only be used by the EM fwk.
>>
>> This would be an alternative to your voltage values.
>> Sounds better to you?
> 
> For opp-microwatt, I do need to know the actual power consumption
> of the core, right?

Correct. You can try to derived that in a way you did and put below.
Although, Dhrystone is a synthetic micro-benchmark with small
impact to data caches, so it will not use much power.

> 
> Full system power consumption while running the in-kernel
> Dhrystones benchmark:
> 
> 800 MHz: avg 4972,55 mW, stdef 20,474 mW
> 1000 MHz: avg 5025,93 mW, stdef 18,644 mW
> 1200 MHz: avg 5059,63 mW, stdef 15,425 mW

Right. From those power values can be try to derive the
'CPU only power' values - assuming only one core was
running the test.

AFAIU you don't have proper DVFS due to missing voltage scaling.

Therefore...
Out of that I got these CPU power values:
800MHz -> 174mW
1000MHz -> 212mW
1200MHz -> 261mW

> 
> The system also has test points across a 0.005 Ohm sense resistor in
> the DVFS power supply line, but no on-board measurement sensor (like
> the MAX9611 on Salvator-X(S)), so I haven't measured anything
> there yet.
> 
>> Do you know from /sys/kernel/debug/energy_model/
>> the current power values?
> 
> With this series applied:
> 
> root@ebisu:~# grep -r . /sys/kernel/debug/energy_model/
> /sys/kernel/debug/energy_model/cpu0/ps:1200000/inefficient:0
> /sys/kernel/debug/energy_model/cpu0/ps:1200000/performance:1024
> /sys/kernel/debug/energy_model/cpu0/ps:1200000/cost:3443
> /sys/kernel/debug/energy_model/cpu0/ps:1200000/power:352643
> /sys/kernel/debug/energy_model/cpu0/ps:1200000/frequency:1200000
> /sys/kernel/debug/energy_model/cpu0/ps:1000000/inefficient:1
> /sys/kernel/debug/energy_model/cpu0/ps:1000000/performance:853
> /sys/kernel/debug/energy_model/cpu0/ps:1000000/cost:3445
> /sys/kernel/debug/energy_model/cpu0/ps:1000000/power:293869
> /sys/kernel/debug/energy_model/cpu0/ps:1000000/frequency:1000000
> /sys/kernel/debug/energy_model/cpu0/ps:800000/inefficient:1
> /sys/kernel/debug/energy_model/cpu0/ps:800000/performance:682
> /sys/kernel/debug/energy_model/cpu0/ps:800000/cost:3447
> /sys/kernel/debug/energy_model/cpu0/ps:800000/power:235095
> /sys/kernel/debug/energy_model/cpu0/ps:800000/frequency:800000
> /sys/kernel/debug/energy_model/cpu0/flags:0x3
> /sys/kernel/debug/energy_model/cpu0/cpus:0-1

Those power values listed above look a bit higher, but they
could be more related to a benchmark which utilized caches
and more parts of the CPU. I don't know if you had chance to
see some of my presentations on Linux conferences, where
I show how much power can vary in different scenarios at
the same frequency...
TLDR; it can be even 1.8x comparing to Dhrystone.

So would say it's OK for you to put either your Dhrystone
power results, or these one from EM dump (probably from
some more heavy benchmark then set into DT coefficient
to derive them in OPP fwk).

Your platform AFAIK is not Heterogeneous, so it won't be used
in EAS w/ that EM. It will be only used for thermal governor
IPA or PowerCap DTPM, which require EM to work.

Regards,
Lukasz

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

Hi Lukasz,

On Mon, Oct 28, 2024 at 2:41 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> On 10/28/24 11:34, Geert Uytterhoeven wrote:
> > On Fri, Oct 25, 2024 at 5:40 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> >> On 10/22/24 14:36, Geert Uytterhoeven wrote:
> >>> On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
> >>> <geert+renesas@glider.be> wrote:
> >>>> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
> >>>>
> >>>>       cpu cpu0: EM: invalid perf. state: -22
> >>>>
> >>>> This happens because the Operating Points Parameters tables do not list
> >>>> voltages, as they are all identical.  Previously, it was assumed they
> >>>> were optional, and unused, when none of the CPU nodes is tied to a
> >>>> regulator using the "cpu-supply" property.  This assumption turned out
> >>>> to be incorrect, causing the reported error message.
> >>>>
> >>>> This RFC patch series fixes this by adding the missing voltages.
> >>>>
> >>>> Note that the Energy Model calculates energy efficiency by dividing the
> >>>> (estimated) CPU power consumption by CPU core clock frequency.  When all
> >>>> voltages have the same value, the former is proportional to clock
> >>>> frequency, and energy efficiency becomes a constant.  Hence all
> >>>> operating points are considered to have the same efficiency, and the
> >>>> Energy Model always picks the one with the highest clock rate (see also
> >>>> [1]).
> >>>>
> >>>> Alternatively, the Energy Model could be changed to silently ignore OPP
> >>>> tables with missing frequencies.  IMHO this is not an unusual case.
> >>>>
> >>>> Which approach should be taken?
> >>>> Thanks for your comments!
> >>>
> >>> Any comments from the Energy Model and PM people?
> >>
> >> My apologies for delay.
> >>
> >> So you had issue with bogus Voltage values and removed them.
> >>
> >> There is another way to setup EM properly, via DT:
> >> "opp-microwatt" [1].
> >>
> >> That micro watt value won't confuse other subsystems, like
> >> your regulator fwk. It will only be used by the EM fwk.
> >>
> >> This would be an alternative to your voltage values.
> >> Sounds better to you?
> >
> > For opp-microwatt, I do need to know the actual power consumption
> > of the core, right?
>
> Correct. You can try to derived that in a way you did and put below.
> Although, Dhrystone is a synthetic micro-benchmark with small
> impact to data caches, so it will not use much power.

Do you have a suggestion for a better load test? stress-ng?

> > Full system power consumption while running the in-kernel
> > Dhrystones benchmark:
> >
> > 800 MHz: avg 4972,55 mW, stdef 20,474 mW
> > 1000 MHz: avg 5025,93 mW, stdef 18,644 mW
> > 1200 MHz: avg 5059,63 mW, stdef 15,425 mW
>
> Right. From those power values can be try to derive the
> 'CPU only power' values - assuming only one core was
> running the test.
>
> AFAIU you don't have proper DVFS due to missing voltage scaling.

Indeed.

> Therefore...
> Out of that I got these CPU power values:
> 800MHz -> 174mW

=> 217.5 µW/MHz

> 1000MHz -> 212mW

=> 212 µW/MHz

> 1200MHz -> 261mW

=> 217.5 µW/MHz.

So 1000 MHz seems to be the most power-efficient.

> > The system also has test points across a 0.005 Ohm sense resistor in
> > the DVFS power supply line, but no on-board measurement sensor (like
> > the MAX9611 on Salvator-X(S)), so I haven't measured anything
> > there yet.

I'll try to do some measurements at these test points.

> >> Do you know from /sys/kernel/debug/energy_model/
> >> the current power values?
> >
> > With this series applied:
> >
> > root@ebisu:~# grep -r . /sys/kernel/debug/energy_model/
> > /sys/kernel/debug/energy_model/cpu0/ps:1200000/inefficient:0
> > /sys/kernel/debug/energy_model/cpu0/ps:1200000/performance:1024
> > /sys/kernel/debug/energy_model/cpu0/ps:1200000/cost:3443
> > /sys/kernel/debug/energy_model/cpu0/ps:1200000/power:352643
> > /sys/kernel/debug/energy_model/cpu0/ps:1200000/frequency:1200000
> > /sys/kernel/debug/energy_model/cpu0/ps:1000000/inefficient:1
> > /sys/kernel/debug/energy_model/cpu0/ps:1000000/performance:853
> > /sys/kernel/debug/energy_model/cpu0/ps:1000000/cost:3445
> > /sys/kernel/debug/energy_model/cpu0/ps:1000000/power:293869
> > /sys/kernel/debug/energy_model/cpu0/ps:1000000/frequency:1000000
> > /sys/kernel/debug/energy_model/cpu0/ps:800000/inefficient:1
> > /sys/kernel/debug/energy_model/cpu0/ps:800000/performance:682
> > /sys/kernel/debug/energy_model/cpu0/ps:800000/cost:3447
> > /sys/kernel/debug/energy_model/cpu0/ps:800000/power:235095
> > /sys/kernel/debug/energy_model/cpu0/ps:800000/frequency:800000
> > /sys/kernel/debug/energy_model/cpu0/flags:0x3
> > /sys/kernel/debug/energy_model/cpu0/cpus:0-1
>
> Those power values listed above look a bit higher, but they
> could be more related to a benchmark which utilized caches
> and more parts of the CPU. I don't know if you had chance to
> see some of my presentations on Linux conferences, where
> I show how much power can vary in different scenarios at
> the same frequency...

Thanks, I will have a look...

> TLDR; it can be even 1.8x comparing to Dhrystone.
>
> So would say it's OK for you to put either your Dhrystone
> power results, or these one from EM dump (probably from
> some more heavy benchmark then set into DT coefficient
> to derive them in OPP fwk).

Given the figures above (212 µW/MHz vs. 217.5 µW/MHz) using rather
coarse measurements are already close or identical, they might end up
being identical, and then we're back at square zero, where EM cannot
do anything?

Thanks!

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

Hi Lukasz,

On Mon, Nov 4, 2024 at 4:15 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Mon, Oct 28, 2024 at 2:41 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> > On 10/28/24 11:34, Geert Uytterhoeven wrote:
> > > On Fri, Oct 25, 2024 at 5:40 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> > >> On 10/22/24 14:36, Geert Uytterhoeven wrote:
> > >>> On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
> > >>> <geert+renesas@glider.be> wrote:
> > >>>> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
> > >>>>
> > >>>>       cpu cpu0: EM: invalid perf. state: -22
> > >>>>
> > >>>> This happens because the Operating Points Parameters tables do not list
> > >>>> voltages, as they are all identical.  Previously, it was assumed they
> > >>>> were optional, and unused, when none of the CPU nodes is tied to a
> > >>>> regulator using the "cpu-supply" property.  This assumption turned out
> > >>>> to be incorrect, causing the reported error message.
> > >>>>
> > >>>> This RFC patch series fixes this by adding the missing voltages.
> > >>>>
> > >>>> Note that the Energy Model calculates energy efficiency by dividing the
> > >>>> (estimated) CPU power consumption by CPU core clock frequency.  When all
> > >>>> voltages have the same value, the former is proportional to clock
> > >>>> frequency, and energy efficiency becomes a constant.  Hence all
> > >>>> operating points are considered to have the same efficiency, and the
> > >>>> Energy Model always picks the one with the highest clock rate (see also
> > >>>> [1]).
> > >>>>
> > >>>> Alternatively, the Energy Model could be changed to silently ignore OPP
> > >>>> tables with missing frequencies.  IMHO this is not an unusual case.
> > >>>>
> > >>>> Which approach should be taken?
> > >>>> Thanks for your comments!
> > >>>
> > >>> Any comments from the Energy Model and PM people?
> > >>
> > >> My apologies for delay.
> > >>
> > >> So you had issue with bogus Voltage values and removed them.
> > >>
> > >> There is another way to setup EM properly, via DT:
> > >> "opp-microwatt" [1].
> > >>
> > >> That micro watt value won't confuse other subsystems, like
> > >> your regulator fwk. It will only be used by the EM fwk.
> > >>
> > >> This would be an alternative to your voltage values.
> > >> Sounds better to you?
> > >
> > > For opp-microwatt, I do need to know the actual power consumption
> > > of the core, right?
> >
> > Correct. You can try to derived that in a way you did and put below.
> > Although, Dhrystone is a synthetic micro-benchmark with small
> > impact to data caches, so it will not use much power.
>
> Do you have a suggestion for a better load test? stress-ng?
>
> > > Full system power consumption while running the in-kernel
> > > Dhrystones benchmark:
> > >
> > > 800 MHz: avg 4972,55 mW, stdef 20,474 mW
> > > 1000 MHz: avg 5025,93 mW, stdef 18,644 mW
> > > 1200 MHz: avg 5059,63 mW, stdef 15,425 mW
> >
> > Right. From those power values can be try to derive the
> > 'CPU only power' values - assuming only one core was
> > running the test.
> >
> > AFAIU you don't have proper DVFS due to missing voltage scaling.
>
> Indeed.
>
> > Therefore...
> > Out of that I got these CPU power values:
> > 800MHz -> 174mW
>
> => 217.5 µW/MHz
>
> > 1000MHz -> 212mW
>
> => 212 µW/MHz
>
> > 1200MHz -> 261mW

BTW, how did you get from my avg mW values above to your CPU power mW
values? I seem to be missing something...

>
> => 217.5 µW/MHz.
>
> So 1000 MHz seems to be the most power-efficient.
>
> > > The system also has test points across a 0.005 Ohm sense resistor in
> > > the DVFS power supply line, but no on-board measurement sensor (like
> > > the MAX9611 on Salvator-X(S)), so I haven't measured anything
> > > there yet.
>
> I'll try to do some measurements at these test points.

So I measured the voltage across the sense resistor, and used that to
calculate the actual power draw:
  A. Idle (1 or 2 CPU cores online doesn't seem to matter):
      -  765 mW @  800 MHz,
      -  786 mW @ 1000 MHz,
      -  807 mW @ 1200 MHz.
  B. Maximum seen during "stress-ng -c 10" with 1 CPU core online:
      -  993 mW @  800 MHz,
      - 1055 mW @ 1000 MHz,
      - 1096 mW @ 1200 MHz.
  C. Maximum seen during "stress-ng -c 10" with 2 CPU cores online:
      - 1179 mW @  800 MHz,
      - 1303 mW @ 1000 MHz,
      - 1386 mW @ 1200 MHz.

As expected, the A-C increase is about the double of the A-B increase,
due to the use of 2 CPU cores.

Thanks again!

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

Hi Lukasz,

On Wed, Nov 13, 2024 at 5:58 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Mon, Nov 4, 2024 at 4:15 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> > On Mon, Oct 28, 2024 at 2:41 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> > > On 10/28/24 11:34, Geert Uytterhoeven wrote:
> > > > On Fri, Oct 25, 2024 at 5:40 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> > > >> On 10/22/24 14:36, Geert Uytterhoeven wrote:
> > > >>> On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
> > > >>> <geert+renesas@glider.be> wrote:
> > > >>>> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
> > > >>>>
> > > >>>>       cpu cpu0: EM: invalid perf. state: -22
> > > >>>>
> > > >>>> This happens because the Operating Points Parameters tables do not list
> > > >>>> voltages, as they are all identical.  Previously, it was assumed they
> > > >>>> were optional, and unused, when none of the CPU nodes is tied to a
> > > >>>> regulator using the "cpu-supply" property.  This assumption turned out
> > > >>>> to be incorrect, causing the reported error message.
> > > >>>>
> > > >>>> This RFC patch series fixes this by adding the missing voltages.
> > > >>>>
> > > >>>> Note that the Energy Model calculates energy efficiency by dividing the
> > > >>>> (estimated) CPU power consumption by CPU core clock frequency.  When all
> > > >>>> voltages have the same value, the former is proportional to clock
> > > >>>> frequency, and energy efficiency becomes a constant.  Hence all
> > > >>>> operating points are considered to have the same efficiency, and the
> > > >>>> Energy Model always picks the one with the highest clock rate (see also
> > > >>>> [1]).
> > > >>>>
> > > >>>> Alternatively, the Energy Model could be changed to silently ignore OPP
> > > >>>> tables with missing frequencies.  IMHO this is not an unusual case.
> > > >>>>
> > > >>>> Which approach should be taken?
> > > >>>> Thanks for your comments!
> > > >>>
> > > >>> Any comments from the Energy Model and PM people?
> > > >>
> > > >> My apologies for delay.
> > > >>
> > > >> So you had issue with bogus Voltage values and removed them.
> > > >>
> > > >> There is another way to setup EM properly, via DT:
> > > >> "opp-microwatt" [1].
> > > >>
> > > >> That micro watt value won't confuse other subsystems, like
> > > >> your regulator fwk. It will only be used by the EM fwk.
> > > >>
> > > >> This would be an alternative to your voltage values.
> > > >> Sounds better to you?
> > > >
> > > > For opp-microwatt, I do need to know the actual power consumption
> > > > of the core, right?
> > >
> > > Correct. You can try to derived that in a way you did and put below.
> > > Although, Dhrystone is a synthetic micro-benchmark with small
> > > impact to data caches, so it will not use much power.
> >
> > Do you have a suggestion for a better load test? stress-ng?
> >
> > > > Full system power consumption while running the in-kernel
> > > > Dhrystones benchmark:
> > > >
> > > > 800 MHz: avg 4972,55 mW, stdef 20,474 mW
> > > > 1000 MHz: avg 5025,93 mW, stdef 18,644 mW
> > > > 1200 MHz: avg 5059,63 mW, stdef 15,425 mW
> > >
> > > Right. From those power values can be try to derive the
> > > 'CPU only power' values - assuming only one core was
> > > running the test.
> > >
> > > AFAIU you don't have proper DVFS due to missing voltage scaling.
> >
> > Indeed.
> >
> > > Therefore...
> > > Out of that I got these CPU power values:
> > > 800MHz -> 174mW
> >
> > => 217.5 µW/MHz
> >
> > > 1000MHz -> 212mW
> >
> > => 212 µW/MHz
> >
> > > 1200MHz -> 261mW
>
> BTW, how did you get from my avg mW values above to your CPU power mW
> values? I seem to be missing something...

Ping...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

Hi Lukasz,

On Tue, Dec 10, 2024 at 3:28 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Wed, Nov 13, 2024 at 5:58 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> > On Mon, Nov 4, 2024 at 4:15 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> > > On Mon, Oct 28, 2024 at 2:41 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> > > > On 10/28/24 11:34, Geert Uytterhoeven wrote:
> > > > > On Fri, Oct 25, 2024 at 5:40 PM Lukasz Luba <lukasz.luba@arm.com> wrote:
> > > > >> On 10/22/24 14:36, Geert Uytterhoeven wrote:
> > > > >>> On Tue, Oct 8, 2024 at 11:14 AM Geert Uytterhoeven
> > > > >>> <geert+renesas@glider.be> wrote:
> > > > >>>> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
> > > > >>>>
> > > > >>>>       cpu cpu0: EM: invalid perf. state: -22
> > > > >>>>
> > > > >>>> This happens because the Operating Points Parameters tables do not list
> > > > >>>> voltages, as they are all identical.  Previously, it was assumed they
> > > > >>>> were optional, and unused, when none of the CPU nodes is tied to a
> > > > >>>> regulator using the "cpu-supply" property.  This assumption turned out
> > > > >>>> to be incorrect, causing the reported error message.
> > > > >>>>
> > > > >>>> This RFC patch series fixes this by adding the missing voltages.
> > > > >>>>
> > > > >>>> Note that the Energy Model calculates energy efficiency by dividing the
> > > > >>>> (estimated) CPU power consumption by CPU core clock frequency.  When all
> > > > >>>> voltages have the same value, the former is proportional to clock
> > > > >>>> frequency, and energy efficiency becomes a constant.  Hence all
> > > > >>>> operating points are considered to have the same efficiency, and the
> > > > >>>> Energy Model always picks the one with the highest clock rate (see also
> > > > >>>> [1]).
> > > > >>>>
> > > > >>>> Alternatively, the Energy Model could be changed to silently ignore OPP
> > > > >>>> tables with missing frequencies.  IMHO this is not an unusual case.
> > > > >>>>
> > > > >>>> Which approach should be taken?
> > > > >>>> Thanks for your comments!
> > > > >>>
> > > > >>> Any comments from the Energy Model and PM people?
> > > > >>
> > > > >> My apologies for delay.
> > > > >>
> > > > >> So you had issue with bogus Voltage values and removed them.
> > > > >>
> > > > >> There is another way to setup EM properly, via DT:
> > > > >> "opp-microwatt" [1].
> > > > >>
> > > > >> That micro watt value won't confuse other subsystems, like
> > > > >> your regulator fwk. It will only be used by the EM fwk.
> > > > >>
> > > > >> This would be an alternative to your voltage values.
> > > > >> Sounds better to you?
> > > > >
> > > > > For opp-microwatt, I do need to know the actual power consumption
> > > > > of the core, right?
> > > >
> > > > Correct. You can try to derived that in a way you did and put below.
> > > > Although, Dhrystone is a synthetic micro-benchmark with small
> > > > impact to data caches, so it will not use much power.
> > >
> > > Do you have a suggestion for a better load test? stress-ng?
> > >
> > > > > Full system power consumption while running the in-kernel
> > > > > Dhrystones benchmark:
> > > > >
> > > > > 800 MHz: avg 4972,55 mW, stdef 20,474 mW
> > > > > 1000 MHz: avg 5025,93 mW, stdef 18,644 mW
> > > > > 1200 MHz: avg 5059,63 mW, stdef 15,425 mW
> > > >
> > > > Right. From those power values can be try to derive the
> > > > 'CPU only power' values - assuming only one core was
> > > > running the test.
> > > >
> > > > AFAIU you don't have proper DVFS due to missing voltage scaling.
> > >
> > > Indeed.
> > >
> > > > Therefore...
> > > > Out of that I got these CPU power values:
> > > > 800MHz -> 174mW
> > >
> > > => 217.5 µW/MHz
> > >
> > > > 1000MHz -> 212mW
> > >
> > > => 212 µW/MHz
> > >
> > > > 1200MHz -> 261mW
> >
> > BTW, how did you get from my avg mW values above to your CPU power mW
> > values? I seem to be missing something...
>
> Ping...

Gentle ping.
Thanks!

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC 0/2] arm64: dts: renesas: Re-add voltages to OPP tables

[Geert Uytterhoeven]

On Tue, 8 Oct 2024 at 11:14, Geert Uytterhoeven <geert+renesas@glider.be> wrote:
> When CONFIG_ENERGY_MODEL=y, an error is printed on RZ/G2E and R-Car E3:
>
>     cpu cpu0: EM: invalid perf. state: -22
>
> This happens because the Operating Points Parameters tables do not list
> voltages, as they are all identical.  Previously, it was assumed they
> were optional, and unused, when none of the CPU nodes is tied to a
> regulator using the "cpu-supply" property.  This assumption turned out
> to be incorrect, causing the reported error message.
>
> This RFC patch series fixes this by adding the missing voltages.
>
> Note that the Energy Model calculates energy efficiency by dividing the
> (estimated) CPU power consumption by CPU core clock frequency.  When all
> voltages have the same value, the former is proportional to clock
> frequency, and energy efficiency becomes a constant.  Hence all
> operating points are considered to have the same efficiency, and the
> Energy Model always picks the one with the highest clock rate (see also
> [1]).
>
> Alternatively, the Energy Model could be changed to silently ignore OPP
> tables with missing frequencies.  IMHO this is not an unusual case.
>
> Which approach should be taken?
> Thanks for your comments!
>
> [1] "PM: EM: Question Potential Issue with EM and OPP Table in cpufreq
>      ondemand Governor"
>     https://lore.kernel.org/all/a2ca883e-122e-43a1-b377-c43956b5b3be@arm.com
>
> Geert Uytterhoeven (2):
>   arm64: dts: renesas: r8a774c0: Re-add voltages to OPP table
>   arm64: dts: renesas: r8a77990: Re-add voltages to OPP table
>
>  arch/arm64/boot/dts/renesas/r8a774c0.dtsi | 3 +++
>  arch/arm64/boot/dts/renesas/r8a77990.dtsi | 3 +++
>  2 files changed, 6 insertions(+)

Queuing in renesas-devel for v6.15.
This can be replaced by an alternative solution later...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds