[RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Tomeu,

Since you asked for fixes to be sent upfront I have kept poking at the
RK3588 NPU, and I ended up implementing devfreq for rocket locally.  It
works, but on the way there I hit a crash class that I could not find
documented anywhere, and I also measured something about the vendor OPP
table that I did not expect.  Both seem worth sharing before I clean any
of it up for posting, so I would rather ask first than send a series you
may not want in this shape.

Cc'ing Jiaxing since he is working on the clocks and on RK3576.


1. The hardware constraint
==========================

An NPU power domain cannot be switched on or off while the NPU compute
clock is above its DT assigned-clock-rate of 200 MHz.

Changing the rate while a domain is already on is fine - I have taken it
to 1 GHz and back many times without a single error.  It is the domain
transition that breaks.

What happens when a domain is moved at a high rate:

  rockchip-pm-domain ...: failed to get ack on domain 'nputop', val=0xa9ffe
  rocket fdab0000.npu: devfreq: cannot power up for rate change: -110

The domain is then wedged: genpd still believes it is on, but the first
MMIO into it raises an asynchronous SError and the box panics.  Captured
over the serial console:

  Kernel panic - not syncing: Asynchronous SError Interrupt
  Comm: rmmod
   _regmap_read
   regmap_read
   rockchip_pd_power
   rockchip_pd_power_off
   _genpd_power_off        <- rollback
   genpd_power_off
   genpd_power_on          <- failed
   genpd_runtime_resume
   device_release_driver

This is not specific to nputop.  I have the same message for 'npu2'
(val=0xa9fff), which matches the DT: all three NPU domains list the NPU
clock among their handshake clocks - rk3588-base.dtsi lines 864, 877 and
885, for RK3588_PD_NPUTOP, RK3588_PD_NPU1 and RK3588_PD_NPU2.

So the clock the domains need for their idle/ack handshake is the same
clock we would be scaling.  My best explanation is that the PLL that
produces it lives inside the domain, so once the domain drops, the clock
state goes with it and the handshake can never complete.  I cannot
confirm that part - reading the PVTPLL registers is documented as
hanging the machine, so I have not tried.  The behaviour itself is
reproducible and cost me four hard hangs before I understood it.

I mention it because it is a trap for anyone adding DVFS here, including
the RK3576 work, and because it is invisible until the first time you
let the NPU idle at a raised clock.


2. What ended up working
========================

Runtime PM callbacks are not enough.  The domains are powered on by the
driver core before probe, powered off from a workqueue after detach, and
system sleep bypasses runtime PM references entirely - so the driver
never sees all the transitions.

What does work is hooking the transitions themselves:
dev_pm_genpd_add_notifier() on all three cores, and on GENPD_NOTIFY_PRE_ON
and GENPD_NOTIFY_PRE_OFF force the clock back to the DT rate, vetoing the
transition with notifier_from_errno() if that fails.  Every path -
runtime PM, system sleep, attach at probe, detach after unbind - goes
through _genpd_power_on()/_genpd_power_off(), so nothing can slip past.
If a transition does happen, the boost cancels itself and says so,
rather than leaving the driver claiming a rate the hardware is not
running.

This has now survived everything that used to kill the box, including
repeated sleep/wake cycles at a raised clock and rmmod while boosted.


3. The numbers, which are the surprising part
=============================================

Measured with MobileNetV1 through Teflon, one inference thread pinned to
one A76, and a bit-exact oracle: sha256 over intermediate tensors on
every iteration, zero tolerance.  The 600, 900 and 1000 MHz rows are
30-minute runs of 275k-280k inferences each and the oracle passed
bit-exact in all three, so none of this is instability; the 200 MHz row
is a shorter control from the same session.

  nominal    supply    throughput
  200 MHz    800 mV     68.5 inf/s      (the current fixed rate)
  600 MHz    800 mV    155.4 inf/s
  900 MHz    850 mV    152.4 inf/s
  1000 MHz   850 mV    152.9 inf/s

600 MHz is the optimum.  900 and 1000 are indistinguishable from each
other and both land about 4% below 600, on a quieter and cooler machine.
The vendor OPP table decoded from the downstream DTB asks for 700 mV up
to 700 MHz, 750 mV at 800, 800 mV at 900 and 850 mV at 1000, and I ran
the top of that range at the voltage it asks for - it does not help.

Backing out an effective clock from the per-chunk time, nominal 600
appears to deliver more than nominal 900 or 1000 do.  I would not lean
on that decode, but the throughput ordering does not depend on it.

Thermals were never a factor: the highest I saw all day was 50.8 degC,
against a critical trip at 115.  This is one board and one model, and
MobileNetV1 is memory-heavy, so a compute-dense network may well behave
differently - but for this workload the top half of the vendor table
buys nothing.

If that holds up elsewhere, an OPP table for rocket probably should not
simply mirror the vendor one.


4. Thermal, separately
======================

While looking at this I noticed npu-thermal has only a critical trip at
115 degC - no passive trip, no cooling map, polling-delay-passive is 0 -
while gpu-thermal, a few lines above in the same file and on the same
tsadc, has both.  That was harmless while the NPU was pinned at 200 MHz
because it could not be slowed down anyway; with DVFS it stops being
harmless.

I have two small patches for that (a #cooling-cells binding update and
the thermal zone itself), but they only make sense once something
registers a cooling device, so they would belong with the driver work
rather than on their own.


5. What I am asking
===================

- Is DVFS for rocket something you want upstream at all, or is it better
  left alone for now?

- Does the genpd-notifier approach look right to you, or is there a
  cleaner hook I have missed?

- Given the measurements, would you want the OPP table to stop at
  600 MHz rather than follow the vendor range?

- If you do want a series, I will need to clean the code up first - it
  still keeps its state file-static rather than in rocket_device, and it
  bypasses the OPP layer for the rate change because our own direct
  writes leave the OPP cache stale.  Both are fixable; I would rather
  know the shape you want before rewriting.

Happy to send the current code as-is off-list if that is easier to
comment on than prose.

Thanks,
Igor

[RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Tomeu,

Since you asked for fixes to be sent upfront I have kept poking at the
RK3588 NPU, and I ended up implementing devfreq for rocket locally.  It
works, but on the way there I hit a crash class that I could not find
documented anywhere, and I also measured something about the vendor OPP
table that I did not expect.  Both seem worth sharing before I clean any
of it up for posting, so I would rather ask first than send a series you
may not want in this shape.

Cc'ing Jiaxing since he is working on the clocks and on RK3576.


1. The hardware constraint
==========================

An NPU power domain cannot be switched on or off while the NPU compute
clock is above its DT assigned-clock-rate of 200 MHz.

Changing the rate while a domain is already on is fine - I have taken it
to 1 GHz and back many times without a single error.  It is the domain
transition that breaks.

What happens when a domain is moved at a high rate:

  rockchip-pm-domain ...: failed to get ack on domain 'nputop', val=0xa9ffe
  rocket fdab0000.npu: devfreq: cannot power up for rate change: -110

The domain is then wedged: genpd still believes it is on, but the first
MMIO into it raises an asynchronous SError and the box panics.  Captured
over the serial console:

  Kernel panic - not syncing: Asynchronous SError Interrupt
  Comm: rmmod
   _regmap_read
   regmap_read
   rockchip_pd_power
   rockchip_pd_power_off
   _genpd_power_off        <- rollback
   genpd_power_off
   genpd_power_on          <- failed
   genpd_runtime_resume
   device_release_driver

This is not specific to nputop.  I have the same message for 'npu2'
(val=0xa9fff), which matches the DT: all three NPU domains list the NPU
clock among their handshake clocks - rk3588-base.dtsi lines 864, 877 and
885, for RK3588_PD_NPUTOP, RK3588_PD_NPU1 and RK3588_PD_NPU2.

So the clock the domains need for their idle/ack handshake is the same
clock we would be scaling.  My best explanation is that the PLL that
produces it lives inside the domain, so once the domain drops, the clock
state goes with it and the handshake can never complete.  I cannot
confirm that part - reading the PVTPLL registers is documented as
hanging the machine, so I have not tried.  The behaviour itself is
reproducible and cost me four hard hangs before I understood it.

I mention it because it is a trap for anyone adding DVFS here, including
the RK3576 work, and because it is invisible until the first time you
let the NPU idle at a raised clock.


2. What ended up working
========================

Runtime PM callbacks are not enough.  The domains are powered on by the
driver core before probe, powered off from a workqueue after detach, and
system sleep bypasses runtime PM references entirely - so the driver
never sees all the transitions.

What does work is hooking the transitions themselves:
dev_pm_genpd_add_notifier() on all three cores, and on GENPD_NOTIFY_PRE_ON
and GENPD_NOTIFY_PRE_OFF force the clock back to the DT rate, vetoing the
transition with notifier_from_errno() if that fails.  Every path -
runtime PM, system sleep, attach at probe, detach after unbind - goes
through _genpd_power_on()/_genpd_power_off(), so nothing can slip past.
If a transition does happen, the boost cancels itself and says so,
rather than leaving the driver claiming a rate the hardware is not
running.

This has now survived everything that used to kill the box, including
repeated sleep/wake cycles at a raised clock and rmmod while boosted.


3. The numbers, which are the surprising part
=============================================

Measured with MobileNetV1 through Teflon, one inference thread pinned to
one A76, and a bit-exact oracle: sha256 over intermediate tensors on
every iteration, zero tolerance.  The 600, 900 and 1000 MHz rows are
30-minute runs of 275k-280k inferences each and the oracle passed
bit-exact in all three, so none of this is instability; the 200 MHz row
is a shorter control from the same session.

  nominal    supply    throughput
  200 MHz    800 mV     68.5 inf/s      (the current fixed rate)
  600 MHz    800 mV    155.4 inf/s
  900 MHz    850 mV    152.4 inf/s
  1000 MHz   850 mV    152.9 inf/s

600 MHz is the optimum.  900 and 1000 are indistinguishable from each
other and both land about 4% below 600, on a quieter and cooler machine.
The vendor OPP table decoded from the downstream DTB asks for 700 mV up
to 700 MHz, 750 mV at 800, 800 mV at 900 and 850 mV at 1000, and I ran
the top of that range at the voltage it asks for - it does not help.

Backing out an effective clock from the per-chunk time, nominal 600
appears to deliver more than nominal 900 or 1000 do.  I would not lean
on that decode, but the throughput ordering does not depend on it.

Thermals were never a factor: the highest I saw all day was 50.8 degC,
against a critical trip at 115.  This is one board and one model, and
MobileNetV1 is memory-heavy, so a compute-dense network may well behave
differently - but for this workload the top half of the vendor table
buys nothing.

If that holds up elsewhere, an OPP table for rocket probably should not
simply mirror the vendor one.


4. Thermal, separately
======================

While looking at this I noticed npu-thermal has only a critical trip at
115 degC - no passive trip, no cooling map, polling-delay-passive is 0 -
while gpu-thermal, a few lines above in the same file and on the same
tsadc, has both.  That was harmless while the NPU was pinned at 200 MHz
because it could not be slowed down anyway; with DVFS it stops being
harmless.

I have two small patches for that (a #cooling-cells binding update and
the thermal zone itself), but they only make sense once something
registers a cooling device, so they would belong with the driver work
rather than on their own.


5. What I am asking
===================

- Is DVFS for rocket something you want upstream at all, or is it better
  left alone for now?

- Does the genpd-notifier approach look right to you, or is there a
  cleaner hook I have missed?

- Given the measurements, would you want the OPP table to stop at
  600 MHz rather than follow the vendor range?

- If you do want a series, I will need to clean the code up first - it
  still keeps its state file-static rather than in rocket_device, and it
  bypasses the OPP layer for the rate change because our own direct
  writes leave the OPP cache stale.  Both are fixable; I would rather
  know the shape you want before rewriting.

Happy to send the current code as-is off-list if that is easier to
comment on than prose.

Thanks,
Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Diederik,

(re-adding the lists, your reply came to me only)

Thanks - that pointer was useful, and it changes what I thought I was
looking at.  Two things came out of following it.

First, a data point that may narrow your search rather than widen it:
this board is not running upstream TF-A.  Its SCMI implementation
identifies itself as Rockchip's own:

  arm-scmi arm-scmi.0.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0

and it still fails, so whatever this is, it is not exclusive to upstream
TF-A BL31.  If rkbin BL31 makes your PineTab2 suspend/resume work, the
difference there may be something more specific than the ack handshake
itself.

Second, going through the archive for that error string, the history is
longer than I realised - Peter Geis' and Sebastian Reichel's pmdomain
work on RK3588, and further back "arm64: dts: rockchip: add hevc power
domain clock to rk3328".  That last one is interesting because the fix
was to add a missing clock to the power domain, i.e. the handshake
needs its clocks.

What I have looks like a neighbouring case rather than the same one.
Here the clocks are not missing - all three NPU domains already list the
NPU clock (rk3588-base.dtsi lines 864, 877 and 885).  It is the rate
that matters: the ack never arrives if the domain is moved while that
clock is above its DT assigned-clock-rate, and the same domain moves
cleanly once the rate is back down.  I have not seen that variant
described anywhere, which is why I wrote it up.

That suggests something you could try, though it is a guess and you know
your board far better than I do: if any clock listed in a domain you are
suspending happens to be at a non-boot rate at that moment - left there
by devfreq, by an assigned-clock-rate, or by whatever ran last - then it
would be the same shape as what I am seeing.  Pinning those clocks to
their boot rates before the transition would be a cheap thing to rule
out.  If it does nothing, at least it is eliminated.

Thanks again for taking the time, and good luck with the PineTab2.

Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Diederik,

(re-adding the lists, your reply came to me only)

Thanks - that pointer was useful, and it changes what I thought I was
looking at.  Two things came out of following it.

First, a data point that may narrow your search rather than widen it:
this board is not running upstream TF-A.  Its SCMI implementation
identifies itself as Rockchip's own:

  arm-scmi arm-scmi.0.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0

and it still fails, so whatever this is, it is not exclusive to upstream
TF-A BL31.  If rkbin BL31 makes your PineTab2 suspend/resume work, the
difference there may be something more specific than the ack handshake
itself.

Second, going through the archive for that error string, the history is
longer than I realised - Peter Geis' and Sebastian Reichel's pmdomain
work on RK3588, and further back "arm64: dts: rockchip: add hevc power
domain clock to rk3328".  That last one is interesting because the fix
was to add a missing clock to the power domain, i.e. the handshake
needs its clocks.

What I have looks like a neighbouring case rather than the same one.
Here the clocks are not missing - all three NPU domains already list the
NPU clock (rk3588-base.dtsi lines 864, 877 and 885).  It is the rate
that matters: the ack never arrives if the domain is moved while that
clock is above its DT assigned-clock-rate, and the same domain moves
cleanly once the rate is back down.  I have not seen that variant
described anywhere, which is why I wrote it up.

That suggests something you could try, though it is a guess and you know
your board far better than I do: if any clock listed in a domain you are
suspending happens to be at a non-boot rate at that moment - left there
by devfreq, by an assigned-clock-rate, or by whatever ran last - then it
would be the same shape as what I am seeing.  Pinning those clocks to
their boot rates before the transition would be a cheap thing to rule
out.  If it does nothing, at least it is eliminated.

Thanks again for taking the time, and good luck with the PineTab2.

Igor

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Diederik de Haas]

On Sat Aug 1, 2026 at 4:40 PM CEST, Igor Paunovic wrote:
> Hi Diederik,
>
> (re-adding the lists, your reply came to me only)
>
> Thanks - that pointer was useful, and it changes what I thought I was
> looking at.  Two things came out of following it.

Then I guess it's useful I add the main part of that pointer to the ML:

  This is all 'above my paygrade', but the 'failed to get ack on domain' is
  a type of error which seems to surface on a regular basis:
  https://lore.kernel.org/linux-rockchip/?q=%22failed+to+get+ack+on+domain%22

(My hope was that it may have given Igor pointers wrt the problem)

> First, a data point that may narrow your search rather than widen it:
> this board is not running upstream TF-A.  Its SCMI implementation
> identifies itself as Rockchip's own:
>
>   arm-scmi arm-scmi.0.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0
>
> and it still fails, so whatever this is, it is not exclusive to upstream
> TF-A BL31.  If rkbin BL31 makes your PineTab2 suspend/resume work, the
> difference there may be something more specific than the ack handshake
> itself.
>
> Second, going through the archive for that error string, the history is
> longer than I realised - Peter Geis' and Sebastian Reichel's pmdomain
> work on RK3588, and further back "arm64: dts: rockchip: add hevc power
> domain clock to rk3328".  That last one is interesting because the fix
> was to add a missing clock to the power domain, i.e. the handshake
> needs its clocks.
>
> What I have looks like a neighbouring case rather than the same one.
> Here the clocks are not missing - all three NPU domains already list the
> NPU clock (rk3588-base.dtsi lines 864, 877 and 885).  It is the rate
> that matters: the ack never arrives if the domain is moved while that
> clock is above its DT assigned-clock-rate, and the same domain moves
> cleanly once the rate is back down.  I have not seen that variant
> described anywhere, which is why I wrote it up.
>
> That suggests something you could try, though it is a guess and you know
> your board far better than I do: if any clock listed in a domain you are
> suspending happens to be at a non-boot rate at that moment - left there
> by devfreq, by an assigned-clock-rate, or by whatever ran last - then it
> would be the same shape as what I am seeing.  Pinning those clocks to
> their boot rates before the transition would be a cheap thing to rule
> out.  If it does nothing, at least it is eliminated.
>
> Thanks again for taking the time, and good luck with the PineTab2.
>
> Igor
>
> _______________________________________________
> Linux-rockchip mailing list
> Linux-rockchip@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-rockchip


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Diederik de Haas]

On Sat Aug 1, 2026 at 4:40 PM CEST, Igor Paunovic wrote:
> Hi Diederik,
>
> (re-adding the lists, your reply came to me only)
>
> Thanks - that pointer was useful, and it changes what I thought I was
> looking at.  Two things came out of following it.

Then I guess it's useful I add the main part of that pointer to the ML:

  This is all 'above my paygrade', but the 'failed to get ack on domain' is
  a type of error which seems to surface on a regular basis:
  https://lore.kernel.org/linux-rockchip/?q=%22failed+to+get+ack+on+domain%22

(My hope was that it may have given Igor pointers wrt the problem)

> First, a data point that may narrow your search rather than widen it:
> this board is not running upstream TF-A.  Its SCMI implementation
> identifies itself as Rockchip's own:
>
>   arm-scmi arm-scmi.0.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0
>
> and it still fails, so whatever this is, it is not exclusive to upstream
> TF-A BL31.  If rkbin BL31 makes your PineTab2 suspend/resume work, the
> difference there may be something more specific than the ack handshake
> itself.
>
> Second, going through the archive for that error string, the history is
> longer than I realised - Peter Geis' and Sebastian Reichel's pmdomain
> work on RK3588, and further back "arm64: dts: rockchip: add hevc power
> domain clock to rk3328".  That last one is interesting because the fix
> was to add a missing clock to the power domain, i.e. the handshake
> needs its clocks.
>
> What I have looks like a neighbouring case rather than the same one.
> Here the clocks are not missing - all three NPU domains already list the
> NPU clock (rk3588-base.dtsi lines 864, 877 and 885).  It is the rate
> that matters: the ack never arrives if the domain is moved while that
> clock is above its DT assigned-clock-rate, and the same domain moves
> cleanly once the rate is back down.  I have not seen that variant
> described anywhere, which is why I wrote it up.
>
> That suggests something you could try, though it is a guess and you know
> your board far better than I do: if any clock listed in a domain you are
> suspending happens to be at a non-boot rate at that moment - left there
> by devfreq, by an assigned-clock-rate, or by whatever ran last - then it
> would be the same shape as what I am seeing.  Pinning those clocks to
> their boot rates before the transition would be a cheap thing to rule
> out.  If it does nothing, at least it is eliminated.
>
> Thanks again for taking the time, and good luck with the PineTab2.
>
> Igor
>
> _______________________________________________
> Linux-rockchip mailing list
> Linux-rockchip@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Jiaxing Hu]

Hi Igor,

> all three NPU domains already list the NPU clock (rk3588-base.dtsi
> lines 864, 877 and 885)

Those lines list CLK_NPU_DSU0, but the clock the driver holds as "npu",
and the one devfreq scales, is <&scmi_clk SCMI_CLK_NPU>. The driver never
holds CLK_NPU_DSU0 at all. They may share a root, but if they do not then
the handshake is not breaking because you scaled the clock it needs, and
the notifier is treating a symptom. Worth a look at clk_summary first.

On RK3576 they really are the same clock ("npu" is CLK_RKNN_DSU0, which
PD_NPUTOP also lists), so that comparison at least is easy here.

Also relevant: the vendor does not scale the handshake side at all. Live
sample from a 6.1.115 BSP during a working inference, captured by
Olaf001au:

  clk_npu 950 MHz, clk_dsu 198 MHz, aclk_cbuf 198 MHz

Compute at 950, everything else left at boot rate. If something similar
exists on RK3588 you may be able to pick a clock that is not in any
domain's list and avoid the constraint rather than veto around it.

Two smaller data points:

We get an async SError from NPU domain power-on here too, which is why my
v3 has the settle delay and the reset cycling. But our DSU0 sits at 594 to
786 MHz permanently and only the cold power-on fails; later transitions at
the same rate are fine. So either your constraint is RK3588 specific or my
settle delay is masking it and my explanation is wrong. No idea which.

You said you had not tried PVTPLL because reading its registers is
supposed to hang. I tried it from the other end on RK3576, routing the NPU
clock through SCMI: zero jobs completed, 83 scheduler timeouts, everything
else unchanged. Not a drop-in for the CRU clock.

On the OPP table, your plateau looks memory bound, and the vendor pinning
aclk_cbuf at 198 while compute runs at 950 points the same way. If so, 600
is where MobileNetV1 stops scaling rather than where the hardware does.
Maybe worth one compute dense model before cutting the table there.

Cheers,
Jiaxing

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Jiaxing Hu]

Hi Igor,

> all three NPU domains already list the NPU clock (rk3588-base.dtsi
> lines 864, 877 and 885)

Those lines list CLK_NPU_DSU0, but the clock the driver holds as "npu",
and the one devfreq scales, is <&scmi_clk SCMI_CLK_NPU>. The driver never
holds CLK_NPU_DSU0 at all. They may share a root, but if they do not then
the handshake is not breaking because you scaled the clock it needs, and
the notifier is treating a symptom. Worth a look at clk_summary first.

On RK3576 they really are the same clock ("npu" is CLK_RKNN_DSU0, which
PD_NPUTOP also lists), so that comparison at least is easy here.

Also relevant: the vendor does not scale the handshake side at all. Live
sample from a 6.1.115 BSP during a working inference, captured by
Olaf001au:

  clk_npu 950 MHz, clk_dsu 198 MHz, aclk_cbuf 198 MHz

Compute at 950, everything else left at boot rate. If something similar
exists on RK3588 you may be able to pick a clock that is not in any
domain's list and avoid the constraint rather than veto around it.

Two smaller data points:

We get an async SError from NPU domain power-on here too, which is why my
v3 has the settle delay and the reset cycling. But our DSU0 sits at 594 to
786 MHz permanently and only the cold power-on fails; later transitions at
the same rate are fine. So either your constraint is RK3588 specific or my
settle delay is masking it and my explanation is wrong. No idea which.

You said you had not tried PVTPLL because reading its registers is
supposed to hang. I tried it from the other end on RK3576, routing the NPU
clock through SCMI: zero jobs completed, 83 scheduler timeouts, everything
else unchanged. Not a drop-in for the CRU clock.

On the OPP table, your plateau looks memory bound, and the vendor pinning
aclk_cbuf at 198 while compute runs at 950 points the same way. If so, 600
is where MobileNetV1 stops scaling rather than where the hardware does.
Maybe worth one compute dense model before cutting the table there.

Cheers,
Jiaxing

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Tomeu, hi Jiaxing,

I need to correct the measurements in my first mail. The conclusions I
drew from them do not survive better methodology, and since two of my
questions to Tomeu were based on those conclusions, I would rather fix
this now than have anyone reason from bad data.

Short version: the numbers were real, but they measured my interrupt
configuration, not the NPU.


1. What was wrong
=================

Each inference generates 41 interrupts on this driver (one per task,
more on that below).  All of them were landing on CPU0, a Cortex-A55
that was also entering its shallow idle state (exit latency 220 us)
between interrupts.  So every inference paid a large, frequency-
independent CPU-side cost, which compressed the differences between
NPU frequencies and produced the inversion I reported.

Three changes, none of them driver code:

  - NPU IRQ affinity moved to an idle A76
  - cpu-sleep (state1) disabled on that core
  - performance cpufreq governors during measurement

With that, the picture changes completely.  Even the fixed 200 MHz
baseline moves from 68.5 to ~91 inf/s - a quarter of the stock
performance was lost to configuration alone.  I should have caught
this; the per-inference interrupt count was in my own telemetry.


2. Corrected numbers
====================

Same rig as before otherwise: MobileNetV1 through Teflon, one inference
thread pinned to one A76, bit-exact sha256 oracle over intermediate
tensors on every deep-check, zero tolerance, oracle passed in every run
quoted here.  Kernel 7.0.0 + local devfreq for rocket.  Ambient
controlled, die temperature logged per row (all rows 42.5-45.3 degC).
Every row is an ~80 s run; key points were replicated 2-5x across the
day, spread within 1%.

  inf/s            800 mV    850 mV    950 mV
  200 MHz           91.1      90.7      90.8
  300 MHz          161.7     170.9     185.1
  400 MHz          187.3     196.2     211.4
  500 MHz          208.8     216.7     226.6
  600 MHz          221.1     227.9     236.5
  700 MHz          227.4     233.0     240.7
  800 MHz             -      232.2     241.9
  900 MHz             -      233.8     241.4
  1000 MHz            -      235.0     242.8

(The 800 mV column stops at 700 because my local safeguard refuses
higher OPPs on that rail.  The identical 200 MHz row across all three
rails is the negative control - at 200 MHz voltage headroom is
irrelevant, and the rig reproduces to 0.5%.)

Corrections to my three claims, in order:

a) "600 MHz is the optimum" - withdrawn.  The curve is monotonic to
   ~700 MHz and flat above it.  There is no inversion; the inversion
   was the A55 paying 41 wakeups per inference.

b) "Voltage does not help" - withdrawn.  At the same nominal clock,
   raising the rail 800 -> 950 mV buys +14% at 300 MHz and +7% at
   600 MHz.  Consistent with the clock being PVT-controlled: more
   voltage lets the PLL deliver a higher actual clock for the same
   nominal request.  (This also means the flat region above 700 is
   partly voltage-limited, not purely memory-limited: the per-chunk
   floor is ~72.5 us at 850 mV but ~69 us at 950 mV.)

c) "Should the OPP table stop at 600?" - wrong question, please ignore
   it.  The plateau is this board's memory (LPDDR4X-2112) plus the
   voltage effect above; boards with LPDDR5 should keep scaling
   further.  The table should carry the full vendor range and let each
   board cap via max_freq / thermal policy.

One methodological note that may be useful to anyone benchmarking this
hardware: the largest error sources I quantified today were, in order -
interrupt placement (+32% at stock clock), a busy desktop session
sharing the SoC (up to 18%), die temperature (~0.5%/degC via the PVT
loop), and an open browser (~1%).  The silicon was the most
reproducible part of the whole setup.


3. Jiaxing's points
===================

> Those lines list CLK_NPU_DSU0, but the clock the driver holds as
> "npu" [...] is <&scmi_clk SCMI_CLK_NPU>. [...] Worth a look at
> clk_summary first.

You are right about the identity, and thanks for pushing me to check
it properly.  Decoded from the live DT: "npu" is SCMI_CLK_NPU (SCMI
clock id 6); aclk/hclk/pclk come from the CRU.  The domains list
CLK_NPU_DSU0, which devfreq never touches.

I then took clk_summary snapshots idle, mid-benchmark at nominal 600,
and mid-benchmark at nominal 1000:

                    idle      during 600   during 1000
  clk_npu_dsu0      250 MHz   250 MHz      250 MHz
  aclk_npu0/1/2     250       250          250
  hclk_npu0/1/2     198       198          198
  pclk_npu_root     100       100          100

So the handshake-side clocks never move - this driver already does
exactly what your vendor sample shows (compute clock scaled, dsu/aclk/
hclk parked at boot rates).  And yet the domain transition still fails
whenever the SCMI compute clock is raised, reproducibly.  So the
constraint tracks the compute clock itself, not the DSU clock, and
"pick a clock outside the domain lists" unfortunately does not exist
here: the only rate anyone scales is already outside them.

Whether the mechanism is a PLL inside the domain remains my hypothesis
- the TRM's NPU chapter is silent on PVTPLL and I still have not risked
reading its registers.  The constraint itself is empirical.

A caveat worth passing on: clk_summary reports SCMI_CLK_NPU at
200 MHz even mid-benchmark at nominal 1000 - the CCF cache is stale for
firmware clocks, so clk_summary genuinely cannot answer this question.
I verified the actual rate two ways: clk_get_rate() (which round-trips
to firmware), and the per-chunk NPU time scaling 236 -> 69 us across
the sweep.

> your plateau looks memory bound [...] Maybe worth one compute dense
> model before cutting the table there.

Fair, and the answer is a few lines up: I am no longer proposing to cut
the table anywhere.  I did run the compute-dense model you asked for:
InceptionV1 (quant), same rig, 850 mV.  One caveat first: on this stack
Teflon leaves ~79% of an Inception inference on the CPU (many more ops
fall back than for MobileNet), so total throughput barely moves with
clock and the honest metric is the NPU-side time per inference, which
the harness separates out:

  nominal     NPU-side per inference    total
  300 MHz     7.2 ms                    43.7 inf/s
  700 MHz     4.16 ms                   50.5 inf/s
  1000 MHz    4.17 ms                   50.2 inf/s

It scales 300 -> 700 and then flattens at exactly the same knee as
MobileNetV1 (whose NPU-side time goes 4.54 -> 3.00 -> 2.97 ms over the
same points).  So on this board the plateau is the platform, not
MobileNetV1's memory profile - consistent with the LPDDR4X explanation,
and it does not change the conclusion: full table, per-board cap.
Boards with faster memory may well keep scaling; mine cannot test that.

(Two incidental data points from the same runs: Inception jobs get
scheduled across two NPU cores here - 33+27 interrupts per inference
against MobileNet's 41 on one core - and the bit-exact oracle passed at
every frequency for this model too.)

On your RK3576 SCMI experiment (zero jobs, 83 timeouts): one cross-data
point - on RK3588 the SCMI/PVTPLL path is what we use successfully for
the GPU (Mali-G610 at 1 GHz via SCMI, measured and stable), and the NPU
compute clock here is SCMI-routed and scales fine while the domain is
up.  So the SCMI plumbing itself can work on this family; whatever
failed on RK3576 may be specific to that port rather than the approach.


4. Thermal, update
==================

The missing NPU cooling chain from my first mail is now running here
end-to-end: #cooling-cells on the NPU node, a passive trip at 85 degC
with a cooling map, and devfreq_cooling registered by the driver - the
kernel bound them together without manual intervention.  So the two DT
patches are ready whenever the driver work lands; they need nothing
beyond what devfreq already provides.


5. Where this leaves the questions to Tomeu
===========================================

The two RFC questions that survive unchanged: whether you want DVFS
for rocket upstream at all, and whether the genpd-notifier shape is
acceptable.  The OPP question is answered above (full table, per-board
cap).  The code cleanup questions stand as written.

Sorry for the noise, and thanks to Jiaxing for the push to verify the
clock tree - the numbers are better for it.

Igor

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Tomeu, hi Jiaxing,

I need to correct the measurements in my first mail. The conclusions I
drew from them do not survive better methodology, and since two of my
questions to Tomeu were based on those conclusions, I would rather fix
this now than have anyone reason from bad data.

Short version: the numbers were real, but they measured my interrupt
configuration, not the NPU.


1. What was wrong
=================

Each inference generates 41 interrupts on this driver (one per task,
more on that below).  All of them were landing on CPU0, a Cortex-A55
that was also entering its shallow idle state (exit latency 220 us)
between interrupts.  So every inference paid a large, frequency-
independent CPU-side cost, which compressed the differences between
NPU frequencies and produced the inversion I reported.

Three changes, none of them driver code:

  - NPU IRQ affinity moved to an idle A76
  - cpu-sleep (state1) disabled on that core
  - performance cpufreq governors during measurement

With that, the picture changes completely.  Even the fixed 200 MHz
baseline moves from 68.5 to ~91 inf/s - a quarter of the stock
performance was lost to configuration alone.  I should have caught
this; the per-inference interrupt count was in my own telemetry.


2. Corrected numbers
====================

Same rig as before otherwise: MobileNetV1 through Teflon, one inference
thread pinned to one A76, bit-exact sha256 oracle over intermediate
tensors on every deep-check, zero tolerance, oracle passed in every run
quoted here.  Kernel 7.0.0 + local devfreq for rocket.  Ambient
controlled, die temperature logged per row (all rows 42.5-45.3 degC).
Every row is an ~80 s run; key points were replicated 2-5x across the
day, spread within 1%.

  inf/s            800 mV    850 mV    950 mV
  200 MHz           91.1      90.7      90.8
  300 MHz          161.7     170.9     185.1
  400 MHz          187.3     196.2     211.4
  500 MHz          208.8     216.7     226.6
  600 MHz          221.1     227.9     236.5
  700 MHz          227.4     233.0     240.7
  800 MHz             -      232.2     241.9
  900 MHz             -      233.8     241.4
  1000 MHz            -      235.0     242.8

(The 800 mV column stops at 700 because my local safeguard refuses
higher OPPs on that rail.  The identical 200 MHz row across all three
rails is the negative control - at 200 MHz voltage headroom is
irrelevant, and the rig reproduces to 0.5%.)

Corrections to my three claims, in order:

a) "600 MHz is the optimum" - withdrawn.  The curve is monotonic to
   ~700 MHz and flat above it.  There is no inversion; the inversion
   was the A55 paying 41 wakeups per inference.

b) "Voltage does not help" - withdrawn.  At the same nominal clock,
   raising the rail 800 -> 950 mV buys +14% at 300 MHz and +7% at
   600 MHz.  Consistent with the clock being PVT-controlled: more
   voltage lets the PLL deliver a higher actual clock for the same
   nominal request.  (This also means the flat region above 700 is
   partly voltage-limited, not purely memory-limited: the per-chunk
   floor is ~72.5 us at 850 mV but ~69 us at 950 mV.)

c) "Should the OPP table stop at 600?" - wrong question, please ignore
   it.  The plateau is this board's memory (LPDDR4X-2112) plus the
   voltage effect above; boards with LPDDR5 should keep scaling
   further.  The table should carry the full vendor range and let each
   board cap via max_freq / thermal policy.

One methodological note that may be useful to anyone benchmarking this
hardware: the largest error sources I quantified today were, in order -
interrupt placement (+32% at stock clock), a busy desktop session
sharing the SoC (up to 18%), die temperature (~0.5%/degC via the PVT
loop), and an open browser (~1%).  The silicon was the most
reproducible part of the whole setup.


3. Jiaxing's points
===================

> Those lines list CLK_NPU_DSU0, but the clock the driver holds as
> "npu" [...] is <&scmi_clk SCMI_CLK_NPU>. [...] Worth a look at
> clk_summary first.

You are right about the identity, and thanks for pushing me to check
it properly.  Decoded from the live DT: "npu" is SCMI_CLK_NPU (SCMI
clock id 6); aclk/hclk/pclk come from the CRU.  The domains list
CLK_NPU_DSU0, which devfreq never touches.

I then took clk_summary snapshots idle, mid-benchmark at nominal 600,
and mid-benchmark at nominal 1000:

                    idle      during 600   during 1000
  clk_npu_dsu0      250 MHz   250 MHz      250 MHz
  aclk_npu0/1/2     250       250          250
  hclk_npu0/1/2     198       198          198
  pclk_npu_root     100       100          100

So the handshake-side clocks never move - this driver already does
exactly what your vendor sample shows (compute clock scaled, dsu/aclk/
hclk parked at boot rates).  And yet the domain transition still fails
whenever the SCMI compute clock is raised, reproducibly.  So the
constraint tracks the compute clock itself, not the DSU clock, and
"pick a clock outside the domain lists" unfortunately does not exist
here: the only rate anyone scales is already outside them.

Whether the mechanism is a PLL inside the domain remains my hypothesis
- the TRM's NPU chapter is silent on PVTPLL and I still have not risked
reading its registers.  The constraint itself is empirical.

A caveat worth passing on: clk_summary reports SCMI_CLK_NPU at
200 MHz even mid-benchmark at nominal 1000 - the CCF cache is stale for
firmware clocks, so clk_summary genuinely cannot answer this question.
I verified the actual rate two ways: clk_get_rate() (which round-trips
to firmware), and the per-chunk NPU time scaling 236 -> 69 us across
the sweep.

> your plateau looks memory bound [...] Maybe worth one compute dense
> model before cutting the table there.

Fair, and the answer is a few lines up: I am no longer proposing to cut
the table anywhere.  I did run the compute-dense model you asked for:
InceptionV1 (quant), same rig, 850 mV.  One caveat first: on this stack
Teflon leaves ~79% of an Inception inference on the CPU (many more ops
fall back than for MobileNet), so total throughput barely moves with
clock and the honest metric is the NPU-side time per inference, which
the harness separates out:

  nominal     NPU-side per inference    total
  300 MHz     7.2 ms                    43.7 inf/s
  700 MHz     4.16 ms                   50.5 inf/s
  1000 MHz    4.17 ms                   50.2 inf/s

It scales 300 -> 700 and then flattens at exactly the same knee as
MobileNetV1 (whose NPU-side time goes 4.54 -> 3.00 -> 2.97 ms over the
same points).  So on this board the plateau is the platform, not
MobileNetV1's memory profile - consistent with the LPDDR4X explanation,
and it does not change the conclusion: full table, per-board cap.
Boards with faster memory may well keep scaling; mine cannot test that.

(Two incidental data points from the same runs: Inception jobs get
scheduled across two NPU cores here - 33+27 interrupts per inference
against MobileNet's 41 on one core - and the bit-exact oracle passed at
every frequency for this model too.)

On your RK3576 SCMI experiment (zero jobs, 83 timeouts): one cross-data
point - on RK3588 the SCMI/PVTPLL path is what we use successfully for
the GPU (Mali-G610 at 1 GHz via SCMI, measured and stable), and the NPU
compute clock here is SCMI-routed and scales fine while the domain is
up.  So the SCMI plumbing itself can work on this family; whatever
failed on RK3576 may be specific to that port rather than the approach.


4. Thermal, update
==================

The missing NPU cooling chain from my first mail is now running here
end-to-end: #cooling-cells on the NPU node, a passive trip at 85 degC
with a cooling map, and devfreq_cooling registered by the driver - the
kernel bound them together without manual intervention.  So the two DT
patches are ready whenever the driver work lands; they need nothing
beyond what devfreq already provides.


5. Where this leaves the questions to Tomeu
===========================================

The two RFC questions that survive unchanged: whether you want DVFS
for rocket upstream at all, and whether the genpd-notifier shape is
acceptable.  The OPP question is answered above (full table, per-board
cap).  The code cleanup questions stand as written.

Sorry for the noise, and thanks to Jiaxing for the push to verify the
clock tree - the numbers are better for it.

Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip