[RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Tomeu,

Since you asked for fixes to be sent upfront I have kept poking at the
RK3588 NPU, and I ended up implementing devfreq for rocket locally.  It
works, but on the way there I hit a crash class that I could not find
documented anywhere, and I also measured something about the vendor OPP
table that I did not expect.  Both seem worth sharing before I clean any
of it up for posting, so I would rather ask first than send a series you
may not want in this shape.

Cc'ing Jiaxing since he is working on the clocks and on RK3576.


1. The hardware constraint
==========================

An NPU power domain cannot be switched on or off while the NPU compute
clock is above its DT assigned-clock-rate of 200 MHz.

Changing the rate while a domain is already on is fine - I have taken it
to 1 GHz and back many times without a single error.  It is the domain
transition that breaks.

What happens when a domain is moved at a high rate:

  rockchip-pm-domain ...: failed to get ack on domain 'nputop', val=0xa9ffe
  rocket fdab0000.npu: devfreq: cannot power up for rate change: -110

The domain is then wedged: genpd still believes it is on, but the first
MMIO into it raises an asynchronous SError and the box panics.  Captured
over the serial console:

  Kernel panic - not syncing: Asynchronous SError Interrupt
  Comm: rmmod
   _regmap_read
   regmap_read
   rockchip_pd_power
   rockchip_pd_power_off
   _genpd_power_off        <- rollback
   genpd_power_off
   genpd_power_on          <- failed
   genpd_runtime_resume
   device_release_driver

This is not specific to nputop.  I have the same message for 'npu2'
(val=0xa9fff), which matches the DT: all three NPU domains list the NPU
clock among their handshake clocks - rk3588-base.dtsi lines 864, 877 and
885, for RK3588_PD_NPUTOP, RK3588_PD_NPU1 and RK3588_PD_NPU2.

So the clock the domains need for their idle/ack handshake is the same
clock we would be scaling.  My best explanation is that the PLL that
produces it lives inside the domain, so once the domain drops, the clock
state goes with it and the handshake can never complete.  I cannot
confirm that part - reading the PVTPLL registers is documented as
hanging the machine, so I have not tried.  The behaviour itself is
reproducible and cost me four hard hangs before I understood it.

I mention it because it is a trap for anyone adding DVFS here, including
the RK3576 work, and because it is invisible until the first time you
let the NPU idle at a raised clock.


2. What ended up working
========================

Runtime PM callbacks are not enough.  The domains are powered on by the
driver core before probe, powered off from a workqueue after detach, and
system sleep bypasses runtime PM references entirely - so the driver
never sees all the transitions.

What does work is hooking the transitions themselves:
dev_pm_genpd_add_notifier() on all three cores, and on GENPD_NOTIFY_PRE_ON
and GENPD_NOTIFY_PRE_OFF force the clock back to the DT rate, vetoing the
transition with notifier_from_errno() if that fails.  Every path -
runtime PM, system sleep, attach at probe, detach after unbind - goes
through _genpd_power_on()/_genpd_power_off(), so nothing can slip past.
If a transition does happen, the boost cancels itself and says so,
rather than leaving the driver claiming a rate the hardware is not
running.

This has now survived everything that used to kill the box, including
repeated sleep/wake cycles at a raised clock and rmmod while boosted.


3. The numbers, which are the surprising part
=============================================

Measured with MobileNetV1 through Teflon, one inference thread pinned to
one A76, and a bit-exact oracle: sha256 over intermediate tensors on
every iteration, zero tolerance.  The 600, 900 and 1000 MHz rows are
30-minute runs of 275k-280k inferences each and the oracle passed
bit-exact in all three, so none of this is instability; the 200 MHz row
is a shorter control from the same session.

  nominal    supply    throughput
  200 MHz    800 mV     68.5 inf/s      (the current fixed rate)
  600 MHz    800 mV    155.4 inf/s
  900 MHz    850 mV    152.4 inf/s
  1000 MHz   850 mV    152.9 inf/s

600 MHz is the optimum.  900 and 1000 are indistinguishable from each
other and both land about 4% below 600, on a quieter and cooler machine.
The vendor OPP table decoded from the downstream DTB asks for 700 mV up
to 700 MHz, 750 mV at 800, 800 mV at 900 and 850 mV at 1000, and I ran
the top of that range at the voltage it asks for - it does not help.

Backing out an effective clock from the per-chunk time, nominal 600
appears to deliver more than nominal 900 or 1000 do.  I would not lean
on that decode, but the throughput ordering does not depend on it.

Thermals were never a factor: the highest I saw all day was 50.8 degC,
against a critical trip at 115.  This is one board and one model, and
MobileNetV1 is memory-heavy, so a compute-dense network may well behave
differently - but for this workload the top half of the vendor table
buys nothing.

If that holds up elsewhere, an OPP table for rocket probably should not
simply mirror the vendor one.


4. Thermal, separately
======================

While looking at this I noticed npu-thermal has only a critical trip at
115 degC - no passive trip, no cooling map, polling-delay-passive is 0 -
while gpu-thermal, a few lines above in the same file and on the same
tsadc, has both.  That was harmless while the NPU was pinned at 200 MHz
because it could not be slowed down anyway; with DVFS it stops being
harmless.

I have two small patches for that (a #cooling-cells binding update and
the thermal zone itself), but they only make sense once something
registers a cooling device, so they would belong with the driver work
rather than on their own.


5. What I am asking
===================

- Is DVFS for rocket something you want upstream at all, or is it better
  left alone for now?

- Does the genpd-notifier approach look right to you, or is there a
  cleaner hook I have missed?

- Given the measurements, would you want the OPP table to stop at
  600 MHz rather than follow the vendor range?

- If you do want a series, I will need to clean the code up first - it
  still keeps its state file-static rather than in rocket_device, and it
  bypasses the OPP layer for the rate change because our own direct
  writes leave the OPP cache stale.  Both are fixable; I would rather
  know the shape you want before rewriting.

Happy to send the current code as-is off-list if that is easier to
comment on than prose.

Thanks,
Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Jiaxing Hu]

Hi Igor,

> all three NPU domains already list the NPU clock (rk3588-base.dtsi
> lines 864, 877 and 885)

Those lines list CLK_NPU_DSU0, but the clock the driver holds as "npu",
and the one devfreq scales, is <&scmi_clk SCMI_CLK_NPU>. The driver never
holds CLK_NPU_DSU0 at all. They may share a root, but if they do not then
the handshake is not breaking because you scaled the clock it needs, and
the notifier is treating a symptom. Worth a look at clk_summary first.

On RK3576 they really are the same clock ("npu" is CLK_RKNN_DSU0, which
PD_NPUTOP also lists), so that comparison at least is easy here.

Also relevant: the vendor does not scale the handshake side at all. Live
sample from a 6.1.115 BSP during a working inference, captured by
Olaf001au:

  clk_npu 950 MHz, clk_dsu 198 MHz, aclk_cbuf 198 MHz

Compute at 950, everything else left at boot rate. If something similar
exists on RK3588 you may be able to pick a clock that is not in any
domain's list and avoid the constraint rather than veto around it.

Two smaller data points:

We get an async SError from NPU domain power-on here too, which is why my
v3 has the settle delay and the reset cycling. But our DSU0 sits at 594 to
786 MHz permanently and only the cold power-on fails; later transitions at
the same rate are fine. So either your constraint is RK3588 specific or my
settle delay is masking it and my explanation is wrong. No idea which.

You said you had not tried PVTPLL because reading its registers is
supposed to hang. I tried it from the other end on RK3576, routing the NPU
clock through SCMI: zero jobs completed, 83 scheduler timeouts, everything
else unchanged. Not a drop-in for the CRU clock.

On the OPP table, your plateau looks memory bound, and the vendor pinning
aclk_cbuf at 198 while compute runs at 950 points the same way. If so, 600
is where MobileNetV1 stops scaling rather than where the hardware does.
Maybe worth one compute dense model before cutting the table there.

Cheers,
Jiaxing

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Tomeu, hi Jiaxing,

I need to correct the measurements in my first mail. The conclusions I
drew from them do not survive better methodology, and since two of my
questions to Tomeu were based on those conclusions, I would rather fix
this now than have anyone reason from bad data.

Short version: the numbers were real, but they measured my interrupt
configuration, not the NPU.


1. What was wrong
=================

Each inference generates 41 interrupts on this driver (one per task,
more on that below).  All of them were landing on CPU0, a Cortex-A55
that was also entering its shallow idle state (exit latency 220 us)
between interrupts.  So every inference paid a large, frequency-
independent CPU-side cost, which compressed the differences between
NPU frequencies and produced the inversion I reported.

Three changes, none of them driver code:

  - NPU IRQ affinity moved to an idle A76
  - cpu-sleep (state1) disabled on that core
  - performance cpufreq governors during measurement

With that, the picture changes completely.  Even the fixed 200 MHz
baseline moves from 68.5 to ~91 inf/s - a quarter of the stock
performance was lost to configuration alone.  I should have caught
this; the per-inference interrupt count was in my own telemetry.


2. Corrected numbers
====================

Same rig as before otherwise: MobileNetV1 through Teflon, one inference
thread pinned to one A76, bit-exact sha256 oracle over intermediate
tensors on every deep-check, zero tolerance, oracle passed in every run
quoted here.  Kernel 7.0.0 + local devfreq for rocket.  Ambient
controlled, die temperature logged per row (all rows 42.5-45.3 degC).
Every row is an ~80 s run; key points were replicated 2-5x across the
day, spread within 1%.

  inf/s            800 mV    850 mV    950 mV
  200 MHz           91.1      90.7      90.8
  300 MHz          161.7     170.9     185.1
  400 MHz          187.3     196.2     211.4
  500 MHz          208.8     216.7     226.6
  600 MHz          221.1     227.9     236.5
  700 MHz          227.4     233.0     240.7
  800 MHz             -      232.2     241.9
  900 MHz             -      233.8     241.4
  1000 MHz            -      235.0     242.8

(The 800 mV column stops at 700 because my local safeguard refuses
higher OPPs on that rail.  The identical 200 MHz row across all three
rails is the negative control - at 200 MHz voltage headroom is
irrelevant, and the rig reproduces to 0.5%.)

Corrections to my three claims, in order:

a) "600 MHz is the optimum" - withdrawn.  The curve is monotonic to
   ~700 MHz and flat above it.  There is no inversion; the inversion
   was the A55 paying 41 wakeups per inference.

b) "Voltage does not help" - withdrawn.  At the same nominal clock,
   raising the rail 800 -> 950 mV buys +14% at 300 MHz and +7% at
   600 MHz.  Consistent with the clock being PVT-controlled: more
   voltage lets the PLL deliver a higher actual clock for the same
   nominal request.  (This also means the flat region above 700 is
   partly voltage-limited, not purely memory-limited: the per-chunk
   floor is ~72.5 us at 850 mV but ~69 us at 950 mV.)

c) "Should the OPP table stop at 600?" - wrong question, please ignore
   it.  The plateau is this board's memory (LPDDR4X-2112) plus the
   voltage effect above; boards with LPDDR5 should keep scaling
   further.  The table should carry the full vendor range and let each
   board cap via max_freq / thermal policy.

One methodological note that may be useful to anyone benchmarking this
hardware: the largest error sources I quantified today were, in order -
interrupt placement (+32% at stock clock), a busy desktop session
sharing the SoC (up to 18%), die temperature (~0.5%/degC via the PVT
loop), and an open browser (~1%).  The silicon was the most
reproducible part of the whole setup.


3. Jiaxing's points
===================

> Those lines list CLK_NPU_DSU0, but the clock the driver holds as
> "npu" [...] is <&scmi_clk SCMI_CLK_NPU>. [...] Worth a look at
> clk_summary first.

You are right about the identity, and thanks for pushing me to check
it properly.  Decoded from the live DT: "npu" is SCMI_CLK_NPU (SCMI
clock id 6); aclk/hclk/pclk come from the CRU.  The domains list
CLK_NPU_DSU0, which devfreq never touches.

I then took clk_summary snapshots idle, mid-benchmark at nominal 600,
and mid-benchmark at nominal 1000:

                    idle      during 600   during 1000
  clk_npu_dsu0      250 MHz   250 MHz      250 MHz
  aclk_npu0/1/2     250       250          250
  hclk_npu0/1/2     198       198          198
  pclk_npu_root     100       100          100

So the handshake-side clocks never move - this driver already does
exactly what your vendor sample shows (compute clock scaled, dsu/aclk/
hclk parked at boot rates).  And yet the domain transition still fails
whenever the SCMI compute clock is raised, reproducibly.  So the
constraint tracks the compute clock itself, not the DSU clock, and
"pick a clock outside the domain lists" unfortunately does not exist
here: the only rate anyone scales is already outside them.

Whether the mechanism is a PLL inside the domain remains my hypothesis
- the TRM's NPU chapter is silent on PVTPLL and I still have not risked
reading its registers.  The constraint itself is empirical.

A caveat worth passing on: clk_summary reports SCMI_CLK_NPU at
200 MHz even mid-benchmark at nominal 1000 - the CCF cache is stale for
firmware clocks, so clk_summary genuinely cannot answer this question.
I verified the actual rate two ways: clk_get_rate() (which round-trips
to firmware), and the per-chunk NPU time scaling 236 -> 69 us across
the sweep.

> your plateau looks memory bound [...] Maybe worth one compute dense
> model before cutting the table there.

Fair, and the answer is a few lines up: I am no longer proposing to cut
the table anywhere.  I did run the compute-dense model you asked for:
InceptionV1 (quant), same rig, 850 mV.  One caveat first: on this stack
Teflon leaves ~79% of an Inception inference on the CPU (many more ops
fall back than for MobileNet), so total throughput barely moves with
clock and the honest metric is the NPU-side time per inference, which
the harness separates out:

  nominal     NPU-side per inference    total
  300 MHz     7.2 ms                    43.7 inf/s
  700 MHz     4.16 ms                   50.5 inf/s
  1000 MHz    4.17 ms                   50.2 inf/s

It scales 300 -> 700 and then flattens at exactly the same knee as
MobileNetV1 (whose NPU-side time goes 4.54 -> 3.00 -> 2.97 ms over the
same points).  So on this board the plateau is the platform, not
MobileNetV1's memory profile - consistent with the LPDDR4X explanation,
and it does not change the conclusion: full table, per-board cap.
Boards with faster memory may well keep scaling; mine cannot test that.

(Two incidental data points from the same runs: Inception jobs get
scheduled across two NPU cores here - 33+27 interrupts per inference
against MobileNet's 41 on one core - and the bit-exact oracle passed at
every frequency for this model too.)

On your RK3576 SCMI experiment (zero jobs, 83 timeouts): one cross-data
point - on RK3588 the SCMI/PVTPLL path is what we use successfully for
the GPU (Mali-G610 at 1 GHz via SCMI, measured and stable), and the NPU
compute clock here is SCMI-routed and scales fine while the domain is
up.  So the SCMI plumbing itself can work on this family; whatever
failed on RK3576 may be specific to that port rather than the approach.


4. Thermal, update
==================

The missing NPU cooling chain from my first mail is now running here
end-to-end: #cooling-cells on the NPU node, a passive trip at 85 degC
with a cooling map, and devfreq_cooling registered by the driver - the
kernel bound them together without manual intervention.  So the two DT
patches are ready whenever the driver work lands; they need nothing
beyond what devfreq already provides.


5. Where this leaves the questions to Tomeu
===========================================

The two RFC questions that survive unchanged: whether you want DVFS
for rocket upstream at all, and whether the genpd-notifier shape is
acceptable.  The OPP question is answered above (full table, per-board
cap).  The code cleanup questions stand as written.

Sorry for the noise, and thanks to Jiaxing for the push to verify the
clock tree - the numbers are better for it.

Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Tomeu Vizoso]

Hi Igor,

On Sun, Aug 2, 2026 at 2:04 PM Igor Paunovic <royalnet026@gmail.com> wrote:
>
> Hi Tomeu, hi Jiaxing,
>
> I need to correct the measurements in my first mail. The conclusions I
> drew from them do not survive better methodology, and since two of my
> questions to Tomeu were based on those conclusions, I would rather fix
> this now than have anyone reason from bad data.
>
> Short version: the numbers were real, but they measured my interrupt
> configuration, not the NPU.
>
>
> 1. What was wrong
> =================
>
> Each inference generates 41 interrupts on this driver (one per task,
> more on that below).  All of them were landing on CPU0, a Cortex-A55
> that was also entering its shallow idle state (exit latency 220 us)
> between interrupts.  So every inference paid a large, frequency-
> independent CPU-side cost, which compressed the differences between
> NPU frequencies and produced the inversion I reported.
>
> Three changes, none of them driver code:
>
>   - NPU IRQ affinity moved to an idle A76
>   - cpu-sleep (state1) disabled on that core
>   - performance cpufreq governors during measurement
>
> With that, the picture changes completely.  Even the fixed 200 MHz
> baseline moves from 68.5 to ~91 inf/s - a quarter of the stock
> performance was lost to configuration alone.  I should have caught
> this; the per-inference interrupt count was in my own telemetry.
>
>
> 2. Corrected numbers
> ====================
>
> Same rig as before otherwise: MobileNetV1 through Teflon, one inference
> thread pinned to one A76, bit-exact sha256 oracle over intermediate
> tensors on every deep-check, zero tolerance, oracle passed in every run
> quoted here.  Kernel 7.0.0 + local devfreq for rocket.  Ambient
> controlled, die temperature logged per row (all rows 42.5-45.3 degC).
> Every row is an ~80 s run; key points were replicated 2-5x across the
> day, spread within 1%.
>
>   inf/s            800 mV    850 mV    950 mV
>   200 MHz           91.1      90.7      90.8
>   300 MHz          161.7     170.9     185.1
>   400 MHz          187.3     196.2     211.4
>   500 MHz          208.8     216.7     226.6
>   600 MHz          221.1     227.9     236.5
>   700 MHz          227.4     233.0     240.7
>   800 MHz             -      232.2     241.9
>   900 MHz             -      233.8     241.4
>   1000 MHz            -      235.0     242.8
>
> (The 800 mV column stops at 700 because my local safeguard refuses
> higher OPPs on that rail.  The identical 200 MHz row across all three
> rails is the negative control - at 200 MHz voltage headroom is
> irrelevant, and the rig reproduces to 0.5%.)
>
> Corrections to my three claims, in order:
>
> a) "600 MHz is the optimum" - withdrawn.  The curve is monotonic to
>    ~700 MHz and flat above it.  There is no inversion; the inversion
>    was the A55 paying 41 wakeups per inference.
>
> b) "Voltage does not help" - withdrawn.  At the same nominal clock,
>    raising the rail 800 -> 950 mV buys +14% at 300 MHz and +7% at
>    600 MHz.  Consistent with the clock being PVT-controlled: more
>    voltage lets the PLL deliver a higher actual clock for the same
>    nominal request.  (This also means the flat region above 700 is
>    partly voltage-limited, not purely memory-limited: the per-chunk
>    floor is ~72.5 us at 850 mV but ~69 us at 950 mV.)
>
> c) "Should the OPP table stop at 600?" - wrong question, please ignore
>    it.  The plateau is this board's memory (LPDDR4X-2112) plus the
>    voltage effect above; boards with LPDDR5 should keep scaling
>    further.  The table should carry the full vendor range and let each
>    board cap via max_freq / thermal policy.
>
> One methodological note that may be useful to anyone benchmarking this
> hardware: the largest error sources I quantified today were, in order -
> interrupt placement (+32% at stock clock), a busy desktop session
> sharing the SoC (up to 18%), die temperature (~0.5%/degC via the PVT
> loop), and an open browser (~1%).  The silicon was the most
> reproducible part of the whole setup.
>
>
> 3. Jiaxing's points
> ===================
>
> > Those lines list CLK_NPU_DSU0, but the clock the driver holds as
> > "npu" [...] is <&scmi_clk SCMI_CLK_NPU>. [...] Worth a look at
> > clk_summary first.
>
> You are right about the identity, and thanks for pushing me to check
> it properly.  Decoded from the live DT: "npu" is SCMI_CLK_NPU (SCMI
> clock id 6); aclk/hclk/pclk come from the CRU.  The domains list
> CLK_NPU_DSU0, which devfreq never touches.
>
> I then took clk_summary snapshots idle, mid-benchmark at nominal 600,
> and mid-benchmark at nominal 1000:
>
>                     idle      during 600   during 1000
>   clk_npu_dsu0      250 MHz   250 MHz      250 MHz
>   aclk_npu0/1/2     250       250          250
>   hclk_npu0/1/2     198       198          198
>   pclk_npu_root     100       100          100
>
> So the handshake-side clocks never move - this driver already does
> exactly what your vendor sample shows (compute clock scaled, dsu/aclk/
> hclk parked at boot rates).  And yet the domain transition still fails
> whenever the SCMI compute clock is raised, reproducibly.  So the
> constraint tracks the compute clock itself, not the DSU clock, and
> "pick a clock outside the domain lists" unfortunately does not exist
> here: the only rate anyone scales is already outside them.
>
> Whether the mechanism is a PLL inside the domain remains my hypothesis
> - the TRM's NPU chapter is silent on PVTPLL and I still have not risked
> reading its registers.  The constraint itself is empirical.
>
> A caveat worth passing on: clk_summary reports SCMI_CLK_NPU at
> 200 MHz even mid-benchmark at nominal 1000 - the CCF cache is stale for
> firmware clocks, so clk_summary genuinely cannot answer this question.
> I verified the actual rate two ways: clk_get_rate() (which round-trips
> to firmware), and the per-chunk NPU time scaling 236 -> 69 us across
> the sweep.
>
> > your plateau looks memory bound [...] Maybe worth one compute dense
> > model before cutting the table there.
>
> Fair, and the answer is a few lines up: I am no longer proposing to cut
> the table anywhere.  I did run the compute-dense model you asked for:
> InceptionV1 (quant), same rig, 850 mV.  One caveat first: on this stack
> Teflon leaves ~79% of an Inception inference on the CPU (many more ops
> fall back than for MobileNet), so total throughput barely moves with
> clock and the honest metric is the NPU-side time per inference, which
> the harness separates out:
>
>   nominal     NPU-side per inference    total
>   300 MHz     7.2 ms                    43.7 inf/s
>   700 MHz     4.16 ms                   50.5 inf/s
>   1000 MHz    4.17 ms                   50.2 inf/s
>
> It scales 300 -> 700 and then flattens at exactly the same knee as
> MobileNetV1 (whose NPU-side time goes 4.54 -> 3.00 -> 2.97 ms over the
> same points).  So on this board the plateau is the platform, not
> MobileNetV1's memory profile - consistent with the LPDDR4X explanation,
> and it does not change the conclusion: full table, per-board cap.
> Boards with faster memory may well keep scaling; mine cannot test that.
>
> (Two incidental data points from the same runs: Inception jobs get
> scheduled across two NPU cores here - 33+27 interrupts per inference
> against MobileNet's 41 on one core - and the bit-exact oracle passed at
> every frequency for this model too.)
>
> On your RK3576 SCMI experiment (zero jobs, 83 timeouts): one cross-data
> point - on RK3588 the SCMI/PVTPLL path is what we use successfully for
> the GPU (Mali-G610 at 1 GHz via SCMI, measured and stable), and the NPU
> compute clock here is SCMI-routed and scales fine while the domain is
> up.  So the SCMI plumbing itself can work on this family; whatever
> failed on RK3576 may be specific to that port rather than the approach.
>
>
> 4. Thermal, update
> ==================
>
> The missing NPU cooling chain from my first mail is now running here
> end-to-end: #cooling-cells on the NPU node, a passive trip at 85 degC
> with a cooling map, and devfreq_cooling registered by the driver - the
> kernel bound them together without manual intervention.  So the two DT
> patches are ready whenever the driver work lands; they need nothing
> beyond what devfreq already provides.
>
>
> 5. Where this leaves the questions to Tomeu
> ===========================================
>
> The two RFC questions that survive unchanged: whether you want DVFS
> for rocket upstream at all,

Sure!

> and whether the genpd-notifier shape is
> acceptable.

To be honest, I don't feel qualified to answer that. If you don't get
an answer here or know whom to ask (you could check git blame), you
can just send your best try to elicit a discussion.

> The OPP question is answered above (full table, per-board
> cap).

Right.

>  The code cleanup questions stand as written.

Regarding code organization, please check how devfreq is used in other
DRM drivers for ideas.

Thanks,

Tomeu

> Sorry for the noise, and thanks to Jiaxing for the push to verify the
> clock tree - the numbers are better for it.
>
> Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Nicolas Dufresne]

[-- Attachment #1.1: Type: text/plain, Size: 9583 bytes --]

Hi Igor,

Le samedi 01 août 2026 à 15:16 +0200, Igor Paunovic a écrit :
> Hi Tomeu,
> 
> Since you asked for fixes to be sent upfront I have kept poking at the
> RK3588 NPU, and I ended up implementing devfreq for rocket locally.  It
> works, but on the way there I hit a crash class that I could not find
> documented anywhere, and I also measured something about the vendor OPP
> table that I did not expect.  Both seem worth sharing before I clean any
> of it up for posting, so I would rather ask first than send a series you
> may not want in this shape.
> 
> Cc'ing Jiaxing since he is working on the clocks and on RK3576.

Tomeu just notified me that you had sent this RFC. Without any precise
commitment, I was also interested in this and have used AI to come up with a
proof of concept. I'm only looking at the RK3588, I suppose all the issues below
related to RK3576 ? The main constraints on RK3588 is that there is one clock
rate for all cores. the PM part allow suspending the cores independently, but a
special case is needed once all core are suspended, since you cannot resume
reliable at any other rate then 200MHz. This part still feels like a hack to me,
I might just hand rewrite it once I have a plan.

https://gitlab.collabora.com/nicolas/linux/-/commits/rock5b-npu-poc-4?ref_type=heads

I have saved my work here, even though its far from ready to be sent to an ML.
You will feel the AI vibe in the comment/commit, this all needs to be rewritten
for human readers. All this work depends on upstream ATF, I have no idea if the
ABI is the same as downstream, and to be honest, I'm not very interested in
that.

Please, feel free to take whatever you like, I'm still working on it, but its
possible my timeline will endup too long for you. Some important note:

**operating-points-v2**

My view of it is that its not that useful for devfreq in general, you mostly
want a max/lower rate effect, since throughput is generally more important. But:

1. I was adding it anyway with the idea that a future thermal (which you report
having) would be able to throttle it in small increment. As you reported, the
chip itself does not seem to heat up, but my view is that if you can balance the
throttle across CPU/GPU and NPU, you may be able to sustain better overall
performance (to be proven).

2. Even though we always hit max rate, having the OPP makes it really easy for
someone with low power constraints change it per board.

The other problem of the OPP implementation is the driver and DTS side. The
bindings seems sane, we don't require it for backward compatibility reason, but
the driver is modified in a way that it will fail if you don't provide it. I
think it would be fixed, and driver should keep working without the OPP in the
DTS. In the DTS, I've removed the assigned clock/freq, to me keeping it would be
toward forward compatibility, and is fine, but I don't always fully grasp all
the subtil trap of DT compatibility. Its something I will want to ask an expert
before hitting the ML.

cheers,
Nicolas

> 
> 
> 1. The hardware constraint
> ==========================
> 
> An NPU power domain cannot be switched on or off while the NPU compute
> clock is above its DT assigned-clock-rate of 200 MHz.
> 
> Changing the rate while a domain is already on is fine - I have taken it
> to 1 GHz and back many times without a single error.  It is the domain
> transition that breaks.
> 
> What happens when a domain is moved at a high rate:
> 
>   rockchip-pm-domain ...: failed to get ack on domain 'nputop', val=0xa9ffe
>   rocket fdab0000.npu: devfreq: cannot power up for rate change: -110
> 
> The domain is then wedged: genpd still believes it is on, but the first
> MMIO into it raises an asynchronous SError and the box panics.  Captured
> over the serial console:
> 
>   Kernel panic - not syncing: Asynchronous SError Interrupt
>   Comm: rmmod
>    _regmap_read
>    regmap_read
>    rockchip_pd_power
>    rockchip_pd_power_off
>    _genpd_power_off        <- rollback
>    genpd_power_off
>    genpd_power_on          <- failed
>    genpd_runtime_resume
>    device_release_driver
> 
> This is not specific to nputop.  I have the same message for 'npu2'
> (val=0xa9fff), which matches the DT: all three NPU domains list the NPU
> clock among their handshake clocks - rk3588-base.dtsi lines 864, 877 and
> 885, for RK3588_PD_NPUTOP, RK3588_PD_NPU1 and RK3588_PD_NPU2.
> 
> So the clock the domains need for their idle/ack handshake is the same
> clock we would be scaling.  My best explanation is that the PLL that
> produces it lives inside the domain, so once the domain drops, the clock
> state goes with it and the handshake can never complete.  I cannot
> confirm that part - reading the PVTPLL registers is documented as
> hanging the machine, so I have not tried.  The behaviour itself is
> reproducible and cost me four hard hangs before I understood it.
> 
> I mention it because it is a trap for anyone adding DVFS here, including
> the RK3576 work, and because it is invisible until the first time you
> let the NPU idle at a raised clock.
> 
> 
> 2. What ended up working
> ========================
> 
> Runtime PM callbacks are not enough.  The domains are powered on by the
> driver core before probe, powered off from a workqueue after detach, and
> system sleep bypasses runtime PM references entirely - so the driver
> never sees all the transitions.
> 
> What does work is hooking the transitions themselves:
> dev_pm_genpd_add_notifier() on all three cores, and on GENPD_NOTIFY_PRE_ON
> and GENPD_NOTIFY_PRE_OFF force the clock back to the DT rate, vetoing the
> transition with notifier_from_errno() if that fails.  Every path -
> runtime PM, system sleep, attach at probe, detach after unbind - goes
> through _genpd_power_on()/_genpd_power_off(), so nothing can slip past.
> If a transition does happen, the boost cancels itself and says so,
> rather than leaving the driver claiming a rate the hardware is not
> running.
> 
> This has now survived everything that used to kill the box, including
> repeated sleep/wake cycles at a raised clock and rmmod while boosted.
> 
> 
> 3. The numbers, which are the surprising part
> =============================================
> 
> Measured with MobileNetV1 through Teflon, one inference thread pinned to
> one A76, and a bit-exact oracle: sha256 over intermediate tensors on
> every iteration, zero tolerance.  The 600, 900 and 1000 MHz rows are
> 30-minute runs of 275k-280k inferences each and the oracle passed
> bit-exact in all three, so none of this is instability; the 200 MHz row
> is a shorter control from the same session.
> 
>   nominal    supply    throughput
>   200 MHz    800 mV     68.5 inf/s      (the current fixed rate)
>   600 MHz    800 mV    155.4 inf/s
>   900 MHz    850 mV    152.4 inf/s
>   1000 MHz   850 mV    152.9 inf/s
> 
> 600 MHz is the optimum.  900 and 1000 are indistinguishable from each
> other and both land about 4% below 600, on a quieter and cooler machine.
> The vendor OPP table decoded from the downstream DTB asks for 700 mV up
> to 700 MHz, 750 mV at 800, 800 mV at 900 and 850 mV at 1000, and I ran
> the top of that range at the voltage it asks for - it does not help.
> 
> Backing out an effective clock from the per-chunk time, nominal 600
> appears to deliver more than nominal 900 or 1000 do.  I would not lean
> on that decode, but the throughput ordering does not depend on it.
> 
> Thermals were never a factor: the highest I saw all day was 50.8 degC,
> against a critical trip at 115.  This is one board and one model, and
> MobileNetV1 is memory-heavy, so a compute-dense network may well behave
> differently - but for this workload the top half of the vendor table
> buys nothing.
> 
> If that holds up elsewhere, an OPP table for rocket probably should not
> simply mirror the vendor one.
> 
> 
> 4. Thermal, separately
> ======================
> 
> While looking at this I noticed npu-thermal has only a critical trip at
> 115 degC - no passive trip, no cooling map, polling-delay-passive is 0 -
> while gpu-thermal, a few lines above in the same file and on the same
> tsadc, has both.  That was harmless while the NPU was pinned at 200 MHz
> because it could not be slowed down anyway; with DVFS it stops being
> harmless.
> 
> I have two small patches for that (a #cooling-cells binding update and
> the thermal zone itself), but they only make sense once something
> registers a cooling device, so they would belong with the driver work
> rather than on their own.
> 
> 
> 5. What I am asking
> ===================
> 
> - Is DVFS for rocket something you want upstream at all, or is it better
>   left alone for now?
> 
> - Does the genpd-notifier approach look right to you, or is there a
>   cleaner hook I have missed?
> 
> - Given the measurements, would you want the OPP table to stop at
>   600 MHz rather than follow the vendor range?
> 
> - If you do want a series, I will need to clean the code up first - it
>   still keeps its state file-static rather than in rocket_device, and it
>   bypasses the OPP layer for the rate change because our own direct
>   writes leave the OPP cache stale.  Both are fixable; I would rather
>   know the shape you want before rewriting.
> 
> Happy to send the current code as-is off-list if that is easier to
> comment on than prose.
> 
> Thanks,
> Igor

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Nicolas,

> I'm only looking at the RK3588, I suppose all the issues below
> related to RK3576 ?

No - everything in the RFC is RK3588 (Orange Pi 5 Plus, all three
cores). RK3576 is Jiaxing's enablement series and has its own set of
problems; nothing I reported came from there.

I went through the four commits on rock5b-npu-poc-4 today. We
converged on the same shape independently, which is encouraging:
200 MHz kept as the suspend rate, a single devfreq instance modelled
on panfrost with busy time aggregated across the three cores, and a
cooling device on top. Your ~2.5x on the SSD pipeline also matches
the 2.58x I measured here with simple_ondemand against the 200 MHz
pin.

The TF-A pointer (rk3588_clk.c, PVTPLL vs normal path) is the most
valuable part for me - it names the mechanism behind the power-on
ack failure I could only demonstrate empirically. I will reference
it in the cover letter once I have checked the firmware source
myself.

Status here: after Tomeu's go-ahead I am preparing the series -
bindings, a full-range OPP table in the DT (300-1000 MHz plus the
200 MHz suspend point, so essentially the table you ended up with),
a safe-rate-on-suspend guard, the devfreq itself, and a hold-all
guard that resumes all cores around any rate change. The guard is
ordered before the devfreq patch so no bisect point has scaling
without it. The one hard dependency is my "request the core clocks
by name" v2, still waiting for pickup.

Agreed on OPP staying optional - the plan in my series is that the
driver keeps working with no OPP table in the DT, which I saw you
intend to fix on your side as well.

One path worth checking in your PoC, because it is the one that
made me write the hold-all guard: a sysfs min_freq/max_freq write
while all three cores are runtime-suspended goes straight to
clk_set_rate, which can select the PVTPLL path while the domain is
off - exactly the case your TF-A reference explains. With the guard
in place I measured that write waking the cores and completing
cleanly.

One difference in test conditions worth keeping in mind: my numbers
are with the vendor bl31 that EDK2 bundles, yours is upstream TF-A.
Comparing SCMI behaviour on both seems wise before either of us
claims anything firmware-specific.

Thank you for the "take whatever you like" - anything I lift will
carry credit, and I will Cc you on the series.

Regards,
Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Igor Paunovic]

Hi Diederik,

(re-adding the lists, your reply came to me only)

Thanks - that pointer was useful, and it changes what I thought I was
looking at.  Two things came out of following it.

First, a data point that may narrow your search rather than widen it:
this board is not running upstream TF-A.  Its SCMI implementation
identifies itself as Rockchip's own:

  arm-scmi arm-scmi.0.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0

and it still fails, so whatever this is, it is not exclusive to upstream
TF-A BL31.  If rkbin BL31 makes your PineTab2 suspend/resume work, the
difference there may be something more specific than the ack handshake
itself.

Second, going through the archive for that error string, the history is
longer than I realised - Peter Geis' and Sebastian Reichel's pmdomain
work on RK3588, and further back "arm64: dts: rockchip: add hevc power
domain clock to rk3328".  That last one is interesting because the fix
was to add a missing clock to the power domain, i.e. the handshake
needs its clocks.

What I have looks like a neighbouring case rather than the same one.
Here the clocks are not missing - all three NPU domains already list the
NPU clock (rk3588-base.dtsi lines 864, 877 and 885).  It is the rate
that matters: the ack never arrives if the domain is moved while that
clock is above its DT assigned-clock-rate, and the same domain moves
cleanly once the rate is back down.  I have not seen that variant
described anywhere, which is why I wrote it up.

That suggests something you could try, though it is a guess and you know
your board far better than I do: if any clock listed in a domain you are
suspending happens to be at a non-boot rate at that moment - left there
by devfreq, by an assigned-clock-rate, or by whatever ran last - then it
would be the same shape as what I am seeing.  Pinning those clocks to
their boot rates before the transition would be a cheap thing to rule
out.  If it does nothing, at least it is eliminated.

Thanks again for taking the time, and good luck with the PineTab2.

Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Diederik de Haas]

On Sat Aug 1, 2026 at 4:40 PM CEST, Igor Paunovic wrote:
> Hi Diederik,
>
> (re-adding the lists, your reply came to me only)
>
> Thanks - that pointer was useful, and it changes what I thought I was
> looking at.  Two things came out of following it.

Then I guess it's useful I add the main part of that pointer to the ML:

  This is all 'above my paygrade', but the 'failed to get ack on domain' is
  a type of error which seems to surface on a regular basis:
  https://lore.kernel.org/linux-rockchip/?q=%22failed+to+get+ack+on+domain%22

(My hope was that it may have given Igor pointers wrt the problem)

> First, a data point that may narrow your search rather than widen it:
> this board is not running upstream TF-A.  Its SCMI implementation
> identifies itself as Rockchip's own:
>
>   arm-scmi arm-scmi.0.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0
>
> and it still fails, so whatever this is, it is not exclusive to upstream
> TF-A BL31.  If rkbin BL31 makes your PineTab2 suspend/resume work, the
> difference there may be something more specific than the ack handshake
> itself.
>
> Second, going through the archive for that error string, the history is
> longer than I realised - Peter Geis' and Sebastian Reichel's pmdomain
> work on RK3588, and further back "arm64: dts: rockchip: add hevc power
> domain clock to rk3328".  That last one is interesting because the fix
> was to add a missing clock to the power domain, i.e. the handshake
> needs its clocks.
>
> What I have looks like a neighbouring case rather than the same one.
> Here the clocks are not missing - all three NPU domains already list the
> NPU clock (rk3588-base.dtsi lines 864, 877 and 885).  It is the rate
> that matters: the ack never arrives if the domain is moved while that
> clock is above its DT assigned-clock-rate, and the same domain moves
> cleanly once the rate is back down.  I have not seen that variant
> described anywhere, which is why I wrote it up.
>
> That suggests something you could try, though it is a guess and you know
> your board far better than I do: if any clock listed in a domain you are
> suspending happens to be at a non-boot rate at that moment - left there
> by devfreq, by an assigned-clock-rate, or by whatever ran last - then it
> would be the same shape as what I am seeing.  Pinning those clocks to
> their boot rates before the transition would be a cheap thing to rule
> out.  If it does nothing, at least it is eliminated.
>
> Thanks again for taking the time, and good luck with the PineTab2.
>
> Igor
>
> _______________________________________________
> Linux-rockchip mailing list
> Linux-rockchip@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-rockchip


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip