Re: [RFC] accel/rocket: DVFS on RK3588 - a hardware constraint, and some numbers

[Jiaxing Hu <gahing@gahingwoo.com>]

Hi Igor,

> One honest gap: I could not yet take the voltage-vs-frequency
> curve [...] Future work.

I took one on RK3576 this week, from the correctness side, and it
turned out to be the answer to a fault I had been chasing for four
days. Short version: solved, and the fix is a voltage.

The fault. Two NPU cores with jobs in flight at the same time, and the
second one writes single words of its output wrong: the right value
plus a bit of the accumulator. Of 64 wrong words I took apart, 63 are
an integer away from the right one and 25 of those by exactly 1024,
and re-reading the same word after invalidating the CPU's cache of the
buffer gave the same wrong value 48 times out of 48, so it is what the
hardware wrote. One core is always exact. It took four days because it
is invisible to a single threaded harness.

The cause. Mainline sets no rate for CLK_RKNN_DSU0 and no board sets
vdd_npu_s0, so my ROCK 4D runs the NPU at 786.432 MHz on 750 mV, while
Rockchip's own table for this NPU asks 800 mV of its 800 MHz step at
the worst leakage bin. Same wall as yours on the rail, so the curve is
one device tree per point. A pass below is 5400 rows of a batched
matrix multiply, each row against the same multiply done one row at a
time:

  786 MHz, 750 mV    11 to 25 wrong rows a pass
  594 MHz, 750 mV    0, 0, 0, 0
  786 MHz, 800 mV    0, 0, 0, 0
  786 MHz, 850 mV    0, 0, 0, 0

The fix, two of them, both measured. Give the rail 800 mV and keep
786 MHz: both cores run, nine models come out identical, and my time
to first token drops about 20%. Or leave the rail alone and clock the
NPU at 594: also exact, and still ahead of one core at 786, because
what 786 MHz costs is the second core. My v12 takes 594 in the SoC
dtsi, since a board that describes no NPU rail has to be correct too,
and my runtime now reads the rail and the clock out of sysfs and only
runs the two cores together inside the vendor's envelope.

What I would do on RK3588. Your rows are one inference thread with a
bit-exact oracle, so they cannot see this class at all. Before the OPP
table is settled, run the oracle with all three cores loaded at 900
and 1000 MHz on 850 mV. And whatever the table ends up as, it needs
the voltage column with the rate: a rate without its rail is what
mainline has today, and it is what corrupts.

One SCMI note, since you are reading rates through it now. On RK3576
an assigned-clock-rates on the SCMI clock in the NPU node hangs the
board before the console comes up, with no kernel output at all. I
have not proven why, but the vendor's driver never sets that rate from
DT either: rockchip_opp_config_clks() returns early when the clock is
an SCMI clock and the device is not runtime active, so the rate is
only ever written from the runtime resume path with the domains
already on. Your EL3 path is a read, so none of this touches it.

Cheers,
Jiaxing