Re: [PATCH] accel/rocket: search every core slot when a core is removed

[Igor Paunovic <royalnet026@gmail.com>]

Hi Jiaxing,

> Moved here from the DVFS thread so that b4 can collect it.
>
> Tested-by: Jiaxing Hu <gahing@gahingwoo.com> # RK3576, two cores

Thank you, and it works: b4 picks it up with the comment attached and
the DKIM check passes. Two cores on a different SoC is worth more to
this patch than anything I can produce on my own board.

Your test also found the edge of something else, and I would rather
tell you before you spend more rounds on it.

You wrote that the cores came back "numbered 0 and 1 both times". That
is the case that works. I rebound the three cores here in all six
orders and only the devicetree order ran clean:

  bind order (slot 0 first)   NPU job timed out         inference
  fdab fdac fdad (devicetree) none                      129.4 inf/s, oracle ok
  fdab fdad fdac              27 fdad                   1.95 inf/s, oracle ok
  fdac fdab fdad              135 fdac, 5 fdab, 1 fdad  oracle rejected output
  fdac fdad fdab              135 fdac, 5 fdad          oracle rejected output
  fdad fdab fdac              136 fdad, 5 fdab          oracle rejected output
  fdad fdac fdab              135 fdad, 1 fdab          oracle rejected output

Counted from the journal since each bind, one six-second inference per
row. The single fdad timeout in row three is on a correctly numbered
core and I cannot account for it; the most likely explanation is a job
still draining from the previous row, since the count starts before the
unbind. Everything else lands on a core whose slot is not its hardware
number, in proportion to the tasks the scheduler gave it, and in both
directions of the mismatch: 5 timeouts on the core in slot 1 whether its
hardware number is above it (row four) or below it (row five).

The cause is in rocket_job_hw_submit():

  extra_bit = 0x10000000 * core->index;

core->index is the slot the core takes in rdev->cores[], which is bind
order. The vendor driver computes that same bit from the hardware
number of the core (rknpu_job.c: REG_WRITE((0xe + 0x10000000 * i), ...)
with i indexing rknpu_dev->base[]). On a normal boot the two agree and
nothing shows. Bind out of that order and every task submitted to a
core whose slot is not its hardware number times out after 500 ms, the
reset does not help, and the inference finishes with wrong output. No
error, no warning, no regulator or clock message - the rail sat at
700000 uV in the runs that passed and in the runs that failed. Only the
bit-exact oracle and the UART log showed it.

The second row is the one I would have missed. The oracle passed there,
because the tensors it checks are computed by the core in slot 0, which
happened to be correct. Throughput was 66 times lower and 27 jobs had
timed out. A single assertion does not catch this.

Two cores make it cheaper to test than three. If you bind the second
core before the first on your ROCK 4D, on your current kernel, I expect
the jobs that land on slot 0 to time out and your nine models to stop
decoding identically. If they do, that is the bug reproduced on a second
SoC by someone who is not me, which is worth more than my six rows.
If they do not, I have the cause wrong and I would like to know that -
please say so on the patch thread rather than here, so it lands where
b4 will pick it up.

The fix is on the list now:

  https://lore.kernel.org/dri-devel/20260905135612.7324-1-royalnet026@gmail.com/

It numbers the cores by their position among the core nodes in the
devicetree. It resolves that through dev->driver->of_match_table
rather than a compatible string, so it works both before and after your
12/14, which moves that enumeration to for_each_matching_node() and
renames the table. All six orders pass afterwards, 124 to 136 inf/s,
zero timeouts. It is a separate patch with Fixes: and Cc: stable, and
it goes out before v2 of the DVFS series.

One more thing your report is useful for: it is the same class as what
you found on RK3576 last week. Wrong result, no complaint from the
hardware. I am starting to think this driver needs a bit-exact check in
whatever test people run against it, because throughput alone will
happily report a healthy number while the output is garbage.

Igor