RK3568 NPU: what triggers the CNA weight-fetch DMA after the executer engages? (mainline rocket vs vendor rknpu)

[Midgy BALON <midgy971@gmail.com>]

Hi Chaoyi, hi all,

Following up on the RK3568 NPU work — thanks again for the AUTO_GATING
bit-31 fix and the MMU-v2 clarification; both are in the driver now.

The mainline rocket accel driver reaches a good state on the RK3568
NPU: it probes, powers the domain, programs the pipeline, and submits a
job cleanly.  One functional problem remains, and I have narrowed it down
to a single hardware step that I don't think I can resolve from outside
Rockchip.

The test is one convolution (conv2d, 16 -> 128 channels, 5x5, stride 2)
run through Teflon/mesa.  On the vendor rknpu driver that conv produces
correct output.  On rocket the job engages the whole pipeline but never
computes: the output tensor comes back uniformly at the quantization
zero-point (every element == 128), and the job times out with no
completion interrupt.

I captured the working vendor rknpu and the stuck rocket running the SAME
conv, on the SAME SoC, in the same boot, and drove both to the identical
engaged hardware state.  The per-unit status and the CNA configuration
registers match exactly:

  - both reach S_STATUS = 0x0c and S_POINTER = 0x1000e (executer bit 16
    set) on all five sub-units (CNA / CMAC / ACCU / DPU / RDMA);
  - PC TASKST = 0xf000 on both;
  - the CNA config is byte-identical: CONV_CON*, DATA_SIZE*,
    WEIGHT_SIZE*, CBUF.

The divergence is entirely in the data-movement counters (the NPU perf
registers).  Read straight off the hardware:

  counter  (perf reg)     vendor rknpu    rocket    what it counts
  -------------------     ------------    ------    ----------------------
  wt_rd    (0x803c)       0x1900          0         CNA weight-fetch DMA
  dt_wr    (0x8034)       0x6400          0         DPU output writes
  dt_rd    (0x8038)       0x32c9          0x8b      feature / regcmd reads

The vendor's counters describe a full convolution: its CNA fetches weights
(wt_rd = 0x1900) and its DPU writes the complete output (dt_wr = 0x6400).
rocket's CNA never fetches weights (wt_rd = 0) and its DPU never writes
(dt_wr = 0); rocket's dt_rd = 0x8b accounts for only the regcmd fetch, so
it never even reads the feature data.  With no weights the MAC array
starves and the DPU writes nothing — which is exactly the uniform
zero-point output we see.

(One aside for anyone who has chased this IP before: TASKST = 0xf000 is
the NORMAL working value — the vendor produces full output at 0xf000.  It
is not a stall.)

I have matched every software-visible surface I can find:

  - Command stream + data.  Byte-exact replay of the vendor's captured
    conv2d regcmd, together with its weight BO and input BO, through the
    rocket UABI — still no compute.  The content is not the gate.

  - state_init.  rk356x_rknpu_config.state_init is NULL on RK3568 (only
    RK3576 has one).  I tried the RK3576 ritual (0x1024 = 0x80000000,
    S_POINTER 0 -> 1 -> 0x1e) on RK3568 anyway — no change.

  - Reset / power / clocks / IOMMU.  rocket mirrors rknpu_soft_reset (srst
    assert/deassert, SCMI/PVTPLL two-step, NoC de-idle at PMU
    0xfdd90000, IOMMU re-establish); clocks match the vendor (SCMI 1 GHz,
    CLK_NPU / aclk 200 MHz); AUTO_GATING uses your bit-31 fix.

  - Non-NPU-block writes (captured live for both drivers in the same
    boot).  The only raw non-NPU writes the vendor makes are the NoC/PMU
    de-idle (NOC_AUTO = 0xfffb, BUS_IDLE_ST = 0x1ea, PWR_STATUS = 0x7f);
    rocket makes them identically.  There is no npu_grf read-margin and no
    memory-repair step on RK3568 (those exist only on RK3576 / RK3588).

With the executer engaged (S_STATUS = 0x0c, S_POINTER bit 16 set) and the
CNA's regcmd config already landed, what actually triggers the CNA's
weight-fetch DMA?  The vendor rknpu reaches wt_rd = 0x1900 from a clean
submit; rocket, on the same SoC with the same engage and the same regcmd,
stays at wt_rd = 0.

My remaining hypotheses are all hardware-internal, and I would value your
read on which (if any) is real:

  1. an ordering/timing requirement between the engage and the
     weight-fetch trigger that a byte-identical but differently-sequenced
     submit does not satisfy;

  2. an internal CNA/CBUF sequencer or credit/handshake that the vendor's
     power-on or first-job path leaves armed, and that is not a
     CPU-visible register;

  3. an undocumented CNA precondition (a CBUF / DCOMP / weight-DMA enable)
     that must be set before the fetch will issue.

Even a pointer of the form "the weight fetch won't start unless X is set"
would unblock the driver.  The rest of the driver is upstream-ready; this
is the only thing between "submits" and "computes", and I would be happy
to run any targeted register capture on the board.

Thanks in advance.
Kind regards
Midgy

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip