* RK3568 NPU: what triggers the CNA weight-fetch DMA after the executer engages? (mainline rocket vs vendor rknpu)
@ 2026-07-21 23:07 ` Midgy BALON
0 siblings, 0 replies; 4+ messages in thread
From: Midgy BALON @ 2026-07-21 23:07 UTC (permalink / raw)
To: chaoyi.chen
Cc: linux-rockchip, Tomeu Vizoso, Simon Xue, Heiko Stuebner,
dri-devel
Hi Chaoyi, hi all,
Following up on the RK3568 NPU work — thanks again for the AUTO_GATING
bit-31 fix and the MMU-v2 clarification; both are in the driver now.
The mainline rocket accel driver reaches a good state on the RK3568
NPU: it probes, powers the domain, programs the pipeline, and submits a
job cleanly. One functional problem remains, and I have narrowed it down
to a single hardware step that I don't think I can resolve from outside
Rockchip.
The test is one convolution (conv2d, 16 -> 128 channels, 5x5, stride 2)
run through Teflon/mesa. On the vendor rknpu driver that conv produces
correct output. On rocket the job engages the whole pipeline but never
computes: the output tensor comes back uniformly at the quantization
zero-point (every element == 128), and the job times out with no
completion interrupt.
I captured the working vendor rknpu and the stuck rocket running the SAME
conv, on the SAME SoC, in the same boot, and drove both to the identical
engaged hardware state. The per-unit status and the CNA configuration
registers match exactly:
- both reach S_STATUS = 0x0c and S_POINTER = 0x1000e (executer bit 16
set) on all five sub-units (CNA / CMAC / ACCU / DPU / RDMA);
- PC TASKST = 0xf000 on both;
- the CNA config is byte-identical: CONV_CON*, DATA_SIZE*,
WEIGHT_SIZE*, CBUF.
The divergence is entirely in the data-movement counters (the NPU perf
registers). Read straight off the hardware:
counter (perf reg) vendor rknpu rocket what it counts
------------------- ------------ ------ ----------------------
wt_rd (0x803c) 0x1900 0 CNA weight-fetch DMA
dt_wr (0x8034) 0x6400 0 DPU output writes
dt_rd (0x8038) 0x32c9 0x8b feature / regcmd reads
The vendor's counters describe a full convolution: its CNA fetches weights
(wt_rd = 0x1900) and its DPU writes the complete output (dt_wr = 0x6400).
rocket's CNA never fetches weights (wt_rd = 0) and its DPU never writes
(dt_wr = 0); rocket's dt_rd = 0x8b accounts for only the regcmd fetch, so
it never even reads the feature data. With no weights the MAC array
starves and the DPU writes nothing — which is exactly the uniform
zero-point output we see.
(One aside for anyone who has chased this IP before: TASKST = 0xf000 is
the NORMAL working value — the vendor produces full output at 0xf000. It
is not a stall.)
I have matched every software-visible surface I can find:
- Command stream + data. Byte-exact replay of the vendor's captured
conv2d regcmd, together with its weight BO and input BO, through the
rocket UABI — still no compute. The content is not the gate.
- state_init. rk356x_rknpu_config.state_init is NULL on RK3568 (only
RK3576 has one). I tried the RK3576 ritual (0x1024 = 0x80000000,
S_POINTER 0 -> 1 -> 0x1e) on RK3568 anyway — no change.
- Reset / power / clocks / IOMMU. rocket mirrors rknpu_soft_reset (srst
assert/deassert, SCMI/PVTPLL two-step, NoC de-idle at PMU
0xfdd90000, IOMMU re-establish); clocks match the vendor (SCMI 1 GHz,
CLK_NPU / aclk 200 MHz); AUTO_GATING uses your bit-31 fix.
- Non-NPU-block writes (captured live for both drivers in the same
boot). The only raw non-NPU writes the vendor makes are the NoC/PMU
de-idle (NOC_AUTO = 0xfffb, BUS_IDLE_ST = 0x1ea, PWR_STATUS = 0x7f);
rocket makes them identically. There is no npu_grf read-margin and no
memory-repair step on RK3568 (those exist only on RK3576 / RK3588).
With the executer engaged (S_STATUS = 0x0c, S_POINTER bit 16 set) and the
CNA's regcmd config already landed, what actually triggers the CNA's
weight-fetch DMA? The vendor rknpu reaches wt_rd = 0x1900 from a clean
submit; rocket, on the same SoC with the same engage and the same regcmd,
stays at wt_rd = 0.
My remaining hypotheses are all hardware-internal, and I would value your
read on which (if any) is real:
1. an ordering/timing requirement between the engage and the
weight-fetch trigger that a byte-identical but differently-sequenced
submit does not satisfy;
2. an internal CNA/CBUF sequencer or credit/handshake that the vendor's
power-on or first-job path leaves armed, and that is not a
CPU-visible register;
3. an undocumented CNA precondition (a CBUF / DCOMP / weight-DMA enable)
that must be set before the fetch will issue.
Even a pointer of the form "the weight fetch won't start unless X is set"
would unblock the driver. The rest of the driver is upstream-ready; this
is the only thing between "submits" and "computes", and I would be happy
to run any targeted register capture on the board.
Thanks in advance.
Kind regards
Midgy
^ permalink raw reply [flat|nested] 4+ messages in thread
* RK3568 NPU: what triggers the CNA weight-fetch DMA after the executer engages? (mainline rocket vs vendor rknpu)
@ 2026-07-21 23:07 ` Midgy BALON
0 siblings, 0 replies; 4+ messages in thread
From: Midgy BALON @ 2026-07-21 23:07 UTC (permalink / raw)
To: chaoyi.chen
Cc: linux-rockchip, Tomeu Vizoso, Simon Xue, Heiko Stuebner,
dri-devel
Hi Chaoyi, hi all,
Following up on the RK3568 NPU work — thanks again for the AUTO_GATING
bit-31 fix and the MMU-v2 clarification; both are in the driver now.
The mainline rocket accel driver reaches a good state on the RK3568
NPU: it probes, powers the domain, programs the pipeline, and submits a
job cleanly. One functional problem remains, and I have narrowed it down
to a single hardware step that I don't think I can resolve from outside
Rockchip.
The test is one convolution (conv2d, 16 -> 128 channels, 5x5, stride 2)
run through Teflon/mesa. On the vendor rknpu driver that conv produces
correct output. On rocket the job engages the whole pipeline but never
computes: the output tensor comes back uniformly at the quantization
zero-point (every element == 128), and the job times out with no
completion interrupt.
I captured the working vendor rknpu and the stuck rocket running the SAME
conv, on the SAME SoC, in the same boot, and drove both to the identical
engaged hardware state. The per-unit status and the CNA configuration
registers match exactly:
- both reach S_STATUS = 0x0c and S_POINTER = 0x1000e (executer bit 16
set) on all five sub-units (CNA / CMAC / ACCU / DPU / RDMA);
- PC TASKST = 0xf000 on both;
- the CNA config is byte-identical: CONV_CON*, DATA_SIZE*,
WEIGHT_SIZE*, CBUF.
The divergence is entirely in the data-movement counters (the NPU perf
registers). Read straight off the hardware:
counter (perf reg) vendor rknpu rocket what it counts
------------------- ------------ ------ ----------------------
wt_rd (0x803c) 0x1900 0 CNA weight-fetch DMA
dt_wr (0x8034) 0x6400 0 DPU output writes
dt_rd (0x8038) 0x32c9 0x8b feature / regcmd reads
The vendor's counters describe a full convolution: its CNA fetches weights
(wt_rd = 0x1900) and its DPU writes the complete output (dt_wr = 0x6400).
rocket's CNA never fetches weights (wt_rd = 0) and its DPU never writes
(dt_wr = 0); rocket's dt_rd = 0x8b accounts for only the regcmd fetch, so
it never even reads the feature data. With no weights the MAC array
starves and the DPU writes nothing — which is exactly the uniform
zero-point output we see.
(One aside for anyone who has chased this IP before: TASKST = 0xf000 is
the NORMAL working value — the vendor produces full output at 0xf000. It
is not a stall.)
I have matched every software-visible surface I can find:
- Command stream + data. Byte-exact replay of the vendor's captured
conv2d regcmd, together with its weight BO and input BO, through the
rocket UABI — still no compute. The content is not the gate.
- state_init. rk356x_rknpu_config.state_init is NULL on RK3568 (only
RK3576 has one). I tried the RK3576 ritual (0x1024 = 0x80000000,
S_POINTER 0 -> 1 -> 0x1e) on RK3568 anyway — no change.
- Reset / power / clocks / IOMMU. rocket mirrors rknpu_soft_reset (srst
assert/deassert, SCMI/PVTPLL two-step, NoC de-idle at PMU
0xfdd90000, IOMMU re-establish); clocks match the vendor (SCMI 1 GHz,
CLK_NPU / aclk 200 MHz); AUTO_GATING uses your bit-31 fix.
- Non-NPU-block writes (captured live for both drivers in the same
boot). The only raw non-NPU writes the vendor makes are the NoC/PMU
de-idle (NOC_AUTO = 0xfffb, BUS_IDLE_ST = 0x1ea, PWR_STATUS = 0x7f);
rocket makes them identically. There is no npu_grf read-margin and no
memory-repair step on RK3568 (those exist only on RK3576 / RK3588).
With the executer engaged (S_STATUS = 0x0c, S_POINTER bit 16 set) and the
CNA's regcmd config already landed, what actually triggers the CNA's
weight-fetch DMA? The vendor rknpu reaches wt_rd = 0x1900 from a clean
submit; rocket, on the same SoC with the same engage and the same regcmd,
stays at wt_rd = 0.
My remaining hypotheses are all hardware-internal, and I would value your
read on which (if any) is real:
1. an ordering/timing requirement between the engage and the
weight-fetch trigger that a byte-identical but differently-sequenced
submit does not satisfy;
2. an internal CNA/CBUF sequencer or credit/handshake that the vendor's
power-on or first-job path leaves armed, and that is not a
CPU-visible register;
3. an undocumented CNA precondition (a CBUF / DCOMP / weight-DMA enable)
that must be set before the fetch will issue.
Even a pointer of the form "the weight fetch won't start unless X is set"
would unblock the driver. The rest of the driver is upstream-ready; this
is the only thing between "submits" and "computes", and I would be happy
to run any targeted register capture on the board.
Thanks in advance.
Kind regards
Midgy
_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: RK3568 NPU: what triggers the CNA weight-fetch DMA after the executer engages? (mainline rocket vs vendor rknpu)
2026-07-21 23:07 ` Midgy BALON
@ 2026-11-20 11:27 ` MidG971
-1 siblings, 0 replies; 4+ messages in thread
From: MidG971 @ 2026-11-20 11:27 UTC (permalink / raw)
To: Chaoyi Chen
Cc: linux-rockchip, Tomeu Vizoso, Simon Xue, Heiko Stuebner,
dri-devel
Hi Chaoyi, hi all,
A gentle follow-up on the thread above from July: no reply yet, and I
wanted to add the new evidence I gathered since, in case it sharpens
the question.
Since then I confirmed the engage/completion split explicitly:
- Loading the vendor rknpu and running ONE vendor job ("priming"),
then unloading it (no power cycle) and loading mainline rocket,
makes rocket's executers engage for the first time (S_POINTER
bit 16 = 0x1000e on all five units, matching the working vendor
signature). So the engage is one-time HW state established by a
*working vendor job*, not by probe/init writes.
- But completion still fails afterwards: even primed, rocket's job
times out with wt_rd = 0 / dt_wr = 0 (vendor: 0x1900 / 0x6400).
- An NIU reset pulse (CRU SOFTRST_CON2 bits 8..10) at resume CLEARS
the primed state (reads drop to 0); nothing we write re-arms it.
I also tested a kiln-style reset edge inside rockchip_pd_power()
at PD power-on (RK3576 kiln 0002 analog) on v7.4-rc3, including a
forced cold power cycle of the PMU island from userspace:
negative -- the pulse does not re-arm the sequencer, and the PD
never actually cold-cycles on this board.
- Cache coherency ruled out (forced dma_sync of all BOs at submit;
live DT has no dma-coherent). CBUF_DBG matches the vendor's engaged
values pre/post job. The board now runs mainline v7.4-rc3 and the
gate reproduces there unchanged.
So the refined question: after a vendor job has left the NPU in the
armed per-power-session state, which step of that job arms the CNA
weight-fetch sequencer - and is there any RK3568-visible precondition
(CBUF/DCOMP/weight-DMA enable, ordering vs. engage) I could replicate
in rocket so it computes without the vendor prime?
Happy to run any targeted capture on the board (RK3568, ROCK 3B).
Thanks again.
Kind regards,
Midgy
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: RK3568 NPU: what triggers the CNA weight-fetch DMA after the executer engages? (mainline rocket vs vendor rknpu)
@ 2026-11-20 11:27 ` MidG971
0 siblings, 0 replies; 4+ messages in thread
From: MidG971 @ 2026-11-20 11:27 UTC (permalink / raw)
To: Chaoyi Chen
Cc: linux-rockchip, Tomeu Vizoso, Simon Xue, Heiko Stuebner,
dri-devel
Hi Chaoyi, hi all,
A gentle follow-up on the thread above from July: no reply yet, and I
wanted to add the new evidence I gathered since, in case it sharpens
the question.
Since then I confirmed the engage/completion split explicitly:
- Loading the vendor rknpu and running ONE vendor job ("priming"),
then unloading it (no power cycle) and loading mainline rocket,
makes rocket's executers engage for the first time (S_POINTER
bit 16 = 0x1000e on all five units, matching the working vendor
signature). So the engage is one-time HW state established by a
*working vendor job*, not by probe/init writes.
- But completion still fails afterwards: even primed, rocket's job
times out with wt_rd = 0 / dt_wr = 0 (vendor: 0x1900 / 0x6400).
- An NIU reset pulse (CRU SOFTRST_CON2 bits 8..10) at resume CLEARS
the primed state (reads drop to 0); nothing we write re-arms it.
I also tested a kiln-style reset edge inside rockchip_pd_power()
at PD power-on (RK3576 kiln 0002 analog) on v7.4-rc3, including a
forced cold power cycle of the PMU island from userspace:
negative -- the pulse does not re-arm the sequencer, and the PD
never actually cold-cycles on this board.
- Cache coherency ruled out (forced dma_sync of all BOs at submit;
live DT has no dma-coherent). CBUF_DBG matches the vendor's engaged
values pre/post job. The board now runs mainline v7.4-rc3 and the
gate reproduces there unchanged.
So the refined question: after a vendor job has left the NPU in the
armed per-power-session state, which step of that job arms the CNA
weight-fetch sequencer - and is there any RK3568-visible precondition
(CBUF/DCOMP/weight-DMA enable, ordering vs. engage) I could replicate
in rocket so it computes without the vendor prime?
Happy to run any targeted capture on the board (RK3568, ROCK 3B).
Thanks again.
Kind regards,
Midgy
_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-20 11:28 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-21 23:07 RK3568 NPU: what triggers the CNA weight-fetch DMA after the executer engages? (mainline rocket vs vendor rknpu) Midgy BALON
2026-07-21 23:07 ` Midgy BALON
2026-11-20 11:27 ` MidG971
2026-11-20 11:27 ` MidG971
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.