Re: [PATCH v2 0/2] accel/rocket: fix shared-device lifecycle on probe failure and unbind

[Igor Paunovic <royalnet026@gmail.com>]

Hi Tomeu,

A ping on this one. It has been on the list since 31 July and 2/2
carries Jiaxing's Reviewed-by.

I am pinging now rather than just waiting because the failure it
describes stopped being a code reading yesterday. I hit it on hardware
while testing Jiaxing's v7 RK3576 series on RK3588, on a tree that did
not have this series applied.

The sequence is the one 2/2 predicts: unbind core 0 while cores 1 and 2
stay bound, then bind it back. What happens without these patches, on
an Orange Pi 5 Plus with all three NPU cores:

  - the returning core takes the index of a core that is still live.
    The driver prints

      rocket fdab0000.npu: Rockchip NPU core 2 version: 1179210309

    for the device that is physically core 0;

  - inference stops being correct. The same MobileNet V1 input that
    classified as "military uniform" before the rebind classifies as
    "toilet tissue" after it, and the oracle hashes change with it;

  - throughput falls from 88.6 to 1.9 inferences per second, and core 0
    stops taking interrupts entirely - 0.00 per inference where it had
    been taking 41.74 - while core 1 absorbs everything;

  - the next unbind then dies. Ten "NPU job timed out" in a row,
    followed by

      Unable to handle kernel paging request at virtual address
      dead000000000122
      pc : destroy_workqueue+0x1b8/0x3e0
      Call trace:
       destroy_workqueue+0x1b8/0x3e0
       drm_sched_fini+0x178/0x1a8 [gpu_sched]
       rocket_job_fini+0x28/0x60 [rocket]
       rocket_core_fini+0x4c/0x78 [rocket]
       rocket_remove+0x6c/0x150 [rocket]
       ... unbind_store

    That is LIST_POISON2 being dereferenced. The sysfs write never
    returns, the task is in uninterruptible sleep and cannot be killed,
    and the machine needs a reboot.

That is the second bullet of 2/2's commit message happening: the bind
reuses the index of a still-live core and overwrites its slot while its
IRQ handler and its DRM scheduler are still active.

With the two patches applied and nothing else changed, the same
sequence - core 2 out and back, core 0 out and back, all three out and
all three back - runs clean. Twelve inference runs across two modules,
one oracle hash for all of them, correct classification every time, and
nothing in dmesg beyond the probe messages.

I am happy to resend with the tag collected if that is easier. There
is also a practical reason to have it in: the three-core RK3588 test
that Jiaxing asked for on his v7 cannot run to completion on a tree
without this, because the core-0 rebind step is what trips it.

Thanks,
Igor

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip