Linux USB
 help / color / mirror / Atom feed
* H618 sunxi/musb driver (likely) looses interrupts -> g_mass_storage I/O errors/resets
@ 2026-09-03 18:55 te36de
  2026-09-04  7:56 ` Lovekesh Solanki
  0 siblings, 1 reply; 3+ messages in thread
From: te36de @ 2026-09-03 18:55 UTC (permalink / raw)
  To: Bin Liu, Greg Kroah-Hartman
  Cc: linux-sunxi, linux-usb, Samuel Holland, Jernej Skrabec

Summary:
  Likely problem in H618 sunxi/musb driver code leading to I/O errors,
resets on g_mass_storage.
  Produced on Orange Pi Zero 3 (OPZ3, H618 based), Linux
6.18.48-current-sunxi64:
    reproduced also with 7.1.12-edge-sunxi64 (problem not fixed in 7).
  set up g_mass_storage on sparse file (no actual I/O required reading
from it), read from host:
    Works reliable for minutes (14...22MByte/sec) when OPZ3 is otherwise idle.
    Creates repeatedly intermitting I/O errors and resets on USB host as soon
    as there is even just some process-level only CPU load on H618
(stress-ng on 2 kernel).
  Problem does not occur on RPI4 using dwc2 (instead of sunxi/musb)
with same tests
  Diagnosis seems to point to lost interrupts.

1. Reproduce by:

1.1 Connect gadget device (Orange PI zero 3 1GB in my case) USB-C port to host.

Note: Given how this powers the gadget device, it may not be
sufficient for gadget devices needing
a lot more current, but OPZ3 stays well below 500 mA under all conditions.

1.2 On gadget device (Orange PI zero 3 1GB):

truncate -s 12T /tmp/empty.img
modprobe g_mass_storage
file=/disk/S2/Shelf.UHD/TMP/avatar/AVATAR_BONUS1.iso stall=0 ro=1
removable=1

1.3. On host device

Determine device name for gadget, in this case /dev/sdb:

[129005.654791] usb 1-1: new high-speed USB device number 9 using xhci_hcd
[129005.785710] usb 1-1: New USB device found, idVendor=0525,
idProduct=a4a5, bcdDevice= 6.18
[129005.785730] usb 1-1: New USB device strings: Mfr=3, Product=4,
SerialNumber=0
[129005.785736] usb 1-1: Product: Mass Storage Gadget
[129005.785740] usb 1-1: Manufacturer: Armbian Linux
6.18.48-current-sunxi64 with musb-hdrc
[129005.787697] usb-storage 1-1:1.0: USB Mass Storage device detected
[129005.788090] usb-storage 1-1:1.0: Quirks match for vid 0525 pid a4a5: 10000
[129005.788244] scsi host2: usb-storage 1-1:1.0
[129006.798208] scsi 2:0:0:0: Direct-Access     Linux    File-Stor
Gadget 0618 PQ: 0 ANSI: 2
[129006.798370] sd 2:0:0:0: Attached scsi generic sg1 type 0
[129006.798560] sd 2:0:0:0: Power-on or device reset occurred
[129006.799139] sd 2:0:0:0: [sdb] Very big device. Trying to use READ
CAPACITY(16).
[129006.799905] sd 2:0:0:0: [sdb] 25769803776 512-byte logical blocks:
(13.2 TB/12.0 TiB)
[129006.800049] sd 2:0:0:0: [sdb] Write Protect is on
[129006.800053] sd 2:0:0:0: [sdb] Mode Sense: 0f 00 80 00
[129006.800181] sd 2:0:0:0: [sdb] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[129006.801447] sd 2:0:0:0: [sdb] Very big device. Trying to use READ
CAPACITY(16).
[129006.821771] sd 2:0:0:0: [sdb] Attached SCSI removable disk

Then start program to read from the disk:

pv -ptera < /dev/sdb > /dev/zero

And in another window check for I/O errors:

dmesg -w

1.3.1 "empty" run:

With nothing else running on the gadget, this should run reliable
without any I/O errors. Throughput
should be beteween 15...22 MByte/sec depending on block size of host
(due to the stop&go nature of BOT
i think).

1.3.2 error run:

In addition to 3.1, start some CPU activity on the gadget, e.g.:

stress-ng --cpu 4 --timeout 6000s

Now observe that after at most a few minutes, I/O error(s) and resets
will happen to the
gadget device, visible on the host, e.g.:

[131887.406689] I/O error, dev sdb, sector 4682752 op 0x0:(READ) flags
0x84700 phys_seg 2 prio class 0
[131887.406725] sd 2:0:0:0: [sdb] tag#0 device offline or changed
[131887.406731] I/O error, dev sdb, sector 4682992 op 0x0:(READ) flags
0x84700 phys_seg 2 prio class 0
[131887.406810] sd 2:0:0:0: [sdb] tag#0 device offline or changed
[131887.406828] I/O error, dev sdb, sector 4683232 op 0x0:(READ) flags
0x80700 phys_seg 1 prio class 0
[131887.416234] sd 2:0:0:0: [sdb] Very big device. Trying to use READ
CAPACITY(16).
[131974.304607] usb 1-1: reset high-speed USB device number 9 using xhci_hcd
[131974.435735] sd 2:0:0:0: [sdb] tag#0 FAILED Result:
hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=31s
[131974.435754] sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00
00 00 60 51 e0 00 00 00 20 00 00
[131974.435759] I/O error, dev sdb, sector 6312416 op 0x0:(READ) flags
0x80700 phys_seg 1 prio class 0
[131974.436075] sd 2:0:0:0: Power-on or device reset occurred
[131974.436097] sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_OK cmd_age=0s
[131974.436105] sd 2:0:0:0: [sdb] tag#0 Sense Key : Unit Attention [current]
[131974.436112] sd 2:0:0:0: [sdb] tag#0 Add. Sense: Power on, reset,
or bus device reset occurred
[131974.436119] sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00
00 00 60 52 00 00 00 00 f0 00 00
(host using larger block size for reads achieve lowe

On OPZ3, these errors happen already with as few as just 2 CPU cores
(out of 4) loaded.

1.4. Notes

- The test from 3. are just to eliminate as many possible sources of
error as possible.
  In the target application, the additional load comes from network
I/O, but i tried to
  simplify the reproduction of the error down to just process level
CPU being used.

- Problem does not happen on RPI4, which is why i think the problem is
not in g_mass_storage/f_mass_storage,
  but in the sunxi/musb driver - RPI4 useses instead dwc2 driver.

- The problem may be happening less often with loweer g_mass_storage
transfer rates,
  but for the target use-case, a speed of 12MByte/sec or more is
required, otherwise
  cheaper (but slower)  gadget devices than H618 would also become an option.

- Problem does not depend on size of file used, happens equally for
smaller file sizes,
  just using such a large file because of intended use-case.

- Lots of parameters that don't change the result when changed
  - "stall=0/1
  - musb_hdrc.use_dma=0
  - capping the host's transfer size (max_sectors_kb=16)
  - different idVendor/idProduct
  - moving the musb IRQ to another CPU via smp_affinity (IRQ 310 is
not shared with any other device)
  - powering the board from a separate supply rather than the host port

2. Troubleshooting

I had claude.ai vibe code me some tracingon the gadget. I can't claim
i understood it really
down to the details, but here is how claude.ai is summarizing it. Rest
of email below
is writeup from claude

Working - one SCSI READ(10), 16 KiB:

  file-storage   do_read() {
  file-storage     kernel_read()             0.03  ms   data from backing file
  file-storage     usb_ep_queue() {
  file-storage       musb_gadget_queue()     0.004 ms   UDC accepts request
  file-storage     }                         0.008 ms
  file-storage   }
  file-storage   sleep_thread() {                       thread waits
    <irq ctx>      sunxi_musb_interrupt() {
    <irq ctx>        musb_interrupt() {
    <irq ctx>          musb_g_tx()           0.010 ms   one packet sent
    <irq ctx>        }                       0.012 ms
    <irq ctx>      }                         0.014 ms
                   ... ~20 such interrupts, one per packet ...
    <irq ctx>      sunxi_musb_interrupt() {
    <irq ctx>        musb_interrupt() {
    <irq ctx>          musb_g_tx() {                    last packet
    <irq ctx>            bulk_in_complete()             request given back
    <irq ctx>          }                     0.010 ms
    <irq ctx>        }                       0.012 ms
    <irq ctx>      }                         0.015 ms
  file-storage   }                           3.8   ms   thread woken

Failing - identical up to the queue, then nothing:

  file-storage   usb_ep_queue() {
  file-storage     musb_gadget_queue()       0.005 ms   UDC accepts request
  file-storage   }                           0.008 ms
  file-storage   sleep_thread() {                       thread waits

                 ... ~30 s of complete silence ...

                 no sunxi_musb_interrupt, no musb_interrupt,
                 no musb_g_tx, no musb_g_rx, no bulk_in_complete

    [host]       SCSI command times out, I/O errors, port reset
    [gadget]     re-enumerates, resumes normally

  Kernel stack of the file-storage thread during the hang:

    [<0>] sleep_thread+0x10c/0x180 [usb_f_mass_storage]
    [<0>] fsg_main_thread+0x10c/0x16b0 [usb_f_mass_storage]
    [<0>] kthread+0x134/0x1f8
    [<0>] ret_from_fork+0x10/0x20

  task state S (interruptible sleep), i.e. it is not blocked on I/O.

  Event counts at the moment of the hang, and the delta over the 2 s
  after the thread parked:

    usb_ep_queue           650      +0
    musb_gadget_queue      650      +0
    sunxi_musb_interrupt 14605      +0
    musb_interrupt       14605      +0
    musb_g_tx            14519      +0
    musb_g_rx               86      +0
    bulk_in_complete       566      +0
    kernel_read            478      +0

What this appears to show
-------------------------

The request is accepted by musb_gadget_queue() and the transfer never
starts: not a completion lost at the end of a transfer that otherwise
ran, but no packet sent at all.

Notably sunxi_musb_interrupt() - the registered handler for the musb
IRQ - does not run even once during the hang.  So the failure is not the
driver mishandling an interrupt; no interrupt is delivered to the handler
at all.  I cannot tell from this whether the controller never raised its
line or whether the interrupt was lost or masked elsewhere.

Since the failure is triggered by CPU load alone, with a sparse backing
file and no I/O of any kind, this looks like a race between queueing a
request and the endpoint's interrupt or FIFO state, rather than anything
in f_mass_storage or the storage path.

^ permalink raw reply	[flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-09-04 15:20 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-03 18:55 H618 sunxi/musb driver (likely) looses interrupts -> g_mass_storage I/O errors/resets te36de
2026-09-04  7:56 ` Lovekesh Solanki
2026-09-04 15:19   ` te36de

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox