MUSB interrupt storm on device removal

["Måns Rullgård" <mans@mansr.com>]

Bin Liu <b-liu@ti.com> writes:

> On Fri, Jan 18, 2019 at 08:15:02PM +0000, Måns Rullgård wrote:
>> Bin Liu <b-liu@ti.com> writes:
>> 
>> > On Mon, Dec 17, 2018 at 09:36:17PM +0000, Måns Rullgård wrote:
>> >> Bin Liu <b-liu@ti.com> writes:
>> >> 
>> >> > On Mon, Dec 17, 2018 at 07:16:08PM +0000, Måns Rullgård wrote:
>> >> >> Bin Liu <b-liu@ti.com> writes:
>> >> >> 
>> >> >> > Hi,
>> >> >> >
>> >> >> > On Mon, Dec 17, 2018 at 03:13:12PM +0000, Måns Rullgård wrote:
>> >> >> >> I have a strange problem with the musb driver in host mode on AM3358
>> >> >> >> (beaglebone) hardware.  If I connect a multi-port serial adapter and
>
> Is this on beaglebone or beagleboneblack?

We've seen it with Beaglebone Enhanced [1] and BeagleCore [2] based
devices.  Both are based on the AM3358 and mostly compatible with the
Beaglebone Black.

[1] https://elinux.org/SanCloud_BeagleBoneEnhanced
[2] http://beaglecore.com/products/bcm1str.html

>> >> >> >> open two or more of the ttys, then unplug the device, an interrupt storm
>> >> >> >> ensues that makes the system completely unresponsive until the watchdog
>> >> >> >> resets it.  Enabling some debug messages, I get this repeated endlessly:
>> >> >> >> 
>> >> >> >> musb-hdrc musb-hdrc.1: usbintr (0) epintr(1c0000)
>> >> >> >> musb-hdrc musb-hdrc.1: end 2 RX proto error
>> >> >> >> musb-hdrc musb-hdrc.1: ... next ep2 RX urb 72aabc13
>> >> >> >> musb-hdrc musb-hdrc.1: <-- hw2 urb 72aabc13 spd3 dev4 ep3in h_addr02 h_port00 bytes 4096
>> >> >> >> musb-hdrc musb-hdrc.1: RXCSR2 := 2020
>> >> >> >> musb-hdrc musb-hdrc.1: end 3 RX proto error
>> >> >> >> musb-hdrc musb-hdrc.1: ... next ep3 RX urb 94c2cc43
>> >> >> >> musb-hdrc musb-hdrc.1: <-- hw3 urb 94c2cc43 spd3 dev4 ep2in h_addr02 h_port00 bytes 4096
>> >> >> >> musb-hdrc musb-hdrc.1: RXCSR3 := 2020
>> >> >> >> musb-hdrc musb-hdrc.1: end 4 RX proto error
>> >> >> >> musb-hdrc musb-hdrc.1: ... next ep4 RX urb cbdc39c6
>> >> >> >> musb-hdrc musb-hdrc.1: <-- hw4 urb cbdc39c6 spd3 dev4 ep5in h_addr02 h_port00 bytes 4096
>> >> >> >> musb-hdrc musb-hdrc.1: RXCSR4 := 2020
>> >> >> >> 
>> >> >> >> This happens with both a two-port FTDI serial adapter and a Simcom GSM
>> >> >> >> modem (Qualcomm based) using the "option" driver.  In both cases, the
>> >> >> >> problem occurs only if two or more of the ttys are opened when the
>> >> >> >> device is unplugged.
>> >> >> >
>> >> >> > Please help me to understand the test case so that I can try to
>> >> >> > replicate the issue - if I connect a multi-port FTDI adapter, use cat
>> >> >> > command to open more than one port then unplug the adapter, I should
>> >> >> > see the flooding debug messages?
>> >> >> 
>> >> >> Yes, that's what I'm seeing.  Obviously, you have to first enable those
>> >> >> debug messages.  Otherwise it just stops dead in its tracks.
>> >> >> 
>> >> >> I forgot to mention, I also tried making the interrupt handler threaded.
>> >> >> This allows the system to remain slightly responsive, but it never
>> >> >> recovers.
>> >> >> 
>> >> >> > On which kernel version do you see the problem?
>> >> >> 
>> >> >> This was on 4.19.9.  I don't see any later commits that look related,
>> >> >> but I can try some other tree if you suggest one.
>> >> >
>> >> > This should be fine. The driver was barely touched recently.
>> >> > I have only a few days left for 2018, let me try to reproduce it first
>> >> > after I am back in Jan.
>> >> 
>> >> No problem.  I suspect most of us will have some time off during the
>> >> holidays.
>> >> 
>> >> If it helps, we first noticed this with a 4.9 (stable) kernel, so it's
>> >> not a recent regression.
>> >
>> > When I tried to reproduce the issue today, something weird happened, the
>> > v4.19.9 kernel I compiled doesn't bring usb1 port on my BBB to host
>> > mode, otg state was stuck in a_idle, DevCtl register is 0x81. The latest
>
> I fixed the the bug which was in the am335x phy driver last week,
>
>> > v5.0-rc1 and somewhere in v4.16 do not have such problem. I tried git
>> > bisect, but it was screwed up somewhere need to start over. I will find
>> > some time for this issue.
>> >
>> > By the way, I don't see the interrupt storm issue with v5.0-rc1 on my
>> > BBB, when unplugged the FTDI 4-port device, the 'proto error' message
>> > only shows once for each port opened, then MUSB_DISCONNECT interrupt
>> > (0x20) happened and everything was torn down.
>> 
>> I tested this with v5.0-rc2.  Now the badness doesn't happen every time,
>> but two or three attempts is still enough to reproduce it.  Also, when
>> it does happen, the system no longer freezes completely.  It is now
>> possible to interrupt the process with the tty open, at which point the
>> kernel prints these messages:
>> 
>> ftdi_sio ttyUSB0: failed to get modem status: -71
>> ftdi_sio ttyUSB0: error from flowcontrol urb
>> 
>> Meanwhile, about 10k interrupts per second have been recorded.  The USB
>> device is still reported as connected, and the port remains unusable.
>
> but I still cannot see the interrupt storm problem in v4.19.9.
> Did you build the kernel with omap2plus_defconfig without any other
> changes?

No, we use a configuration suitable for our devices.  It shouldn't
differ in any way that matters though.