USB lockup on OMAP3530

USB lockup on OMAP3530

[Andreas Hartmetz]

Hi,

I have a problem with the USB OTG port on my Beagle Board revision B6 - the OTG
port is the only USB port on that device so it's critical that it works.
Everybody on this list probably knows this hardware, I'll just say that it has
an ARM Cortex-A8 CPU (OMAP3530) with a built-in USB interface. The driver is
musb_hdrc.
I'm using this interface in host mode (no gadget mode support compiled in,
though this doesn't seem to make a difference) to attach necessary peripherals
via a powered USB hub. The make and model of hub do not play a role, I've tried
several.

The problem:
Whenever the following conditions are met all hardware on the port "disappears"
after a few minutes, cutting off the board from network and storage:
- Network traffic over USB (doesn't matter if it's regular 100 Mbps Ethernet
  or a WLAN stick)
- Disk "traffic" over USB - I use a 2,5" disk in a USB enclosure
- CPU load - not sure if this is necessary
The only fix seems to be a reboot.

This happens to me when compiling something on the board (easier than
cross-compiling) with the help of Icecream, which is a kind of distcc on
steroids. On #beagle on Freenode IRC I've found somebody with the same problem
when scp-ing a large file (several hundred megabytes) off or onto the USB disk.
I can reproduce that as well.
I have spent a lot of time trying to find fixes and/or workarounds but nothing
worked so far except using the "validation kernel" recommended to test the
hardware e.g. after receipt:
http://code.google.com/p/beagleboard/wiki/BeagleBoardDiagnostics.
Fittingly, the link to the kernel image is currently broken. The board usually
locked itself up after about a day when using that ancient kernel, so it's not
an option either, and probably quite unmaintained and buggy in many other areas.

So far I have tried many versions of the linux-omap and linux-omap-pm kernel,
from about 2.6.30 to the latest git version. They all exhibit the USB OTG death
bug.
I've used kernels with openembedded patches and without, currently without.
Yesterday I discovered the musb_hdrc.fifo_mode parameter and played around
with it. I also modified the given configurations. Result:
- FIFO configurations including .mode = BUF_DOUBLE don't work at all - no
  devices work.
- the USB death bug is not fixed by:
  - using only one endpoint
  - using no TXRX entries but only separate RX and TX
    (every endpoint gets a TX and an RX entry though)
  - using a large number of endpoints with same maxpacket value
... still no solution.

Enabling debug output of the musb_hdrc driver (yes I've also compiled in debug
messages) is not very practical due to the high volume of messages; also, when
the bug occurs nothing special is printed. The first error usually comes from
the memory manager / filesystem complaining that it can't do "IO to offline
device", i.e. the disappeared external harddisk (which contains the swapfile).

I would *really* appreciate somebody looking into this because this currently
makes the hardware as useful as a brick for me. I can supply debug output and
test patches.

Cheers,
Andreas

Re: USB lockup on OMAP3530

[Robert Nelson]

On Tue, Jan 26, 2010 at 4:30 PM, Andreas Hartmetz <ahartmetz@gmail.com> wrote:
> Hi,
>
> I have a problem with the USB OTG port on my Beagle Board revision B6 - the OTG
> port is the only USB port on that device so it's critical that it works.
> Everybody on this list probably knows this hardware, I'll just say that it has
> an ARM Cortex-A8 CPU (OMAP3530) with a built-in USB interface. The driver is
> musb_hdrc.
> I'm using this interface in host mode (no gadget mode support compiled in,
> though this doesn't seem to make a difference) to attach necessary peripherals
> via a powered USB hub. The make and model of hub do not play a role, I've tried
> several.
>
> The problem:
> Whenever the following conditions are met all hardware on the port "disappears"
> after a few minutes, cutting off the board from network and storage:
> - Network traffic over USB (doesn't matter if it's regular 100 Mbps Ethernet
>  or a WLAN stick)
> - Disk "traffic" over USB - I use a 2,5" disk in a USB enclosure
> - CPU load - not sure if this is necessary
> The only fix seems to be a reboot.
>
> This happens to me when compiling something on the board (easier than
> cross-compiling) with the help of Icecream, which is a kind of distcc on
> steroids. On #beagle on Freenode IRC I've found somebody with the same problem
> when scp-ing a large file (several hundred megabytes) off or onto the USB disk.
> I can reproduce that as well.
> I have spent a lot of time trying to find fixes and/or workarounds but nothing
> worked so far except using the "validation kernel" recommended to test the
> hardware e.g. after receipt:
> http://code.google.com/p/beagleboard/wiki/BeagleBoardDiagnostics.
> Fittingly, the link to the kernel image is currently broken. The board usually
> locked itself up after about a day when using that ancient kernel, so it's not
> an option either, and probably quite unmaintained and buggy in many other areas.
>
> So far I have tried many versions of the linux-omap and linux-omap-pm kernel,
> from about 2.6.30 to the latest git version. They all exhibit the USB OTG death
> bug.
> I've used kernels with openembedded patches and without, currently without.
> Yesterday I discovered the musb_hdrc.fifo_mode parameter and played around
> with it. I also modified the given configurations. Result:
> - FIFO configurations including .mode = BUF_DOUBLE don't work at all - no
>  devices work.
> - the USB death bug is not fixed by:
>  - using only one endpoint
>  - using no TXRX entries but only separate RX and TX
>    (every endpoint gets a TX and an RX entry though)
>  - using a large number of endpoints with same maxpacket value
> ... still no solution.
>
> Enabling debug output of the musb_hdrc driver (yes I've also compiled in debug
> messages) is not very practical due to the high volume of messages; also, when
> the bug occurs nothing special is printed. The first error usually comes from
> the memory manager / filesystem complaining that it can't do "IO to offline
> device", i.e. the disappeared external harddisk (which contains the swapfile).
>
> I would *really* appreciate somebody looking into this because this currently
> makes the hardware as useful as a brick for me. I can supply debug output and
> test patches.
>
> Cheers,
> Andreas

Hi Andreas,

Give this patch a try:
http://bazaar.launchpad.net/~beagleboard-kernel/%2Bjunk/2.6-stable/annotate/head%3A/patches/musb/fifo-change.patch

It's in Angstrom's 2.6.29, my 2.6.31/32... And a real fix just went
into mainline 2.6.33-rc5, with it my otg port is solid as a rock and
can transfer large amounts of data between shared devices on the
otg/musb port...

http://rcn-ee.homeip.net:81/dl/farm/log/2.6.32.6-x6.0_beagle-128mb-0_musb-stress-test.txt

Regards,

-- 
Robert Nelson
http://www.rcn-ee.com/
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: USB lockup on OMAP3530

[Andreas Hartmetz]

On Wednesday 27 January 2010 00:13:48 Robert Nelson wrote:
> On Tue, Jan 26, 2010 at 4:30 PM, Andreas Hartmetz <ahartmetz@gmail.com> 
wrote:
> > Hi,
> > 
> > I have a problem with the USB OTG port on my Beagle Board revision B6 -
> > the OTG port is the only USB port on that device so it's critical that
> > it works. Everybody on this list probably knows this hardware, I'll just
> > say that it has an ARM Cortex-A8 CPU (OMAP3530) with a built-in USB
> > interface. The driver is musb_hdrc.
> > I'm using this interface in host mode (no gadget mode support compiled
> > in, though this doesn't seem to make a difference) to attach necessary
> > peripherals via a powered USB hub. The make and model of hub do not play
> > a role, I've tried several.
> > 
> > The problem:
> > Whenever the following conditions are met all hardware on the port
> > "disappears" after a few minutes, cutting off the board from network and
> > storage: - Network traffic over USB (doesn't matter if it's regular 100
> > Mbps Ethernet or a WLAN stick)
> > - Disk "traffic" over USB - I use a 2,5" disk in a USB enclosure
> > - CPU load - not sure if this is necessary
> > The only fix seems to be a reboot.
> > 
> > This happens to me when compiling something on the board (easier than
> > cross-compiling) with the help of Icecream, which is a kind of distcc on
> > steroids. On #beagle on Freenode IRC I've found somebody with the same
> > problem when scp-ing a large file (several hundred megabytes) off or
> > onto the USB disk. I can reproduce that as well.
> > I have spent a lot of time trying to find fixes and/or workarounds but
> > nothing worked so far except using the "validation kernel" recommended
> > to test the hardware e.g. after receipt:
> > http://code.google.com/p/beagleboard/wiki/BeagleBoardDiagnostics.
> > Fittingly, the link to the kernel image is currently broken. The board
> > usually locked itself up after about a day when using that ancient
> > kernel, so it's not an option either, and probably quite unmaintained
> > and buggy in many other areas.
> > 
> > So far I have tried many versions of the linux-omap and linux-omap-pm
> > kernel, from about 2.6.30 to the latest git version. They all exhibit
> > the USB OTG death bug.
> > I've used kernels with openembedded patches and without, currently
> > without. Yesterday I discovered the musb_hdrc.fifo_mode parameter and
> > played around with it. I also modified the given configurations. Result:
> > - FIFO configurations including .mode = BUF_DOUBLE don't work at all - no
> >  devices work.
> > - the USB death bug is not fixed by:
> >  - using only one endpoint
> >  - using no TXRX entries but only separate RX and TX
> >    (every endpoint gets a TX and an RX entry though)
> >  - using a large number of endpoints with same maxpacket value
> > ... still no solution.
> > 
> > Enabling debug output of the musb_hdrc driver (yes I've also compiled in
> > debug messages) is not very practical due to the high volume of
> > messages; also, when the bug occurs nothing special is printed. The
> > first error usually comes from the memory manager / filesystem
> > complaining that it can't do "IO to offline device", i.e. the
> > disappeared external harddisk (which contains the swapfile).
> > 
> > I would *really* appreciate somebody looking into this because this
> > currently makes the hardware as useful as a brick for me. I can supply
> > debug output and test patches.
> > 
> > Cheers,
> > Andreas
> 
> Hi Andreas,
> 
> Give this patch a try:
> http://bazaar.launchpad.net/~beagleboard-kernel/%2Bjunk/2.6-stable/annotate
> /head%3A/patches/musb/fifo-change.patch
> 
> It's in Angstrom's 2.6.29, my 2.6.31/32... And a real fix just went
> into mainline 2.6.33-rc5, with it my otg port is solid as a rock and
> can transfer large amounts of data between shared devices on the
> otg/musb port...
> 
> http://rcn-ee.homeip.net:81/dl/farm/log/2.6.32.6-x6.0_beagle-128mb-0_musb-s
> tress-test.txt
> 
> Regards,

Nice try, but no success.

I've tried the fifo-change patch against the pm branch (with 
musb_hdrc.fifo_mode=4 obviously, otherwise it wouldn't do anything) and also 
the 2.6.32.6-x6.0 kernel from you. Nothing is fixed for me. With your kernel I 
have an additional problem I don't have with pm: it takes several tries to 
bring up the USB WLAN interface to a working state, with the rt73usb driver.
Basically I start and kill wpa_supplicant, unload the module a few times and 
whatnot until it works. DHCP seems to work almost every time but no IP or ICMP 
packets come through. I know it sounds strange.
When it works ping response times range from one to ten seconds (?!) whereas 
they are in the several milliseconds range with pm.
I tried your latest kernel with and without musb_hdrc.fifo_mode=4 to be sure.
Which fix in mainline do you mean btw?
As long as no particular commit fixes the bug I consider it fixed by accident, 
and it may break again by accident.
I suspect that your stress test is not stressful enough. Try scp or anything 
that loads the CPU heavily while network and disk I/O is going on.
Maybe throw in some swapping with swap on the external disk for good measure, 
git gc seems to work well for that.
If all else fails I can publish a filesystem image with a distributed build 
ready to run.

Andreas

RE: USB lockup on OMAP3530

[Gadiyar, Anand]

<snip>

> > > 
> > > So far I have tried many versions of the linux-omap and linux-omap-pm
> > > kernel, from about 2.6.30 to the latest git version. They all exhibit
> > > the USB OTG death bug.
> > > I've used kernels with openembedded patches and without, currently
> > > without. Yesterday I discovered the musb_hdrc.fifo_mode parameter and
> > > played around with it. I also modified the given configurations. Result:
> > > - FIFO configurations including .mode = BUF_DOUBLE don't work at all - no
> > >  devices work.
> > > - the USB death bug is not fixed by:
> > >  - using only one endpoint
> > >  - using no TXRX entries but only separate RX and TX
> > >    (every endpoint gets a TX and an RX entry though)
> > >  - using a large number of endpoints with same maxpacket value
> > > ... still no solution.
> > > 
> > > Enabling debug output of the musb_hdrc driver (yes I've also compiled in
> > > debug messages) is not very practical due to the high volume of
> > > messages; also, when the bug occurs nothing special is printed. The
> > > first error usually comes from the memory manager / filesystem
> > > complaining that it can't do "IO to offline device", i.e. the
> > > disappeared external harddisk (which contains the swapfile).
> > > 
> > > I would *really* appreciate somebody looking into this because this
> > > currently makes the hardware as useful as a brick for me. I can supply
> > > debug output and test patches.

Would you care to try these two patches please?

http://marc.info/?l=linux-usb&m=125957336716915&w=2

And

The dependency patch:
http://marc.info/?l=linux-usb&m=125879861004131&w=2


Let me know if these fix the bug for you.

- Anand

Re: USB lockup on OMAP3530

[Michael Trimarchi]

Hi

Gadiyar, Anand wrote:
> <snip>
> 
>>>> So far I have tried many versions of the linux-omap and linux-omap-pm
>>>> kernel, from about 2.6.30 to the latest git version. They all exhibit
>>>> the USB OTG death bug.
>>>> I've used kernels with openembedded patches and without, currently
>>>> without. Yesterday I discovered the musb_hdrc.fifo_mode parameter and
>>>> played around with it. I also modified the given configurations. Result:
>>>> - FIFO configurations including .mode = BUF_DOUBLE don't work at all - no
>>>>  devices work.
>>>> - the USB death bug is not fixed by:
>>>>  - using only one endpoint
>>>>  - using no TXRX entries but only separate RX and TX
>>>>    (every endpoint gets a TX and an RX entry though)
>>>>  - using a large number of endpoints with same maxpacket value
>>>> ... still no solution.
>>>>
>>>> Enabling debug output of the musb_hdrc driver (yes I've also compiled in
>>>> debug messages) is not very practical due to the high volume of
>>>> messages; also, when the bug occurs nothing special is printed. The
>>>> first error usually comes from the memory manager / filesystem
>>>> complaining that it can't do "IO to offline device", i.e. the
>>>> disappeared external harddisk (which contains the swapfile).
>>>>
>>>> I would *really* appreciate somebody looking into this because this
>>>> currently makes the hardware as useful as a brick for me. I can supply
>>>> debug output and test patches.
> 
> Would you care to try these two patches please?
> 
> http://marc.info/?l=linux-usb&m=125957336716915&w=2
> 
> And
> 
> The dependency patch:
> http://marc.info/?l=linux-usb&m=125879861004131&w=2
> 
> 
> Let me know if these fix the bug for you.
> 

Use an hub with external power and connect the hub to the OTG and the device to the HUB.
Is it the same?


> - Anand
> --
> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: USB lockup on OMAP3530

[Andreas Hartmetz]

On Thu, Feb 11, 2010 at 7:57 PM, Michael Trimarchi
<michael@panicking.kicks-ass.org> wrote:
>
> Use an hub with external power and connect the hub to the OTG and the device
> to the HUB.
> Is it the same?
>
Uhm... what do you think how I attached a hard disk and a network
device to the Beagle's USB OTG port (ONE port) that can only supply
100 mA?
Of course I have everything connected to a powered hub that is
connected to the OTG port.
Note that my bug report is not about USB devices not working at all.

Re: USB lockup on OMAP3530

[Michael Trimarchi]

Hi

Andreas Hartmetz wrote:
> On Thu, Feb 11, 2010 at 7:57 PM, Michael Trimarchi
> <michael@panicking.kicks-ass.org> wrote:
>> Use an hub with external power and connect the hub to the OTG and the device
>> to the HUB.
>> Is it the same?
>>
> Uhm... what do you think how I attached a hard disk and a network
> device to the Beagle's USB OTG port (ONE port) that can only supply
> 100 mA?
> Of course I have everything connected to a powered hub that is
> connected to the OTG port.
> Note that my bug report is not about USB devices not working at all.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Sorry for that :), I haven't read all the message. But I had the same problem
using a camera with isoc transfer. BTW, I have the two patches applied and that
one from openembeeded doens't work.

Michael