Kernel crash during USB device enumeration or MSC mounting

Kernel crash during USB device enumeration or MSC mounting

[Dan Halbert]

Hi -- I was referred to this list by Kai-Heng Feng after filing an issue 
on the Ubuntu bug tracker here:

   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1871143.

Many further details are in that report, including "lsusb -v", "lspci 
-v", dmesg, and call trace output.

We are seeing kernel crashes from certain composite USB devices that 
present as MSC, CDC, HID, and MIDI. The devices are microcontroller 
boards running CircuitPython 
(https://github.com/adafruit/circuitpython), which uses the TinyUSB USB 
stack: https://github.com/hathach/tinyusb.

The board is commanded to forcibly reformat its MSC FAT12 filesystem and 
then reset. This causes a USB reset and re-enumeration, which can often 
(though not all the time) cause a kernel crash.

The crashes may be some kind of use-after-free bug in the kernel USB driver.

We have seen this or similar crashes caused by several different boards 
that use different microcontrollers. The hosts are x64, including Dell 
Intel-chipset desktops and an older Mac Mini running Linux (also Intel 
chipset).

The ports in use can be either USB2 or USB3. Sometimes interposing a 
USB2 hub helps.

The boards and underlying software are from Adafruit. I work for 
Adafruit on CircuitPython, and the maintainer of TinyUSB also works for 
Adafrui, so we can provide further help to track this down. We can 
provide Beagle 12 USB traces if that would be helpful.

I have reproduced these crashes using kernels as far back as 4.10 and up 
to 5.7, so it is a long-standing issue.

We're very happy to provide further information. See the Launchpad bug 
report for more details.

Dan

Re: Kernel crash during USB device enumeration or MSC mounting

[Greg KH]

On Sat, Jul 04, 2020 at 11:15:29AM -0400, Dan Halbert wrote:
> Hi -- I was referred to this list by Kai-Heng Feng after filing an issue on
> the Ubuntu bug tracker here:
> 
>   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1871143.
> 
> Many further details are in that report, including "lsusb -v", "lspci -v",
> dmesg, and call trace output.

Where is the kernel crash in those logs?  I don't seem to be able to
find the correct one, can you just send it here so we can read and
comment on it in the email thread?  I see something odd with the fat
driver, are you sure you are creating a correct filesystem image for it
to handle?

> We are seeing kernel crashes from certain composite USB devices that present
> as MSC, CDC, HID, and MIDI. The devices are microcontroller boards running
> CircuitPython (https://github.com/adafruit/circuitpython), which uses the
> TinyUSB USB stack: https://github.com/hathach/tinyusb.
> 
> The board is commanded to forcibly reformat its MSC FAT12 filesystem and
> then reset. This causes a USB reset and re-enumeration, which can often
> (though not all the time) cause a kernel crash.

What do you mean by "reset"?  Are you disconnecting the device from the
USB bus causing it to electronically be removed and then added back?  Or
doing something else?

> The crashes may be some kind of use-after-free bug in the kernel USB driver.

Maybe, but the traces seem _very_ short on the stuff on the web page,
can you send us the full traceback from the crash?

> We have seen this or similar crashes caused by several different boards that
> use different microcontrollers. The hosts are x64, including Dell
> Intel-chipset desktops and an older Mac Mini running Linux (also Intel
> chipset).

That's not good, but might mean it is a filesystem issue, not a USB
issue.

thanks,

greg k-h

Re: Kernel crash during USB device enumeration or MSC mounting

[Greg KH]

On Sat, Jul 04, 2020 at 05:34:29PM +0200, Greg KH wrote:
> On Sat, Jul 04, 2020 at 11:15:29AM -0400, Dan Halbert wrote:
> > Hi -- I was referred to this list by Kai-Heng Feng after filing an issue on
> > the Ubuntu bug tracker here:
> > 
> >   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1871143.
> > 
> > Many further details are in that report, including "lsusb -v", "lspci -v",
> > dmesg, and call trace output.
> 
> Where is the kernel crash in those logs?  I don't seem to be able to
> find the correct one, can you just send it here so we can read and
> comment on it in the email thread?  I see something odd with the fat
> driver, are you sure you are creating a correct filesystem image for it
> to handle?
> 
> > We are seeing kernel crashes from certain composite USB devices that present
> > as MSC, CDC, HID, and MIDI. The devices are microcontroller boards running
> > CircuitPython (https://github.com/adafruit/circuitpython), which uses the
> > TinyUSB USB stack: https://github.com/hathach/tinyusb.
> > 
> > The board is commanded to forcibly reformat its MSC FAT12 filesystem and
> > then reset. This causes a USB reset and re-enumeration, which can often
> > (though not all the time) cause a kernel crash.
> 
> What do you mean by "reset"?  Are you disconnecting the device from the
> USB bus causing it to electronically be removed and then added back?  Or
> doing something else?
> 
> > The crashes may be some kind of use-after-free bug in the kernel USB driver.
> 
> Maybe, but the traces seem _very_ short on the stuff on the web page,
> can you send us the full traceback from the crash?
> 
> > We have seen this or similar crashes caused by several different boards that
> > use different microcontrollers. The hosts are x64, including Dell
> > Intel-chipset desktops and an older Mac Mini running Linux (also Intel
> > chipset).
> 
> That's not good, but might mean it is a filesystem issue, not a USB
> issue.

Ok, this trace:
	https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1871143/comments/4

seems to point at the filesystem code.  Are you sure you are creating
the image correctly?

If you unplug and plug the device back in with that same filesystem
image, does it work, or still crash?

thanks,

greg k-h

Re: Kernel crash during USB device enumeration or MSC mounting

[Dan Halbert]

On 7/4/20 11:36 AM, Greg KH wrote

Thanks for your quick reply, Greg.

> Where is the kernel crash in those logs?  I don't seem to be able to
> find the correct one, can you just send it here so we can read and
> comment on it in the email thread?  I see something odd with the fat
> driver, are you sure you are creating a correct filesystem image for it
> to handle?

There are two crashes described in that report. In the first post, I 
mentioned that the full kdump crash dump is here:
https://drive.google.com/open?id=1ka3ySSccEMRSmnYgI9X16IndPDmIKTD6
Is that what you are looking for?

The first trace seems USB-related, and is in comment #2 (you have to 
click Read More...),
and starts out:

[76707.692760] Call Trace:
[76707.692767] ? usb_alloc_urb+0x29/0x60
[76707.692772] usb_alloc_urb+0x29/0x60
[76707.692779] acm_probe+0x6da/0xdb0 [cdc_acm]
...

The second trace is in comment #4, and seems filesystem-related:

[40866.349358] Call Trace:
[40866.349363] ? kernfs_fop_open+0xc2/0x3b0
[40866.349366] kernfs_fop_open+0xc2/0x3b0
[40866.349368] ? kernfs_fop_read+0x1b0/0x1b0
[40866.349372] do_dentry_open+0x143/0x3a0
[40866.349373] vfs_open+0x2d/0x30
[40866.349376] do_last+0x194/0x900
[40866.349392] path_openat+0x8d/0x290
...

> What do you mean by "reset"?  Are you disconnecting the device from the
> USB bus causing it to electronically be removed and then added back?  Or
> doing something else?

The microcontroller is doing a hard reset, which causes an electrical 
USB reset, if I understand correctly. We have also occasionally seen 
crashes on plug-in (and maybe unplug, though I have not seen that 
personally).

> ... the traces seem _very_ short on the stuff on the web page,
> can you send us the full traceback from the crash?

I have the kdump stuff, but I'm not sure how to give you a full 
traceback. Do you want the whole contents of the /var/crash dirs?

>That's not good, but might mean it is a filesystem issue, not a USB
>issue.

Yes, possibly, though it might be an interaction between the filesystem 
code and USB, since I'm seeing different traces. We have not seen these 
crashes on some other motherboards, including some Intel laptop chipsets 
and some AMD chipsets.

> Ok, this trace:
> 	https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1871143/comments/4
> 
> seems to point at the filesystem code.  Are you sure you are creating
> the image correctly?
> 
> If you unplug and plug the device back in with that same filesystem
> image, does it work, or still crash?

Yes, it works fine. It might be something about the abruptness of the 
disconnect and the speed with which it tries to reconnect. It is 
intermittent, and doesn't always cause a crash.

We've been using these devices and their filesystems for years 
successfully for several years. That doesn't mean the filesystem code 
on-board is perfect, but we shouldn't be able to crash the kernel.

Thanks,
Dan

Re: Kernel crash during USB device enumeration or MSC mounting

[Alan Stern]

On Sat, Jul 04, 2020 at 05:36:07PM +0200, Greg KH wrote:
> On Sat, Jul 04, 2020 at 05:34:29PM +0200, Greg KH wrote:
> > On Sat, Jul 04, 2020 at 11:15:29AM -0400, Dan Halbert wrote:
> > > Hi -- I was referred to this list by Kai-Heng Feng after filing an issue on
> > > the Ubuntu bug tracker here:
> > > 
> > >   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1871143.
> > > 
> > > Many further details are in that report, including "lsusb -v", "lspci -v",
> > > dmesg, and call trace output.
> > 
> > Where is the kernel crash in those logs?  I don't seem to be able to
> > find the correct one, can you just send it here so we can read and
> > comment on it in the email thread?  I see something odd with the fat
> > driver, are you sure you are creating a correct filesystem image for it
> > to handle?
> > 
> > > We are seeing kernel crashes from certain composite USB devices that present
> > > as MSC, CDC, HID, and MIDI. The devices are microcontroller boards running
> > > CircuitPython (https://github.com/adafruit/circuitpython), which uses the
> > > TinyUSB USB stack: https://github.com/hathach/tinyusb.
> > > 
> > > The board is commanded to forcibly reformat its MSC FAT12 filesystem and
> > > then reset. This causes a USB reset and re-enumeration, which can often
> > > (though not all the time) cause a kernel crash.
> > 
> > What do you mean by "reset"?  Are you disconnecting the device from the
> > USB bus causing it to electronically be removed and then added back?  Or
> > doing something else?
> > 
> > > The crashes may be some kind of use-after-free bug in the kernel USB driver.
> > 
> > Maybe, but the traces seem _very_ short on the stuff on the web page,
> > can you send us the full traceback from the crash?
> > 
> > > We have seen this or similar crashes caused by several different boards that
> > > use different microcontrollers. The hosts are x64, including Dell
> > > Intel-chipset desktops and an older Mac Mini running Linux (also Intel
> > > chipset).
> > 
> > That's not good, but might mean it is a filesystem issue, not a USB
> > issue.
> 
> Ok, this trace:
> 	https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1871143/comments/4
> 
> seems to point at the filesystem code.  Are you sure you are creating
> the image correctly?
> 
> If you unplug and plug the device back in with that same filesystem
> image, does it work, or still crash?

It looks to me as though the user is reformatting the USB drive on the 
microcontroller while it is mounted on the host.  At least, the log 
message:

[40864.285807] FAT-fs (sdc1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

seems to indicate that a FAT filesystem was mounted, and the circuitpython 
command storage.erase_filesystem() completely reinitializes the device's 
filesystem.

This is definitely a user error: It is forbidden for a device to 
manipulate data that it is exporting to a host as a USB drive.  And it's 
not surprising that doing so would cause the host to crash.

If you really want to call storage.erase_filesystem() while the device is 
connected to the host, you should at least unmount the drive on the host 
beforehand.

Alan Stern

Re: Kernel crash during USB device enumeration or MSC mounting

[Dan Halbert]

On 7/4/20 12:09 PM, Alan Stern wrote:
> It looks to me as though the user is reformatting the USB drive on the
> microcontroller while it is mounted on the host.  At least, the log
> message:
> 
> [40864.285807] FAT-fs (sdc1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
> 
> seems to indicate that a FAT filesystem was mounted, and the circuitpython
> command storage.erase_filesystem() completely reinitializes the device's
> filesystem.
> 
> This is definitely a user error: It is forbidden for a device to
> manipulate data that it is exporting to a host as a USB drive.  And it's
> not surprising that doing so would cause the host to crash.
> 
> If you really want to call storage.erase_filesystem() while the device is
> connected to the host, you should at least unmount the drive on the host
> beforehand.

Thanks for your observations, which are very helpful in my thinking 
about this. That's correct, that code does reformat the filesystem, but 
it also does a USB reset, so the host should ideally not make an 
assumption that its idea of what is on the drive is correct when it 
re-enumerates.

We cannot tell the host from the MSC side that we want to unmount 
cleanly from the drive. We could disconnect from USB first.

I just confirmed with one of our users that he has also seen crashes on 
simple unplug of the mounted device.

I would hope that the host would not crash when the MSC device does 
something untoward, and that it would be more robust. It is a potential 
kernel attack mechanism otherwise.

The first crash trace I mentioned in my reply to Greg does seem to be 
inside the USB stack, not in the filesystem code. It's possible there 
are two problems here, or it's possible the filesystem code gets 
confused and is making the USB stack confused as well.

Dan

Re: Kernel crash during USB device enumeration or MSC mounting

[Alan Stern]

On Sat, Jul 04, 2020 at 12:33:38PM -0400, Dan Halbert wrote:
> On 7/4/20 12:09 PM, Alan Stern wrote:
> > It looks to me as though the user is reformatting the USB drive on the
> > microcontroller while it is mounted on the host.  At least, the log
> > message:
> > 
> > [40864.285807] FAT-fs (sdc1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
> > 
> > seems to indicate that a FAT filesystem was mounted, and the circuitpython
> > command storage.erase_filesystem() completely reinitializes the device's
> > filesystem.
> > 
> > This is definitely a user error: It is forbidden for a device to
> > manipulate data that it is exporting to a host as a USB drive.  And it's
> > not surprising that doing so would cause the host to crash.
> > 
> > If you really want to call storage.erase_filesystem() while the device is
> > connected to the host, you should at least unmount the drive on the host
> > beforehand.
> 
> Thanks for your observations, which are very helpful in my thinking about
> this. That's correct, that code does reformat the filesystem, but it also
> does a USB reset, so the host should ideally not make an assumption that its
> idea of what is on the drive is correct when it re-enumerates.

The host sees the reformat as it takes place, and that can mess up the 
host.  The fact that there is a disconnect afterward limits the window 
but does not eliminate it.

> We cannot tell the host from the MSC side that we want to unmount cleanly
> from the drive. We could disconnect from USB first.

That would be a good idea.

Still, what reason is there for reformatting the storage while it is 
connected to a host?  That's equivalent to a real disk drive deciding to 
erase itself while it is in use.  Why not do the reformat before the 
device is attached to the host?

Or why not have the host do the reformat instead of doing it on the 
device?

> I just confirmed with one of our users that he has also seen crashes on
> simple unplug of the mounted device.

That could be a separate issue.  I haven't seen reports of things like 
that for quite a long time.  Is it reproducible?  If it is, you (or your 
user) ought to be able to get the same result by unplugging an ordinary 
USB flash drive while it is mounted.

> I would hope that the host would not crash when the MSC device does
> something untoward, and that it would be more robust. It is a potential
> kernel attack mechanism otherwise.

Have you heard of Bad USB?  Yes, there are a lot of attack mechanisms 
here.  The fact is, operating systems tend to trust the contents of 
attached disk drives.  If you don't want the host to trust the contents of 
the device's Mass Storage interface, don't allow the host to mount it.

One of the most fundamental assumptions computers make about attached disk 
drives is that they don't spontaneously change their contents.  When you 
violate that assumption, almost anything can happen.

> The first crash trace I mentioned in my reply to Greg does seem to be inside
> the USB stack, not in the filesystem code. It's possible there are two
> problems here, or it's possible the filesystem code gets confused and is
> making the USB stack confused as well.

Whenever one subsystem in the kernel gets out of whack, it can fairly 
easily corrupt the entire kernel.  You can't conclude anything just from 
the immediate appearance of a bug.

Anyway, the first crash in in your reply to Greg _wasn't_ in the USB 
stack:

[76707.692717] general protection fault: 0000 [#1] SMP PTI
[76707.692723] CPU: 3 PID: 75883 Comm: kworker/3:1 Kdump: loaded Not tainted 5.4.0-21-generic #25-Ubuntu
[76707.692725] Hardware name: Dell Inc. OptiPlex 7010/0WR7PY, BIOS A29 06/28/2018
[76707.692732] Workqueue: usb_hub_wq hub_event
[76707.692738] RIP: 0010:__kmalloc+0xa5/0x270

As you can see, the problem occurred in __kmalloc(), which is part of the 
memory management subsystem -- not the USB stack.

Alan Stern