BUG when removing USB flash drive

BUG when removing USB flash drive

[Alberto Bertogli]

Hi!

I've just hit a bug after removing a Kingston USB flash drive.

I have removed it several (probably more than 40) times today without
problems until now. I'm removing it while doing streaming write()s on
the device (using /dev/sda) because I'm testing some things with the
drive. This was just like all the other times, so it looks like a race
somewhere.

This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is
compiled with both SMP and preempt).

After the BUG the kernel doesn't detect the device anymore.

Please let me know if you need any more information or if I can give you a
hand with testing.

Thanks a lot,
		Alberto


kernel BUG at drivers/usb/storage/usb.c:848!
invalid operand: 0000 [#1]
PREEMPT SMP
CPU:    1
EIP:    0060:[<c030843c>]    Not tainted
EFLAGS: 00010202   (2.6.6)
EIP is at usb_stor_release_resources+0xa3/0xcd
eax: 00002832   ebx: c639e200   ecx: c04bd768   edx: debf2888
esi: c04c6780   edi: c308ac00   ebp: c308ac24   esp: dededeb0
ds: 007b   es: 007b   ss: 0068
Process khubd (pid: 10, threadinfo=dedec000 task=dedef770)
Stack: debf2888 debf2854 c02e9f48 c639e200 debf2854 de40a670 debf2864
c04c67a0
       c0272926 debf2864 debf288c debf2864 c308acd0 c0272a4d debf2864
debf28bc
       debf2864 c308acd0 c0271b06 debf2864 debf2864 c308ac00 c0271b57
debf2864
Call Trace:
 [<c02e9f48>] usb_unbind_interface+0x7a/0x7c
 [<c0272926>] device_release_driver+0x64/0x66
 [<c0272a4d>] bus_remove_device+0x56/0x98
 [<c0271b06>] device_del+0x5f/0x9d
 [<c0271b57>] device_unregister+0x13/0x23
 [<c02eff98>] usb_disable_device+0x71/0xac
 [<c02eaa32>] usb_disconnect+0x9c/0xeb
 [<c02ecb1d>] hub_port_connect_change+0x274/0x279
 [<c02ec509>] hub_port_status+0x45/0xb0
 [<c02ece0b>] hub_events+0x2e9/0x364
 [<c02eceb3>] hub_thread+0x2d/0xe8
 [<c0105f86>] ret_from_fork+0x6/0x14
 [<c011a3a1>] default_wake_function+0x0/0x12
 [<c02ece86>] hub_thread+0x0/0xe8
 [<c0104271>] kernel_thread_helper+0x5/0xb

Code: 0f 0b 50 03 05 a6 43 c0 e9 7c ff ff ff c7 80 e0 01 00 00 00


Five seconds later, this comes out:

 <1>Unable to handle kernel paging request at virtual address 31a875c8
 printing eip:
c0307b2b
*pde = 00000000
Oops: 0002 [#2]
PREEMPT SMP
CPU:    0
EIP:    0060:[<c0307b2b>]    Not tainted
EFLAGS: 00010002   (2.6.6)
EIP is at usb_stor_control_thread+0x14d/0x2c9
eax: 31a875c8   ebx: c639e200   ecx: c639e200   edx: 00002003
esi: c639e600   edi: d3c84000   ebp: d3c84000   esp: d3c85f9c
ds: 007b   es: 007b   ss: 0068
Process usb-storage (pid: 10290, threadinfo=d3c84000 task=c76b0130)
Stack: dec00800 c639e200 c76b0698 c639e310 c639e324 c76b0130 c13e5ca0
dededda8
       c0105f86 dedef770 c03079de 00000000 c639e200 00000000 00000000
00000000
       00000000 c03079de 00000000 00000000 00000000 c0104271 c639e200
00000000
Call Trace:
 [<c0105f86>] ret_from_fork+0x6/0x14
 [<c03079de>] usb_stor_control_thread+0x0/0x2c9
 [<c03079de>] usb_stor_control_thread+0x0/0x2c9
 [<c0104271>] kernel_thread_helper+0x5/0xb

Code: f0 fe 08 0f 88 38 0b 00 00 8b 83 b8 00 00 00 81 b8 50 01 00
 <6>note: usb-storage[10290] exited with preempt_count 1

Re: [linux-usb-devel] BUG when removing USB flash drive

[Alan Stern]

On Thu, 13 May 2004, Alberto Bertogli wrote:

> Hi!
> 
> I've just hit a bug after removing a Kingston USB flash drive.
> 
> I have removed it several (probably more than 40) times today without
> problems until now. I'm removing it while doing streaming write()s on
> the device (using /dev/sda) because I'm testing some things with the
> drive. This was just like all the other times, so it looks like a race
> somewhere.
> 
> This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is
> compiled with both SMP and preempt).
> 
> After the BUG the kernel doesn't detect the device anymore.
> 
> Please let me know if you need any more information or if I can give you a
> hand with testing.
> 
> Thanks a lot,
> 		Alberto
> 
> 
> kernel BUG at drivers/usb/storage/usb.c:848!

This BUG happened because the SCSI layer was still using the drive after 
usb-storage had called scsi_remove_host().

This is an aspect of SCSI midlayer <-> low-level driver interaction that
I'm not very clear about.  How long after the host is removed can the
midlayer continue to use it?  How does the driver know when the midlayer
is finished using the host so the driver can exit (and unload from
memory)?

Alan Stern

Re: [linux-usb-devel] BUG when removing USB flash drive

[Mike Anderson]

Alan Stern [stern@rowland.harvard.edu] wrote:
> This BUG happened because the SCSI layer was still using the drive after 
> usb-storage had called scsi_remove_host().

In this case the scsi_remove_host is being called in a unexpected
disconnect case. Any IOs in flight will be canceled in the mid-layer
(i.e., they will not be chased down with calls to eh_abort_handler).

> 
> This is an aspect of SCSI midlayer <-> low-level driver interaction that
> I'm not very clear about.  How long after the host is removed can the
> midlayer continue to use it?  How does the driver know when the midlayer
> is finished using the host so the driver can exit (and unload from
> memory)?

The LLDD queuecommand should not be called after the return of
scsi_remove_host (unless we have a bug). The LLDD should not free its
resources until the release function is called on the struct device
passed in during the scsi_add_host call. This release function could be
called post the scsi_host_dev_release function being called as it does a
put on this struct device. This chain of release calls is undefined as a
hotplug goes out on the remove calls, but if user space never umounts /
closes the device things will not fully cleanup.

-andmike
--
Michael Anderson
andmike@us.ibm.com

Re: [linux-usb-devel] BUG when removing USB flash drive

[Alan Stern]

On Thu, 13 May 2004, Mike Anderson wrote:

> The LLDD queuecommand should not be called after the return of
> scsi_remove_host (unless we have a bug). The LLDD should not free its
> resources until the release function is called on the struct device
> passed in during the scsi_add_host call.

Ah.  We have a problem here, because usb-storage doesn't own that struct 
device.  It is owned by the USB core, and usb-storage has no way to know 
when its release function has been called.

Even worse, the release function may never be called at all.  The device
itself may still exist and be attached to the system even though the
usb-storage driver has been unbound from it.

There must be many other LLDDs with the same problem.  Any driver that 
manages a PCI host controller, for example.  The struct device it receives 
from the PCI layer isn't owned by the LLDD, yet that's what it must pass 
to scsi_add_host().

How should we handle this?

Alan Stern

Re: [linux-usb-devel] BUG when removing USB flash drive

[Mike Anderson]

Alan Stern [stern@rowland.harvard.edu] wrote:
> On Thu, 13 May 2004, Mike Anderson wrote:
> 
> > The LLDD queuecommand should not be called after the return of
> > scsi_remove_host (unless we have a bug). The LLDD should not free its
> > resources until the release function is called on the struct device
> > passed in during the scsi_add_host call.
> 
> Ah.  We have a problem here, because usb-storage doesn't own that struct 
> device.  It is owned by the USB core, and usb-storage has no way to know 
> when its release function has been called.
> 
> Even worse, the release function may never be called at all.  The device
> itself may still exist and be attached to the system even though the
> usb-storage driver has been unbound from it.
> 
> There must be many other LLDDs with the same problem.  Any driver that 
> manages a PCI host controller, for example.  The struct device it receives 
> from the PCI layer isn't owned by the LLDD, yet that's what it must pass 
> to scsi_add_host().
> 
> How should we handle this?
> 

I was incorrect in my previous statement. Well we may hold a reference
count on the struct device passed into the scsi_add_host until all
references on the host instance are gone and there maybe a positive ref
count on the hostt module until all devices are closed a instance of
LLDD should be able to do cleanup post return of scsi_remove_host. This
needs to be true or SCSI mid-layer would not honor the requirement for
the pci remove call.

I believe in the past you or someone else has mentioned that post the
return of scsi_remove_host a Scsi_Host instance still has a hostt and a
transportt which may be dangerous if all code paths are not correctly
handling a host in the SHOST_DEL state.

I guess we still have this current issue on what is happening in the
usb storage that it believes SCSI mid is still using the device. Is
something not cleaning up, or is something in SCSI mid not honoring the
SHOST_DEL state.

-andmike
--
Michael Anderson
andmike@us.ibm.com

Re: [linux-usb-devel] BUG when removing USB flash drive

[Alan Stern]

On Fri, 14 May 2004, Mike Anderson wrote:

> I was incorrect in my previous statement. Well we may hold a reference
> count on the struct device passed into the scsi_add_host until all
> references on the host instance are gone and there maybe a positive ref
> count on the hostt module until all devices are closed a instance of
> LLDD should be able to do cleanup post return of scsi_remove_host. This
> needs to be true or SCSI mid-layer would not honor the requirement for
> the pci remove call.
> 
> I believe in the past you or someone else has mentioned that post the
> return of scsi_remove_host a Scsi_Host instance still has a hostt and a
> transportt which may be dangerous if all code paths are not correctly
> handling a host in the SHOST_DEL state.

Yes, that was me.

> I guess we still have this current issue on what is happening in the
> usb storage that it believes SCSI mid is still using the device. Is
> something not cleaning up, or is something in SCSI mid not honoring the
> SHOST_DEL state.

usb-storage checks the state of its kernel thread after calling
scsi_remove_host(), while preparing to kill the thread.  This can only
be done while the thread is idle, not processing any queued commands or
error handler requests.  The BUG in Alberto's log was triggered because
the thread was not idle.  Unfortunately the log doesn't say whether it was
executing a queued command or an error handler request at the time.

It does appear to have been some sort of race, as Alberto mentioned in his 
original message:

http://marc.theaimsgroup.com/?l=linux-usb-devel&m=108449593410567&w=2

It might not be easy to find out where the race is occurring.

Alan Stern

Re: [linux-usb-devel] BUG when removing USB flash drive

[James Bottomley]

On Fri, 2004-05-14 at 14:33, Alan Stern wrote:
> usb-storage checks the state of its kernel thread after calling
> scsi_remove_host(), while preparing to kill the thread.  This can only
> be done while the thread is idle, not processing any queued commands or
> error handler requests.  The BUG in Alberto's log was triggered because
> the thread was not idle.  Unfortunately the log doesn't say whether it was
> executing a queued command or an error handler request at the time.

The idea behind scsi_remove_host() was that it was going to be called
from the module_exit() routine.  This means that the reference count on
the LLD has already dropped to zero and it's safe for removal.

If you call scsi_remove_host() outside of the module_exit() code then
what it does is remove all hosts from visibility (and makes it
impossible to open or do anything with them), but it cannot revoke
outstanding references (which may continue to use the device).  The
mid-layer takes the devices through CANCEL and then DEL.  When
scsi_remove_host() returns, you are guaranteed (barring any state model
cockups) that the devices on the LLD with outstanding references are all
in the DEL state.  It will also block the device from above (no commands
will be sent down).  Any currently outstanding commands will be errored
from the mid-layer; however, it will do nothing about outstanding
commands in the LLD, you have to deal with them yourself.

The way this is supposed to work is that when you get a disconnection,
you start refusing commands (and you also discard any active ones). 
Then you call scsi_remove_host().  We discard all current commands

So, if the usb storage LLD isn't discarding the commands, then they may
be active.

James

Re: [linux-usb-devel] BUG when removing USB flash drive

[Alan Stern]

On 14 May 2004, James Bottomley wrote:

> The idea behind scsi_remove_host() was that it was going to be called
> from the module_exit() routine.  This means that the reference count on
> the LLD has already dropped to zero and it's safe for removal.
> 
> If you call scsi_remove_host() outside of the module_exit() code then
> what it does is remove all hosts from visibility (and makes it
> impossible to open or do anything with them), but it cannot revoke
> outstanding references (which may continue to use the device).  The
> mid-layer takes the devices through CANCEL and then DEL.  When
> scsi_remove_host() returns, you are guaranteed (barring any state model
> cockups) that the devices on the LLD with outstanding references are all
> in the DEL state.  It will also block the device from above (no commands
> will be sent down).  Any currently outstanding commands will be errored
> from the mid-layer; however, it will do nothing about outstanding
> commands in the LLD, you have to deal with them yourself.
> 
> The way this is supposed to work is that when you get a disconnection,
> you start refusing commands (and you also discard any active ones). 
> Then you call scsi_remove_host().  We discard all current commands
> 
> So, if the usb storage LLD isn't discarding the commands, then they may
> be active.

Yes, that would explain it.  We do arrange to abort the current command, 
but this is done before calling scsi_remove_host() when it should be done 
after.  And we don't wait for the kernel thread to become idle afterwards.

I'll work on a patch.  Testing it won't be easy, though; it's hard to 
prove that a race _isn't_ happening!

Alan Stern

Re: [linux-usb-devel] BUG when removing USB flash drive

[Alberto Bertogli]

On Fri, May 14, 2004 at 05:13:38PM -0400, Alan Stern wrote:
> I'll work on a patch.  Testing it won't be easy, though; it's hard to 
> prove that a race _isn't_ happening!

I'm going to have to repeat this tests a few more times (in fact, I'll
boot -mm2 in a couple of minutes to see if I can reproduce the bug there),
so if you want me to try some patch let me know. As you say, I can't
_prove_ it, but at least I can confirm I can't hit it =)

Thanks,
		Alberto

Re: [linux-usb-devel] BUG when removing USB flash drive

[Alan Stern]

I took a look at scsi_host_cancel(), scsi_device_cancel(), and a few other 
routines.  Three things stood out:

	When a host is removed, currently executing requests are 
	aborted within the midlevel but the low-level driver is not
	informed of this.  Presumably it shouldn't care since it
	ought to cancel its own outstanding I/O?

	There's a race between scsi_host_cancel()/scsi_device_cancel()
	and scsi_times_out().  If the timer fires before 
	scsi_device_cancel() can delete it, and scsi_host_cancel()
	tests the SHOST_RECOVERY bit before the error handler manages
	to set it, then you will remove a host that's still in error
	recovery.

	There's no protection against accessing the host template
	via procfs, if a user process pins the file during
	scsi_remove_host().  You probably are already aware of this.

Alan Stern

Re: BUG when removing USB flash drive

[Greg KH]

On Thu, May 13, 2004 at 09:41:32PM -0300, Alberto Bertogli wrote:
> 
> This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is
> compiled with both SMP and preempt).

Can you test the latest -mm tree and see if it fixes this problem for
you?

thanks,

greg k-h

Re: BUG when removing USB flash drive

[Alberto Bertogli]

On Fri, May 14, 2004 at 04:03:01AM -0700, Greg KH wrote:
> On Thu, May 13, 2004 at 09:41:32PM -0300, Alberto Bertogli wrote:
> > 
> > This is a stock 2.6.6 kernel, on a Pentium 4 with HT (the kernel is
> > compiled with both SMP and preempt).
> 
> Can you test the latest -mm tree and see if it fixes this problem for
> you?

Sure. I've just tested -mm2; sorry it took so long but I had a couple of
other things going on.

Sadly, it locks up hard (no sysrq or anything) after booting, right in the
login prompt. Sometimes I got as far as typing the username and pressing
enter, others just frozen without even being able to press a key.

So I removed the SMT scheduler because I thought it might be related, and
it didn't even got to boot at all, it seem to freeze somewhere after
the initial load, but I can't see anything because the screen goes black
(even after I disabled framebuffer, SMP and preempt just in case).

I'll see if I can get a working config tomorrow morning so I can get to do
this tests (and report that lockup too).


Thanks a lot,
		Alberto