Re: oops with USB Storage on 2.6.14

Re: oops with USB Storage on 2.6.14

[Andrew Morton]

Masanari Iida <standby24x7@gmail.com> wrote:
>
> Hello,
> I updated my system's kernel from 2.6.13.2 to 2.6.14,
> then it oops when I connect my Digital Camera via USB connection
> as USB storage device.
> I went back to 2.6.14-rc1, still the same panic happen.
> 2.6.13.2 and before, the kernel has been worked as expected.
> 
> CPU Intel P4(2.4Ghz)
> USB Device   Pentax Optio S40.
> 
> Unable to handle kernel paging request at virtual address dc9d1f4c
>  printing eip:
> c02b44cc
> *pde = 00073067
> *pte = 1c9d1000
> Oops: 0000 [#1]
> SMP DEBUG_PAGEALLOC
> Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> ipt_recent iptable
> _filter ip_tables video rtc
> CPU:    1
> EIP:    0060:[<c02b44cc>]    Not tainted VLI
> EFLAGS: 00010286   (2.6.14)
> EIP is at scsi_run_queue+0xc/0xd0
> eax: 00000001   ebx: dc9d1e3c   ecx: d6b67910   edx: dc9d1e3c
> esi: d5048eb0   edi: dc9d1e3c   ebp: c1507e98   esp: c1507e84
> ds: 007b   es: 007b   ss: 0068
> Process ksoftirqd/1 (pid: 6, threadinfo=c1506000 task=dfe2dad0)
> Stack: 00000292 de3a7bf8 dc9d1e3c d5048eb0 dc9d1e3c c1507ea8 c02b4612 dc9d1e3c
>        da51bf60 c1507ecc c02b473f d5048eb0 00000000 00000024 00000286 00000001
>        d5048eb0 00000000 c1507f10 c02b4b2e d5048eb0 00000000 00000024 00000001
> 
> Call Trace:
>  [<c0103abf>] show_stack+0x7f/0xa0
>  [<c0103c72>] show_registers+0x162/0x1d0
>  [<c0103e90>] die+0x100/0x1a0
>  [<c039d7ae>] do_page_fault+0x31e/0x640
>  [<c0103763>] error_code+0x4f/0x54
>  [<c02b4612>] scsi_next_command+0x22/0x30
>  [<c02b473f>] scsi_end_request+0xcf/0xf0
>  [<c02b4b2e>] scsi_io_completion+0x26e/0x470
>  [<c02b4fc7>] scsi_generic_done+0x37/0x50
>  [<c02af9e5>] scsi_finish_command+0x85/0xa0
>  [<c02af89c>] scsi_softirq+0xcc/0x140
>  [<c0122085>] __do_softirq+0xd5/0xf0
>  [<c01220d8>] do_softirq+0x38/0x40
>  [<c0122685>] ksoftirqd+0x95/0xe0
>  [<c0131cfa>] kthread+0xba/0xc0
>  [<c0100ecd>] kernel_thread_helper+0x5/0x18
> Code: f0 8b 42 44 e8 16 7f 0e 00 89 45 ec 89 1c 24 e8 6b b7 ff ff eb aa 89 f6 8d
>  bc 27 00 00 00 00 55 89 e5 57 56 53 83 ec 08 8b 55 08 <8b> 82 10 01 00 00 8b 38
>  f6 80 85 01 00 00 80 0f 85 9e 00 00 00
>   <0>Kernel panic - not syncing: Fatal exception in interrupt
> 

Either a scsi bug or a USB bug.  Either way, regressions like this are a
top priority.

Could you please try disabling CONFIG_DEBUG_PAGEALLOC and retest?  If that
works OK, it's probably a use-after-free.

Thanks. 

Re: oops with USB Storage on 2.6.14

[Andrew Morton]

Masanari Iida <standby24x7@gmail.com> wrote:
>

You removed linux-scis and linux-usb from cc.  Please retain them.

> >
> > Could you please try disabling CONFIG_DEBUG_PAGEALLOC and retest?  If that
> > works OK, it's probably a use-after-free.
> >
> Hello Andrew,
> 
> I did disabled CONFIG_DEBUG_PAGEALLOC and re-tested on 2.6.14-rc1.
> Now the oops didn't happen when I connect digital camera to the USB.

So the first oops was probably use-after-free.

> I could mount the camera as USB storage.
> But oops still happen when I turned the camera power off.
> (This oops didn't halt my system, BTW)
> 
> # Unable to handle kernel paging request at virtual address 6b6b6bb3
>   printing eip:
> c02b88ca
> *pde = 00000000
> Oops: 0002 [#1]
> SMP
> Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> ipt_recent iptable_filter ip_tables video rtc
> CPU:    0
> EIP:    0060:[<c02b88ca>]    Not tainted VLI
> EFLAGS: 00010296   (2.6.14-rc1)
> EIP is at scsi_remove_device+0x3a/0x50
> eax: 00000001   ebx: dee9f478   ecx: 00000000   edx: 6b6b6b6b
> esi: de6655c4   edi: de6655bc   ebp: dfd4fdb4   esp: dfd4fda4
> ds: 007b   es: 007b   ss: 0068
> Process khubd (pid: 15, threadinfo=dfd4e000 task=dff1f030)
> Stack: dee9f478 00000066 dee9f478 de6655c4 dfd4fdcc c02b89a1 dee9f478 de6655c8
>        de213c98 de6655c4 dfd4fde8 c02b8a16 de213c84 dfd4fde8 00000282 de6655c8
>        de6655c8 dfd4fe04 c02b7655 de213c98 de6655cc de6655c4 de6656e0 de213a8c
> Call Trace:
>  [<c0103aaf>] show_stack+0x7f/0xa0
>  [<c0103c62>] show_registers+0x162/0x1d0
>  [<c0103e74>] die+0xf4/0x1a0
>  [<c039cfce>] do_page_fault+0x31e/0x640
>  [<c0103753>] error_code+0x4f/0x54
>  [<c02b89a1>] __scsi_remove_target+0xc1/0xe0
>  [<c02b8a16>] scsi_remove_target+0x26/0x60
>  [<c02b7655>] scsi_forget_host+0x45/0x70
>  [<c02afd97>] scsi_remove_host+0x57/0xa0
>  [<c02e5145>] quiesce_and_remove_host+0x75/0xb0
>  [<c02e55fd>] storage_disconnect+0x1d/0x2c
>  [<c02c8286>] usb_unbind_interface+0x86/0x90
>  [<c025617b>] __device_release_driver+0x8b/0x90
>  [<c02561b6>] device_release_driver+0x36/0x50
>  [<c02557c9>] bus_remove_device+0x79/0x90
>  [<c0254525>] device_del+0x35/0x70
>  [<c02d00fb>] usb_disable_device+0xfb/0x130
>  [<c02caa46>] usb_disconnect+0xc6/0x180
>  [<c02cbe2f>] hub_port_connect_change+0x3cf/0x400
>  [<c02cc12e>] hub_events+0x2ce/0x410
>  [<c02cc285>] hub_thread+0x15/0xf0
>   [<c0131aaa>] kthread+0xba/0xc0
>  [<c0100ecd>] kernel_thread_helper+0x5/0x18
> Code: 5d 08 89 75 fc 8b 33 89 44 24 04 c7 04 24 e8 a8 3b c0 e8 9a 14
> e6 ff f0 ff 4e 48 0f 88 a8 04 00 00 89 1c
>   24 e8 38 ff ff ff 8b 13 <f0> ff 42 48 0f 8e a1 04 00 00 8b 5d f8 8b
> 75 fc 89 ec 5d c3 89
> 
> If you need some more test, let me know.
> In that case, please specify which version of kernel you want me to test.
> 

OK, thanks.  This is a different bug.  Presumably in USB.

Re: [linux-usb-devel] Re: oops with USB Storage on 2.6.14

[Alan Stern]

On Sun, 30 Oct 2005, Andrew Morton wrote:

> Masanari Iida <standby24x7@gmail.com> wrote:

> > Hello Andrew,
> > 
> > I did disabled CONFIG_DEBUG_PAGEALLOC and re-tested on 2.6.14-rc1.
> > Now the oops didn't happen when I connect digital camera to the USB.
> 
> So the first oops was probably use-after-free.
> 
> > I could mount the camera as USB storage.
> > But oops still happen when I turned the camera power off.
> > (This oops didn't halt my system, BTW)
> > 
> > # Unable to handle kernel paging request at virtual address 6b6b6bb3
> >   printing eip:
> > c02b88ca
> > *pde = 00000000
> > Oops: 0002 [#1]
> > SMP
> > Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> > ipt_recent iptable_filter ip_tables video rtc
> > CPU:    0
> > EIP:    0060:[<c02b88ca>]    Not tainted VLI
> > EFLAGS: 00010296   (2.6.14-rc1)
> > EIP is at scsi_remove_device+0x3a/0x50

> > If you need some more test, let me know.
> > In that case, please specify which version of kernel you want me to test.
> > 
> 
> OK, thanks.  This is a different bug.  Presumably in USB.

This was fixed in later releases of 2.6.14-rc.

I wasn't able to reproduce the original problem, even after setting 
CONFIG_DEBUG_PAGEALLOC.

Alan Stern

Re: [linux-usb-devel] Re: oops with USB Storage on 2.6.14

[Masanari Iida]

On 11/1/05, Alan Stern <stern@rowland.harvard.edu> wrote:
> On Sun, 30 Oct 2005, Andrew Morton wrote:
>
> > Masanari Iida <standby24x7@gmail.com> wrote:
>
> > > Hello Andrew,
> > >
> > > I did disabled CONFIG_DEBUG_PAGEALLOC and re-tested on 2.6.14-rc1.
> > > Now the oops didn't happen when I connect digital camera to the USB.
> >
> > So the first oops was probably use-after-free.
> >
> > > I could mount the camera as USB storage.
> > > But oops still happen when I turned the camera power off.
> > > (This oops didn't halt my system, BTW)
> > >
> > > # Unable to handle kernel paging request at virtual address 6b6b6bb3
> > >   printing eip:
> > > c02b88ca
> > > *pde = 00000000
> > > Oops: 0002 [#1]
> > > SMP
> > > Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> > > ipt_recent iptable_filter ip_tables video rtc
> > > CPU:    0
> > > EIP:    0060:[<c02b88ca>]    Not tainted VLI
> > > EFLAGS: 00010296   (2.6.14-rc1)
> > > EIP is at scsi_remove_device+0x3a/0x50
>
> > > If you need some more test, let me know.
> > > In that case, please specify which version of kernel you want me to test.
> > >
> >
> > OK, thanks.  This is a different bug.  Presumably in USB.
>
> This was fixed in later releases of 2.6.14-rc.
>
> I wasn't able to reproduce the original problem, even after setting
> CONFIG_DEBUG_PAGEALLOC.
>
> Alan Stern
>
Alan,

Confirm the " scsi_remove_device " oops didn't happen on 2.4.14.
Talking about the original PANIC, as I have a workaround
(CONFIG_DEBUG_PAGEALLOC disabled),  I agree to close my report, now.

Thank you.

Masanari Iida

Re: [linux-usb-devel] Re: oops with USB Storage on 2.6.14

[Andrew Morton]

Masanari Iida <standby24x7@gmail.com> wrote:
>
> On 11/1/05, Alan Stern <stern@rowland.harvard.edu> wrote:
> > On Sun, 30 Oct 2005, Andrew Morton wrote:
> >
> > > Masanari Iida <standby24x7@gmail.com> wrote:
> >
> > > > Hello Andrew,
> > > >
> > > > I did disabled CONFIG_DEBUG_PAGEALLOC and re-tested on 2.6.14-rc1.
> > > > Now the oops didn't happen when I connect digital camera to the USB.
> > >
> > > So the first oops was probably use-after-free.
> > >
> > > > I could mount the camera as USB storage.
> > > > But oops still happen when I turned the camera power off.
> > > > (This oops didn't halt my system, BTW)
> > > >
> > > > # Unable to handle kernel paging request at virtual address 6b6b6bb3
> > > >   printing eip:
> > > > c02b88ca
> > > > *pde = 00000000
> > > > Oops: 0002 [#1]
> > > > SMP
> > > > Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> > > > ipt_recent iptable_filter ip_tables video rtc
> > > > CPU:    0
> > > > EIP:    0060:[<c02b88ca>]    Not tainted VLI
> > > > EFLAGS: 00010296   (2.6.14-rc1)
> > > > EIP is at scsi_remove_device+0x3a/0x50
> >
> > > > If you need some more test, let me know.
> > > > In that case, please specify which version of kernel you want me to test.
> > > >
> > >
> > > OK, thanks.  This is a different bug.  Presumably in USB.
> >
> > This was fixed in later releases of 2.6.14-rc.
> >
> > I wasn't able to reproduce the original problem, even after setting
> > CONFIG_DEBUG_PAGEALLOC.
> >
> > Alan Stern
> >
> Alan,
> 
> Confirm the " scsi_remove_device " oops didn't happen on 2.4.14.

2.6.14, I assume.

> Talking about the original PANIC, as I have a workaround
> (CONFIG_DEBUG_PAGEALLOC disabled),  I agree to close my report, now.

That's not a valid workaround.  We're touching freed, unallocated or simply
wild memory and that is a bad bug.

Re: oops with USB Storage on 2.6.14

[Andrew Morton]

Masanari Iida <standby24x7@gmail.com> wrote:
>
> Hello,
> I updated my system's kernel from 2.6.13.2 to 2.6.14,
> then it oops when I connect my Digital Camera via USB connection
> as USB storage device.
> I went back to 2.6.14-rc1, still the same panic happen.
> 2.6.13.2 and before, the kernel has been worked as expected.
> 
> CPU Intel P4(2.4Ghz)
> USB Device   Pentax Optio S40.
> 
> Unable to handle kernel paging request at virtual address dc9d1f4c
>  printing eip:
> c02b44cc
> *pde = 00073067
> *pte = 1c9d1000
> Oops: 0000 [#1]
> SMP DEBUG_PAGEALLOC
> Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> ipt_recent iptable
> _filter ip_tables video rtc
> CPU:    1
> EIP:    0060:[<c02b44cc>]    Not tainted VLI
> EFLAGS: 00010286   (2.6.14)
> EIP is at scsi_run_queue+0xc/0xd0
> eax: 00000001   ebx: dc9d1e3c   ecx: d6b67910   edx: dc9d1e3c
> esi: d5048eb0   edi: dc9d1e3c   ebp: c1507e98   esp: c1507e84
> ds: 007b   es: 007b   ss: 0068
> Process ksoftirqd/1 (pid: 6, threadinfo=c1506000 task=dfe2dad0)
> Stack: 00000292 de3a7bf8 dc9d1e3c d5048eb0 dc9d1e3c c1507ea8 c02b4612 dc9d1e3c
>        da51bf60 c1507ecc c02b473f d5048eb0 00000000 00000024 00000286 00000001
>        d5048eb0 00000000 c1507f10 c02b4b2e d5048eb0 00000000 00000024 00000001
> 
> Call Trace:
>  [<c0103abf>] show_stack+0x7f/0xa0
>  [<c0103c72>] show_registers+0x162/0x1d0
>  [<c0103e90>] die+0x100/0x1a0
>  [<c039d7ae>] do_page_fault+0x31e/0x640
>  [<c0103763>] error_code+0x4f/0x54
>  [<c02b4612>] scsi_next_command+0x22/0x30
>  [<c02b473f>] scsi_end_request+0xcf/0xf0
>  [<c02b4b2e>] scsi_io_completion+0x26e/0x470
>  [<c02b4fc7>] scsi_generic_done+0x37/0x50
>  [<c02af9e5>] scsi_finish_command+0x85/0xa0
>  [<c02af89c>] scsi_softirq+0xcc/0x140
>  [<c0122085>] __do_softirq+0xd5/0xf0
>  [<c01220d8>] do_softirq+0x38/0x40
>  [<c0122685>] ksoftirqd+0x95/0xe0
>  [<c0131cfa>] kthread+0xba/0xc0
>  [<c0100ecd>] kernel_thread_helper+0x5/0x18
> Code: f0 8b 42 44 e8 16 7f 0e 00 89 45 ec 89 1c 24 e8 6b b7 ff ff eb aa 89 f6 8d
>  bc 27 00 00 00 00 55 89 e5 57 56 53 83 ec 08 8b 55 08 <8b> 82 10 01 00 00 8b 38
>  f6 80 85 01 00 00 80 0f 85 9e 00 00 00
>   <0>Kernel panic - not syncing: Fatal exception in interrupt
> 

Has there been any progress on this?

If not, can you please test the latest snapshot from
ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots and if it still fails, raise a bug at bugzilla.kernel.org?

Thanks.

Re: oops with USB Storage on 2.6.14

[Masanari Iida]

On 11/8/05, Andrew Morton <akpm@osdl.org> wrote:
> Masanari Iida <standby24x7@gmail.com> wrote:
> >
> > Hello,
> > I updated my system's kernel from 2.6.13.2 to 2.6.14,
> > then it oops when I connect my Digital Camera via USB connection
> > as USB storage device.
> > I went back to 2.6.14-rc1, still the same panic happen.
> > 2.6.13.2 and before, the kernel has been worked as expected.
> >
> > CPU Intel P4(2.4Ghz)
> > USB Device   Pentax Optio S40.
> >
> > Unable to handle kernel paging request at virtual address dc9d1f4c
> >  printing eip:
> > c02b44cc
> > *pde = 00073067
> > *pte = 1c9d1000
> > Oops: 0000 [#1]
> > SMP DEBUG_PAGEALLOC
> > Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> > ipt_recent iptable
> > _filter ip_tables video rtc
> > CPU:    1
> > EIP:    0060:[<c02b44cc>]    Not tainted VLI
> > EFLAGS: 00010286   (2.6.14)
> > EIP is at scsi_run_queue+0xc/0xd0
> > eax: 00000001   ebx: dc9d1e3c   ecx: d6b67910   edx: dc9d1e3c
> > esi: d5048eb0   edi: dc9d1e3c   ebp: c1507e98   esp: c1507e84
> > ds: 007b   es: 007b   ss: 0068
> > Process ksoftirqd/1 (pid: 6, threadinfo=c1506000 task=dfe2dad0)
> > Stack: 00000292 de3a7bf8 dc9d1e3c d5048eb0 dc9d1e3c c1507ea8 c02b4612 dc9d1e3c
> >        da51bf60 c1507ecc c02b473f d5048eb0 00000000 00000024 00000286 00000001
> >        d5048eb0 00000000 c1507f10 c02b4b2e d5048eb0 00000000 00000024 00000001
> >
> > Call Trace:
> >  [<c0103abf>] show_stack+0x7f/0xa0
> >  [<c0103c72>] show_registers+0x162/0x1d0
> >  [<c0103e90>] die+0x100/0x1a0
> >  [<c039d7ae>] do_page_fault+0x31e/0x640
> >  [<c0103763>] error_code+0x4f/0x54
> >  [<c02b4612>] scsi_next_command+0x22/0x30
> >  [<c02b473f>] scsi_end_request+0xcf/0xf0
> >  [<c02b4b2e>] scsi_io_completion+0x26e/0x470
> >  [<c02b4fc7>] scsi_generic_done+0x37/0x50
> >  [<c02af9e5>] scsi_finish_command+0x85/0xa0
> >  [<c02af89c>] scsi_softirq+0xcc/0x140
> >  [<c0122085>] __do_softirq+0xd5/0xf0
> >  [<c01220d8>] do_softirq+0x38/0x40
> >  [<c0122685>] ksoftirqd+0x95/0xe0
> >  [<c0131cfa>] kthread+0xba/0xc0
> >  [<c0100ecd>] kernel_thread_helper+0x5/0x18
> > Code: f0 8b 42 44 e8 16 7f 0e 00 89 45 ec 89 1c 24 e8 6b b7 ff ff eb aa 89 f6 8d
> >  bc 27 00 00 00 00 55 89 e5 57 56 53 83 ec 08 8b 55 08 <8b> 82 10 01 00 00 8b 38
> >  f6 80 85 01 00 00 80 0f 85 9e 00 00 00
> >   <0>Kernel panic - not syncing: Fatal exception in interrupt
> >
>
> Has there been any progress on this?
>
> If not, can you please test the latest snapshot from
> ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots and if it still fails, raise a bug at bugzilla.kernel.org?
>
> Thanks.
>

Hello again, Andrew,

I have tested on 2.6.14-git10 with CONFIG_DEBUG_PAGEALLOC=y.
The original oops with USB Storage (Camera)  is fixed now.
Thank you.

Masanari

RE: oops with USB Storage on 2.6.14

[goggin, edward]

I've run into a bug like this several times using 2.6.14-rc4 while
testing dm-multipath's reaction to uevents generated by forcing 
fiber channel transport failures -- which leads to the scsi device
being detached and the queuedata pointer in the device's queue being
reset in scsi_device_dev_release.  The fix I've used is below and
it seems to work well for me.  I was going to place this patch on
dm-devel today or tomorrow anyway.

drivers/scsi/scsi_lib.c:scsi_next_command()
Call scsi_device_get and scsi_device_put around the calls to
scsi_put_command
and scsi_run_queue so that the scsi host structure will not be de-allocated
between scsi_put_command and scsi_run_queue.

*** ../base/linux-2.6.14-rc4/drivers/scsi/scsi_lib.c	Mon Oct 10 20:19:19
2005
--- drivers/scsi/scsi_lib.c	Thu Nov  3 13:30:03 2005
***************
*** 592,601 ****
  
  void scsi_next_command(struct scsi_cmnd *cmd)
  {
! 	struct request_queue *q = cmd->device->request_queue;
  
  	scsi_put_command(cmd);
  	scsi_run_queue(q);
  }
  
  void scsi_run_host_queues(struct Scsi_Host *shost)
--- 592,611 ----
  
  void scsi_next_command(struct scsi_cmnd *cmd)
  {
! 	struct scsi_device *sdev = cmd->device;
! 	struct request_queue *q = sdev->request_queue;
! 
! 	// need to hold a reference on the device before we let go of the
cmd
! 	if (scsi_device_get(sdev)) {
! 		scsi_put_command(cmd);
! 		return;		// maybe sdev_state == SDEV_CANCEL, SDEV_DEL
! 	}
  
  	scsi_put_command(cmd);
  	scsi_run_queue(q);
+ 
+ 	// ok to remove device now
+ 	scsi_device_put(sdev);
  }
  
  void scsi_run_host_queues(struct Scsi_Host *shost)
  

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org 
> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Andrew Morton
> Sent: Monday, November 07, 2005 11:41 PM
> To: Masanari Iida
> Cc: linux-kernel@vger.kernel.org; 
> linux-usb-devel@lists.sourceforge.net; linux-scsi@vger.kernel.org
> Subject: Re: oops with USB Storage on 2.6.14
> 
> Masanari Iida <standby24x7@gmail.com> wrote:
> >
> > Hello,
> > I updated my system's kernel from 2.6.13.2 to 2.6.14,
> > then it oops when I connect my Digital Camera via USB connection
> > as USB storage device.
> > I went back to 2.6.14-rc1, still the same panic happen.
> > 2.6.13.2 and before, the kernel has been worked as expected.
> > 
> > CPU Intel P4(2.4Ghz)
> > USB Device   Pentax Optio S40.
> > 
> > Unable to handle kernel paging request at virtual address dc9d1f4c
> >  printing eip:
> > c02b44cc
> > *pde = 00073067
> > *pte = 1c9d1000
> > Oops: 0000 [#1]
> > SMP DEBUG_PAGEALLOC
> > Modules linked in: autofs e100 ipt_LOG ipt_state ip_conntrack
> > ipt_recent iptable
> > _filter ip_tables video rtc
> > CPU:    1
> > EIP:    0060:[<c02b44cc>]    Not tainted VLI
> > EFLAGS: 00010286   (2.6.14)
> > EIP is at scsi_run_queue+0xc/0xd0
> > eax: 00000001   ebx: dc9d1e3c   ecx: d6b67910   edx: dc9d1e3c
> > esi: d5048eb0   edi: dc9d1e3c   ebp: c1507e98   esp: c1507e84
> > ds: 007b   es: 007b   ss: 0068
> > Process ksoftirqd/1 (pid: 6, threadinfo=c1506000 task=dfe2dad0)
> > Stack: 00000292 de3a7bf8 dc9d1e3c d5048eb0 dc9d1e3c 
> c1507ea8 c02b4612 dc9d1e3c
> >        da51bf60 c1507ecc c02b473f d5048eb0 00000000 
> 00000024 00000286 00000001
> >        d5048eb0 00000000 c1507f10 c02b4b2e d5048eb0 
> 00000000 00000024 00000001
> > 
> > Call Trace:
> >  [<c0103abf>] show_stack+0x7f/0xa0
> >  [<c0103c72>] show_registers+0x162/0x1d0
> >  [<c0103e90>] die+0x100/0x1a0
> >  [<c039d7ae>] do_page_fault+0x31e/0x640
> >  [<c0103763>] error_code+0x4f/0x54
> >  [<c02b4612>] scsi_next_command+0x22/0x30
> >  [<c02b473f>] scsi_end_request+0xcf/0xf0
> >  [<c02b4b2e>] scsi_io_completion+0x26e/0x470
> >  [<c02b4fc7>] scsi_generic_done+0x37/0x50
> >  [<c02af9e5>] scsi_finish_command+0x85/0xa0
> >  [<c02af89c>] scsi_softirq+0xcc/0x140
> >  [<c0122085>] __do_softirq+0xd5/0xf0
> >  [<c01220d8>] do_softirq+0x38/0x40
> >  [<c0122685>] ksoftirqd+0x95/0xe0
> >  [<c0131cfa>] kthread+0xba/0xc0
> >  [<c0100ecd>] kernel_thread_helper+0x5/0x18
> > Code: f0 8b 42 44 e8 16 7f 0e 00 89 45 ec 89 1c 24 e8 6b b7 
> ff ff eb aa 89 f6 8d
> >  bc 27 00 00 00 00 55 89 e5 57 56 53 83 ec 08 8b 55 08 <8b> 
> 82 10 01 00 00 8b 38
> >  f6 80 85 01 00 00 80 0f 85 9e 00 00 00
> >   <0>Kernel panic - not syncing: Fatal exception in interrupt
> > 
> 
> Has there been any progress on this?
> 
> If not, can you please test the latest snapshot from
> ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots and if 
> it still fails, raise a bug at bugzilla.kernel.org?
> 
> Thanks.
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: oops with USB Storage on 2.6.14

[Rolf Eike Beer]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

Am Dienstag, 8. November 2005 17:24 schrieb goggin, edward:
>I've run into a bug like this several times using 2.6.14-rc4 while
>testing dm-multipath's reaction to uevents generated by forcing
>fiber channel transport failures -- which leads to the scsi device
>being detached and the queuedata pointer in the device's queue being
>reset in scsi_device_dev_release.  The fix I've used is below and
>it seems to work well for me.  I was going to place this patch on
>dm-devel today or tomorrow anyway.
>
>drivers/scsi/scsi_lib.c:scsi_next_command()
>Call scsi_device_get and scsi_device_put around the calls to
>scsi_put_command
>and scsi_run_queue so that the scsi host structure will not be de-allocated
>between scsi_put_command and scsi_run_queue.
>
>*** ../base/linux-2.6.14-rc4/drivers/scsi/scsi_lib.c	Mon Oct 10 20:19:19
>2005
>--- drivers/scsi/scsi_lib.c	Thu Nov  3 13:30:03 2005
>***************
>*** 592,601 ****

Your patch is linewrapped. Also please use unified diff format, good choice 
for diff options is "-Naurp".

Eike

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: oops with USB Storage on 2.6.14

[James Bottomley]

On Tue, 2005-11-08 at 11:24 -0500, goggin, edward wrote:
> ! 	struct scsi_device *sdev = cmd->device;
> ! 	struct request_queue *q = sdev->request_queue;
> ! 
> ! 	// need to hold a reference on the device before we let go of the
> cmd
> ! 	if (scsi_device_get(sdev)) {
> ! 		scsi_put_command(cmd);
> ! 		return;		// maybe sdev_state == SDEV_CANCEL, SDEV_DEL
> ! 	}
>   
>   	scsi_put_command(cmd);
>   	scsi_run_queue(q);
> + 
> + 	// ok to remove device now
> + 	scsi_device_put(sdev);

This is the right idea, I think, but not necessarily the right fix.
scsi_device_get() will fail if the device is going offline, but we would
still need to run the queues.

try this sequence instead:

get_device(&sdev->sdev_gendev);
scsi_put_command(cmd);
scsi_run_queue(q);
put_device(&sdev->sdev_gendev);

James




-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
linux-usb-devel@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

RE: oops with USB Storage on 2.6.14

[goggin, edward]

Good point.

I tested your suggested patch fix and it works well in my test use case.

BTW, I've got test code in my just tested version of your patch to read the
kref of the sdev_gendev between the calls to scsi_run_queue and
put_device to increase the likelihood that the use case has actually
been tested.

> -----Original Message-----
> From: James Bottomley [mailto:James.Bottomley@SteelEye.com] 
> Sent: Tuesday, November 08, 2005 12:02 PM
> To: goggin, edward
> Cc: 'Andrew Morton'; Masanari Iida; 
> linux-kernel@vger.kernel.org; 
> linux-usb-devel@lists.sourceforge.net; linux-scsi@vger.kernel.org
> Subject: RE: oops with USB Storage on 2.6.14
> 
> On Tue, 2005-11-08 at 11:24 -0500, goggin, edward wrote:
> > ! 	struct scsi_device *sdev = cmd->device;
> > ! 	struct request_queue *q = sdev->request_queue;
> > ! 
> > ! 	// need to hold a reference on the device before we let 
> go of the
> > cmd
> > ! 	if (scsi_device_get(sdev)) {
> > ! 		scsi_put_command(cmd);
> > ! 		return;		// maybe sdev_state == 
> SDEV_CANCEL, SDEV_DEL
> > ! 	}
> >   
> >   	scsi_put_command(cmd);
> >   	scsi_run_queue(q);
> > + 
> > + 	// ok to remove device now
> > + 	scsi_device_put(sdev);
> 
> This is the right idea, I think, but not necessarily the right fix.
> scsi_device_get() will fail if the device is going offline, 
> but we would
> still need to run the queues.
> 
> try this sequence instead:
> 
> get_device(&sdev->sdev_gendev);
> scsi_put_command(cmd);
> scsi_run_queue(q);
> put_device(&sdev->sdev_gendev);
> 
> James
> 
> 


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
linux-usb-devel@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

RE: oops with USB Storage on 2.6.14

[goggin, edward]

Thanks!  Here's a better one.

--- ../base/linux-2.6.14-rc4/drivers/scsi/scsi_lib.c	2005-10-10
20:19:19.000000000 -0500
+++ drivers/scsi/scsi_lib.c	2005-11-07 04:46:23.000000000 -0600
@@ -592,10 +592,17 @@ static void scsi_requeue_command(struct 
 
 void scsi_next_command(struct scsi_cmnd *cmd)
 {
-	struct request_queue *q = cmd->device->request_queue;
+	struct scsi_device *sdev = cmd->device;
+	struct request_queue *q = sdev->request_queue;
+
+	/* need to hold a reference on the device before we let go of the
cmd */
+	get_device(&sdev->sdev_gendev);
 
 	scsi_put_command(cmd);
 	scsi_run_queue(q);
+
+	/* ok to remove device now */
+	put_device(&sdev->sdev_gendev);
 }
 
 void scsi_run_host_queues(struct Scsi_Host *shost)



 

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org 
> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Rolf Eike Beer
> Sent: Tuesday, November 08, 2005 11:38 AM
> To: goggin, edward
> Cc: 'Andrew Morton'; Masanari Iida; 
> linux-kernel@vger.kernel.org; 
> linux-usb-devel@lists.sourceforge.net; linux-scsi@vger.kernel.org
> Subject: Re: oops with USB Storage on 2.6.14
> 
> Am Dienstag, 8. November 2005 17:24 schrieb goggin, edward:
> >I've run into a bug like this several times using 2.6.14-rc4 while
> >testing dm-multipath's reaction to uevents generated by forcing
> >fiber channel transport failures -- which leads to the scsi device
> >being detached and the queuedata pointer in the device's queue being
> >reset in scsi_device_dev_release.  The fix I've used is below and
> >it seems to work well for me.  I was going to place this patch on
> >dm-devel today or tomorrow anyway.
> >
> >drivers/scsi/scsi_lib.c:scsi_next_command()
> >Call scsi_device_get and scsi_device_put around the calls to
> >scsi_put_command
> >and scsi_run_queue so that the scsi host structure will not 
> be de-allocated
> >between scsi_put_command and scsi_run_queue.
> >
> >*** ../base/linux-2.6.14-rc4/drivers/scsi/scsi_lib.c	Mon Oct 
> 10 20:19:19
> >2005
> >--- drivers/scsi/scsi_lib.c	Thu Nov  3 13:30:03 2005
> >***************
> >*** 592,601 ****
> 
> Your patch is linewrapped. Also please use unified diff 
> format, good choice 
> for diff options is "-Naurp".
> 
> Eike
> 

RE: oops with USB Storage on 2.6.14

[James Bottomley]

On Tue, 2005-11-08 at 15:02 -0500, goggin, edward wrote:
> Thanks!  Here's a better one.

It's line wrapped, but I fixed that up.

James

Re: oops with USB Storage on 2.6.14

[Patrick Mansfield]

On Tue, Nov 08, 2005 at 04:08:43PM -0500, James Bottomley wrote:
> On Tue, 2005-11-08 at 15:02 -0500, goggin, edward wrote:
> > Thanks!  Here's a better one.
> 
> It's line wrapped, but I fixed that up.

What code path triggered this?

I mean we get a ref to the sdev in the upper level driver opens, scan, and
sd flush. So where are we not getting a ref? 

Shouldn't the get be done at a higher level?

-- Patrick Mansfield

Re: oops with USB Storage on 2.6.14

[James Bottomley]

On Tue, 2005-11-08 at 13:33 -0800, Patrick Mansfield wrote:
> I mean we get a ref to the sdev in the upper level driver opens, scan, and
> sd flush. So where are we not getting a ref? 
> 
> Shouldn't the get be done at a higher level?

Actually, no, because of the way we run the queues for the next command.

If this is a sd_sync_cache() or something for the last possible command
on the device, the process may have a reference to the device, but as
soon as we call end_that_request_last(), they may be racing to release
it.  The bug is triggered when we get into scsi_next_command() with us
holding the only remaining reference to the device.

James

RE: oops with USB Storage on 2.6.14

[goggin, edward]

> -----Original Message-----
> From: Patrick Mansfield [mailto:patmans@us.ibm.com] 
> Sent: Tuesday, November 08, 2005 4:33 PM
> To: James Bottomley
> Cc: goggin, edward; 'Rolf Eike Beer'; 'Andrew Morton'; 
> Masanari Iida; linux-kernel@vger.kernel.org; 
> linux-usb-devel@lists.sourceforge.net; linux-scsi@vger.kernel.org
> Subject: Re: oops with USB Storage on 2.6.14
> 
> On Tue, Nov 08, 2005 at 04:08:43PM -0500, James Bottomley wrote:
> > On Tue, 2005-11-08 at 15:02 -0500, goggin, edward wrote:
> > > Thanks!  Here's a better one.
> > 
> > It's line wrapped, but I fixed that up.
> 
> What code path triggered this?

I was testing multipath responsiveness to FC transport failures
by inducing scsi target device removal by writing into the
delete attribute of the scsi device kobject for scsi target
devices managed by multipathd.  The test is simple and involves no
user block read/write IO to the multipath mapped or target devices.
After initial multipath discovery is complete, the only IO going on
to target devices is periodic test paths, for my devices this
amounts to issuing an EVPD page 0xc0 inquiry to each target
device every 10 seconds.

My kernel call stack at the time of panic looks very similar to
the one originally reported by Masanari Iida.  I've shown
Masanari's kernel stack trace below.

> Call Trace:
>  [<c0103abf>] show_stack+0x7f/0xa0
>  [<c0103c72>] show_registers+0x162/0x1d0
>  [<c0103e90>] die+0x100/0x1a0
>  [<c039d7ae>] do_page_fault+0x31e/0x640
>  [<c0103763>] error_code+0x4f/0x54
>  [<c02b4612>] scsi_next_command+0x22/0x30
>  [<c02b473f>] scsi_end_request+0xcf/0xf0
>  [<c02b4b2e>] scsi_io_completion+0x26e/0x470
>  [<c02b4fc7>] scsi_generic_done+0x37/0x50
>  [<c02af9e5>] scsi_finish_command+0x85/0xa0
>  [<c02af89c>] scsi_softirq+0xcc/0x140
>  [<c0122085>] __do_softirq+0xd5/0xf0
>  [<c01220d8>] do_softirq+0x38/0x40
>  [<c0122685>] ksoftirqd+0x95/0xe0
>  [<c0131cfa>] kthread+0xba/0xc0
>  [<c0100ecd>] kernel_thread_helper+0x5/0x18

The scsi command being terminated by scsi_end_request is
an inquiry issued by the multipathd target device testing
thread.

> 
> I mean we get a ref to the sdev in the upper level driver 
> opens, scan, and
> sd flush. So where are we not getting a ref?

Good question.

The ref to the sdev obtained by the device scan has been
dropped by device_del() called from scsi_remove_device()
since the scsi_device has been removed via sysfs control.

The ref held by the dm open of the target device has been
closed when multipathd updates the multipath map to not
include the target device being removed.

The ref held by the multipathd initiated open of the target
device for purposes of issuing a test IO gets removed as
soon as the multipathd test thread is notified of the
completion of its test SG_IO ioctl via the scsi_end_request()
call on the inquiry request I mentioned earlier.
Soon after this point, the target device is closed by
multipathd since there is no more need for test IOs to
be issued to that target device.  Note that this is the last
ref held on the target scsi device by opens or scans and
that it is highly possible on an SMP host for this ref to be
released BEFORE the scsi_end_request() actually returns to
its soft interrupt stack.

At this point, the only refs held on the target scsi device
are from ones for active scsi commands for that device
or for an invocation of scsi_request_fn() servicing the
device's queue.  If the queue is not actively being
serviced and this is the last active command for the
device, the call to scsi_put_command() from
scsi_next_command() will free the memory for both the
scsi device and its request queue will be freed in
scsi_release_dev_release() when the device's kobject's
kref count goes to zero.

> 
> Shouldn't the get be done at a higher level?

As you can see, there are plenty of gets being done at
higher levels.

BTW, I have since reproduced this problem without
multipath at all, just two simple concurrently executing
processes -- one issues an ioctl to a scsi device
although any IO type would likely do) and closes its file
descriptor while the second one removes the device via sysfs.  
It seems like the prerequisite sequence of events are

open device
issue io to device
device gets reaped
io completes up to scsi_end_request()
device is closed
scsi_put_command() reduces device kref count to zero, device is freed
scsi_next_command() can reference freed scsi device memory

> 
> -- Patrick Mansfield
> 


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
linux-usb-devel@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel