2.6.17-mm5 -- netconsole failed to send full trace

2.6.17-mm5 -- netconsole failed to send full trace

[Miles Lane]

I get a system lockup on my laptop every time I remove my Linksys USB
10/100 Ethernet adapter.  Unfortunately, my laptop has no serial port,
so debugging this kernel is difficult.  I tried netconsole tonight,
but only got:

BUG: unable to handle kernel paging request at virtual address 6b6b6b6b
 printing eip:
c101a44d
*pde = 00000000
Oops: 0000 [#1]
4K_STACKS PREEMPT
last sysfs file: /block/hda/hda9/stat
Modules linked in: netconsole binfmt_misc i915 drm ipv6
speedstep_centrino cpufreq_powersave cpufreq_performance
cpufreq_conservative video thermal button nls_ascii nls_cp437 vfat fat
nls_utf8 ntfs nls_base md_mod sr_mod sbp2 scsi_mod parport_pc lp
parport rtl8150 snd_intel8x0 snd_ac97_codec snd_ac97_bus snd_pcm_oss
snd_mixer_oss snd_pcm snd_timer snd evdev soundcore ipw2200
intel_agpc3 89 e5 53 fa 61 0b 01 1d 14 c1 c7 14 3c 00 00 00 b0 1a 00
eb <8b> 8b 43 85 75

What should I try now?  I somehow doubt that I can make the kernel
send info to a USB tty port.

Thanks,
         Miles

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Andrew Morton]

On Mon, 3 Jul 2006 01:31:36 -0700
"Miles Lane" <miles.lane@gmail.com> wrote:

> I get a system lockup on my laptop every time I remove my Linksys USB
> 10/100 Ethernet adapter.  Unfortunately, my laptop has no serial port,
> so debugging this kernel is difficult.  I tried netconsole tonight,
> but only got:
> 
> BUG: unable to handle kernel paging request at virtual address 6b6b6b6b

Nice.

>  printing eip:
> c101a44d
> *pde = 00000000
> Oops: 0000 [#1]
> 4K_STACKS PREEMPT
> last sysfs file: /block/hda/hda9/stat
> Modules linked in: netconsole binfmt_misc i915 drm ipv6
> speedstep_centrino cpufreq_powersave cpufreq_performance
> cpufreq_conservative video thermal button nls_ascii nls_cp437 vfat fat
> nls_utf8 ntfs nls_base md_mod sr_mod sbp2 scsi_mod parport_pc lp
> parport rtl8150 snd_intel8x0 snd_ac97_codec snd_ac97_bus snd_pcm_oss
> snd_mixer_oss snd_pcm snd_timer snd evdev soundcore ipw2200
> intel_agpc3 89 e5 53 fa 61 0b 01 1d 14 c1 c7 14 3c 00 00 00 b0 1a 00
> eb <8b> 8b 43 85 75
> 
> What should I try now?  I somehow doubt that I can make the kernel
> send info to a USB tty port.
> 

Can you get it to happen on the VGA console, make a record of the display? 
(Digital photo is good).

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Andrew Morton]

On Mon, 3 Jul 2006 07:04:05 -0700
"Miles Lane" <miles.lane@gmail.com> wrote:

> On 7/3/06, Andrew Morton <akpm@osdl.org> wrote:
> > On Mon, 3 Jul 2006 01:31:36 -0700
> > "Miles Lane" <miles.lane@gmail.com> wrote:
> >
> > > I get a system lockup on my laptop every time I remove my Linksys USB
> > > 10/100 Ethernet adapter.  Unfortunately, my laptop has no serial port,
> > > so debugging this kernel is difficult.  I tried netconsole tonight,
> > > but only got:
> > >
> > > BUG: unable to handle kernel paging request at virtual address 6b6b6b6b
> >
> > Nice.
> >
> > >  printing eip:
> > > c101a44d
> > > *pde = 00000000
> > > Oops: 0000 [#1]
> > > 4K_STACKS PREEMPT
> > > last sysfs file: /block/hda/hda9/stat
> > > Modules linked in: netconsole binfmt_misc i915 drm ipv6
> > > speedstep_centrino cpufreq_powersave cpufreq_performance
> > > cpufreq_conservative video thermal button nls_ascii nls_cp437 vfat fat
> > > nls_utf8 ntfs nls_base md_mod sr_mod sbp2 scsi_mod parport_pc lp
> > > parport rtl8150 snd_intel8x0 snd_ac97_codec snd_ac97_bus snd_pcm_oss
> > > snd_mixer_oss snd_pcm snd_timer snd evdev soundcore ipw2200
> > > intel_agpc3 89 e5 53 fa 61 0b 01 1d 14 c1 c7 14 3c 00 00 00 b0 1a 00
> > > eb <8b> 8b 43 85 75
> > >
> > > What should I try now?  I somehow doubt that I can make the kernel
> > > send info to a USB tty port.
> > >
> >
> > Can you get it to happen on the VGA console, make a record of the display?
> > (Digital photo is good).
> >
> >
> 

OK, thanks.  I uploaded that to
http://www.zip.com.au/~akpm/linux/patches/stuff/00003.jpg.

So we have a use-after-free in tasklet_action(), as a consequence of
unplugging a USB ethernet adapter.

The post-2.6.17 changes in drivers/usb/net/ are relatively modest, but
certainly enough to cause lifetime problems.  Do you know whether mainline
has the same bug?  And are you able to identify at which kernel version
this started to happen?

Thanks.

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[David Brownell]

On Tuesday 04 July 2006 10:22 pm, Miles Lane wrote:

> > So we have a use-after-free in tasklet_action(), as a consequence of
> > unplugging a USB ethernet adapter.
> 
> So far, all the kernels have crashed (back to Ubuntu's 2.6.15). 

Erm, exactly which USB ethernet adapter?  That would seem to be a
critical bit of info that's somehow been omitted...

If it's the rtl8150 driver, that would be Petko's ...

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Miles Lane]

On 7/4/06, David Brownell <david-b@pacbell.net> wrote:
> On Tuesday 04 July 2006 10:22 pm, Miles Lane wrote:
>
> > > So we have a use-after-free in tasklet_action(), as a consequence of
> > > unplugging a USB ethernet adapter.
> >
> > So far, all the kernels have crashed (back to Ubuntu's 2.6.15).
>
> Erm, exactly which USB ethernet adapter?  That would seem to be a
> critical bit of info that's somehow been omitted...
>
> If it's the rtl8150 driver, that would be Petko's ...

Linksys EtherFast 10/100 Compact Network Adapter (model USB100M).
Yes, the rtl8150 driver loads when I insert the adapter.

Thanks,
         Miles

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Miles Lane]

On 7/4/06, Miles Lane <miles.lane@gmail.com> wrote:
> On 7/4/06, David Brownell <david-b@pacbell.net> wrote:
> > On Tuesday 04 July 2006 10:22 pm, Miles Lane wrote:
> >
> > > > So we have a use-after-free in tasklet_action(), as a consequence of
> > > > unplugging a USB ethernet adapter.
> > >
> > > So far, all the kernels have crashed (back to Ubuntu's 2.6.15).
> >
> > Erm, exactly which USB ethernet adapter?  That would seem to be a
> > critical bit of info that's somehow been omitted...
> >
> > If it's the rtl8150 driver, that would be Petko's ...
>
> Linksys EtherFast 10/100 Compact Network Adapter (model USB100M).
> Yes, the rtl8150 driver loads when I insert the adapter.

Can someone tell me when the udev/hal support went into the kernel?
I tried compiling a 2.6.12, but when I booted it, Ubuntu 6.06 couldn't
run udev/hal, so my devices didn't get configured.  I loaded rtl8150
using modprobe, but no ethX device was created, so I couldn't test the
card.  I am attempting to determine how far back in the kernel history
I have to go to have this adapter not crash the system when I remove it.
It didn't crash 2.6.12, but it doesn't count if the driver isn't successfully
associated with the hardware, I suspect.

Thanks,
        Miles

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[David Brownell]

On Wednesday 05 July 2006 9:39 am, Miles Lane wrote:
> On 7/4/06, Miles Lane <miles.lane@gmail.com> wrote:
> > On 7/4/06, David Brownell <david-b@pacbell.net> wrote:
> > > On Tuesday 04 July 2006 10:22 pm, Miles Lane wrote:
> > >
> > > > > So we have a use-after-free in tasklet_action(), as a consequence of
> > > > > unplugging a USB ethernet adapter.
> > > >
> > > > So far, all the kernels have crashed (back to Ubuntu's 2.6.15).
> > >
> > > Erm, exactly which USB ethernet adapter?  That would seem to be a
> > > critical bit of info that's somehow been omitted...
> > >
> > > If it's the rtl8150 driver, that would be Petko's ...
> >
> > Linksys EtherFast 10/100 Compact Network Adapter (model USB100M).
> > Yes, the rtl8150 driver loads when I insert the adapter.

So you should contact the maintainer of that driver, yes?


> Can someone tell me when the udev/hal support went into the kernel?

Older versions of udev may not work with newer kernels, and vice versa;
I've never taken HAL apart.

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Miles Lane]

Hi Petko,

David Brownell pointed out that you are the author of this driver (rtl8150).
My laptop is crashing every time I remove the Linksys EtherFast 10/100
Compact Network Adapter (model USB100M) from the USB port.

Here's a link to the discussion thus far:
http://groups.google.com/group/linux.kernel/tree/browse_frm/thread/8c93e310c7b71242/a8a1e3edb1601906?rnum=1&q=miles+lane&_done=%2Fgroup%2Flinux.kernel%2Fbrowse_frm%2Fthread%2F8c93e310c7b71242%2Fc8a8ba47c49c39fc%3Ftvc%3D1%26q%3Dmiles+lane%26#doc_a8a1e3edb1601906

Here's the stacktrace:
http://www.zip.com.au/~akpm/linux/patches/stuff/00003.jpg

I have reproduced the bug with vanilla 2.6.17.  I am currently working my
back through kernel versions to try to isolate the responsible patches.

         Miles

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Andrew Morton]

On Wed, 5 Jul 2006 19:34:52 -0700
"Miles Lane" <miles.lane@gmail.com> wrote:

> On 7/5/06, Miles Lane <miles.lane@gmail.com> wrote:
> > Hi Petko,
> >
> > David Brownell pointed out that you are the author of this driver (rtl8150).
> > My laptop is crashing every time I remove the Linksys EtherFast 10/100
> > Compact Network Adapter (model USB100M) from the USB port.
> >
> > Here's a link to the discussion thus far:
> > http://groups.google.com/group/linux.kernel/tree/browse_frm/thread/8c93e310c7b71242/a8a1e3edb1601906?rnum=1&q=miles+lane&_done=%2Fgroup%2Flinux.kernel%2Fbrowse_frm%2Fthread%2F8c93e310c7b71242%2Fc8a8ba47c49c39fc%3Ftvc%3D1%26q%3Dmiles+lane%26#doc_a8a1e3edb1601906
> >
> > Here's the stacktrace:
> > http://www.zip.com.au/~akpm/linux/patches/stuff/00003.jpg
> >
> > I have reproduced the bug with vanilla 2.6.17.  I am currently working my
> > back through kernel versions to try to isolate the responsible patches.
> 
> 2.6.15 is the first kernel earliest kernel that seems to work with Ubuntu 6.06's
> implementation of hal / udev / dbus.  It does set up the adapter successfully.
> 
> I was able to reproduce the crash with 2.6.15.  I have attached a screenshot
> of the stacktrace.  It may help, since it differs quite a bit from the one for
> 2.6.17-mm5.

The attachment will be too large to make it onto most mailing lists.  I put
a copy here: http://www.zip.com.au/~akpm/linux/patches/stuff/00005.jpg

> BTW, should I join linux-usb-devel and CC that list?  Also, should I take
> this discussion off of LKML?

Nah, spread it around.  Who knows, somoene might actually fix the bug ;)

(cc added)

(Summary: oops in tasklet_action after hot-unplugging a usbnet-driven adapter)

> Here's the detection of the RTL8150:
> 
> usb 1-2: new full speed USB device using uhci_hcd and address 3
> usb 1-2: ep0 maxpacket = 8
> usb 1-2: default language 0x0409
> usb 1-2: new device strings: Mfr=1, Product=2, SerialNumber=3
> usb 1-2: Product: Linksys USB LAN Adapter
> usb 1-2: Manufacturer: Linksys
> usb 1-2: SerialNumber: 5322
> usb 1-2: hotplug
> usb 1-2: adding 1-2:1.0 (config #1, interface 0)
> usb 1-2:1.0: hotplug
> hub 1-0:1.0: state 5 ports 2 chg 0000 evt 0004
> eth1: RealTek RTL8139 at 0xf9060800, 00:c0:9f:95:18:1b, IRQ 10
> eth1:  Identified 8139 chip type 'RTL-8100B/8139D'
> 
> Bus 002 Device 002: ID 0bda:8150 Realtek Semiconductor Corp. RTL8150
> Fast Ethernet Adapter
> Device Descriptor:
>   bLength                18
>   bDescriptorType         1
>   bcdUSB               1.10
>   bDeviceClass            0 (Defined at Interface level)
>   bDeviceSubClass         0
>   bDeviceProtocol         0
>   bMaxPacketSize0         8
>   idVendor           0x0bda Realtek Semiconductor Corp.
>   idProduct          0x8150 RTL8150 Fast Ethernet Adapter
>   bcdDevice            1.00
>   iManufacturer           1 Linksys
>   iProduct                2 Linksys USB LAN Adapter
>   iSerial                 3 5322
>   bNumConfigurations      1
>   Configuration Descriptor:
>     bLength                 9
>     bDescriptorType         2
>     wTotalLength           39
>     bNumInterfaces          1
>     bConfigurationValue     1
>     iConfiguration          0
>     bmAttributes         0xa0
>       (Bus Powered)
>       Remote Wakeup
>     MaxPower              120mA
>     Interface Descriptor:
>       bLength                 9
>       bDescriptorType         4
>       bInterfaceNumber        0
>       bAlternateSetting       0
>       bNumEndpoints           3
>       bInterfaceClass       255 Vendor Specific Class
>       bInterfaceSubClass      0
>       bInterfaceProtocol    255
>       iInterface              0
>       Endpoint Descriptor:
>         bLength                 7
>         bDescriptorType         5
>         bEndpointAddress     0x81  EP 1 IN
>         bmAttributes            2
>           Transfer Type            Bulk
>           Synch Type               None
>           Usage Type               Data
>         wMaxPacketSize     0x0040  1x 64 bytes
>         bInterval               0
>       Endpoint Descriptor:
>         bLength                 7
>         bDescriptorType         5
>         bEndpointAddress     0x02  EP 2 OUT
>         bmAttributes            2
>           Transfer Type            Bulk
>           Synch Type               None
>           Usage Type               Data
>         wMaxPacketSize     0x0040  1x 64 bytes
>         bInterval               0
>       Endpoint Descriptor:
>         bLength                 7
>         bDescriptorType         5
>         bEndpointAddress     0x83  EP 3 IN
>         bmAttributes            3
>           Transfer Type            Interrupt
>           Synch Type               None
>           Usage Type               Data
>         wMaxPacketSize     0x0008  1x 8 bytes
>         bInterval               1
> Device Status:     0x0000
>   (Bus Powered)
> 

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Andrew Morton]

On Wed, 5 Jul 2006 19:42:29 -0700
Andrew Morton <akpm@osdl.org> wrote:

> On Wed, 5 Jul 2006 19:34:52 -0700
> "Miles Lane" <miles.lane@gmail.com> wrote:
> 
> > On 7/5/06, Miles Lane <miles.lane@gmail.com> wrote:
> > > Hi Petko,
> > >
> > > David Brownell pointed out that you are the author of this driver (rtl8150).
> > > My laptop is crashing every time I remove the Linksys EtherFast 10/100
> > > Compact Network Adapter (model USB100M) from the USB port.
> > >
> > > Here's a link to the discussion thus far:
> > > http://groups.google.com/group/linux.kernel/tree/browse_frm/thread/8c93e310c7b71242/a8a1e3edb1601906?rnum=1&q=miles+lane&_done=%2Fgroup%2Flinux.kernel%2Fbrowse_frm%2Fthread%2F8c93e310c7b71242%2Fc8a8ba47c49c39fc%3Ftvc%3D1%26q%3Dmiles+lane%26#doc_a8a1e3edb1601906
> > >
> > > Here's the stacktrace:
> > > http://www.zip.com.au/~akpm/linux/patches/stuff/00003.jpg
> > >
> > > I have reproduced the bug with vanilla 2.6.17.  I am currently working my
> > > back through kernel versions to try to isolate the responsible patches.
> > 
> > 2.6.15 is the first kernel earliest kernel that seems to work with Ubuntu 6.06's
> > implementation of hal / udev / dbus.  It does set up the adapter successfully.
> > 
> > I was able to reproduce the crash with 2.6.15.  I have attached a screenshot
> > of the stacktrace.  It may help, since it differs quite a bit from the one for
> > 2.6.17-mm5.
> 
> The attachment will be too large to make it onto most mailing lists.  I put
> a copy here: http://www.zip.com.au/~akpm/linux/patches/stuff/00005.jpg
> 
> > BTW, should I join linux-usb-devel and CC that list?  Also, should I take
> > this discussion off of LKML?
> 
> Nah, spread it around.  Who knows, somoene might actually fix the bug ;)

I don't suppose it's this easy?

--- a/drivers/usb/net/rtl8150.c~a
+++ a/drivers/usb/net/rtl8150.c
@@ -909,6 +909,7 @@ static void rtl8150_disconnect(struct us
 	usb_set_intfdata(intf, NULL);
 	if (dev) {
 		set_bit(RTL8150_UNPLUG, &dev->flags);
+		tasklet_kill(&dev->tl);
 		tasklet_disable(&dev->tl);
 		unregister_netdev(dev->netdev);
 		unlink_all_urbs(dev);
_

Re: 2.6.17-mm5 -- netconsole failed to send full trace

[Andrew Morton]

On Wed, 5 Jul 2006 19:46:14 -0700
Andrew Morton <akpm@osdl.org> wrote:

> On Wed, 5 Jul 2006 19:42:29 -0700
> Andrew Morton <akpm@osdl.org> wrote:
> 
> > On Wed, 5 Jul 2006 19:34:52 -0700
> > "Miles Lane" <miles.lane@gmail.com> wrote:
> > 
> > > On 7/5/06, Miles Lane <miles.lane@gmail.com> wrote:
> > > > Hi Petko,
> > > >
> > > > David Brownell pointed out that you are the author of this driver (rtl8150).
> > > > My laptop is crashing every time I remove the Linksys EtherFast 10/100
> > > > Compact Network Adapter (model USB100M) from the USB port.
> > > >
> > > > Here's a link to the discussion thus far:
> > > > http://groups.google.com/group/linux.kernel/tree/browse_frm/thread/8c93e310c7b71242/a8a1e3edb1601906?rnum=1&q=miles+lane&_done=%2Fgroup%2Flinux.kernel%2Fbrowse_frm%2Fthread%2F8c93e310c7b71242%2Fc8a8ba47c49c39fc%3Ftvc%3D1%26q%3Dmiles+lane%26#doc_a8a1e3edb1601906
> > > >
> > > > Here's the stacktrace:
> > > > http://www.zip.com.au/~akpm/linux/patches/stuff/00003.jpg
> > > >
> > > > I have reproduced the bug with vanilla 2.6.17.  I am currently working my
> > > > back through kernel versions to try to isolate the responsible patches.
> > > 
> > > 2.6.15 is the first kernel earliest kernel that seems to work with Ubuntu 6.06's
> > > implementation of hal / udev / dbus.  It does set up the adapter successfully.
> > > 
> > > I was able to reproduce the crash with 2.6.15.  I have attached a screenshot
> > > of the stacktrace.  It may help, since it differs quite a bit from the one for
> > > 2.6.17-mm5.
> > 
> > The attachment will be too large to make it onto most mailing lists.  I put
> > a copy here: http://www.zip.com.au/~akpm/linux/patches/stuff/00005.jpg
> > 
> > > BTW, should I join linux-usb-devel and CC that list?  Also, should I take
> > > this discussion off of LKML?
> > 
> > Nah, spread it around.  Who knows, somoene might actually fix the bug ;)
> 
> I don't suppose it's this easy?
> 
> --- a/drivers/usb/net/rtl8150.c~a
> +++ a/drivers/usb/net/rtl8150.c
> @@ -909,6 +909,7 @@ static void rtl8150_disconnect(struct us
>  	usb_set_intfdata(intf, NULL);
>  	if (dev) {
>  		set_bit(RTL8150_UNPLUG, &dev->flags);
> +		tasklet_kill(&dev->tl);
>  		tasklet_disable(&dev->tl);
>  		unregister_netdev(dev->netdev);
>  		unlink_all_urbs(dev);
> _
> 

Better:

--- a/drivers/usb/net/rtl8150.c~a
+++ a/drivers/usb/net/rtl8150.c
@@ -910,6 +910,7 @@ static void rtl8150_disconnect(struct us
 	if (dev) {
 		set_bit(RTL8150_UNPLUG, &dev->flags);
 		tasklet_disable(&dev->tl);
+		tasklet_kill(&dev->tl);
 		unregister_netdev(dev->netdev);
 		unlink_all_urbs(dev);
 		free_all_urbs(dev);
_

But I suspect it won't help.