Re: ext3-related crash in 2.6.20-rc1

Re: ext3-related crash in 2.6.20-rc1

[Andrew Morton]

On Sun, 24 Dec 2006 00:43:05 +0100
Pavel Machek <pavel@ucw.cz> wrote:

> Hi!
> 
> I got this nasty oops while playing with debugger. Not sure if that is
> related; it also might be something with bluetooth; I already know it
> corrupts memory during suspend, perhaps it corrupts memory in some
> error path?
> 
>  
> 
> 								Pavel
> 
> 
> l2cap_recv_acldata: Unexpected continuation frame (len 0)
> l2cap_recv_acldata: Unexpected continuation frame (len 0)
> l2cap_recv_acldata: Unexpected continuation frame (len 0)
> l2cap_recv_acldata: Unexpected continuation frame (len 0)
> l2cap_recv_acldata: Unexpected continuation frame (len 0)
> l2cap_recv_acldata: Unexpected continuation frame (len 0)
> l2cap_recv_acldata: Unexpected continuation frame (len 0)
> PM: Removing info for bluetooth:acl00803715A329
> e1000: eth0: e1000_watchdog: NIC Link is Down
> e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex
> e1000: eth0: e1000_watchdog: 10/100 speed: disabling TSO
> e1000: eth0: e1000_watchdog: NIC Link is Down
> e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex
> e1000: eth0: e1000_watchdog: 10/100 speed: disabling TSO
> ------------[ cut here ]------------
> kernel BUG at fs/buffer.c:1235!

get thee to fs/buffer.c:1235.  You'll see that someone somewhere forgot to
reenable local interrupts.

Were you using gdb at the time?  A fix for something like that was merged
into mainline yesterday.

The slab errors which you're reporting in later emails will almost surely
be unrelated to this.

Re: bluetooth memory corruption (was Re: ext3-related crash in 2.6.20-rc1)

[Andrew Morton]

On Sun, 24 Dec 2006 00:55:01 +0100
Pavel Machek <pavel@ucw.cz> wrote:

> PM: Removing info for No Bus:usbdev3.15_ep81
> PM: Removing info for No Bus:usbdev3.15_ep82
> PM: Removing info for No Bus:usbdev3.15_ep02
> slab error in verify_redzone_free(): cache `size-512': memory outside object was overwritten
>  [<c016a1b8>] cache_free_debugcheck+0x128/0x1d0
>  [<c04b58e3>] hci_usb_close+0xf3/0x160
>  [<c016b530>] kfree+0x50/0xa0
>  [<c04b58e3>] hci_usb_close+0xf3/0x160
>  [<c04b59be>] hci_usb_disconnect+0x2e/0x90
>  [<c0454f23>] usb_disable_interface+0x53/0x70
>  [<c04576f8>] usb_unbind_interface+0x38/0x80
>  [<c032f908>] __device_release_driver+0x68/0xb0
>  [<c032fc3e>] device_release_driver+0x1e/0x40
>  [<c032f1db>] bus_remove_device+0x8b/0xa0
>  [<c032dbc9>] device_del+0x159/0x1c0
>  [<c04559ad>] usb_disable_device+0x4d/0x100
>  [<c044fe8a>] usb_disconnect+0x9a/0x110
>  [<c0452405>] hub_thread+0x355/0xbd0
>  [<c061426e>] schedule+0x2de/0x8f0
>  [<c013c640>] autoremove_wake_function+0x0/0x50
>  [<c04520b0>] hub_thread+0x0/0xbd0
>  [<c013c58c>] kthread+0xec/0xf0
>  [<c013c4a0>] kthread+0x0/0xf0
>  [<c0103be7>] kernel_thread_helper+0x7/0x10
>  =======================

yes, this one looks like memory scribblage in bluetooth.  The
buffer.c assertion failure should now be fixed, please verify.

Re: ptrace() memory corruption?

[Andrew Morton]

On Sun, 24 Dec 2006 12:51:16 +0059
Jiri Slaby <jirislaby@gmail.com> wrote:

> [   58.674780] EFLAGS: 00010046   (2.6.20-rc1-mm1 #207)

please, test 2.6.20-rc2.  We applied a fix for this.

Re: ptrace() memory corruption?

[Jiri Slaby]

Andrew Morton wrote:
> On Sun, 24 Dec 2006 12:51:16 +0059
> Jiri Slaby <jirislaby@gmail.com> wrote:
> 
>> [   58.674780] EFLAGS: 00010046   (2.6.20-rc1-mm1 #207)
> 
> please, test 2.6.20-rc2.  We applied a fix for this.

It's working now.

thanks,
-- 
http://www.fi.muni.cz/~xslaby/            Jiri Slaby
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Re: [Bluez-devel] bluetooth memory corruption (was Re: ext3-related crash in 2.6.20-rc1)

[Marcel Holtmann]

Hi Pavel,

> > I got this nasty oops while playing with debugger. Not sure if that is
> > related; it also might be something with bluetooth; I already know it
> > corrupts memory during suspend, perhaps it corrupts memory in some
> > error path?
> 
> Okay, I spoke too soon. bluetooth & suspend memory corruption was
> _way_ harder to reproduce than expected. Took me 5-or-so-suspend
> cycles... so it is probably unrelated to the previous crash.

can you try to reproduce this with 2.6.20-rc2 as well.

Regards

Marcel



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

Re: bluetooth memory corruption (was Re: ext3-related crash in 2.6.20-rc1)

[Pavel Machek]

Hi!

> > PM: Removing info for No Bus:usbdev3.15_ep81
> > PM: Removing info for No Bus:usbdev3.15_ep82
> > PM: Removing info for No Bus:usbdev3.15_ep02
> > slab error in verify_redzone_free(): cache `size-512': memory outside object was overwritten
> >  [<c016a1b8>] cache_free_debugcheck+0x128/0x1d0
> >  [<c04b58e3>] hci_usb_close+0xf3/0x160
> >  [<c016b530>] kfree+0x50/0xa0
> >  [<c04b58e3>] hci_usb_close+0xf3/0x160
> >  [<c04b59be>] hci_usb_disconnect+0x2e/0x90
> >  [<c0454f23>] usb_disable_interface+0x53/0x70
> >  [<c04576f8>] usb_unbind_interface+0x38/0x80
> >  [<c032f908>] __device_release_driver+0x68/0xb0
> >  [<c032fc3e>] device_release_driver+0x1e/0x40
> >  [<c032f1db>] bus_remove_device+0x8b/0xa0
> >  [<c032dbc9>] device_del+0x159/0x1c0
> >  [<c04559ad>] usb_disable_device+0x4d/0x100
> >  [<c044fe8a>] usb_disconnect+0x9a/0x110
> >  [<c0452405>] hub_thread+0x355/0xbd0
> >  [<c061426e>] schedule+0x2de/0x8f0
> >  [<c013c640>] autoremove_wake_function+0x0/0x50
> >  [<c04520b0>] hub_thread+0x0/0xbd0
> >  [<c013c58c>] kthread+0xec/0xf0
> >  [<c013c4a0>] kthread+0x0/0xf0
> >  [<c0103be7>] kernel_thread_helper+0x7/0x10
> >  =======================
> 
> yes, this one looks like memory scribblage in bluetooth.  The
> buffer.c assertion failure should now be fixed, please verify.

I can confirm buffer.c assertion to be fixed (yes, I was using gdb at
that time).
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [Bluez-devel] bluetooth memory corruption (was Re: ext3-related crash in 2.6.20-rc1)

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 7401 bytes --]

On Sun 2006-12-24 15:39:23, Marcel Holtmann wrote:
> Hi Pavel,
> 
> > > I got this nasty oops while playing with debugger. Not sure if that is
> > > related; it also might be something with bluetooth; I already know it
> > > corrupts memory during suspend, perhaps it corrupts memory in some
> > > error path?
> > 
> > Okay, I spoke too soon. bluetooth & suspend memory corruption was
> > _way_ harder to reproduce than expected. Took me 5-or-so-suspend
> > cycles... so it is probably unrelated to the previous crash.
> 
> can you try to reproduce this with 2.6.20-rc2 as well.

Yep, here it is, reproduced on 6-th-or-so suspend.

bluetooth may need to be actively used in order for this to trigger;
connecting to the net over my cellphone seems to work okay.

(Full logs in attachment).

								Pavel

Linux version 2.6.20-rc2 (pavel@amd) (gcc version 4.0.4 20060507
(prerelease) (Debian 4.0.3-3)) #383 SMP Fri Dec 22 11:30:05 CET 2006
...
system 00:00: resuming
pnp 00:01: resuming
system 00:02: resuming
pnp 00:03: resuming
pnp 00:04: resuming
pnp 00:05: resuming
pnp 00:06: resuming
pnp 00:07: resuming
i8042 kbd 00:08: resuming
pnp: Device 00:08 does not support activation.
i8042 aux 00:09: resuming
pnp: Device 00:09 does not support activation.
pnp 00:0a: resuming
pnp 00:0b: resuming
platform bluetooth: resuming
pcspkr pcspkr: resuming
vesafb vesafb.0: resuming
serial8250 serial8250: resuming
usb usb1: resuming
usb usb3: resuming
ata2: SATA link down (SStatus 0 SControl 0)
ata3: SATA link down (SStatus 0 SControl 0)
ata4: SATA link down (SStatus 0 SControl 0)
hub 1-0:1.0: resuming
hub 3-0:1.0: resuming
i8042 i8042: resuming
atkbd serio0: resuming
psmouse serio1: resuming
usb usb4: resuming
usb usb5: resuming
hub 4-0:1.0: resuming
hub 5-0:1.0: resuming
usb usb2: resuming
hub 2-0:1.0: resuming
mmcblk mmc0:cc53: resuming
sd 0:0:0:0: resuming
usb 3-2: resuming
 usbdev3.8_ep00: PM: resume from 0, parent 3-2 still 2
usb 3-2:1.0: PM: resume from 2, parent 3-2 still 2
usb 3-2:1.0: resuming
 usbdev3.8_ep81: PM: resume from 0, parent 3-2:1.0 still 2
 usbdev3.8_ep02: PM: resume from 0, parent 3-2:1.0 still 2
 usbdev3.8_ep83: PM: resume from 0, parent 3-2:1.0 still 2
usb 3-1: resuming
 usbdev3.9_ep00: PM: resume from 0, parent 3-1 still 2
hci_usb 3-1:1.0: PM: resume from 2, parent 3-1 still 2
hci_usb 3-1:1.0: resuming
 hci0: PM: resume from 0, parent 3-1:1.0 still 2
 usbdev3.9_ep81: PM: resume from 0, parent 3-1:1.0 still 2
 usbdev3.9_ep82: PM: resume from 0, parent 3-1:1.0 still 2
 usbdev3.9_ep02: PM: resume from 0, parent 3-1:1.0 still 2
hci_usb 3-1:1.1: PM: resume from 2, parent 3-1 still 2
hci_usb 3-1:1.1: resuming
 usbdev3.9_ep83: PM: resume from 0, parent 3-1:1.1 still 2
 usbdev3.9_ep03: PM: resume from 0, parent 3-1:1.1 still 2
usb 3-1:1.2: PM: resume from 2, parent 3-1 still 2
usb 3-1:1.2: resuming
 usbdev3.9_ep84: PM: resume from 0, parent 3-1:1.2 still 2
 usbdev3.9_ep04: PM: resume from 0, parent 3-1:1.2 still 2
usb 3-1:1.3: PM: resume from 2, parent 3-1 still 2
usb 3-1:1.3: resuming
Restarting tasks ... <3>__tx_submit: hci0 tx submit failed urb f765d1bc type 2 err -19
usb 3-1: USB disconnect, address 9
PM: Removing info for No Bus:usbdev3.9_ep81
PM: Removing info for No Bus:usbdev3.9_ep82
PM: Removing info for No Bus:usbdev3.9_ep02
slab error in verify_redzone_free(): cache `size-512': memory outside object was overwritten
 [<c016a298>] cache_free_debugcheck+0x128/0x1d0
 [<c04b08d3>] hci_usb_close+0xf3/0x160
 [<c016b610>] kfree+0x50/0xa0
 [<c04b08d3>] hci_usb_close+0xf3/0x160
 [<c04b09ae>] hci_usb_disconnect+0x2e/0x90
 [<c044fed3>] usb_disable_interface+0x53/0x70
 [<c04526a8>] usb_unbind_interface+0x38/0x80
 [<c032a8b8>] __device_release_driver+0x68/0xb0
 [<c032abee>] device_release_driver+0x1e/0x40
 [<c032a18b>] bus_remove_device+0x8b/0xa0
 [<c0328b79>] device_del+0x159/0x1c0
 [<c045095d>] usb_disable_device+0x4d/0x100
 [<c044ae3a>] usb_disconnect+0x9a/0x110
 [<c044d3b5>] hub_thread+0x355/0xbd0
 [<c060f53e>] schedule+0x2de/0x8f0
 [<c013c680>] autoremove_wake_function+0x0/0x50
 [<c044d060>] hub_thread+0x0/0xbd0
 [<c013c5cc>] kthread+0xec/0xf0
 [<c013c4e0>] kthread+0x0/0xf0
 [<c0103be7>] kernel_thread_helper+0x7/0x10
 =======================
f70a2720: redzone 1:0x5a5a5a5a, redzone 2:0xc0545e9e.
------------[ cut here ]------------
kernel BUG at mm/slab.c:2878!
invalid opcode: 0000 [#1]
SMP 
Modules linked in:
CPU:    0
EIP:    0060:[<c016a322>]    Not tainted VLI
EFLAGS: 00010012   (2.6.20-rc2 #383)
EIP is at cache_free_debugcheck+0x1b2/0x1d0
eax: f70a271c   ebx: f70a20f8   ecx: 00052c00   edx: 0000020c
esi: c20df680   edi: f70a2720   ebp: 5a5a5a5a   esp: c2313e30
ds: 007b   es: 007b   ss: 0068
Process khubd (pid: 304, ti=c2312000 task=c2257030 task.ti=c2312000)
Stack: c06aedf0 f70a2720 5a5a5a5a c0545e9e c04b08d3 f70a20c0 c20df680 c20d9164 
       f70a2724 00000286 c016b610 f653e8d8 f653e8c4 c2134ba0 0000000c c04b08d3 
       c2134b5c c2134b8c f62e0a54 c2134ad0 00000001 c2134ad0 f62e0a54 c07dbee0 
Call Trace:
 [<c0545e9e>] sock_alloc_send_skb+0x16e/0x1c0
 [<c04b08d3>] hci_usb_close+0xf3/0x160
 [<c016b610>] kfree+0x50/0xa0
 [<c04b08d3>] hci_usb_close+0xf3/0x160
 [<c04b09ae>] hci_usb_disconnect+0x2e/0x90
 [<c044fed3>] usb_disable_interface+0x53/0x70
 [<c04526a8>] usb_unbind_interface+0x38/0x80
 [<c032a8b8>] __device_release_driver+0x68/0xb0
 [<c032abee>] device_release_driver+0x1e/0x40
 [<c032a18b>] bus_remove_device+0x8b/0xa0
 [<c0328b79>] device_del+0x159/0x1c0
 [<c045095d>] usb_disable_device+0x4d/0x100
 [<c044ae3a>] usb_disconnect+0x9a/0x110
 [<c044d3b5>] hub_thread+0x355/0xbd0
 [<c060f53e>] schedule+0x2de/0x8f0
 [<c013c680>] autoremove_wake_function+0x0/0x50
 [<c044d060>] hub_thread+0x0/0xbd0
 [<c013c5cc>] kthread+0xec/0xf0
 [<c013c4e0>] kthread+0x0/0xf0
 [<c0103be7>] kernel_thread_helper+0x7/0x10
 =======================
Code: f0 2c 5a 75 8b b9 05 df 6a c0 89 f2 b8 88 98 61 c0 e8 73 f4 ff ff eb 89 81 fb a5 c2 0f 17 0f 85 6c ff ff ff 90 8d 74 26 00 eb 8e <0f> 0b eb fe 0f 0b eb fe 8d b6 00 00 00 00 0f 0b eb fe 8b 52 0c 
EIP: [<c016a322>] cache_free_debugcheck+0x1b2/0x1d0 SS:ESP 0068:c2313e30
 <7>PM: Adding info for No Bus:vcs63
PM: Adding info for No Bus:vcsa63
PM: Removing info for No Bus:vcs63
PM: Removing info for No Bus:vcsa63
done.
Enabling non-boot CPUs ...
SMP alternatives: switching to SMP code
Booting processor 1/1 eip 3000
Initializing CPU#1
Calibrating delay using timer specific routine.. 3657.64 BogoMIPS (lpj=18288234)
CPU: After generic identify, caps: bfe9fbff 00100000 00000000 00000000 0000c1a9 00000000 00000000
monitor/mwait feature present.
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU: After all inits, caps: bfe9fbff 00100000 00000000 00002940 0000c1a9 00000000 00000000
CPU1: Intel Genuine Intel(R) CPU           T2400  @ 1.83GHz stepping 08
PM: Adding info for No Bus:msr1
CPU1 is up
ata1: waiting for device to spin up (8 secs)
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex
e1000: eth0: e1000_watchdog: 10/100 speed: disabling TSO
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.00: configured for UDMA/100
SCSI device sda: 117210240 512-byte hdwr sectors (60012 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: oops4.bz2 --]
[-- Type: application/octet-stream, Size: 15482 bytes --]

[-- Attachment #3: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #4: Type: text/plain, Size: 164 bytes --]

_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

Re: [Bluez-devel] bluetooth memory corruption (was Re: ext3-related crash in 2.6.20-rc1)

[Marcel Holtmann]

Hi Pavel,

> > > > > I got this nasty oops while playing with debugger. Not sure if that is
> > > > > related; it also might be something with bluetooth; I already know it
> > > > > corrupts memory during suspend, perhaps it corrupts memory in some
> > > > > error path?
> > > > 
> > > > Okay, I spoke too soon. bluetooth & suspend memory corruption was
> > > > _way_ harder to reproduce than expected. Took me 5-or-so-suspend
> > > > cycles... so it is probably unrelated to the previous crash.
> > > 
> > > can you try to reproduce this with 2.6.20-rc2 as well.
> > 
> > (reproduced in another mail).
> > 
> >         _urb_queue_tail(__pending_q(husb, _urb->type), _urb);
> >         err = usb_submit_urb(urb, GFP_ATOMIC);
> >         if (err) {
> >                 BT_ERR("%s tx submit failed urb %p type %d err %d",
> >                                 husb->hdev->name, urb, _urb->type, err);
> >                 _urb_unlink(_urb);
> > 
> >                 ~~~~~~~~~~~~~~~~~~
> > 	 	 Do we need to remove urb from pending_q here?
> > 
> >                 _urb_queue_tail(__completed_q(husb, _urb->type), _urb);
> >         } else
> >                 atomic_inc(__pending_tx(husb, _urb->type));
> > 
> 
> Any news? Should I convert above idea to a patch? Or should I make
> bluetooth suspend() routine return error so corruption is impossible
> to hit?

to be honest, I have no idea. This code is way to ugly anyway.

Regards

Marcel



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel