Re: Can someone please try...

[Pavel Roskin <proski@gnu.org>]

Hello, Michael!

I did more testing, and the results are following.  It looks like the
oopses and panics on i386 were triggered by 4k stacks.  x86_64 doesn't
have this option.

Now that I enabled other debug options on both platforms. but not 4k
stacks, I'm seeing exactly the same problem on each platform.  When run
initially, wpa_supplicant connects with no problems (except very poor
reception of the data packets, but it's another story).  If interrupted
and restarted, wpa_supplicant reconnects, but I'm getting messages like
this (i386):

Slab corruption: start=cfdaece0, len=1024
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02d70c2>](skb_release_data+0x7b/0x7f)
000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Prev obj: start=cfdae8d4, len=1024
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c026ea5a>](device_create+0x2c/0x98)
000: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
010: ad 4e ad de ff ff ff ff ff ff ff ff 10 3a 6d c0
Next obj: start=cfdaf0ec, len=1024
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0165730>](expand_files+0x95/0x2c2)
000: 78 55 39 c7 78 55 39 c7 78 55 39 c7 88 da 52 df
010: d8 18 3b c7 00 00 00 00 00 00 00 00 00 00 00 00

and this (x86_64):

Slab corruption: start=ffff81000ec8a198, len=1024
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<ffffffff8042e916>](skb_release_data+0x94/0x99)
000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Next obj: start=ffff81000ec8a5b0, len=1024
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<ffffffff803be6e9>](device_create+0x5f/0x110)
000: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

I can restart wpa_supplicant again, and it would show similar messages.
The first "Last user" is inevitably skb_release_data.

I have no idea how to deal with it.  I think I need a stack trace at the
time when skb_release_data is called.

This is a stack trace at the time when slab corruption is detected.
It's actually incorrect closer to the top, perhaps from gcc
optimizations for static functions.

Slab corruption: start=ffff8100066f81d8, len=1024

Call Trace:
 [<ffffffff80218636>] vsnprintf+0x338/0x5a8
 [<ffffffff8020713d>] check_poison_obj+0x69/0x1ae
 [<ffffffff803c3ff2>] _request_firmware+0x8f/0x326
 [<ffffffff803c3ff2>] _request_firmware+0x8f/0x326


 [<ffffffff8020c09a>] cache_alloc_debugcheck_after+0x32/0x1a2
 [<ffffffff803c3ff2>] _request_firmware+0x8f/0x326
 [<ffffffff802aaae2>] kmem_cache_zalloc+0xaf/0xd8
 [<ffffffff803c3ff2>] _request_firmware+0x8f/0x326
 [<ffffffff880111ea>] :bcm43xx_d80211:bcm43xx_phy_init_tssi2dbm_table
+0xf0/0x2ca
 [<ffffffff803c432a>] request_firmware+0xe/0x10
 [<ffffffff88007d75>] :bcm43xx_d80211:bcm43xx_chip_init+0x96/0xaba
 [<ffffffff8020a03d>] kmem_cache_alloc+0xaf/0xbe
 [<ffffffff88009c97>] :bcm43xx_d80211:bcm43xx_wireless_core_init
+0x4de/0xa3d
 [<ffffffff8800b4e8>] :bcm43xx_d80211:bcm43xx_add_interface+0x64/0xde
 [<ffffffff8046eaa0>] ieee80211_open+0x1c7/0x2cc
 [<ffffffff804330da>] dev_open+0x36/0x76
 [<ffffffff8043185b>] dev_change_flags+0x5d/0x122
 [<ffffffff8045a1a3>] devinet_ioctl+0x259/0x5e8
 [<ffffffff8045a7f2>] inet_ioctl+0x71/0x8f
 [<ffffffff8042a395>] sock_ioctl+0x1db/0x1fd
 [<ffffffff8023bfa7>] do_ioctl+0x1b/0x50
 [<ffffffff8022c9b2>] vfs_ioctl+0x22a/0x23c
 [<ffffffff80289975>] trace_hardirqs_on+0x124/0x14e
 [<ffffffff802459a2>] sys_ioctl+0x42/0x65
 [<ffffffff8025531e>] system_call+0x7e/0x83

Anyway, I could narrow down this message to the first kzalloc() call in
fw_register_device(), file drivers/base/firmware_class.c.  This only
seems to confirm my suspicion that the actual corruption happened before
this point.  We are just hitting it when trying to allocate more memory.

Help with debugging this problem will be appreciated.  I've never hunted
down such problems, especially in kernel space.

-- 
Regards,
Pavel Roskin