even more breakage in wireless-dev.git

even more breakage in wireless-dev.git

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 3277 bytes --]

Hah, I get to complain too.

http://thread.gmane.org/gmane.linux.network/52202 still exists:

[  441.994183] bcm43xx_d80211: Radio turned off
[  442.310739] eth1: skb_queue not empty
[  442.313338] Unable to handle kernel paging request for data at address 0x6b6b6b73

[... warn_on() stuff removed ...]

[  442.314609] Oops: Kernel access of bad area, sig: 11 [#1]
[  442.314624]
[  442.314635] Modules linked in: binfmt_misc hci_usb rfcomm l2cap bluetooth fuse af_packet snd_powermac configfs nls_utf8 hfsplus nls_base dm_snapshot dm_mirror sha256 joydev eth1394 snd_aoa_codec_tas snd_aoa_fabric_layout appletouch snd_aoa usbhid pcmcia snd_aoa_i2sbus snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd ieee80211softmac ieee80211 ieee80211_crypt arc4 rc80211_simple soundcore ohci1394 ieee1394 ehci_hcd snd_aoa_soundbus ohci_hcd yenta_socket rsrc_nonstatic firmware_class usbcore ssb
pcmcia_core 80211 cfg80211 uninorth_agp agpgart evdev unix
[  442.315368] NIP: C0030ADC LR: C0030CB4 CTR: C0030A5C
[  442.315386] REGS: c037dbc0 TRAP: 0300   Not tainted  (2.6.20-g3ff05f36-dirty)
[  442.315401] MSR: 00009032 <EE,ME,IR,DR>  CR: 42040444  XER: 20000000
[  442.315477] DAR: 6B6B6B73, DSISR: 40000000
[  442.315494] TASK = effa8720[481] 'udevd' THREAD: c037c000
[  442.315509] GPR00: 00009032 C037DC70 EFFA8720 C077DA48 C056D780 00000001 00000000 C0428774
[  442.315596] GPR08: 00000000 00000000 00000000 6B6B6B6B 22040442 1002714C 28204422 00000000
[  442.315692] GPR16: 100FB7C8 100D0000 100B0000 100D0000 00000007 0000000F 00000000 000001A4
[  442.315787] GPR24: C0780000 00000000 C077F458 00000001 C0780000 C0570000 00000001 6B6B6B6B
[  442.315881] NIP [C0030ADC] tasklet_action+0x80/0xec

[rest just posted for completeness]

[  442.315910] LR [C0030CB4] __do_softirq+0x80/0xf4
[  442.315937] Call Trace:
[  442.315952] [C037DC70] [F220F804] ohci_irq+0x68/0x270 [ohci_hcd] (unreliable)
[  442.316003] [C037DC90] [C0030CB4] __do_softirq+0x80/0xf4
[  442.316039] [C037DCC0] [C00069A8] do_softirq+0x58/0x5c
[  442.316081] [C037DCD0] [C0030A4C] irq_exit+0x48/0x58
[  442.316114] [C037DCE0] [C0006A54] do_IRQ+0xa8/0xc8
[  442.316383] [C037DCF0] [C0011A28] ret_from_except+0x0/0x14
[  442.316420] --- Exception: 501 at down_write+0xc/0x64
[  442.316453]     LR = notify_change+0x1f8/0x284
[  442.316468] [C037DDB0] [C02C6D28] __mutex_lock_slowpath+0x110/0x1ac (unreliable)
[  442.316513] [C037DDD0] [C009BC70] notify_change+0x38/0x284
[  442.316550] [C037DE00] [C0082868] do_truncate+0x5c/0x84
[  442.316586] [C037DE50] [C008E080] may_open+0x148/0x200
[  442.316618] [C037DE70] [C0090548] open_namei+0x88/0x6b4
[  442.316653] [C037DEC0] [C0082480] do_filp_open+0x30/0x78
[  442.316689] [C037DF20] [C0082520] do_sys_open+0x58/0xf8
[  442.316724] [C037DF40] [C0011380] ret_from_syscall+0x0/0x38
[  442.316760] --- Exception: c01 at 0xff205ac
[  442.316788]     LR = 0xffb3d70
[  442.316801] Instruction dump:
[  442.316824] 7d40f078 7c00492d 40a2fff4 71400001 41820084 806b0010 816b000c 7d6903a6
[  442.316903] 4e800421 2f9f0000 7febfb78 419e0054 <800b0008> 392b0004 83eb0000 2f800000
[  442.316984]  <0>Kernel panic - not syncing: Fatal exception in interrupt
[  442.317035]  <0>Rebooting in 180 seconds..

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: even more breakage in wireless-dev.git

[Johannes Berg]

[-- Attachment #1.1: Type: text/plain, Size: 2685 bytes --]

On Sat, 2007-02-17 at 11:23 +0100, Johannes Berg wrote:
> Hah, I get to complain too.

and I also give some debugging info........

> [  441.994183] bcm43xx_d80211: Radio turned off
> [  442.310739] eth1: skb_queue not empty
> [  442.313338] Unable to handle kernel paging request for data at address 0x6b6b6b73

> [  442.315881] NIP [C0030ADC] tasklet_action+0x80/0xec

ok so I was guessing that we free a used tasklet, and I'm right.
Attached is a debug patch that gave me the following log instead of the
oops:

[  281.135208] bcm43xx_d80211: Radio turned off
[  281.371838] eth1: skb_queue not empty
[  281.372089] attempt to free scheduled tasklet c0506188 in object c0506000!
[  281.372107] will leak memory instead!
[  281.372122] Call Trace:
[  281.372138] [E8FF9CD0] [C0009024] show_stack+0x3c/0x194 (unreliable)
[  281.372199] [E8FF9D00] [C0030900] __tasklet_free_check+0x70/0x98
[  281.372244] [E8FF9D20] [C007F670] kfree+0x78/0x120
[  281.372288] [E8FF9D40] [F208A104] cfg80211_dev_free+0x24/0x38 [cfg80211]
[  281.372358] [E8FF9D60] [F208A580] wiphy_class_dev_release+0x14/0x24 [cfg80211]
[  281.372406] [E8FF9D70] [C01D04D8] class_dev_release+0x74/0x9c
[  281.372450] [E8FF9D90] [C016ED7C] kobject_cleanup+0x70/0xd4
[  281.372490] [E8FF9DB0] [C016FD30] kref_put+0x54/0x6c
[  281.372537] [E8FF9DC0] [C016ECFC] kobject_put+0x24/0x34
[  281.372573] [E8FF9DD0] [C01CF5EC] class_device_put+0x1c/0x2c
[  281.372613] [E8FF9DE0] [F208A0D0] wiphy_free+0x14/0x24 [cfg80211]
[  281.372655] [E8FF9DF0] [F224AACC] ieee80211_free_hw+0x24/0x38 [80211]
[  281.372744] [E8FF9E10] [F24836F8] bcm43xx_wireless_exit+0x24/0x38 [bcm43xx_d80211]
[  281.372813] [E8FF9E30] [F2081464] ssb_device_remove+0x34/0x54 [ssb]
[  281.372863] [E8FF9E40] [C01CEA08] __device_release_driver+0x88/0xc0
[  281.372910] [E8FF9E50] [C01CF180] driver_detach+0x110/0x140
[  281.372949] [E8FF9E70] [C01CE49C] bus_remove_driver+0x8c/0xc8
[  281.372987] [E8FF9E90] [C01CF208] driver_unregister+0x18/0x34
[  281.373025] [E8FF9EB0] [F2081CBC] ssb_driver_unregister+0x14/0x24 [ssb]
[  281.373078] [E8FF9EC0] [F249B7F4] bcm43xx_exit+0x18/0x35e4 [bcm43xx_d80211]
[  281.373126] [E8FF9ED0] [C004CC98] sys_delete_module+0x1ac/0x210
[  281.373173] [E8FF9F40] [C0011380] ret_from_syscall+0x0/0x38
[  281.373223] --- Exception: c01 at 0xff6ae18
[  281.373253]     LR = 0x10001214

As you can see (compile with debug info and get the struct sizes etc.),
the tasklet in question must be 

	struct tasklet_struct tasklet;

in struct ieee80211_local. Now to figure out why it is scheduled while
we do ieee80211_free_hw, but I think that's good enough for me today.


johannes

[-- Attachment #1.2: tasklet-free-debug.patch --]
[-- Type: text/x-vhdl, Size: 1887 bytes --]

---
 kernel/softirq.c |   23 +++++++++++++++++++++++
 mm/slab.c        |    6 +++++-
 2 files changed, 28 insertions(+), 1 deletion(-)

--- wireless-dev.orig/kernel/softirq.c	2007-02-17 11:35:52.381777421 +0100
+++ wireless-dev/kernel/softirq.c	2007-02-17 11:59:54.431777421 +0100
@@ -353,6 +353,29 @@ void fastcall __tasklet_schedule(struct 
 
 EXPORT_SYMBOL(__tasklet_schedule);
 
+int fastcall __tasklet_free_check(char *from, unsigned long len)
+{
+	struct tasklet_struct *t;
+	unsigned long start = (unsigned long) from;
+	int ret = 0;
+
+	/* really should check all CPUs but I only have one
+	 * right now so I'm fine. hah! */
+	t = __get_cpu_var(tasklet_vec).list;
+	while (t) {
+		unsigned long tul = (unsigned long)t;
+		if (tul >= start && tul < (start+len)) {
+			printk(KERN_ERR "attempt to free scheduled tasklet %p in object %p!\n", t, from);
+			printk(KERN_ERR "will leak memory instead!\n");
+			dump_stack();
+			ret = -EBUSY;
+		}
+		t = t->next;
+	}
+
+	return ret;
+}
+
 void fastcall __tasklet_hi_schedule(struct tasklet_struct *t)
 {
 	unsigned long flags;
--- wireless-dev.orig/mm/slab.c	2007-02-17 11:36:56.181777421 +0100
+++ wireless-dev/mm/slab.c	2007-02-17 11:45:17.161777421 +0100
@@ -3747,6 +3747,9 @@ EXPORT_SYMBOL(kmem_cache_free);
  * Don't free memory not originally allocated by kmalloc()
  * or you will run into trouble.
  */
+
+extern int fastcall __tasklet_free_check(char *from, unsigned long length);
+
 void kfree(const void *objp)
 {
 	struct kmem_cache *c;
@@ -3758,7 +3761,8 @@ void kfree(const void *objp)
 	kfree_debugcheck(objp);
 	c = virt_to_cache(objp);
 	debug_check_no_locks_freed(objp, obj_size(c));
-	__cache_free(c, (void *)objp);
+	if (__tasklet_free_check(objp, obj_size(c)) == 0)
+		__cache_free(c, (void *)objp);
 	local_irq_restore(flags);
 }
 EXPORT_SYMBOL(kfree);

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: even more breakage in wireless-dev.git

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1820 bytes --]

And I still get this...


[ 3328.408677] freeing scheduled tasklet dd126294 in object dd126000!<3> descheduled instead.
[ 3328.408685] Call Trace:
[ 3328.408690] [DB015CB0] [C0008D6C] show_stack+0x3c/0x194 (unreliable)
[ 3328.408711] [DB015CE0] [C002FA8C] __tasklet_free_check+0x98/0xbc
[ 3328.408725] [DB015D10] [C008037C] kfree+0x78/0x114
[ 3328.408735] [DB015D30] [F20EF244] cfg80211_dev_free+0x2c/0x40 [cfg80211]
[ 3328.408773] [DB015D50] [F20EF754] wiphy_dev_release+0x14/0x24 [cfg80211]
[ 3328.408786] [DB015D60] [C01D0264] device_release+0x28/0x88
[ 3328.408802] [DB015D70] [C0172BD8] kobject_cleanup+0x70/0xac
[ 3328.408819] [DB015D90] [C0173C44] kref_put+0x54/0x6c
[ 3328.408828] [DB015DA0] [C0172B58] kobject_put+0x24/0x34
[ 3328.408838] [DB015DB0] [C01D04E4] put_device+0x1c/0x2c
[ 3328.408848] [DB015DC0] [F20EF208] wiphy_free+0x14/0x24 [cfg80211]
[ 3328.408860] [DB015DD0] [F252CF40] ieee80211_free_hw+0x24/0x38 [mac80211]
[ 3328.408914] [DB015DF0] [F27905E0] bcm43xx_wireless_exit+0x24/0x38 [bcm43xx_mac80211]
[ 3328.408953] [DB015E10] [F2217604] ssb_device_remove+0x38/0x58 [ssb]
[ 3328.408976] [DB015E20] [C01D34D8] __device_release_driver+0x88/0xc8
[ 3328.408988] [DB015E30] [C01D3C98] driver_detach+0x110/0x140
[ 3328.408997] [DB015E50] [C01D2F6C] bus_remove_driver+0x8c/0xc8
[ 3328.409007] [DB015E70] [C01D3D20] driver_unregister+0x18/0x34
[ 3328.409016] [DB015E90] [F2217F5C] ssb_driver_unregister+0x14/0x24 [ssb]
[ 3328.409031] [DB015EA0] [F27A8340] bcm43xx_exit+0x18/0x3620 [bcm43xx_mac80211]
[ 3328.409049] [DB015EB0] [C004CD24] delete_module+0x11c/0x1c0
[ 3328.409059] [DB015EE0] [C004CE58] sys_delete_module+0x90/0xc8
[ 3328.409068] [DB015F40] [C0010BD8] ret_from_syscall+0x0/0x38
[ 3328.409082] --- Exception: c01 at 0xff6ae18
[ 3328.409096]     LR = 0x10001214


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]