* kref refcounting breakage in mainline @ 2007-03-02 8:58 Andrew Morton 2007-03-03 5:48 ` Greg KH 2007-03-06 0:25 ` Greg KH 0 siblings, 2 replies; 16+ messages in thread From: Andrew Morton @ 2007-03-02 8:58 UTC (permalink / raw) To: Greg KH, Tejun Heo, Kay Sievers; +Cc: linux-kernel, Adrian Bunk -mm has a debugging patch which warns when atomic_dec_and_test() takes an atomic_t negative (ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.20/2.6.20-mm2/broken-out/detect-atomic-counter-underflows.patch). When it is applied to current mainline, a simple `rmmod ipw2200' gives: [ 75.825072] BUG: atomic counter underflow at: [ 75.825180] [<c01c6eb4>] kref_put+0x66/0x82 [ 75.825278] [<c022e4d4>] bus_remove_driver+0x66/0x75 [ 75.825383] [<c022ee2c>] driver_unregister+0x8/0x13 [ 75.825484] [<c01d7add>] pci_unregister_driver+0xc/0x45 [ 75.825593] [<c0132147>] sys_delete_module+0x157/0x17c [ 75.825703] [<c013c663>] audit_syscall_entry+0x10d/0x137 [ 75.825818] [<c0103b14>] syscall_call+0x7/0xb [ 75.825913] [<c02d0000>] xfrm4_dst_destroy+0xe/0xd5 This didn't happen in 2.6.20-mm2, so this bug was introduced by a patch which was not in the -mm lineup twelve days ago. Presumably the effect of this is a memory leak or a use-after-free. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-02 8:58 kref refcounting breakage in mainline Andrew Morton @ 2007-03-03 5:48 ` Greg KH 2007-03-06 0:25 ` Greg KH 1 sibling, 0 replies; 16+ messages in thread From: Greg KH @ 2007-03-03 5:48 UTC (permalink / raw) To: Andrew Morton; +Cc: Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Fri, Mar 02, 2007 at 12:58:33AM -0800, Andrew Morton wrote: > > -mm has a debugging patch which warns when atomic_dec_and_test() takes an > atomic_t negative > (ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.20/2.6.20-mm2/broken-out/detect-atomic-counter-underflows.patch). > > > When it is applied to current mainline, a simple `rmmod ipw2200' gives: > > [ 75.825072] BUG: atomic counter underflow at: > [ 75.825180] [<c01c6eb4>] kref_put+0x66/0x82 > [ 75.825278] [<c022e4d4>] bus_remove_driver+0x66/0x75 > [ 75.825383] [<c022ee2c>] driver_unregister+0x8/0x13 > [ 75.825484] [<c01d7add>] pci_unregister_driver+0xc/0x45 > [ 75.825593] [<c0132147>] sys_delete_module+0x157/0x17c > [ 75.825703] [<c013c663>] audit_syscall_entry+0x10d/0x137 > [ 75.825818] [<c0103b14>] syscall_call+0x7/0xb > [ 75.825913] [<c02d0000>] xfrm4_dst_destroy+0xe/0xd5 > > This didn't happen in 2.6.20-mm2, so this bug was introduced by a patch > which was not in the -mm lineup twelve days ago. > > Presumably the effect of this is a memory leak or a use-after-free. Ugh. I'll add it to my local tree here and try to bisect to find the problem. thanks for the pointer, greg k-h ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-02 8:58 kref refcounting breakage in mainline Andrew Morton 2007-03-03 5:48 ` Greg KH @ 2007-03-06 0:25 ` Greg KH 2007-03-06 5:43 ` Mike Galbraith 2007-03-06 12:11 ` Mel Gorman 1 sibling, 2 replies; 16+ messages in thread From: Greg KH @ 2007-03-06 0:25 UTC (permalink / raw) To: Andrew Morton, Mike Galbraith Cc: Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Fri, Mar 02, 2007 at 12:58:33AM -0800, Andrew Morton wrote: > > -mm has a debugging patch which warns when atomic_dec_and_test() takes an > atomic_t negative > (ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.20/2.6.20-mm2/broken-out/detect-atomic-counter-underflows.patch). > > > When it is applied to current mainline, a simple `rmmod ipw2200' gives: > > [ 75.825072] BUG: atomic counter underflow at: > [ 75.825180] [<c01c6eb4>] kref_put+0x66/0x82 > [ 75.825278] [<c022e4d4>] bus_remove_driver+0x66/0x75 > [ 75.825383] [<c022ee2c>] driver_unregister+0x8/0x13 > [ 75.825484] [<c01d7add>] pci_unregister_driver+0xc/0x45 > [ 75.825593] [<c0132147>] sys_delete_module+0x157/0x17c > [ 75.825703] [<c013c663>] audit_syscall_entry+0x10d/0x137 > [ 75.825818] [<c0103b14>] syscall_call+0x7/0xb > [ 75.825913] [<c02d0000>] xfrm4_dst_destroy+0xe/0xd5 > > This didn't happen in 2.6.20-mm2, so this bug was introduced by a patch > which was not in the -mm lineup twelve days ago. > > Presumably the effect of this is a memory leak or a use-after-free. Ok, after a zillion bisects, I've tracked this down to: commit 63ce18cfe685115ff8d341bae4c9204a79043cf0 Author: Mike Galbraith <efault@gmx.de> Date: Wed Feb 21 12:45:35 2007 -0800 driver core: refcounting fix Fix a reference counting bug exposed by commit 725522b5453dd680412f2b6463a988e4fd148757. If driver.mod_name exists, we take a reference in module_add_driver(), and never release it. Undo that reference in module_remove_driver(). Signed-off-by: Mike Galbraith <efault@gmx.de> Cc: Kay Sievers <kay.sievers@vrfy.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de> Mike, I've reverted this patch, and I don't see any references leaking. And, as your patch released the reference on the driver, and the module_add_driver() call would not grab a reference to the driver, only the module kobject, I don't see what you were trying to fix with this patch. Do you have a test case that this fixes? Otherwise, I'll just revert it. thanks, greg k-h ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-06 0:25 ` Greg KH @ 2007-03-06 5:43 ` Mike Galbraith 2007-03-06 21:04 ` Greg KH 2007-03-06 12:11 ` Mel Gorman 1 sibling, 1 reply; 16+ messages in thread From: Mike Galbraith @ 2007-03-06 5:43 UTC (permalink / raw) To: Greg KH; +Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Mon, 2007-03-05 at 16:25 -0800, Greg KH wrote: > Mike, I've reverted this patch, and I don't see any references leaking. > And, as your patch released the reference on the driver, and the > module_add_driver() call would not grab a reference to the driver, only > the module kobject, I don't see what you were trying to fix with this > patch. > > Do you have a test case that this fixes? What it fixed for me was the hard hang reported below. http://lkml.org/lkml/2007/2/16/96 -Mike ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-06 5:43 ` Mike Galbraith @ 2007-03-06 21:04 ` Greg KH 2007-03-07 5:38 ` Mike Galbraith 0 siblings, 1 reply; 16+ messages in thread From: Greg KH @ 2007-03-06 21:04 UTC (permalink / raw) To: Mike Galbraith Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Tue, Mar 06, 2007 at 06:43:22AM +0100, Mike Galbraith wrote: > On Mon, 2007-03-05 at 16:25 -0800, Greg KH wrote: > > > Mike, I've reverted this patch, and I don't see any references leaking. > > And, as your patch released the reference on the driver, and the > > module_add_driver() call would not grab a reference to the driver, only > > the module kobject, I don't see what you were trying to fix with this > > patch. > > > > Do you have a test case that this fixes? > > What it fixed for me was the hard hang reported below. > > http://lkml.org/lkml/2007/2/16/96 What specific module are you trying to unload that causes the hang? I think it might just be a problem with that module, and not with all others. So, I'm going to revert your patch and work to try to find the real cause of this problem. thanks, greg k-h ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-06 21:04 ` Greg KH @ 2007-03-07 5:38 ` Mike Galbraith 2007-03-10 15:44 ` Mike Galbraith 0 siblings, 1 reply; 16+ messages in thread From: Mike Galbraith @ 2007-03-07 5:38 UTC (permalink / raw) To: Greg KH; +Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Tue, 2007-03-06 at 13:04 -0800, Greg KH wrote: > On Tue, Mar 06, 2007 at 06:43:22AM +0100, Mike Galbraith wrote: > > On Mon, 2007-03-05 at 16:25 -0800, Greg KH wrote: > > > > > Mike, I've reverted this patch, and I don't see any references leaking. > > > And, as your patch released the reference on the driver, and the > > > module_add_driver() call would not grab a reference to the driver, only > > > the module kobject, I don't see what you were trying to fix with this > > > patch. > > > > > > Do you have a test case that this fixes? > > > > What it fixed for me was the hard hang reported below. > > > > http://lkml.org/lkml/2007/2/16/96 > > What specific module are you trying to unload that causes the hang? I > think it might just be a problem with that module, and not with all > others. It's ipmi_si that's hanging, waits for completion that never comes. > So, I'm going to revert your patch and work to try to find the real > cause of this problem. Yeah, my stab at it seems busted. I'll take another poke at it to see if I can find out why (post 725522b5453dd680412f2b6463a988e4fd148757) I'm left with a reference. -Mike ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-07 5:38 ` Mike Galbraith @ 2007-03-10 15:44 ` Mike Galbraith 2007-03-10 16:03 ` Mike Galbraith 2007-03-15 5:27 ` Greg KH 0 siblings, 2 replies; 16+ messages in thread From: Mike Galbraith @ 2007-03-10 15:44 UTC (permalink / raw) To: Greg KH; +Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Wed, 2007-03-07 at 06:39 +0100, Mike Galbraith wrote: > On Tue, 2007-03-06 at 13:04 -0800, Greg KH wrote: > > On Tue, Mar 06, 2007 at 06:43:22AM +0100, Mike Galbraith wrote: > > > On Mon, 2007-03-05 at 16:25 -0800, Greg KH wrote: > > > > > > > Mike, I've reverted this patch, and I don't see any references leaking. > > > > And, as your patch released the reference on the driver, and the > > > > module_add_driver() call would not grab a reference to the driver, only > > > > the module kobject, I don't see what you were trying to fix with this > > > > patch. > > > > > > > > Do you have a test case that this fixes? > > > > > > What it fixed for me was the hard hang reported below. > > > > > > http://lkml.org/lkml/2007/2/16/96 > > > > What specific module are you trying to unload that causes the hang? I > > think it might just be a problem with that module, and not with all > > others. > > It's ipmi_si that's hanging, waits for completion that never comes. > > > So, I'm going to revert your patch and work to try to find the real > > cause of this problem. > > Yeah, my stab at it seems busted. I'll take another poke at it to see > if I can find out why (post 725522b5453dd680412f2b6463a988e4fd148757) > I'm left with a reference. Ok, stab #2. My reference count woes stem from module_remove_driver() not removing the link created in module_add_driver(). With the below, my box boots fine. Since I obviously know spit about driver layer glue, I'll just call this one a diagnostic, and head for the hills :) --- linux-2.6.20-rc3/kernel/module.c.org 2007-03-10 15:16:47.000000000 +0100 +++ linux-2.6.20-rc3/kernel/module.c 2007-03-10 15:43:09.000000000 +0100 @@ -2411,14 +2411,28 @@ void module_remove_driver(struct device_ return; sysfs_remove_link(&drv->kobj, "module"); - if (drv->owner && drv->owner->mkobj.drivers_dir) { - driver_name = make_driver_name(drv); - if (driver_name) { - sysfs_remove_link(drv->owner->mkobj.drivers_dir, + driver_name = make_driver_name(drv); + if (!driver_name) + return; + if (drv->owner && drv->owner->mkobj.drivers_dir) + sysfs_remove_link(drv->owner->mkobj.drivers_dir, driver_name); - kfree(driver_name); - } + else if (drv->mod_name) { + struct module_kobject *mk; + struct kobject *mkobj; + + /* Lookup built-in module entry in /sys/modules */ + mkobj = kset_find_obj(&module_subsys.kset, drv->mod_name); + if (!mkobj) + goto out_free; + mk = container_of(mkobj, struct module_kobject, kobj); + module_create_drivers_dir(mk); + sysfs_remove_link(mk->drivers_dir, driver_name); + /* Release reference taken via lookup */ + kobject_put(mkobj); } +out_free: + kfree(driver_name); } EXPORT_SYMBOL(module_remove_driver); #endif ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-10 15:44 ` Mike Galbraith @ 2007-03-10 16:03 ` Mike Galbraith 2007-03-15 5:27 ` Greg KH 1 sibling, 0 replies; 16+ messages in thread From: Mike Galbraith @ 2007-03-10 16:03 UTC (permalink / raw) To: Greg KH; +Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk P.S. forgot to include diagnostic log. Kobject c0644890 is the source of my woes. Printk's come below WARN_ON(is_ipmi_si_kobj). Post-tinker log is huge, and probably not interesting. [ 30.397160] kobject ipmi_devintf: registering. parent: <NULL>, set: module [ 30.404033] kobject_uevent_env [ 30.407098] fill_kobj_path: path = '/module/ipmi_devintf' [ 30.412524] BUG: at lib/kobject.c:448 kobject_get() [ 30.417402] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 30.422564] [<c01057b2>] show_trace+0x12/0x14 [ 30.427031] [<c0105856>] dump_stack+0x16/0x18 [ 30.431501] [<c02d3c2b>] kobject_get+0x66/0x87 [ 30.436056] [<c02d3f1e>] kobject_shadow_add+0x10/0x1e8 [ 30.441312] [<c02d4100>] kobject_add+0xa/0xc [ 30.445695] [<c06804c0>] kernel_param_sysfs_setup+0x4d/0x7f [ 30.451376] [<c068067a>] param_sysfs_init+0x188/0x1c3 [ 30.456538] [<c06725bc>] init+0x144/0x26c [ 30.460661] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 30.465907] ======================= [ 30.469486] get: c18f65c0 count after get is 2 [ 30.473927] kobject ipmi_si: registering. parent: <NULL>, set: module [ 30.480372] kobject_uevent_env [ 30.483430] fill_kobj_path: path = '/module/ipmi_si' ...... [ 73.266556] bus platform: add driver ipmi [ 73.278013] kobject ipmi: registering. parent: <NULL>, set: drivers [ 73.291847] kobject_uevent_env [ 73.302358] fill_kobj_path: path = '/bus/platform/drivers/ipmi' [ 73.315943] ipmi message handler version 39.1 [ 73.327839] ipmi device interface [ 73.338524] device class 'ipmi': registering [ 73.350158] subsystem ipmi: registering [ 73.361309] kobject ipmi: registering. parent: <NULL>, set: class [ 73.374841] bus platform: add driver ipmi_si [ 73.386442] BUG: at lib/kobject.c:448 kobject_get() [ 73.398617] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.411079] [<c01057b2>] show_trace+0x12/0x14 [ 73.422780] [<c0105856>] dump_stack+0x16/0x18 [ 73.434324] [<c02d3c2b>] kobject_get+0x66/0x87 [ 73.445860] [<c02d3f1e>] kobject_shadow_add+0x10/0x1e8 [ 73.458088] [<c02d4100>] kobject_add+0xa/0xc [ 73.469286] [<c02d4248>] kobject_register+0x22/0xb3 [ 73.480986] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 73.492592] [<c035b109>] driver_register+0x54/0x84 [ 73.504101] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.515335] [<c06725bc>] init+0x144/0x26c [ 73.525822] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.537452] ======================= [ 73.547290] get: c064475c count after get is 2 [ 73.557969] kobject ipmi_si: registering. parent: <NULL>, set: drivers [ 73.570825] kobject_uevent_env [ 73.580011] fill_kobj_path: path = '/bus/platform/drivers/ipmi_si' [ 73.592622] BUG: at lib/kobject.c:242 kobject_register() [ 73.604312] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.615822] [<c01057b2>] show_trace+0x12/0x14 [ 73.626551] [<c0105856>] dump_stack+0x16/0x18 [ 73.637211] [<c02d429f>] kobject_register+0x79/0xb3 [ 73.648325] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 73.659297] [<c035b109>] driver_register+0x54/0x84 [ 73.670164] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.680756] [<c06725bc>] init+0x144/0x26c [ 73.690699] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.701854] ======================= [ 73.711295] register: c064475c count now is 2 error 0 [ 73.722218] BUG: at lib/kobject.c:448 kobject_get() [ 73.732911] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.743927] [<c01057b2>] show_trace+0x12/0x14 [ 73.754112] [<c0105856>] dump_stack+0x16/0x18 [ 73.764253] [<c02d3c2b>] kobject_get+0x66/0x87 [ 73.774532] [<c035b157>] get_driver+0x11/0x18 [ 73.784725] [<c035b194>] driver_create_file+0xf/0x32 [ 73.795559] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 73.806343] [<c035b109>] driver_register+0x54/0x84 [ 73.817055] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.827595] [<c06725bc>] init+0x144/0x26c [ 73.837468] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.848544] ======================= [ 73.857932] get: c064475c count after get is 3 [ 73.868223] BUG: at lib/kobject.c:494 kobject_put() [ 73.878943] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.890019] [<c01057b2>] show_trace+0x12/0x14 [ 73.900239] [<c0105856>] dump_stack+0x16/0x18 [ 73.910294] [<c02d3ae7>] kobject_put+0x69/0x82 [ 73.920304] [<c035b144>] put_driver+0xb/0xd [ 73.929909] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 73.940362] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 73.950686] [<c035b109>] driver_register+0x54/0x84 [ 73.960966] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.971055] [<c06725bc>] init+0x144/0x26c [ 73.980478] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.991119] ======================= [ 74.000050] put: c064475c count before put is 3 [ 74.010026] BUG: at lib/kobject.c:448 kobject_get() [ 74.020367] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.031011] [<c01057b2>] show_trace+0x12/0x14 [ 74.040867] [<c0105856>] dump_stack+0x16/0x18 [ 74.050602] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.060353] [<c035b157>] get_driver+0x11/0x18 [ 74.070034] [<c035b194>] driver_create_file+0xf/0x32 [ 74.080418] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 74.090742] [<c035b109>] driver_register+0x54/0x84 [ 74.100979] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 74.111041] [<c06725bc>] init+0x144/0x26c [ 74.120431] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.131047] ======================= [ 74.139949] get: c064475c count after get is 3 [ 74.149807] BUG: at lib/kobject.c:494 kobject_put() [ 74.160111] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.170739] [<c01057b2>] show_trace+0x12/0x14 [ 74.180569] [<c0105856>] dump_stack+0x16/0x18 [ 74.190286] [<c02d3ae7>] kobject_put+0x69/0x82 [ 74.200019] [<c035b144>] put_driver+0xb/0xd [ 74.209503] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 74.219982] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 74.230312] [<c035b109>] driver_register+0x54/0x84 [ 74.240550] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 74.250614] [<c06725bc>] init+0x144/0x26c [ 74.260001] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.270618] ======================= [ 74.279530] put: c064475c count before put is 3 [ 74.289481] IPMI System Interface driver. [ 74.298864] bus pci: add driver ipmi_si [ 74.308007] BUG: at lib/kobject.c:448 kobject_get() [ 74.318260] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.328861] [<c01057b2>] show_trace+0x12/0x14 [ 74.338637] [<c0105856>] dump_stack+0x16/0x18 [ 74.348372] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.358245] [<c02d3f1e>] kobject_shadow_add+0x10/0x1e8 [ 74.368837] [<c02d4100>] kobject_add+0xa/0xc [ 74.378528] [<c02d4248>] kobject_register+0x22/0xb3 [ 74.388817] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 74.399027] [<c035b109>] driver_register+0x54/0x84 [ 74.409229] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.420004] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.430196] [<c06725bc>] init+0x144/0x26c [ 74.439611] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.450184] ======================= [ 74.459045] get: c0644890 count after get is 2 [ 74.468796] kobject ipmi_si: registering. parent: <NULL>, set: drivers [ 74.480769] kobject_uevent_env [ 74.489210] fill_kobj_path: path = '/bus/pci/drivers/ipmi_si' [ 74.500675] BUG: at lib/kobject.c:242 kobject_register() [ 74.511779] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.522812] [<c01057b2>] show_trace+0x12/0x14 [ 74.533082] [<c0105856>] dump_stack+0x16/0x18 [ 74.543269] [<c02d429f>] kobject_register+0x79/0xb3 [ 74.554015] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 74.564685] [<c035b109>] driver_register+0x54/0x84 [ 74.575363] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.586595] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.597161] [<c06725bc>] init+0x144/0x26c [ 74.606974] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.617989] ======================= [ 74.627292] register: c0644890 count now is 2 error 0 [ 74.638150] BUG: at lib/kobject.c:448 kobject_get() [ 74.648812] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.659872] [<c01057b2>] show_trace+0x12/0x14 [ 74.670237] [<c0105856>] dump_stack+0x16/0x18 [ 74.680554] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.690962] [<c02d434c>] kset_find_obj+0x73/0x77 [ 74.701563] [<c01442c5>] module_add_driver+0x64/0x7d [ 74.712578] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 74.723405] [<c035b109>] driver_register+0x54/0x84 [ 74.734247] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.745636] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.756393] [<c06725bc>] init+0x144/0x26c [ 74.766429] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.777654] ======================= [ 74.787103] get: c18f65c0 count after get is 3 [ 74.797461] BUG: at lib/kobject.c:448 kobject_get() [ 74.808209] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.819241] [<c01057b2>] show_trace+0x12/0x14 [ 74.829427] [<c0105856>] dump_stack+0x16/0x18 [ 74.839446] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.849465] [<c01b4812>] sysfs_create_link+0xd3/0x151 [ 74.860178] [<c0144285>] module_add_driver+0x24/0x7d [ 74.870866] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 74.881370] [<c035b109>] driver_register+0x54/0x84 [ 74.891850] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.902918] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.913354] [<c06725bc>] init+0x144/0x26c [ 74.923019] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.933904] ======================= [ 74.943042] get: c18f65c0 count after get is 4 [ 74.953046] BUG: at lib/kobject.c:448 kobject_get() [ 74.963497] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.974262] [<c01057b2>] show_trace+0x12/0x14 [ 74.984220] [<c0105856>] dump_stack+0x16/0x18 [ 74.994120] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.004167] [<c02d3f4a>] kobject_shadow_add+0x3c/0x1e8 [ 75.014941] [<c02d4100>] kobject_add+0xa/0xc [ 75.024787] [<c02d4248>] kobject_register+0x22/0xb3 [ 75.035216] [<c02d4431>] kobject_add_dir+0x4f/0x6e [ 75.045511] [<c01442d9>] module_add_driver+0x78/0x7d [ 75.055991] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 75.066262] [<c035b109>] driver_register+0x54/0x84 [ 75.076489] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.087297] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.097483] [<c06725bc>] init+0x144/0x26c [ 75.106862] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.117454] ======================= [ 75.126306] get: c18f65c0 count after get is 5 [ 75.136022] kobject drivers: registering. parent: ipmi_si, set: <NULL> [ 75.148068] kobject_uevent_env [ 75.156609] kobject filter function caused the event to drop! [ 75.168050] BUG: at lib/kobject.c:448 kobject_get() [ 75.178631] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.189595] [<c01057b2>] show_trace+0x12/0x14 [ 75.199789] [<c0105856>] dump_stack+0x16/0x18 [ 75.209905] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.220072] [<c01b4812>] sysfs_create_link+0xd3/0x151 [ 75.230853] [<c01442a5>] module_add_driver+0x44/0x7d [ 75.241531] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 75.252030] [<c035b109>] driver_register+0x54/0x84 [ 75.262455] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.273444] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.283813] [<c06725bc>] init+0x144/0x26c [ 75.293382] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.304189] ======================= [ 75.313370] get: c0644890 count after get is 3 [ 75.323512] BUG: at lib/kobject.c:448 kobject_get() [ 75.334155] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.345215] [<c01057b2>] show_trace+0x12/0x14 [ 75.355547] [<c0105856>] dump_stack+0x16/0x18 [ 75.365827] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.376185] [<c035b157>] get_driver+0x11/0x18 [ 75.386420] [<c035b194>] driver_create_file+0xf/0x32 [ 75.397291] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 75.408056] [<c035b109>] driver_register+0x54/0x84 [ 75.418682] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.429904] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.440480] [<c06725bc>] init+0x144/0x26c [ 75.450257] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.461238] ======================= [ 75.470402] get: c0644890 count after get is 4 [ 75.480404] BUG: at lib/kobject.c:494 kobject_put() [ 75.490866] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.501613] [<c01057b2>] show_trace+0x12/0x14 [ 75.511530] [<c0105856>] dump_stack+0x16/0x18 [ 75.521420] [<c02d3ae7>] kobject_put+0x69/0x82 [ 75.531449] [<c035b144>] put_driver+0xb/0xd [ 75.541209] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 75.551845] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 75.562324] [<c035b109>] driver_register+0x54/0x84 [ 75.572682] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.583628] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.593942] [<c06725bc>] init+0x144/0x26c [ 75.603469] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.614233] ======================= [ 75.623258] put: c0644890 count before put is 4 [ 75.633227] BUG: at lib/kobject.c:448 kobject_get() [ 75.643575] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.654272] [<c01057b2>] show_trace+0x12/0x14 [ 75.664153] [<c0105856>] dump_stack+0x16/0x18 [ 75.673991] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.683977] [<c035b157>] get_driver+0x11/0x18 [ 75.693884] [<c035b194>] driver_create_file+0xf/0x32 [ 75.704423] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 75.714893] [<c035b109>] driver_register+0x54/0x84 [ 75.725245] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.736189] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.746496] [<c06725bc>] init+0x144/0x26c [ 75.756023] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.766778] ======================= [ 75.775786] get: c0644890 count after get is 4 [ 75.785652] BUG: at lib/kobject.c:494 kobject_put() [ 75.795990] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.806687] [<c01057b2>] show_trace+0x12/0x14 [ 75.816577] [<c0105856>] dump_stack+0x16/0x18 [ 75.826416] [<c02d3ae7>] kobject_put+0x69/0x82 [ 75.836390] [<c035b144>] put_driver+0xb/0xd [ 75.846107] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 75.856744] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 75.867222] [<c035b109>] driver_register+0x54/0x84 [ 75.877589] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.888545] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.898868] [<c06725bc>] init+0x144/0x26c [ 75.908412] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.919193] ======================= [ 75.928228] put: c0644890 count before put is 4 [ 75.949537] ipmi_si: pci_unregister_driver() [ 75.959258] bus pci: remove driver ipmi_si [ 75.968766] BUG: at lib/kobject.c:448 kobject_get() [ 75.979081] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.989802] [<c01057b2>] show_trace+0x12/0x14 [ 75.999832] [<c0105856>] dump_stack+0x16/0x18 [ 76.009799] [<c02d3c2b>] kobject_get+0x66/0x87 [ 76.019863] [<c035b157>] get_driver+0x11/0x18 [ 76.029822] [<c035b16c>] driver_remove_file+0xe/0x27 [ 76.040389] [<c035a019>] bus_remove_driver+0x3a/0xa4 [ 76.050989] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.061432] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.072312] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.082589] [<c06725bc>] init+0x144/0x26c [ 76.092090] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.102742] ======================= [ 76.111690] get: c0644890 count after get is 4 [ 76.121537] BUG: at lib/kobject.c:494 kobject_put() [ 76.131808] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.142477] [<c01057b2>] show_trace+0x12/0x14 [ 76.152463] [<c0105856>] dump_stack+0x16/0x18 [ 76.162397] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.172408] [<c035b144>] put_driver+0xb/0xd [ 76.182134] [<c035b181>] driver_remove_file+0x23/0x27 [ 76.192759] [<c035a019>] bus_remove_driver+0x3a/0xa4 [ 76.203309] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.213726] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.224578] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.234840] [<c06725bc>] init+0x144/0x26c [ 76.244316] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.254951] ======================= [ 76.263853] put: c0644890 count before put is 4 [ 76.273754] BUG: at lib/kobject.c:448 kobject_get() [ 76.284057] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.294677] [<c01057b2>] show_trace+0x12/0x14 [ 76.304462] [<c0105856>] dump_stack+0x16/0x18 [ 76.314213] [<c02d3c2b>] kobject_get+0x66/0x87 [ 76.324122] [<c035b157>] get_driver+0x11/0x18 [ 76.333942] [<c035b16c>] driver_remove_file+0xe/0x27 [ 76.344405] [<c035a025>] bus_remove_driver+0x46/0xa4 [ 76.354882] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.365243] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.376110] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.386356] [<c06725bc>] init+0x144/0x26c [ 76.395830] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.406544] ======================= [ 76.415524] get: c0644890 count after get is 4 [ 76.425362] BUG: at lib/kobject.c:494 kobject_put() [ 76.435659] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.446287] [<c01057b2>] show_trace+0x12/0x14 [ 76.456107] [<c0105856>] dump_stack+0x16/0x18 [ 76.465876] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.475800] [<c035b144>] put_driver+0xb/0xd [ 76.485457] [<c035b181>] driver_remove_file+0x23/0x27 [ 76.496030] [<c035a025>] bus_remove_driver+0x46/0xa4 [ 76.506536] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.516911] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.527772] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.538017] [<c06725bc>] init+0x144/0x26c [ 76.547492] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.558194] ======================= [ 76.567160] put: c0644890 count before put is 4 [ 76.577094] BUG: at lib/kobject.c:494 kobject_put() [ 76.587399] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.598051] [<c01057b2>] show_trace+0x12/0x14 [ 76.607880] [<c0105856>] dump_stack+0x16/0x18 [ 76.617676] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.627617] [<c01b27a3>] sysfs_hash_and_remove+0x116/0x11f [ 76.638712] [<c01b473d>] sysfs_remove_link+0xb/0xd [ 76.649096] [<c0144151>] module_remove_driver+0x18/0x57 [ 76.659972] [<c035a06b>] bus_remove_driver+0x8c/0xa4 [ 76.670600] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.681138] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.692128] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.702470] [<c06725bc>] init+0x144/0x26c [ 76.712092] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.722899] ======================= [ 76.731916] put: c18f65c0 count before put is 5 [ 76.741944] kobject ipmi_si: unregistering [ 76.751496] kobject_uevent_env [ 76.759873] fill_kobj_path: path = '/bus/pci/drivers/ipmi_si' [ 76.771235] BUG: at lib/kobject.c:494 kobject_put() [ 76.781811] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.792782] [<c01057b2>] show_trace+0x12/0x14 [ 76.803009] [<c0105856>] dump_stack+0x16/0x18 [ 76.813168] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.823379] [<c02d3bc2>] unlink+0x3b/0x3e [ 76.833121] [<c02d3caf>] kobject_del+0x16/0x19 [ 76.843272] [<c02d3ce6>] kobject_unregister+0x34/0xa1 [ 76.854002] [<c035a073>] bus_remove_driver+0x94/0xa4 [ 76.864671] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.875185] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.886124] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.896439] [<c06725bc>] init+0x144/0x26c [ 76.905974] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.916676] ======================= [ 76.925694] put: c0644890 count before put is 3 [ 76.935799] BUG: at lib/kobject.c:430 kobject_unregister() [ 76.946990] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.957893] [<c01057b2>] show_trace+0x12/0x14 [ 76.968033] [<c0105856>] dump_stack+0x16/0x18 [ 76.978063] [<c02d3d3a>] kobject_unregister+0x88/0xa1 [ 76.988811] [<c035a073>] bus_remove_driver+0x94/0xa4 [ 76.999498] [<c035b0a8>] driver_unregister+0xb/0x18 [ 77.010064] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 77.021089] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 77.031524] [<c06725bc>] init+0x144/0x26c [ 77.041179] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 77.051980] ======================= [ 77.061057] unregister: c0644890 count before put is 2 [ 77.071811] BUG: at lib/kobject.c:494 kobject_put() [ 77.082360] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 77.093203] [<c01057b2>] show_trace+0x12/0x14 [ 77.103189] [<c0105856>] dump_stack+0x16/0x18 [ 77.113053] [<c02d3ae7>] kobject_put+0x69/0x82 [ 77.123022] [<c02d3d09>] kobject_unregister+0x57/0xa1 [ 77.133655] [<c035a073>] bus_remove_driver+0x94/0xa4 [ 77.144247] [<c035b0a8>] driver_unregister+0xb/0x18 [ 77.154753] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 77.165770] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 77.176205] [<c06725bc>] init+0x144/0x26c [ 77.185880] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 77.196791] ======================= [ 77.205972] put: c0644890 count before put is 2 ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-10 15:44 ` Mike Galbraith 2007-03-10 16:03 ` Mike Galbraith @ 2007-03-15 5:27 ` Greg KH 2007-03-15 7:53 ` Mike Galbraith 1 sibling, 1 reply; 16+ messages in thread From: Greg KH @ 2007-03-15 5:27 UTC (permalink / raw) To: Mike Galbraith Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Sat, Mar 10, 2007 at 04:44:06PM +0100, Mike Galbraith wrote: > On Wed, 2007-03-07 at 06:39 +0100, Mike Galbraith wrote: > > On Tue, 2007-03-06 at 13:04 -0800, Greg KH wrote: > > > On Tue, Mar 06, 2007 at 06:43:22AM +0100, Mike Galbraith wrote: > > > > On Mon, 2007-03-05 at 16:25 -0800, Greg KH wrote: > > > > > > > > > Mike, I've reverted this patch, and I don't see any references leaking. > > > > > And, as your patch released the reference on the driver, and the > > > > > module_add_driver() call would not grab a reference to the driver, only > > > > > the module kobject, I don't see what you were trying to fix with this > > > > > patch. > > > > > > > > > > Do you have a test case that this fixes? > > > > > > > > What it fixed for me was the hard hang reported below. > > > > > > > > http://lkml.org/lkml/2007/2/16/96 > > > > > > What specific module are you trying to unload that causes the hang? I > > > think it might just be a problem with that module, and not with all > > > others. > > > > It's ipmi_si that's hanging, waits for completion that never comes. > > > > > So, I'm going to revert your patch and work to try to find the real > > > cause of this problem. > > > > Yeah, my stab at it seems busted. I'll take another poke at it to see > > if I can find out why (post 725522b5453dd680412f2b6463a988e4fd148757) > > I'm left with a reference. > > Ok, stab #2. > > My reference count woes stem from module_remove_driver() not removing > the link created in module_add_driver(). With the below, my box boots > fine. Since I obviously know spit about driver layer glue, I'll just > call this one a diagnostic, and head for the hills :) Does ipmi_si not have a "owner"? Ah, that makes sense, not all modules do... > --- linux-2.6.20-rc3/kernel/module.c.org 2007-03-10 15:16:47.000000000 +0100 > +++ linux-2.6.20-rc3/kernel/module.c 2007-03-10 15:43:09.000000000 +0100 > @@ -2411,14 +2411,28 @@ void module_remove_driver(struct device_ > return; > > sysfs_remove_link(&drv->kobj, "module"); > - if (drv->owner && drv->owner->mkobj.drivers_dir) { > - driver_name = make_driver_name(drv); > - if (driver_name) { > - sysfs_remove_link(drv->owner->mkobj.drivers_dir, > + driver_name = make_driver_name(drv); > + if (!driver_name) > + return; > + if (drv->owner && drv->owner->mkobj.drivers_dir) > + sysfs_remove_link(drv->owner->mkobj.drivers_dir, > driver_name); > - kfree(driver_name); > - } > + else if (drv->mod_name) { > + struct module_kobject *mk; > + struct kobject *mkobj; > + > + /* Lookup built-in module entry in /sys/modules */ > + mkobj = kset_find_obj(&module_subsys.kset, drv->mod_name); > + if (!mkobj) > + goto out_free; > + mk = container_of(mkobj, struct module_kobject, kobj); > + module_create_drivers_dir(mk); > + sysfs_remove_link(mk->drivers_dir, driver_name); > + /* Release reference taken via lookup */ > + kobject_put(mkobj); > } > +out_free: > + kfree(driver_name); > } > EXPORT_SYMBOL(module_remove_driver); > #endif That's pretty good for not knowing much about the subject matter here. But can you try this version instead? It should work a bit better than yours. thanks for your patience, greg k-h Subject: modules: fix reference counting logic for drivers without module pointers. We weren't dropping the sysfs link for the module driver name if we didn't happen to have the "owner" pointer in the driver. Based on a patch from Mike Galbraith <efault@gmx.de> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de> --- kernel/module.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) --- a/kernel/module.c +++ b/kernel/module.c @@ -2405,20 +2405,30 @@ EXPORT_SYMBOL(module_add_driver); void module_remove_driver(struct device_driver *drv) { + struct module_kobject *mk = NULL; + struct kobject *mkobj = NULL; char *driver_name; if (!drv) return; sysfs_remove_link(&drv->kobj, "module"); - if (drv->owner && drv->owner->mkobj.drivers_dir) { - driver_name = make_driver_name(drv); - if (driver_name) { - sysfs_remove_link(drv->owner->mkobj.drivers_dir, - driver_name); - kfree(driver_name); - } + driver_name = make_driver_name(drv); + if (!driver_name) + return; + + if (drv->owner && drv->owner->mkobj.drivers_dir) + mk = &drv->owner->mkobj; + else { + /* Lookup built-in module entry in /sys/modules */ + mkobj = kset_find_obj(&module_subsys.kset, drv->mod_name); + if (!mkobj) + return; + mk = container_of(mkobj, struct module_kobject, kobj); } + sysfs_remove_link(mk->drivers_dir, driver_name); + kobject_put(mkobj); + kfree(driver_name); } EXPORT_SYMBOL(module_remove_driver); #endif ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-15 5:27 ` Greg KH @ 2007-03-15 7:53 ` Mike Galbraith 2007-03-15 8:06 ` Greg KH 0 siblings, 1 reply; 16+ messages in thread From: Mike Galbraith @ 2007-03-15 7:53 UTC (permalink / raw) To: Greg KH; +Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Wed, 2007-03-14 at 22:27 -0700, Greg KH wrote: > But can you try this version instead? It exploded in strcmp(). Unfortunately, the full oops didn't make it to either console or serial console. [ 30.783048] ipmi message handler version 39.1 [ 30.787632] ipmi device interface [ 30.791166] IPMI System Interface driver. [ 30.816961] BUG: unable to handle kernel NULL pointer dereference at virtual address 00000000 [ 30.832700] printing eip: [ 30.842383] c02d4098 [ 30.851458] *pde = 00000000 [ 30.861089] Oops: 0000 [#1] [ 30.870585] PREEMPT SMP [ 30.879724] Modules linked in: [ 30.889288] CPU: 1 [ 30.889290] EIP: 0060:[<c02d4098>] Not tainted VLI [ 30.889292] EFLAGS: 00010282 (2.6.21-rc3-smp #81) I did this... --- kernel/module.c.org 2007-03-15 07:20:15.000000000 +0100 +++ kernel/module.c 2007-03-15 08:13:36.000000000 +0100 @@ -2419,15 +2419,19 @@ void module_remove_driver(struct device_ if (drv->owner && drv->owner->mkobj.drivers_dir) mk = &drv->owner->mkobj; - else { + else if (drv->mod_name) { /* Lookup built-in module entry in /sys/modules */ mkobj = kset_find_obj(&module_subsys.kset, drv->mod_name); if (!mkobj) - return; + goto out_free; mk = container_of(mkobj, struct module_kobject, kobj); + } else { + WARN_ON(1); + goto out_free; } sysfs_remove_link(mk->drivers_dir, driver_name); kobject_put(mkobj); +out_free: kfree(driver_name); } EXPORT_SYMBOL(module_remove_driver); ...and it booted. [ 24.670410] ipmi message handler version 39.1 [ 24.675000] ipmi device interface [ 24.678542] IPMI System Interface driver. [ 24.703956] BUG: at kernel/module.c:2429 module_remove_driver() [ 24.716837] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 24.728909] [<c01057b2>] show_trace+0x12/0x14 [ 24.740239] [<c0105856>] dump_stack+0x16/0x18 [ 24.751469] [<c01441ce>] module_remove_driver+0xa5/0xa7 [ 24.763584] [<c035a014>] bus_remove_driver+0x6d/0x82 [ 24.775390] [<c035a978>] driver_unregister+0xb/0x18 [ 24.787019] [<c034c8c2>] init_ipmi_si+0x7a9/0x7c1 [ 24.798450] [<c06705bc>] init+0x144/0x26c [ 24.809129] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 24.820916] ======================= [ 24.830926] ipmi_si: Unable to find any System Interface(s) [ 24.842952] IPMI Watchdog: driver initialized 24.853749] Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot. -Mike ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-15 7:53 ` Mike Galbraith @ 2007-03-15 8:06 ` Greg KH 2007-03-15 8:32 ` Mike Galbraith 0 siblings, 1 reply; 16+ messages in thread From: Greg KH @ 2007-03-15 8:06 UTC (permalink / raw) To: Mike Galbraith Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Thu, Mar 15, 2007 at 08:53:07AM +0100, Mike Galbraith wrote: > On Wed, 2007-03-14 at 22:27 -0700, Greg KH wrote: > > > But can you try this version instead? > > It exploded in strcmp(). Unfortunately, the full oops didn't make it to > either console or serial console. > > [ 30.783048] ipmi message handler version 39.1 > [ 30.787632] ipmi device interface > [ 30.791166] IPMI System Interface driver. > [ 30.816961] BUG: unable to handle kernel NULL pointer dereference at virtual address 00000000 > [ 30.832700] printing eip: > [ 30.842383] c02d4098 > [ 30.851458] *pde = 00000000 > [ 30.861089] Oops: 0000 [#1] > [ 30.870585] PREEMPT SMP > [ 30.879724] Modules linked in: > [ 30.889288] CPU: 1 > [ 30.889290] EIP: 0060:[<c02d4098>] Not tainted VLI > [ 30.889292] EFLAGS: 00010282 (2.6.21-rc3-smp #81) Gah, that just happened to me too, sorry for not booting with that change before sending it to you. > I did this... > > --- kernel/module.c.org 2007-03-15 07:20:15.000000000 +0100 > +++ kernel/module.c 2007-03-15 08:13:36.000000000 +0100 > @@ -2419,15 +2419,19 @@ void module_remove_driver(struct device_ > > if (drv->owner && drv->owner->mkobj.drivers_dir) > mk = &drv->owner->mkobj; > - else { > + else if (drv->mod_name) { > /* Lookup built-in module entry in /sys/modules */ > mkobj = kset_find_obj(&module_subsys.kset, drv->mod_name); > if (!mkobj) > - return; > + goto out_free; > mk = container_of(mkobj, struct module_kobject, kobj); > + } else { > + WARN_ON(1); > + goto out_free; > } > sysfs_remove_link(mk->drivers_dir, driver_name); > kobject_put(mkobj); > +out_free: > kfree(driver_name); > } > EXPORT_SYMBOL(module_remove_driver); > > ...and it booted. That's good. But why don't we have a module name for this driver? And if we don't have a module name, why would there be a symlink to remove? That's what is keeping your module from unloading, right? > [ 24.670410] ipmi message handler version 39.1 > [ 24.675000] ipmi device interface > [ 24.678542] IPMI System Interface driver. > [ 24.703956] BUG: at kernel/module.c:2429 module_remove_driver() > [ 24.716837] [<c0105086>] show_trace_log_lvl+0x1a/0x30 > [ 24.728909] [<c01057b2>] show_trace+0x12/0x14 > [ 24.740239] [<c0105856>] dump_stack+0x16/0x18 > [ 24.751469] [<c01441ce>] module_remove_driver+0xa5/0xa7 > [ 24.763584] [<c035a014>] bus_remove_driver+0x6d/0x82 > [ 24.775390] [<c035a978>] driver_unregister+0xb/0x18 > [ 24.787019] [<c034c8c2>] init_ipmi_si+0x7a9/0x7c1 > [ 24.798450] [<c06705bc>] init+0x144/0x26c > [ 24.809129] [<c0104cfb>] kernel_thread_helper+0x7/0x1c > [ 24.820916] ======================= > [ 24.830926] ipmi_si: Unable to find any System Interface(s) > [ 24.842952] IPMI Watchdog: driver initialized > 24.853749] Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot. With the above change, it all works correctly? If the ipmi driver is loaded, what does the /sys/module/MODULE_NAME/ tree look like (replacing MODULE_NAME with whatever the module name really is, sorry, I don't know)? thanks, greg k-h ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-15 8:06 ` Greg KH @ 2007-03-15 8:32 ` Mike Galbraith 2007-03-15 9:39 ` Mike Galbraith [not found] ` <1173953960.6624.45.camel@Homer.simpson.net> 0 siblings, 2 replies; 16+ messages in thread From: Mike Galbraith @ 2007-03-15 8:32 UTC (permalink / raw) To: Greg KH; +Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Thu, 2007-03-15 at 01:06 -0700, Greg KH wrote: > That's good. But why don't we have a module name for this driver? > > And if we don't have a module name, why would there be a symlink to > remove? That's what is keeping your module from unloading, right? Ya got me, but according to my debug logs, what's causing my lockup is the reference we add while making the symlink when we hit... if (driver_name) in module_add_driver(). Maybe we go through there twice, once with a name, and once without? Dunno. > > [ 24.670410] ipmi message handler version 39.1 > > [ 24.675000] ipmi device interface > > [ 24.678542] IPMI System Interface driver. > > [ 24.703956] BUG: at kernel/module.c:2429 module_remove_driver() > > [ 24.716837] [<c0105086>] show_trace_log_lvl+0x1a/0x30 > > [ 24.728909] [<c01057b2>] show_trace+0x12/0x14 > > [ 24.740239] [<c0105856>] dump_stack+0x16/0x18 > > [ 24.751469] [<c01441ce>] module_remove_driver+0xa5/0xa7 > > [ 24.763584] [<c035a014>] bus_remove_driver+0x6d/0x82 > > [ 24.775390] [<c035a978>] driver_unregister+0xb/0x18 > > [ 24.787019] [<c034c8c2>] init_ipmi_si+0x7a9/0x7c1 > > [ 24.798450] [<c06705bc>] init+0x144/0x26c > > [ 24.809129] [<c0104cfb>] kernel_thread_helper+0x7/0x1c > > [ 24.820916] ======================= > > [ 24.830926] ipmi_si: Unable to find any System Interface(s) > > [ 24.842952] IPMI Watchdog: driver initialized > > 24.853749] Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot. > > With the above change, it all works correctly? I don't know about _correctly_, but my diag patch _boots_, as does your patch plus my addon diag bits. > If the ipmi driver is loaded, what does the /sys/module/MODULE_NAME/ > tree look like (replacing MODULE_NAME with whatever the module name > really is, sorry, I don't know)? Well, I will never see that, because ipmi_si finds no interfaces, so always backs out. After backout, with my patch and yours + my addons, I have there leftovers. root@Homer: ls -R /sys/module/ipmi_si /sys/module/ipmi_si: drivers parameters /sys/module/ipmi_si/drivers: /sys/module/ipmi_si/parameters: bt_debug hotmod kcs_debug smic_debug ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-15 8:32 ` Mike Galbraith @ 2007-03-15 9:39 ` Mike Galbraith [not found] ` <1173953960.6624.45.camel@Homer.simpson.net> 1 sibling, 0 replies; 16+ messages in thread From: Mike Galbraith @ 2007-03-15 9:39 UTC (permalink / raw) To: Greg KH; +Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Thu, 2007-03-15 at 09:32 +0100, Mike Galbraith wrote: > On Thu, 2007-03-15 at 01:06 -0700, Greg KH wrote: > > > That's good. But why don't we have a module name for this driver? > > > > And if we don't have a module name, why would there be a symlink to > > remove? That's what is keeping your module from unloading, right? > > Ya got me, but according to my debug logs, what's causing my lockup is > the reference we add while making the symlink when we hit... > if (driver_name) in module_add_driver(). Maybe we go through there > twice, once with a name, and once without? Dunno. I found the log (i think), and even with the patched kernel, gdb still says... (gdb) list *module_add_driver+0x44 0xc01442e5 is in module_add_driver (kernel/module.c:2401). 2396 driver_name = make_driver_name(drv); 2397 if (driver_name) { 2398 module_create_drivers_dir(mk); 2399 no_warn = sysfs_create_link(mk->drivers_dir, &drv->kobj, 2400 driver_name); 2401 kfree(driver_name); 2402 } 2403 } 2404 EXPORT_SYMBOL(module_add_driver); 2405 (gdb) See: kobject filter function caused the event to drop! (erm, that spot caused some hmm action here. if that drop is bad, i have another one as well.) [ 30.397160] kobject ipmi_devintf: registering. parent: <NULL>, set: module [ 30.404033] kobject_uevent_env [ 30.407098] fill_kobj_path: path = '/module/ipmi_devintf' [ 30.412524] BUG: at lib/kobject.c:448 kobject_get() [ 30.417402] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 30.422564] [<c01057b2>] show_trace+0x12/0x14 [ 30.427031] [<c0105856>] dump_stack+0x16/0x18 [ 30.431501] [<c02d3c2b>] kobject_get+0x66/0x87 [ 30.436056] [<c02d3f1e>] kobject_shadow_add+0x10/0x1e8 [ 30.441312] [<c02d4100>] kobject_add+0xa/0xc [ 30.445695] [<c06804c0>] kernel_param_sysfs_setup+0x4d/0x7f [ 30.451376] [<c068067a>] param_sysfs_init+0x188/0x1c3 [ 30.456538] [<c06725bc>] init+0x144/0x26c [ 30.460661] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 30.465907] ======================= [ 30.469486] get: c18f65c0 count after get is 2 [ 30.473927] kobject ipmi_si: registering. parent: <NULL>, set: module [ 30.480372] kobject_uevent_env [ 30.483430] fill_kobj_path: path = '/module/ipmi_si' ...... [ 73.266556] bus platform: add driver ipmi [ 73.278013] kobject ipmi: registering. parent: <NULL>, set: drivers [ 73.291847] kobject_uevent_env [ 73.302358] fill_kobj_path: path = '/bus/platform/drivers/ipmi' [ 73.315943] ipmi message handler version 39.1 [ 73.327839] ipmi device interface [ 73.338524] device class 'ipmi': registering [ 73.350158] subsystem ipmi: registering [ 73.361309] kobject ipmi: registering. parent: <NULL>, set: class [ 73.374841] bus platform: add driver ipmi_si [ 73.386442] BUG: at lib/kobject.c:448 kobject_get() [ 73.398617] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.411079] [<c01057b2>] show_trace+0x12/0x14 [ 73.422780] [<c0105856>] dump_stack+0x16/0x18 [ 73.434324] [<c02d3c2b>] kobject_get+0x66/0x87 [ 73.445860] [<c02d3f1e>] kobject_shadow_add+0x10/0x1e8 [ 73.458088] [<c02d4100>] kobject_add+0xa/0xc [ 73.469286] [<c02d4248>] kobject_register+0x22/0xb3 [ 73.480986] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 73.492592] [<c035b109>] driver_register+0x54/0x84 [ 73.504101] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.515335] [<c06725bc>] init+0x144/0x26c [ 73.525822] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.537452] ======================= [ 73.547290] get: c064475c count after get is 2 [ 73.557969] kobject ipmi_si: registering. parent: <NULL>, set: drivers [ 73.570825] kobject_uevent_env [ 73.580011] fill_kobj_path: path = '/bus/platform/drivers/ipmi_si' [ 73.592622] BUG: at lib/kobject.c:242 kobject_register() [ 73.604312] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.615822] [<c01057b2>] show_trace+0x12/0x14 [ 73.626551] [<c0105856>] dump_stack+0x16/0x18 [ 73.637211] [<c02d429f>] kobject_register+0x79/0xb3 [ 73.648325] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 73.659297] [<c035b109>] driver_register+0x54/0x84 [ 73.670164] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.680756] [<c06725bc>] init+0x144/0x26c [ 73.690699] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.701854] ======================= [ 73.711295] register: c064475c count now is 2 error 0 [ 73.722218] BUG: at lib/kobject.c:448 kobject_get() [ 73.732911] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.743927] [<c01057b2>] show_trace+0x12/0x14 [ 73.754112] [<c0105856>] dump_stack+0x16/0x18 [ 73.764253] [<c02d3c2b>] kobject_get+0x66/0x87 [ 73.774532] [<c035b157>] get_driver+0x11/0x18 [ 73.784725] [<c035b194>] driver_create_file+0xf/0x32 [ 73.795559] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 73.806343] [<c035b109>] driver_register+0x54/0x84 [ 73.817055] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.827595] [<c06725bc>] init+0x144/0x26c [ 73.837468] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.848544] ======================= [ 73.857932] get: c064475c count after get is 3 [ 73.868223] BUG: at lib/kobject.c:494 kobject_put() [ 73.878943] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 73.890019] [<c01057b2>] show_trace+0x12/0x14 [ 73.900239] [<c0105856>] dump_stack+0x16/0x18 [ 73.910294] [<c02d3ae7>] kobject_put+0x69/0x82 [ 73.920304] [<c035b144>] put_driver+0xb/0xd [ 73.929909] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 73.940362] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 73.950686] [<c035b109>] driver_register+0x54/0x84 [ 73.960966] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 73.971055] [<c06725bc>] init+0x144/0x26c [ 73.980478] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 73.991119] ======================= [ 74.000050] put: c064475c count before put is 3 [ 74.010026] BUG: at lib/kobject.c:448 kobject_get() [ 74.020367] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.031011] [<c01057b2>] show_trace+0x12/0x14 [ 74.040867] [<c0105856>] dump_stack+0x16/0x18 [ 74.050602] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.060353] [<c035b157>] get_driver+0x11/0x18 [ 74.070034] [<c035b194>] driver_create_file+0xf/0x32 [ 74.080418] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 74.090742] [<c035b109>] driver_register+0x54/0x84 [ 74.100979] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 74.111041] [<c06725bc>] init+0x144/0x26c [ 74.120431] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.131047] ======================= [ 74.139949] get: c064475c count after get is 3 [ 74.149807] BUG: at lib/kobject.c:494 kobject_put() [ 74.160111] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.170739] [<c01057b2>] show_trace+0x12/0x14 [ 74.180569] [<c0105856>] dump_stack+0x16/0x18 [ 74.190286] [<c02d3ae7>] kobject_put+0x69/0x82 [ 74.200019] [<c035b144>] put_driver+0xb/0xd [ 74.209503] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 74.219982] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 74.230312] [<c035b109>] driver_register+0x54/0x84 [ 74.240550] [<c034c486>] init_ipmi_si+0x4d/0x829 [ 74.250614] [<c06725bc>] init+0x144/0x26c [ 74.260001] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.270618] ======================= [ 74.279530] put: c064475c count before put is 3 [ 74.289481] IPMI System Interface driver. [ 74.298864] bus pci: add driver ipmi_si [ 74.308007] BUG: at lib/kobject.c:448 kobject_get() [ 74.318260] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.328861] [<c01057b2>] show_trace+0x12/0x14 [ 74.338637] [<c0105856>] dump_stack+0x16/0x18 [ 74.348372] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.358245] [<c02d3f1e>] kobject_shadow_add+0x10/0x1e8 [ 74.368837] [<c02d4100>] kobject_add+0xa/0xc [ 74.378528] [<c02d4248>] kobject_register+0x22/0xb3 [ 74.388817] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 74.399027] [<c035b109>] driver_register+0x54/0x84 [ 74.409229] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.420004] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.430196] [<c06725bc>] init+0x144/0x26c [ 74.439611] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.450184] ======================= [ 74.459045] get: c0644890 count after get is 2 [ 74.468796] kobject ipmi_si: registering. parent: <NULL>, set: drivers [ 74.480769] kobject_uevent_env [ 74.489210] fill_kobj_path: path = '/bus/pci/drivers/ipmi_si' [ 74.500675] BUG: at lib/kobject.c:242 kobject_register() [ 74.511779] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.522812] [<c01057b2>] show_trace+0x12/0x14 [ 74.533082] [<c0105856>] dump_stack+0x16/0x18 [ 74.543269] [<c02d429f>] kobject_register+0x79/0xb3 [ 74.554015] [<c035a13e>] bus_add_driver+0x77/0x1ae [ 74.564685] [<c035b109>] driver_register+0x54/0x84 [ 74.575363] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.586595] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.597161] [<c06725bc>] init+0x144/0x26c [ 74.606974] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.617989] ======================= [ 74.627292] register: c0644890 count now is 2 error 0 [ 74.638150] BUG: at lib/kobject.c:448 kobject_get() [ 74.648812] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.659872] [<c01057b2>] show_trace+0x12/0x14 [ 74.670237] [<c0105856>] dump_stack+0x16/0x18 [ 74.680554] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.690962] [<c02d434c>] kset_find_obj+0x73/0x77 [ 74.701563] [<c01442c5>] module_add_driver+0x64/0x7d [ 74.712578] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 74.723405] [<c035b109>] driver_register+0x54/0x84 [ 74.734247] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.745636] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.756393] [<c06725bc>] init+0x144/0x26c [ 74.766429] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.777654] ======================= [ 74.787103] get: c18f65c0 count after get is 3 [ 74.797461] BUG: at lib/kobject.c:448 kobject_get() [ 74.808209] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.819241] [<c01057b2>] show_trace+0x12/0x14 [ 74.829427] [<c0105856>] dump_stack+0x16/0x18 [ 74.839446] [<c02d3c2b>] kobject_get+0x66/0x87 [ 74.849465] [<c01b4812>] sysfs_create_link+0xd3/0x151 [ 74.860178] [<c0144285>] module_add_driver+0x24/0x7d [ 74.870866] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 74.881370] [<c035b109>] driver_register+0x54/0x84 [ 74.891850] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 74.902918] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 74.913354] [<c06725bc>] init+0x144/0x26c [ 74.923019] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 74.933904] ======================= [ 74.943042] get: c18f65c0 count after get is 4 [ 74.953046] BUG: at lib/kobject.c:448 kobject_get() [ 74.963497] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 74.974262] [<c01057b2>] show_trace+0x12/0x14 [ 74.984220] [<c0105856>] dump_stack+0x16/0x18 [ 74.994120] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.004167] [<c02d3f4a>] kobject_shadow_add+0x3c/0x1e8 [ 75.014941] [<c02d4100>] kobject_add+0xa/0xc [ 75.024787] [<c02d4248>] kobject_register+0x22/0xb3 [ 75.035216] [<c02d4431>] kobject_add_dir+0x4f/0x6e [ 75.045511] [<c01442d9>] module_add_driver+0x78/0x7d [ 75.055991] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 75.066262] [<c035b109>] driver_register+0x54/0x84 [ 75.076489] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.087297] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.097483] [<c06725bc>] init+0x144/0x26c [ 75.106862] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.117454] ======================= [ 75.126306] get: c18f65c0 count after get is 5 [ 75.136022] kobject drivers: registering. parent: ipmi_si, set: <NULL> [ 75.148068] kobject_uevent_env [ 75.156609] kobject filter function caused the event to drop! [ 75.168050] BUG: at lib/kobject.c:448 kobject_get() [ 75.178631] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.189595] [<c01057b2>] show_trace+0x12/0x14 [ 75.199789] [<c0105856>] dump_stack+0x16/0x18 [ 75.209905] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.220072] [<c01b4812>] sysfs_create_link+0xd3/0x151 <== here [ 75.230853] [<c01442a5>] module_add_driver+0x44/0x7d [ 75.241531] [<c035a170>] bus_add_driver+0xa9/0x1ae [ 75.252030] [<c035b109>] driver_register+0x54/0x84 [ 75.262455] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.273444] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.283813] [<c06725bc>] init+0x144/0x26c [ 75.293382] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.304189] ======================= [ 75.313370] get: c0644890 count after get is 3 [ 75.323512] BUG: at lib/kobject.c:448 kobject_get() [ 75.334155] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.345215] [<c01057b2>] show_trace+0x12/0x14 [ 75.355547] [<c0105856>] dump_stack+0x16/0x18 [ 75.365827] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.376185] [<c035b157>] get_driver+0x11/0x18 [ 75.386420] [<c035b194>] driver_create_file+0xf/0x32 [ 75.397291] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 75.408056] [<c035b109>] driver_register+0x54/0x84 [ 75.418682] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.429904] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.440480] [<c06725bc>] init+0x144/0x26c [ 75.450257] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.461238] ======================= [ 75.470402] get: c0644890 count after get is 4 [ 75.480404] BUG: at lib/kobject.c:494 kobject_put() [ 75.490866] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.501613] [<c01057b2>] show_trace+0x12/0x14 [ 75.511530] [<c0105856>] dump_stack+0x16/0x18 [ 75.521420] [<c02d3ae7>] kobject_put+0x69/0x82 [ 75.531449] [<c035b144>] put_driver+0xb/0xd [ 75.541209] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 75.551845] [<c035a221>] bus_add_driver+0x15a/0x1ae [ 75.562324] [<c035b109>] driver_register+0x54/0x84 [ 75.572682] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.583628] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.593942] [<c06725bc>] init+0x144/0x26c [ 75.603469] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.614233] ======================= [ 75.623258] put: c0644890 count before put is 4 [ 75.633227] BUG: at lib/kobject.c:448 kobject_get() [ 75.643575] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.654272] [<c01057b2>] show_trace+0x12/0x14 [ 75.664153] [<c0105856>] dump_stack+0x16/0x18 [ 75.673991] [<c02d3c2b>] kobject_get+0x66/0x87 [ 75.683977] [<c035b157>] get_driver+0x11/0x18 [ 75.693884] [<c035b194>] driver_create_file+0xf/0x32 [ 75.704423] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 75.714893] [<c035b109>] driver_register+0x54/0x84 [ 75.725245] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.736189] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.746496] [<c06725bc>] init+0x144/0x26c [ 75.756023] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.766778] ======================= [ 75.775786] get: c0644890 count after get is 4 [ 75.785652] BUG: at lib/kobject.c:494 kobject_put() [ 75.795990] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.806687] [<c01057b2>] show_trace+0x12/0x14 [ 75.816577] [<c0105856>] dump_stack+0x16/0x18 [ 75.826416] [<c02d3ae7>] kobject_put+0x69/0x82 [ 75.836390] [<c035b144>] put_driver+0xb/0xd [ 75.846107] [<c035b1b0>] driver_create_file+0x2b/0x32 [ 75.856744] [<c035a233>] bus_add_driver+0x16c/0x1ae [ 75.867222] [<c035b109>] driver_register+0x54/0x84 [ 75.877589] [<c02e2999>] __pci_register_driver+0x7e/0xb3 [ 75.888545] [<c034c830>] init_ipmi_si+0x3f7/0x829 [ 75.898868] [<c06725bc>] init+0x144/0x26c [ 75.908412] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 75.919193] ======================= [ 75.928228] put: c0644890 count before put is 4 [ 75.949537] ipmi_si: pci_unregister_driver() [ 75.959258] bus pci: remove driver ipmi_si [ 75.968766] BUG: at lib/kobject.c:448 kobject_get() [ 75.979081] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 75.989802] [<c01057b2>] show_trace+0x12/0x14 [ 75.999832] [<c0105856>] dump_stack+0x16/0x18 [ 76.009799] [<c02d3c2b>] kobject_get+0x66/0x87 [ 76.019863] [<c035b157>] get_driver+0x11/0x18 [ 76.029822] [<c035b16c>] driver_remove_file+0xe/0x27 [ 76.040389] [<c035a019>] bus_remove_driver+0x3a/0xa4 [ 76.050989] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.061432] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.072312] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.082589] [<c06725bc>] init+0x144/0x26c [ 76.092090] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.102742] ======================= [ 76.111690] get: c0644890 count after get is 4 [ 76.121537] BUG: at lib/kobject.c:494 kobject_put() [ 76.131808] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.142477] [<c01057b2>] show_trace+0x12/0x14 [ 76.152463] [<c0105856>] dump_stack+0x16/0x18 [ 76.162397] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.172408] [<c035b144>] put_driver+0xb/0xd [ 76.182134] [<c035b181>] driver_remove_file+0x23/0x27 [ 76.192759] [<c035a019>] bus_remove_driver+0x3a/0xa4 [ 76.203309] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.213726] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.224578] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.234840] [<c06725bc>] init+0x144/0x26c [ 76.244316] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.254951] ======================= [ 76.263853] put: c0644890 count before put is 4 [ 76.273754] BUG: at lib/kobject.c:448 kobject_get() [ 76.284057] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.294677] [<c01057b2>] show_trace+0x12/0x14 [ 76.304462] [<c0105856>] dump_stack+0x16/0x18 [ 76.314213] [<c02d3c2b>] kobject_get+0x66/0x87 [ 76.324122] [<c035b157>] get_driver+0x11/0x18 [ 76.333942] [<c035b16c>] driver_remove_file+0xe/0x27 [ 76.344405] [<c035a025>] bus_remove_driver+0x46/0xa4 [ 76.354882] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.365243] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.376110] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.386356] [<c06725bc>] init+0x144/0x26c [ 76.395830] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.406544] ======================= [ 76.415524] get: c0644890 count after get is 4 [ 76.425362] BUG: at lib/kobject.c:494 kobject_put() [ 76.435659] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.446287] [<c01057b2>] show_trace+0x12/0x14 [ 76.456107] [<c0105856>] dump_stack+0x16/0x18 [ 76.465876] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.475800] [<c035b144>] put_driver+0xb/0xd [ 76.485457] [<c035b181>] driver_remove_file+0x23/0x27 [ 76.496030] [<c035a025>] bus_remove_driver+0x46/0xa4 [ 76.506536] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.516911] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.527772] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.538017] [<c06725bc>] init+0x144/0x26c [ 76.547492] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.558194] ======================= [ 76.567160] put: c0644890 count before put is 4 [ 76.577094] BUG: at lib/kobject.c:494 kobject_put() [ 76.587399] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.598051] [<c01057b2>] show_trace+0x12/0x14 [ 76.607880] [<c0105856>] dump_stack+0x16/0x18 [ 76.617676] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.627617] [<c01b27a3>] sysfs_hash_and_remove+0x116/0x11f [ 76.638712] [<c01b473d>] sysfs_remove_link+0xb/0xd [ 76.649096] [<c0144151>] module_remove_driver+0x18/0x57 [ 76.659972] [<c035a06b>] bus_remove_driver+0x8c/0xa4 [ 76.670600] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.681138] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.692128] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.702470] [<c06725bc>] init+0x144/0x26c [ 76.712092] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.722899] ======================= [ 76.731916] put: c18f65c0 count before put is 5 [ 76.741944] kobject ipmi_si: unregistering [ 76.751496] kobject_uevent_env [ 76.759873] fill_kobj_path: path = '/bus/pci/drivers/ipmi_si' [ 76.771235] BUG: at lib/kobject.c:494 kobject_put() [ 76.781811] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.792782] [<c01057b2>] show_trace+0x12/0x14 [ 76.803009] [<c0105856>] dump_stack+0x16/0x18 [ 76.813168] [<c02d3ae7>] kobject_put+0x69/0x82 [ 76.823379] [<c02d3bc2>] unlink+0x3b/0x3e [ 76.833121] [<c02d3caf>] kobject_del+0x16/0x19 [ 76.843272] [<c02d3ce6>] kobject_unregister+0x34/0xa1 [ 76.854002] [<c035a073>] bus_remove_driver+0x94/0xa4 [ 76.864671] [<c035b0a8>] driver_unregister+0xb/0x18 [ 76.875185] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 76.886124] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 76.896439] [<c06725bc>] init+0x144/0x26c [ 76.905974] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 76.916676] ======================= [ 76.925694] put: c0644890 count before put is 3 [ 76.935799] BUG: at lib/kobject.c:430 kobject_unregister() [ 76.946990] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 76.957893] [<c01057b2>] show_trace+0x12/0x14 [ 76.968033] [<c0105856>] dump_stack+0x16/0x18 [ 76.978063] [<c02d3d3a>] kobject_unregister+0x88/0xa1 [ 76.988811] [<c035a073>] bus_remove_driver+0x94/0xa4 [ 76.999498] [<c035b0a8>] driver_unregister+0xb/0x18 [ 77.010064] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 77.021089] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 77.031524] [<c06725bc>] init+0x144/0x26c [ 77.041179] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 77.051980] ======================= [ 77.061057] unregister: c0644890 count before put is 2 [ 77.071811] BUG: at lib/kobject.c:494 kobject_put() [ 77.082360] [<c0105086>] show_trace_log_lvl+0x1a/0x30 [ 77.093203] [<c01057b2>] show_trace+0x12/0x14 [ 77.103189] [<c0105856>] dump_stack+0x16/0x18 [ 77.113053] [<c02d3ae7>] kobject_put+0x69/0x82 [ 77.123022] [<c02d3d09>] kobject_unregister+0x57/0xa1 [ 77.133655] [<c035a073>] bus_remove_driver+0x94/0xa4 [ 77.144247] [<c035b0a8>] driver_unregister+0xb/0x18 [ 77.154753] [<c02e282c>] pci_unregister_driver+0x13/0x73 [ 77.165770] [<c034cc34>] init_ipmi_si+0x7fb/0x829 [ 77.176205] [<c06725bc>] init+0x144/0x26c [ 77.185880] [<c0104cfb>] kernel_thread_helper+0x7/0x1c [ 77.196791] ======================= [ 77.205972] put: c0644890 count before put is 2 ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <1173953960.6624.45.camel@Homer.simpson.net>]
* Re: kref refcounting breakage in mainline [not found] ` <1173953960.6624.45.camel@Homer.simpson.net> @ 2007-03-15 14:54 ` Greg KH 2007-03-19 23:41 ` Randy Dunlap 0 siblings, 1 reply; 16+ messages in thread From: Greg KH @ 2007-03-15 14:54 UTC (permalink / raw) To: Mike Galbraith Cc: Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Thu, Mar 15, 2007 at 11:19:20AM +0100, Mike Galbraith wrote: > On Thu, 2007-03-15 at 01:06 -0700, Greg KH wrote: > > > That's good. But why don't we have a module name for this driver? > > > > And if we don't have a module name, why would there be a symlink to > > remove? That's what is keeping your module from unloading, right? > > You keep saying "module", and that's making me a bit nervous ;-) > > Just to be sure we're not talking past each other, when you say module, > don't mean the modprobe kind... i hope. This "module" as in driver is > compiled in. (said that before, but you may have missed it) Ahh, that changes everything here, thanks for letting me know, I had missed this. The problem is that the module_init() is failing, yet this isn't really a module, it's built into the kernel. So some of the module teardown logic is dieing when it thinks that we really have a full module structure here (owner and such). I'll look at this further tomorrow, as I'm travelling pretty much all day today, sorry. thanks, greg k-h ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-15 14:54 ` Greg KH @ 2007-03-19 23:41 ` Randy Dunlap 0 siblings, 0 replies; 16+ messages in thread From: Randy Dunlap @ 2007-03-19 23:41 UTC (permalink / raw) To: Greg KH Cc: Mike Galbraith, Andrew Morton, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk On Thu, 15 Mar 2007 07:54:14 -0700 Greg KH wrote: > On Thu, Mar 15, 2007 at 11:19:20AM +0100, Mike Galbraith wrote: > > On Thu, 2007-03-15 at 01:06 -0700, Greg KH wrote: > > > > > That's good. But why don't we have a module name for this driver? > > > > > > And if we don't have a module name, why would there be a symlink to > > > remove? That's what is keeping your module from unloading, right? > > > > You keep saying "module", and that's making me a bit nervous ;-) > > > > Just to be sure we're not talking past each other, when you say module, > > don't mean the modprobe kind... i hope. This "module" as in driver is > > compiled in. (said that before, but you may have missed it) > > Ahh, that changes everything here, thanks for letting me know, I had > missed this. > > The problem is that the module_init() is failing, yet this isn't really > a module, it's built into the kernel. So some of the module teardown > logic is dieing when it thinks that we really have a full module > structure here (owner and such). Urgh, it's not a "loadable" module, but it's still a logical module. > I'll look at this further tomorrow, as I'm travelling pretty much all > day today, sorry. --- ~Randy *** Remember to use Documentation/SubmitChecklist when testing your code *** ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: kref refcounting breakage in mainline 2007-03-06 0:25 ` Greg KH 2007-03-06 5:43 ` Mike Galbraith @ 2007-03-06 12:11 ` Mel Gorman 1 sibling, 0 replies; 16+ messages in thread From: Mel Gorman @ 2007-03-06 12:11 UTC (permalink / raw) To: Greg KH Cc: Andrew Morton, Mike Galbraith, Tejun Heo, Kay Sievers, linux-kernel, Adrian Bunk, ipslinux On (05/03/07 16:25), Greg KH didst pronounce: > On Fri, Mar 02, 2007 at 12:58:33AM -0800, Andrew Morton wrote: > > > > -mm has a debugging patch which warns when atomic_dec_and_test() takes an > > atomic_t negative > > (ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.20/2.6.20-mm2/broken-out/detect-atomic-counter-underflows.patch). > > > > > > When it is applied to current mainline, a simple `rmmod ipw2200' gives: > > > > [ 75.825072] BUG: atomic counter underflow at: > > [ 75.825180] [<c01c6eb4>] kref_put+0x66/0x82 > > [ 75.825278] [<c022e4d4>] bus_remove_driver+0x66/0x75 > > [ 75.825383] [<c022ee2c>] driver_unregister+0x8/0x13 > > [ 75.825484] [<c01d7add>] pci_unregister_driver+0xc/0x45 > > [ 75.825593] [<c0132147>] sys_delete_module+0x157/0x17c > > [ 75.825703] [<c013c663>] audit_syscall_entry+0x10d/0x137 > > [ 75.825818] [<c0103b14>] syscall_call+0x7/0xb > > [ 75.825913] [<c02d0000>] xfrm4_dst_destroy+0xe/0xd5 > > > > This didn't happen in 2.6.20-mm2, so this bug was introduced by a patch > > which was not in the -mm lineup twelve days ago. > > > > Presumably the effect of this is a memory leak or a use-after-free. > > Ok, after a zillion bisects, I've tracked this down to: > commit 63ce18cfe685115ff8d341bae4c9204a79043cf0 > Author: Mike Galbraith <efault@gmx.de> > Date: Wed Feb 21 12:45:35 2007 -0800 > > driver core: refcounting fix > > Fix a reference counting bug exposed by commit > 725522b5453dd680412f2b6463a988e4fd148757. If driver.mod_name exists, we > take a reference in module_add_driver(), and never release it. Undo that > reference in module_remove_driver(). > > Signed-off-by: Mike Galbraith <efault@gmx.de> > Cc: Kay Sievers <kay.sievers@vrfy.org> > Signed-off-by: Andrew Morton <akpm@linux-foundation.org> > Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de> > > On http://test.kernel.org, elm3b132 is showing a similar error message for the IPS driver on 2.6.21-rc2-mm1. There is no such device in the machine so this is a normal error path for no devices found. The warning looks like BUG: atomic counter underflow at: [<c027ccb4>] kref_put+0x7d/0x9d [<c02a4e38>] bus_remove_driver+0x36/0x41 [<c02a5997>] driver_unregister+0xb/0x13 [<c0286bdd>] pci_unregister_driver+0xb/0x13 [<c04c653a>] ips_module_init+0x41/0x57 [<c04b2bd2>] do_initcalls+0x58/0xf5 [<c01353d5>] register_irq_proc+0x75/0x92 [<c04b2c97>] init+0x0/0x8b [<c04b2ce0>] init+0x49/0x8b [<c01030ff>] kernel_thread_helper+0x7/0x10 This is essentially identical to the warning on ipw2200. It occurs whether the driver is compiled into the kernel or as a module. Reverting Mike's patch fixes the problem - or at least the warning has disappeared. However, I've cc'd the IPS maintainers so they can confirm they are calling pci_unregister_driver() correctly. I note that many drivers in drivers/scsi do not call pci_unregister_driver() in the module_init path. ipw2200 also calls pci_unregister_driver() in the module_init path when an error is encountered. So, maybe this is an error in how the drivers use pci_[un]register_driver() instead of a problem with Mike's patch? > Mike, I've reverted this patch, and I don't see any references leaking. > And, as your patch released the reference on the driver, and the > module_add_driver() call would not grab a reference to the driver, only > the module kobject, I don't see what you were trying to fix with this > patch. > > Do you have a test case that this fixes? > > Otherwise, I'll just revert it. > > thanks, > > greg k-h > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- -- Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2007-03-19 23:43 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-03-02 8:58 kref refcounting breakage in mainline Andrew Morton
2007-03-03 5:48 ` Greg KH
2007-03-06 0:25 ` Greg KH
2007-03-06 5:43 ` Mike Galbraith
2007-03-06 21:04 ` Greg KH
2007-03-07 5:38 ` Mike Galbraith
2007-03-10 15:44 ` Mike Galbraith
2007-03-10 16:03 ` Mike Galbraith
2007-03-15 5:27 ` Greg KH
2007-03-15 7:53 ` Mike Galbraith
2007-03-15 8:06 ` Greg KH
2007-03-15 8:32 ` Mike Galbraith
2007-03-15 9:39 ` Mike Galbraith
[not found] ` <1173953960.6624.45.camel@Homer.simpson.net>
2007-03-15 14:54 ` Greg KH
2007-03-19 23:41 ` Randy Dunlap
2007-03-06 12:11 ` Mel Gorman
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox