2.6.25 sysdev API problem

2.6.25 sysdev API problem

[Mikael Pettersson]

In kernels up to and including 2.6.24, it was possible to
register-then-unregister a sysdev_class/sys_device pair
multiple times. Starting with the 2.6.24-git1 kernel, doing
so causes a warning

kobject (f88e96c8): tried to init an initialized object, something is seriously wrong

the second time the class/device pair is registered, followed
soon thereafter by random BUG()s and a kernel panic.

I'm seeing this problem with an out-of-tree but open-source driver
(the "perfctr" performance-monitoring counters driver), but similar
sysdev usage patterns can also be found in oprofile and kvm (though
I don't use them so I don't know if they have this problem or not).

Below is a simple driver module that illustrates the problem.
Inserting it into a 2.6.25-rc4 kernel first causes

foo_sysfs_init                                                             
foo_sysfs_exit                                                             
foo_sysfs_init                                                             
kobject (f88e96c8): tried to init an initialized object, something is seriously wrong
Pid: 2078, comm: insmod Not tainted 2.6.25-rc4 #1                          
 [<c01c3926>] kobject_init+0x76/0x80                                       
 [<c01c3db3>] kobject_init_and_add+0x13/0x40                               
 [<c02143cf>] sysdev_register+0x5f/0xd0                                    
 [<f88e904f>] foo_open_use_close+0x2f/0x80 [foo]                           
 [<f88eb00a>] foo_init+0xa/0xd [foo]                                       
 [<c013ca3c>] sys_init_module+0x11c/0x1ad0                                 
 [<c0151210>] handle_mm_fault+0x100/0x5f0                                  
 [<c02142b0>] sysdev_unregister+0x0/0x60                                   
 [<c0160f9e>] filp_close+0x3e/0x70                                         
 [<c0102f2a>] sysenter_past_esp+0x5f/0x85                                  
 =======================                                                   
foo_sysfs_exit

followed soon thereafter by 

------------[ cut here ]------------
Kernel BUG at c015f795 [verbose debug info unavailable]
invalid opcode: 0000 [#1] SMP 
Modules linked in: foo sunrpc af_packet snd_hda_intel snd_seq_oss snd_seq_midi_evee

Pid: 2112, comm: tcsh Not tainted (2.6.25-rc4 #1)
EIP: 0060:[<c015f795>] EFLAGS: 00010082 CPU: 1
EIP is at cache_alloc_refill+0x1c5/0x520
EAX: 00000001 EBX: 0000000c ECX: f78046c0 EDX: f7812bc0
ESI: ffffffff EDI: f686d3a0 EBP: f78236c0 ESP: f6a29f34
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process tcsh (pid: 2112, ti=f6a28000 task=f7b41aa0 task.ti=f6a28000)
Stack: 000000d0 000000d0 f7812bc8 f7812bd0 00000000 000000d0 f78046c0 f7817e00 
       f7812bc0 f71848c0 bfa03b78 00000001 f7169234 f735395c fffffff4 f78046c0 
       00000282 000000d0 c015f5c2 fffffff4 09815808 b7fc3ff4 f6a28000 c016ab08 
Call Trace:
 [<c015f5c2>] kmem_cache_alloc+0x82/0x90
 [<c016ab08>] getname+0x28/0xc0
 [<c010137e>] sys_execve+0xe/0x60
 [<c0102f2a>] sysenter_past_esp+0x5f/0x85
 [<c02a0000>] fn_hash_dump+0xd0/0x240
 =======================
Code: 00 83 c4 38 5b 5e 5f 5d c3 8b 7a 10 c7 42 34 01 00 00 00 39 7c 24 0c 74 b8 8 
EIP: [<c015f795>] cache_alloc_refill+0x1c5/0x520 SS:ESP 0068:f6a29f34
---[ end trace aeb1b82666c8fe26 ]---

The BUG() that finally kills the kernel often occurs in cache_alloc_refill(),
but it can also occur in other places.

/Mikael

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/sysdev.h>
#include <linux/version.h>

MODULE_LICENSE("GPL");

static int foo_suspend(struct sys_device *dev, pm_message_t state) { return 0; }

static int foo_resume(struct sys_device *dev) { return 0; }

static struct sysdev_class foo_sysdev_class = {
#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,24)
	.name		= "foo",
#else
	set_kset_name("foo"),
#endif
	.resume		= foo_resume,
	.suspend	= foo_suspend,
};

static struct sys_device foo_sys_device = {
	.id	= 0,
	.cls	= &foo_sysdev_class,
};

static int foo_sysfs_init(void)
{
	int err;

	printk("%s\n", __FUNCTION__);
	err = sysdev_class_register(&foo_sysdev_class);
	if (!err)
		err = sysdev_register(&foo_sys_device);
	if (err)
		printk("%s failed, err %d\n", __FUNCTION__, err);
	return err;
}

static void foo_sysfs_exit(void)
{
	printk("%s\n", __FUNCTION__);
	sysdev_unregister(&foo_sys_device);
	sysdev_class_unregister(&foo_sysdev_class);
}

static void foo_open_use_close(void)
{
	if (foo_sysfs_init() == 0)
		foo_sysfs_exit();
}

int __init foo_init(void)
{
	foo_open_use_close();
	foo_open_use_close();
	return 0;
}

void __exit foo_exit(void) { }

module_init(foo_init)
module_exit(foo_exit)

Re: 2.6.25 sysdev API problem

[Greg KH]

On Sat, Mar 08, 2008 at 04:56:39PM +0100, Mikael Pettersson wrote:
> In kernels up to and including 2.6.24, it was possible to
> register-then-unregister a sysdev_class/sys_device pair
> multiple times. Starting with the 2.6.24-git1 kernel, doing
> so causes a warning
> 
> kobject (f88e96c8): tried to init an initialized object, something is seriously wrong

This is a warning only, I have a patch queued up to fix this.  I've
included it below.

> the second time the class/device pair is registered, followed
> soon thereafter by random BUG()s and a kernel panic.

That's odd.  I don't think that is related, but it might be.  Can you
try the patch and let me know if it still happens?

thanks,

greg k-h


From: Balaji Rao <balajirrao@gmail.com>
Date: Thu, 6 Mar 2008 22:23:18 +0530
Subject: kobjects: mark cleaned up kobjects as unitialized
To: gregkh@suse.de
Cc: <kvm-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org
Message-ID: <200803062223.18857.balajirrao@gmail.com>
Content-Disposition: inline


When I remove only the kvm-intel module without removing the kvm module
itself, I get an error saying that a kobject is trying to be
reinitialized. Its because of the fact that kvm reuses a kobject in
kvm_init when calling sysdev_register.

This patch fixes kobject_cleanup by marking the kobject as uninitialized
when we cleanup to allow kobjects to be reused.

Signed-off-by: Balaji Rao <balajirrao@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>

---
 lib/kobject.c |    3 +++
 1 file changed, 3 insertions(+)

--- a/lib/kobject.c
+++ b/lib/kobject.c
@@ -577,6 +577,9 @@ static void kobject_cleanup(struct kobje
 		pr_debug("kobject: '%s': free name\n", name);
 		kfree(name);
 	}
+
+	/* Set the state to uninitialized */
+	kobj->state_initialized = 0;
 }
 
 static void kobject_release(struct kref *kref)

Re: 2.6.25 sysdev API problem

[Mikael Pettersson]

Greg KH writes:
 > On Sat, Mar 08, 2008 at 04:56:39PM +0100, Mikael Pettersson wrote:
 > > In kernels up to and including 2.6.24, it was possible to
 > > register-then-unregister a sysdev_class/sys_device pair
 > > multiple times. Starting with the 2.6.24-git1 kernel, doing
 > > so causes a warning
 > > 
 > > kobject (f88e96c8): tried to init an initialized object, something is seriously wrong
 > 
 > This is a warning only, I have a patch queued up to fix this.  I've
 > included it below.
 > 
 > > the second time the class/device pair is registered, followed
 > > soon thereafter by random BUG()s and a kernel panic.
 > 
 > That's odd.  I don't think that is related, but it might be.  Can you
 > try the patch and let me know if it still happens?

The patch silenced the warning, but I still got a BUG() in
cache_alloc_refill() shortly after loading the test module.

(The machine needs some activity before the BUG() happens,
a few sync;dmesg;ps commands suffice for me.)

If I change the test module to do the sysdev_register stuff
once in module_init and the unregister stuff once in module_exit,
as opposed to once per "session" using the device, then things
work fine and I can't crash the kernel even if I repeatedly
insmod and rmmod the module. This indicates that some other part
of the sysdev object's state, apart from the state_initialized
flag, must also be cleaned up in the unregister() path.

I'll continue investigating this tomorrow.

/Mikael

Re: 2.6.25 sysdev API problem

[Balaji Rao]

On Sunday 09 March 2008 12:17:17 am Greg KH wrote:
> On Sat, Mar 08, 2008 at 04:56:39PM +0100, Mikael Pettersson wrote:
> > In kernels up to and including 2.6.24, it was possible to
> > register-then-unregister a sysdev_class/sys_device pair
> > multiple times. Starting with the 2.6.24-git1 kernel, doing
> > so causes a warning
> >
> > kobject (f88e96c8): tried to init an initialized object, something is
> > seriously wrong
>
> This is a warning only, I have a patch queued up to fix this.  I've
> included it below.
>
> > the second time the class/device pair is registered, followed
> > soon thereafter by random BUG()s and a kernel panic.
>
> That's odd.  I don't think that is related, but it might be.  Can you
> try the patch and let me know if it still happens?
>
> thanks,
>
> greg k-h
>
>
> From: Balaji Rao <balajirrao@gmail.com>
> Date: Thu, 6 Mar 2008 22:23:18 +0530
> Subject: kobjects: mark cleaned up kobjects as unitialized
> To: gregkh@suse.de
> Cc: <kvm-devel@lists.sourceforge.net>, linux-kernel@vger.kernel.org
> Message-ID: <200803062223.18857.balajirrao@gmail.com>
> Content-Disposition: inline
>
>
> When I remove only the kvm-intel module without removing the kvm module
> itself, I get an error saying that a kobject is trying to be
> reinitialized. Its because of the fact that kvm reuses a kobject in
> kvm_init when calling sysdev_register.
>
> This patch fixes kobject_cleanup by marking the kobject as uninitialized
> when we cleanup to allow kobjects to be reused.
>
> Signed-off-by: Balaji Rao <balajirrao@gmail.com>
> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
>
> ---
>  lib/kobject.c |    3 +++
>  1 file changed, 3 insertions(+)
>
> --- a/lib/kobject.c
> +++ b/lib/kobject.c
> @@ -577,6 +577,9 @@ static void kobject_cleanup(struct kobje
>  		pr_debug("kobject: '%s': free name\n", name);
>  		kfree(name);
>  	}
> +
> +	/* Set the state to uninitialized */
> +	kobj->state_initialized = 0;
>  }
>
>  static void kobject_release(struct kref *kref)
Hi Mikael,

The above patch causes slab corruptions as shown by this thread.
http://lkml.org/lkml/2008/3/8/94

I've posted a test fix there. Please watch this thread for updates.

-- 
regards,
balaji rao
3rd year,
Dept. of Mechanical Engineering,
National Institute of Technology, Karnataka