Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Weng Meiling]

When the last child kobject was deleted, it's parent kobject will be deleted,
when removing the parent kobject if the parent kobject's sd has been set NULL
and still not been removed from it's kset's list, at the same time another one
trigger an device adding event, the function get_parent_device() will get the
parent object from the kset's list for kobject_add(), but this time parent
kobject's sd has been NULL. This race will make the sysfs_create_dir() return
ENOENT, the new kobject will be failed to added into sysfs and trigger BUG()
when creating attribute group under the new device's directory. So move the
kobject removal from kset's list before kobj->sd=NULL.

The race situation:

path0(remove parent kobj, e.g:/sys/devices/virtual/block/)   path1(register a new device)

kobject_del(){                                               get_device_parent(){
...                                                          ...
sysfs_remove_dir(kobj);  //kobj->sd=NULL                     spin_lock(&dev->class->p->glue_dirs.list_lock);
...                                                  <===    list_for_each_entry(k, &dev->class->p->glue_dirs.list, entry)
kobj_kset_leave(kobj);   //remove kobj from kset list        ...
}                                                           }


We had triggered the bug, the detail message link:
https://lkml.org/lkml/2014/10/13/40

Signed-off-by: Weng Meiling <wengmeiling.weng@huawei.com>
---
 lib/kobject.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/kobject.c b/lib/kobject.c
index 58751bb..af2b7bb 100644
--- a/lib/kobject.c
+++ b/lib/kobject.c
@@ -560,12 +560,13 @@ void kobject_del(struct kobject *kobj)
 	if (!kobj)
 		return;

+	kobj_kset_leave(kobj);
+
 	sd = kobj->sd;
 	sysfs_remove_dir(kobj);
 	sysfs_put(sd);

 	kobj->state_in_sysfs = 0;
-	kobj_kset_leave(kobj);
 	kobject_put(kobj->parent);
 	kobj->parent = NULL;
 }
-- 
1.8.2.2

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Weng Meiling]

Hi，

Would you please give me some of your views on this issue? Any suggestion is appreciative.

Thanks!

Weng Meiling

On 2014/10/15 14:42, Weng Meiling wrote:
> When the last child kobject was deleted, it's parent kobject will be deleted,
> when removing the parent kobject if the parent kobject's sd has been set NULL
> and still not been removed from it's kset's list, at the same time another one
> trigger an device adding event, the function get_parent_device() will get the
> parent object from the kset's list for kobject_add(), but this time parent
> kobject's sd has been NULL. This race will make the sysfs_create_dir() return
> ENOENT, the new kobject will be failed to added into sysfs and trigger BUG()
> when creating attribute group under the new device's directory. So move the
> kobject removal from kset's list before kobj->sd=NULL.
> 
> The race situation:
> 
> path0(remove parent kobj, e.g:/sys/devices/virtual/block/)   path1(register a new device)
> 
> kobject_del(){                                               get_device_parent(){
> ...                                                          ...
> sysfs_remove_dir(kobj);  //kobj->sd=NULL                     spin_lock(&dev->class->p->glue_dirs.list_lock);
> ...                                                  <===    list_for_each_entry(k, &dev->class->p->glue_dirs.list, entry)
> kobj_kset_leave(kobj);   //remove kobj from kset list        ...
> }                                                           }
> 
> 
> We had triggered the bug, the detail message link:
> https://lkml.org/lkml/2014/10/13/40
> 
> Signed-off-by: Weng Meiling <wengmeiling.weng@huawei.com>
> ---
>  lib/kobject.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/kobject.c b/lib/kobject.c
> index 58751bb..af2b7bb 100644
> --- a/lib/kobject.c
> +++ b/lib/kobject.c
> @@ -560,12 +560,13 @@ void kobject_del(struct kobject *kobj)
>  	if (!kobj)
>  		return;
> 
> +	kobj_kset_leave(kobj);
> +
>  	sd = kobj->sd;
>  	sysfs_remove_dir(kobj);
>  	sysfs_put(sd);
> 
>  	kobj->state_in_sysfs = 0;
> -	kobj_kset_leave(kobj);
>  	kobject_put(kobj->parent);
>  	kobj->parent = NULL;
>  }
> 

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Frans Klaver]

On Thu, Oct 16, 2014 at 3:56 AM, Weng Meiling
<wengmeiling.weng@huawei.com> wrote:
>
> Would you please give me some of your views on this issue? Any suggestion is appreciative.

It'll come. Be patient.

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Weng Meiling]

On 2014/10/16 15:07, Frans Klaver wrote:
> On Thu, Oct 16, 2014 at 3:56 AM, Weng Meiling
> <wengmeiling.weng@huawei.com> wrote:
>>
>> Would you please give me some of your views on this issue? Any suggestion is appreciative.
> 
> It'll come. Be patient.
> 
> .
> 
yeah, maybe I'm too impatient :). But this is important for me.
Thanks for your reply.

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Greg KH]

On Thu, Oct 16, 2014 at 03:23:53PM +0800, Weng Meiling wrote:
> On 2014/10/16 15:07, Frans Klaver wrote:
> > On Thu, Oct 16, 2014 at 3:56 AM, Weng Meiling
> > <wengmeiling.weng@huawei.com> wrote:
> >>
> >> Would you please give me some of your views on this issue? Any suggestion is appreciative.
> > 
> > It'll come. Be patient.
> > 
> > .
> > 
> yeah, maybe I'm too impatient :)

*plonk*

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Weng Meiling]

On 2014/10/16 17:13, Greg KH wrote:
> On Thu, Oct 16, 2014 at 03:23:53PM +0800, Weng Meiling wrote:
>> On 2014/10/16 15:07, Frans Klaver wrote:
>>> On Thu, Oct 16, 2014 at 3:56 AM, Weng Meiling
>>> <wengmeiling.weng@huawei.com> wrote:
>>>>
>>>> Would you please give me some of your views on this issue? Any suggestion is appreciative.
>>>
>>> It'll come. Be patient.
>>>
>>> .
>>>
>> yeah, maybe I'm too impatient :)
> 
> *plonk*
> 
> .
> 

The former patch still can't fix the race. Tejun had commited a fix patch about
race in get_device_parent():

commit 77d3d7c1d561f49f755d7390f0764dff90765974
Author: Tejun Heo <tj@kernel.org>
Date:   Fri Feb 5 17:57:02 2010 +0900

    driver-core: fix race condition in get_device_parent()

    sysfs is creating several devices in cuse class concurrently and with
    CONFIG_SYSFS_DEPRECATED turned off, it triggers the following oops.

     BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
     IP: [<ffffffff81158b0a>] sysfs_addrm_start+0x4a/0xf0
     PGD 75bb067 PUD 75be067 PMD 0
     Oops: 0000 [#1] PREEMPT SMP
     last sysfs file: /sys/devices/system/cpu/cpu7/topology/core_siblings
     CPU 1
     Modules linked in: cuse fuse
     Pid: 4737, comm: osspd Not tainted 2.6.31-work #77
     RIP: 0010:[<ffffffff81158b0a>]  [<ffffffff81158b0a>] sysfs_addrm_start+0x4a/0xf0
     RSP: 0018:ffff88000042f8f8  EFLAGS: 00010296
     RAX: ffff88000042ffd8 RBX: 0000000000000000 RCX: 0000000000000000
     RDX: 0000000000000000 RSI: ffff880007eef660 RDI: 0000000000000001
     RBP: ffff88000042f918 R08: 0000000000000000 R09: 0000000000000000
     R10: 0000000000000001 R11: ffffffff81158b0a R12: ffff88000042f928
     R13: 00000000fffffff4 R14: 0000000000000000 R15: ffff88000042f9a0
     FS:  00007fe93905a950(0000) GS:ffff880008600000(0000) knlGS:0000000000000000
     CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     CR2: 0000000000000038 CR3: 00000000077c9000 CR4: 00000000000006e0
     DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     Process osspd (pid: 4737, threadinfo ffff88000042e000, task ffff880007eef040)
     Stack:
      ffff880005da10e8 0000000011cc8d6e ffff88000042f928 ffff880003d28a28
     <0> ffff88000042f988 ffffffff811592d7 0000000000000000 0000000000000000
     <0> 0000000000000000 0000000000000000 ffff88000042f958 0000000011cc8d6e
     Call Trace:
      [<ffffffff811592d7>] create_dir+0x67/0xe0
      [<ffffffff811593a8>] sysfs_create_dir+0x58/0xb0
      [<ffffffff8128ca7c>] ? kobject_add_internal+0xcc/0x220
      [<ffffffff812942e1>] ? vsnprintf+0x3c1/0xb90
      [<ffffffff8128cab7>] kobject_add_internal+0x107/0x220
      [<ffffffff8128cd37>] kobject_add_varg+0x47/0x80
      [<ffffffff8128ce53>] kobject_add+0x53/0x90
      [<ffffffff81357d84>] device_add+0xd4/0x690
      [<ffffffff81356c2b>] ? dev_set_name+0x4b/0x70
      [<ffffffffa001a884>] cuse_process_init_reply+0x2b4/0x420 [cuse]
      ...

    The problem is that kobject_add_internal() first adds a kobject to the
    kset and then try to create sysfs directory for it.  If the creation
    fails, it remove the kobject from the kset.  get_device_parent()
    accesses class_dirs kset while only holding class_dirs.list_lock to
    see whether the cuse class dir exists.  But when it exists, it may not
    have finished initialization yet or may fail and get removed soon.  In
    the above case, the former happened so the second one ends up trying
    to create subdirectory under NULL sysfs_dirent.

    Fix it by grabbing a mutex in get_device_parent().

    Signed-off-by: Tejun Heo <tj@kernel.org>
    Reported-by: Colin Guthrie <cguthrie@mandriva.org>
    Cc: stable <stable@kernel.org>
    Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>


But this patch didn't cover our situation. So maybe the following patch is more
suitable to fix the race:

Subject: [PATCH] kobject: fix the race between kobject_put and get_device_parent

Signed-off-by: Yijing Wang <wangyijing@huawei.com>
Signed-off-by: Weng Meiling <wengmeiling.weng@huawei.com>
---
 drivers/base/core.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index 28b808c..645eacf 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -724,12 +724,12 @@ class_dir_create_and_add(struct class *class, struct kobject *parent_kobj)
 	return &dir->kobj;
 }

+static DEFINE_MUTEX(gdp_mutex);

 static struct kobject *get_device_parent(struct device *dev,
 					 struct device *parent)
 {
 	if (dev->class) {
-		static DEFINE_MUTEX(gdp_mutex);
 		struct kobject *kobj = NULL;
 		struct kobject *parent_kobj;
 		struct kobject *k;
@@ -793,7 +793,9 @@ static void cleanup_glue_dir(struct device *dev, struct kobject *glue_dir)
 	    glue_dir->kset != &dev->class->p->glue_dirs)
 		return;

+	mutex_lock(&gdp_mutex);
 	kobject_put(glue_dir);
+	mutex_unlock(&gdp_mutex);
 }

 static void cleanup_device_parent(struct device *dev)
-- 
1.8.2.2

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Tejun Heo]

Hello,

Sorry about the delay.

On Wed, Oct 22, 2014 at 04:07:44PM +0800, Weng Meiling wrote:
> diff --git a/drivers/base/core.c b/drivers/base/core.c
> index 28b808c..645eacf 100644
> --- a/drivers/base/core.c
> +++ b/drivers/base/core.c
> @@ -724,12 +724,12 @@ class_dir_create_and_add(struct class *class, struct kobject *parent_kobj)
>  	return &dir->kobj;
>  }
> 
> +static DEFINE_MUTEX(gdp_mutex);
> 
>  static struct kobject *get_device_parent(struct device *dev,
>  					 struct device *parent)
>  {
>  	if (dev->class) {
> -		static DEFINE_MUTEX(gdp_mutex);
>  		struct kobject *kobj = NULL;
>  		struct kobject *parent_kobj;
>  		struct kobject *k;
> @@ -793,7 +793,9 @@ static void cleanup_glue_dir(struct device *dev, struct kobject *glue_dir)
>  	    glue_dir->kset != &dev->class->p->glue_dirs)
>  		return;
> 
> +	mutex_lock(&gdp_mutex);
>  	kobject_put(glue_dir);
> +	mutex_unlock(&gdp_mutex);

So, yeah, this looks like a correct approach; however, do we even need
to clear the glue directories?  What's the downside of just keeping
them around once created?

Thanks.

-- 
tejun

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Yijing Wang]

On 2014/11/5 3:15, Tejun Heo wrote:
> Hello,
> 
> Sorry about the delay.

Thank you for your review and comments.

> 
> On Wed, Oct 22, 2014 at 04:07:44PM +0800, Weng Meiling wrote:
>> diff --git a/drivers/base/core.c b/drivers/base/core.c
>> index 28b808c..645eacf 100644
>> --- a/drivers/base/core.c
>> +++ b/drivers/base/core.c
>> @@ -724,12 +724,12 @@ class_dir_create_and_add(struct class *class, struct kobject *parent_kobj)
>>  	return &dir->kobj;
>>  }
>>
>> +static DEFINE_MUTEX(gdp_mutex);
>>
>>  static struct kobject *get_device_parent(struct device *dev,
>>  					 struct device *parent)
>>  {
>>  	if (dev->class) {
>> -		static DEFINE_MUTEX(gdp_mutex);
>>  		struct kobject *kobj = NULL;
>>  		struct kobject *parent_kobj;
>>  		struct kobject *k;
>> @@ -793,7 +793,9 @@ static void cleanup_glue_dir(struct device *dev, struct kobject *glue_dir)
>>  	    glue_dir->kset != &dev->class->p->glue_dirs)
>>  		return;
>>
>> +	mutex_lock(&gdp_mutex);
>>  	kobject_put(glue_dir);
>> +	mutex_unlock(&gdp_mutex);
> 
> So, yeah, this looks like a correct approach; however, do we even need
> to clear the glue directories? 

Yes. In our platform, the glue directories /sys/devices/virtual/block will be removed once the last child device call device_del().

> What's the downside of just keeping
> them around once created?

Without the fix, kernel will report the bug as bellow:

We tested it in 3.4 kernel, but this issue still exists in latest kernel.

<2>[ 3965.441912] kernel BUG at /usr/src/packages/BUILD/kernel-default-3.4.24.19/linux-3.4/fs/sysfs/group.c:65!
<4>[ 3965.441915] invalid opcode: 0000 [#1] SMP
<4>[ 3965.443707] calling kbox_sync :begin
<4>[ 3965.447262] sync kbox :begin
<4>[ 3965.450122] open all redirect device :begin
<4>[ 3965.454276] open all redirect device :end
<4>[ 3965.458257] flush kbox regions :begin
<4>[ 3965.461897] kbox region (die) is writing into (biosnvram), action is 202
<4>[ 3965.468556] test write len : 2009
<4>[ 3965.471847] first start addr : ffff88007e706000
......
<4>[ 3965.686743]  [<ffffffff811a677e>] sysfs_create_group+0xe/0x10
<4>[ 3965.686748]  [<ffffffff810cfb04>] blk_trace_init_sysfs+0x14/0x20
<4>[ 3965.686753]  [<ffffffff811fcabb>] blk_register_queue+0x3b/0x120
<4>[ 3965.686756]  [<ffffffff812030bc>] add_disk+0x1cc/0x490
<4>[ 3965.686770]  [<ffffffffa36a67c6>] SDM_BLKAddBlkDisk+0x86/0x290 [sdm]
<4>[ 3965.686780]  [<ffffffffa36a6a55>] SDM_BLKRegisterThread+0x85/0x330 [sdm]
<4>[ 3965.686801]  [<ffffffffa00dfb7e>] LVOS_SchedWorkThread+0xde/0x190 [vos]
<4>[ 3965.686806]  [<ffffffff8144a8a4>] kernel_thread_helper+0x4/0x10
<4>[ 3965.686821]  [<ffffffffa00dfaa0>] ? LVOS_CreateSchedWorkThread+0xd0/0xd0 [vos]
<4>[ 3965.686825]  [<ffffffff8144a8a0>] ? gs_change+0x13/0x13

Hi Tejun, if you think it make sense, I will resend it to make Greg easy review and apply.


Thanks!
Yijing.

> 
> Thanks.
> 


-- 
Thanks!
Yijing

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Tejun Heo]

Hello,

On Wed, Nov 05, 2014 at 10:01:39AM +0800, Yijing Wang wrote:
> > So, yeah, this looks like a correct approach; however, do we even need
> > to clear the glue directories? 
> 
> Yes. In our platform, the glue directories
> /sys/devices/virtual/block will be removed once the last child
> device call device_del().

What I mean is why are we even trying to remove the parent directory?
Can't we just keep them around once created?

Thanks.

-- 
tejun

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Yijing Wang]

On 2014/11/5 11:13, Tejun Heo wrote:
> Hello,
> 
> On Wed, Nov 05, 2014 at 10:01:39AM +0800, Yijing Wang wrote:
>>> So, yeah, this looks like a correct approach; however, do we even need
>>> to clear the glue directories? 
>>
>> Yes. In our platform, the glue directories
>> /sys/devices/virtual/block will be removed once the last child
>> device call device_del().
> 
> What I mean is why are we even trying to remove the parent directory?
> Can't we just keep them around once created?

We remove the parent directory because parent ref count is equal to children
device numebr, every child device will get/add parent's ref count when child
device be created.

Keep the parent directory looks good to me,  we could only add kobject_get(&parent)
after the parent dir be created.

....
		/* or create a new class-directory at the parent device */
		k = class_dir_create_and_add(dev->class, parent_kobj);
		/* do not emit an uevent for this simple "glue" directory */
		kobject_get(k);  <--------add parent ref count for first child device.
		mutex_unlock(&gdp_mutex);
		return k;
....

> 
> Thanks.
> 


-- 
Thanks!
Yijing

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Tejun Heo]

Hello,

On Wed, Nov 05, 2014 at 11:27:39AM +0800, Yijing Wang wrote:
> Keep the parent directory looks good to me,  we could only add kobject_get(&parent)
> after the parent dir be created.
> 
> ....
> 		/* or create a new class-directory at the parent device */
> 		k = class_dir_create_and_add(dev->class, parent_kobj);
> 		/* do not emit an uevent for this simple "glue" directory */
> 		kobject_get(k);  <--------add parent ref count for first child device.

The created directory would already have the base ref.  I don't think
you need the above.  Just never put the parent once created.

Greg, how does this sound to you?

Thanks.

-- 
tejun

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Greg KH]

On Tue, Nov 04, 2014 at 10:29:43PM -0500, Tejun Heo wrote:
> Hello,
> 
> On Wed, Nov 05, 2014 at 11:27:39AM +0800, Yijing Wang wrote:
> > Keep the parent directory looks good to me,  we could only add kobject_get(&parent)
> > after the parent dir be created.
> > 
> > ....
> > 		/* or create a new class-directory at the parent device */
> > 		k = class_dir_create_and_add(dev->class, parent_kobj);
> > 		/* do not emit an uevent for this simple "glue" directory */
> > 		kobject_get(k);  <--------add parent ref count for first child device.
> 
> The created directory would already have the base ref.  I don't think
> you need the above.  Just never put the parent once created.
> 
> Greg, how does this sound to you?

It makes sense, but I don't understand, what "parent" directory is going
away and causing problems?

thanks,

greg k-h

Re: Subject: [PATCH] kobject: fix the race between kobject_del and get_device_parent

[Yijing Wang]

On 2014/11/5 11:52, Greg KH wrote:
> On Tue, Nov 04, 2014 at 10:29:43PM -0500, Tejun Heo wrote:
>> Hello,
>>
>> On Wed, Nov 05, 2014 at 11:27:39AM +0800, Yijing Wang wrote:
>>> Keep the parent directory looks good to me,  we could only add kobject_get(&parent)
>>> after the parent dir be created.
>>>
>>> ....
>>> 		/* or create a new class-directory at the parent device */
>>> 		k = class_dir_create_and_add(dev->class, parent_kobj);
>>> 		/* do not emit an uevent for this simple "glue" directory */
>>> 		kobject_get(k);  <--------add parent ref count for first child device.
>>
>> The created directory would already have the base ref.  I don't think
>> you need the above.  Just never put the parent once created.
>>
>> Greg, how does this sound to you?
> 
> It makes sense, but I don't understand, what "parent" directory is going
> away and causing problems?

Hi Greg,
   We have some devices be created under /sys/devices/virtual/block, i.e

linux-rh5885:/sys/devices/virtual/block # ls
DEV1 	DEV2 	DEV3

the "block" directory is created after the first child device DEV1 created.

Then we frequently add DEV1,DEV2,DEV3 and remove DEV1,DEV2,DEV3 (this is a stress test).

Then some happens as bellow.

path 1 (Add child device)								path 2 (Remove child device)

device_add()										device_del()
	get_device_parent()									cleanup_device_parent()
		find existent parent dir("block")							cleanup_glue_dir()
		if nothing found, create it.									kobject_put(glue_dir);
		...piece code..
		list_for_each_entry(k, &dev->class->p->glue_dirs.list, entry)
			if (k->parent == parent_kobj) {
				kobj = kobject_get(k);
				break;
			}	
		......
		k = class_dir_create_and_add(dev->class, parent_kobj);

In the path 2, if the child device is the last child device under "/sys/devices/virtual/block",
the parent dir "block" ref count will be decreased to 0. So if path 2 call cleanup_glue_dir() but before kobject_put(glue_dir);
the path 1 child device will find the parent glue dir "block" from the glue_dirs.list, but unfortunately, before the path1 child
device call kobject_get(k), the path 2 remove the parent glue dir, then finally, path 1 reports WARNING and BUG_ON.

The first child device created the glue dir, but not get its ref count, maybe it's a problem.

Kernel Calltrace as bellow:
--------------------
<4>[ 3965.441471] WARNING: at /usr/src/packages/BUILD/kernel-default-3.4.24.19/linux-3.4/include/linux/kref.h:41 kobject_get+0x33/0x40()
<4>[ 3965.441474] Hardware name: Romley
<4>[ 3965.441475] Modules linked in: isd_iop(O) isd_xda(O) ivs_edft(O) ivs_xnet(O) ivs_emp(O) isd_rds(O) isd_idm(O) isd_dft(O) isd_base(O) sdm(O) isd_startwork(O) isd_fid(O) isd_cmm(O) isd_ibc(O) isd_lib(O) xve_hab(PO) xve_net(PO) xve_cls_msg_filter(PO) xve_dscp(PO) pagepool(PO) quota(PO) iod(O) cmm(PO) util(PO) intel_t10(PO) itest_nid(PO) dmi(PO) bsp_adapter(PO) iscsi_sw(PO) iscsi_prot(O) iscsi_seg(PO) iscsi_comm(PO) iscsi_initiator(PO) pciehp(PO) pcieaer(PO) pciecore(PO) quark(O) sal(O) foe(O) lfcoe(O) libfc(O) ib_uverbs(O) ibtgt(PO) ib_srpt(O) ib_cm(O) ib_sa(O) mlx4_ib(O) ib_umad(O) ib_mad(O) mlx4_core(O) ib_core(O) drvtom(O) cxgb4(O) drvtoecore(O) fcdrv(PO) unflowlevel_ioc(PO) unflowlevel(PO) unfcommon(O) fcportft(PO) mpa(O) drvmml(PO) scsi_transport_fc scsi_tgt memtest(PO) drv_iosubsys_ini(O) iocount(O) bsp_mml(PO) agetty_query(PO) cpufreq_powersave nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_limit xt_tcpudp xt_multiport nf_conntrack_ipv4 nf_defrag_ip!
v4 xt_st
ate nf_conntrack usr_cache(O) af_packet acpi_cpufreq sg mperf processor thermal_sys hwmon iptable_filter ip_tables x_tables ixgbe(O) igb(O) bonding(O) tg(O) 8021q e1000e(O) netmgmt(O) dal(PO) dca usb_storage(O) uhci_hcd ehci_hcd usbcore(O) usb_common sata_mml(O) ahci(O) libata(O) satahp(O) drvframe(PO) sd_mod crc_t10dif scsi_mod agetty_interface(PO) os_flush_mgt_ip_config(PO) ipmi_devintf ipmi_msghandler raid1 ext3 jbd mbcache nvram_printk(PO) os_feeddog(PO) os_debug(O) osp_proc(PO) os_rnvramdev(PO) vos(O) zlib_deflate os_die_handler(O) os_oom_handler(O) os_panic_handler(O) bsp(PO) biosnvram_driver(O) kbox(O)
<4>[ 3965.441603] Pid: 318, comm: sched_work Tainted: P           O 3.4.24.19-0.11-default #1
<4>[ 3965.441605] Call Trace:
<4>[ 3965.441611]  [<ffffffff8103717a>] warn_slowpath_common+0x7a/0xb0
<4>[ 3965.441615]  [<ffffffff810371c5>] warn_slowpath_null+0x15/0x20
<4>[ 3965.441618]  [<ffffffff81215963>] kobject_get+0x33/0x40
<4>[ 3965.441624]  [<ffffffff812d1e45>] get_device_parent.isra.11+0x135/0x1f0
<4>[ 3965.441627]  [<ffffffff812d22d4>] device_add+0xd4/0x6d0
<4>[ 3965.441631]  [<ffffffff812d0dbc>] ? dev_set_name+0x3c/0x40
<4>[ 3965.441634]  [<ffffffff812030ac>] add_disk+0x1bc/0x490
<4>[ 3965.441648]  [<ffffffffa36a67c6>] SDM_BLKAddBlkDisk+0x86/0x290 [sdm]
<4>[ 3965.441658]  [<ffffffffa36a6a55>] SDM_BLKRegisterThread+0x85/0x330 [sdm]
<4>[ 3965.441680]  [<ffffffffa00dfb7e>] LVOS_SchedWorkThread+0xde/0x190 [vos]
<4>[ 3965.441686]  [<ffffffff8144a8a4>] kernel_thread_helper+0x4/0x10
<4>[ 3965.441700]  [<ffffffffa00dfaa0>] ? LVOS_CreateSchedWorkThread+0xd0/0xd0 [vos]
<4>[ 3965.441705]  [<ffffffff8144a8a0>] ? gs_change+0x13/0x13
<4>[ 3965.441707] ---[ end trace 814bfedb5cb27305 ]---

<4>[ 3965.441713] WARNING: at /usr/src/packages/BUILD/kernel-default-3.4.24.19/linux-3.4/lib/kobject.c:202 kobject_add_internal+0x11f/0x280()
<4>[ 3965.441716] Hardware name: Romley
<4>[ 3965.441718] kobject_add_internal failed for sd-1a (error: -2 parent: ?qF)
<4>[ 3965.441720] Modules linked in: isd_iop(O) isd_xda(O) ivs_edft(O) ivs_xnet(O) ivs_emp(O) isd_rds(O) isd_idm(O) isd_dft(O) isd_base(O) sdm(O) isd_startwork(O) isd_fid(O) isd_cmm(O) isd_ibc(O) isd_lib(O) xve_hab(PO) xve_net(PO) xve_cls_msg_filter(PO) xve_dscp(PO) pagepool(PO) quota(PO) iod(O) cmm(PO) util(PO) intel_t10(PO) itest_nid(PO) dmi(PO) bsp_adapter(PO) iscsi_sw(PO)
<1>[ 3965.441741] [00002][INFO][ISD][SDM_BLKUnRegisterThread,204][992864]Bd 2 Index 0 removed.
<4>[ 3965.441744]  iscsi_prot(O) iscsi_seg(PO) iscsi_comm(PO) iscsi_initiator(PO) pciehp(PO) pcieaer(PO) pciecore(PO) quark(O) sal(O) foe(O) lfcoe(O) libfc(O) ib_uverbs(O) ibtgt(PO) ib_srpt(O) ib_cm(O) ib_sa(O) mlx4_ib(O) ib_umad(O) ib_mad(O) mlx4_core(O) ib_core(O) drvtom(O) cxgb4(O) drvtoecore(O) fcdrv(PO) unflowlevel_ioc(PO) unflowlevel(PO) unfcommon(O) fcportft(PO) mpa(O) drvmml(PO) scsi_transport_fc scsi_tgt memtest(PO) drv_iosubsys_ini(O) iocount(O) bsp_mml(PO) agetty_query(PO) cpufreq_powersave nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_limit xt_tcpudp xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack usr_cache(O) af_packet acpi_cpufreq sg mperf processor thermal_sys hwmon iptable_filter ip_tables x_tables ixgbe(O) igb(O) bonding(O) tg(O) 8021q e1000e(O) netmgmt(O) dal(PO) dca usb_storage(O) uhci_hcd ehci_hcd usbcore(O) usb_common sata_mml(O) ahci(O) libata(O) satahp(O) drvframe(PO) sd_mod crc_t10dif scsi_mod agetty_interface(PO) !
os_flush
_mgt_ip_config(PO) ipmi_devintf ipmi_msghandler raid1 ext3 jbd mbcache nvram_printk(PO) os_feeddog(PO) os_debug(O) osp_proc(PO) os_rnvramdev(PO) vos(O) zlib_deflate os_die_handler(O) os_oom_handler(O) os_panic_handler(O) bsp(PO) biosnvram_driver(O) kbox(O)
<4>[ 3965.441815] Pid: 318, comm: sched_work Tainted: P        W  O 3.4.24.19-0.11-default #1
<4>[ 3965.441817] Call Trace:
<4>[ 3965.441820]  [<ffffffff8103717a>] warn_slowpath_common+0x7a/0xb0
<4>[ 3965.441823]  [<ffffffff81037251>] warn_slowpath_fmt+0x41/0x50
<4>[ 3965.441826]  [<ffffffff81215e0f>] kobject_add_internal+0x11f/0x280
<4>[ 3965.441830]  [<ffffffff81216267>] kobject_add+0x67/0xc0
<4>[ 3965.441833]  [<ffffffff812d2305>] device_add+0x105/0x6d0
<4>[ 3965.441836]  [<ffffffff812d0dbc>] ? dev_set_name+0x3c/0x40
<4>[ 3965.441839]  [<ffffffff812030ac>] add_disk+0x1bc/0x490
<4>[ 3965.441849]  [<ffffffffa36a67c6>] SDM_BLKAddBlkDisk+0x86/0x290 [sdm]
<4>[ 3965.441858]  [<ffffffffa36a6a55>] SDM_BLKRegisterThread+0x85/0x330 [sdm]
<4>[ 3965.441874]  [<ffffffffa00dfb7e>] LVOS_SchedWorkThread+0xde/0x190 [vos]
<4>[ 3965.441879]  [<ffffffff8144a8a4>] kernel_thread_helper+0x4/0x10
<4>[ 3965.441892]  [<ffffffffa00dfaa0>] ? LVOS_CreateSchedWorkThread+0xd0/0xd0 [vos]
<4>[ 3965.441897]  [<ffffffff8144a8a0>] ? gs_change+0x13/0x13
<4>[ 3965.441899] ---[ end trace 814bfedb5cb27306 ]---


<2>[ 3965.441912] kernel BUG at /usr/src/packages/BUILD/kernel-default-3.4.24.19/linux-3.4/fs/sysfs/group.c:65!
<4>[ 3965.441915] invalid opcode: 0000 [#1] SMP
<4>[ 3965.443707] calling kbox_sync :begin
<4>[ 3965.447262] sync kbox :begin
<4>[ 3965.450122] open all redirect device :begin
<4>[ 3965.454276] open all redirect device :end
<4>[ 3965.458257] flush kbox regions :begin
<4>[ 3965.461897] kbox region (die) is writing into (biosnvram), action is 202
<4>[ 3965.468556] test write len : 2009
<4>[ 3965.471847] first start addr : ffff88007e706000
<1>[ 3965.474542] [00002][INFO][ISD][ISD_EMMProcessBDMEvent,3778][992873]SCSI_DISK_EVENT_OUT BDM id is 2
<1>[ 3965.474546] [00002][INFO][ISD][ISD_EMMLunCheckSame,2131][992873]Find same ramdisk in list! BdmID(0x2)
<1>[ 3965.474549] [00002][INFO][ISD][ISD_EMMProcessDiskOut,3099][992873]No need to add ramdisk(DiskType:0xfc) to topo!
<1>[ 3965.474555] [00002][WARN][ISD][SDM_DEVDiskEventHandle,927][992873]Disk(disk id=2) type is peripheral lun, skip handle.
<4>[ 3965.515140] second start addr : ffff88007e706000
<4>[ 3965.519727] cur_index : 0, offset : 0, third start addr : ffff88007e706000
<4>[ 3965.526557] first length : 2009
<4>[ 3965.529676] cur_index : 1, record_number : 1, total_number : 4
<4>[ 3965.535471] kbox region (die) has been written into (biosnvram)
<4>[ 3965.541351] dev biosnvram is dirty
<4>[ 3965.544732] kbox region (console) is writing into (biosnvram), action is 202
<4>[ 3965.551733] test write len : 1211
<4>[ 3965.555022] first start addr : ffff88007e906000
<4>[ 3965.559521] second start addr : ffff88007e906000
<4>[ 3965.564107] cur_index : 0, offset : 0, third start addr : ffff88007e906000
<4>[ 3965.570935] first length : 1211
<4>[ 3965.574053] cur_index : 1, record_number : 1, total_number : 16
<4>[ 3965.579932] kbox region (console) has been written into (biosnvram)
<4>[ 3965.586155] dev biosnvram is dirty
<4>[ 3965.589531] kbox region (message) is writing into (biosnvram), action is 202
<4>[ 3965.596534] test write len : 65566
<4>[ 3965.599910] first start addr : ffff88007e806000
<4>[ 3965.604410] second start addr : ffff88007e806000
<4>[ 3965.608997] cur_index : 0, offset : 0, third start addr : ffff88007e806000
<4>[ 3965.615827] first length : 65566
<4>[ 3965.619040] cur_index : 1, record_number : 1, total_number : 4
<4>[ 3965.624835] kbox region (message) has been written into (biosnvram)
<4>[ 3965.631059] dev biosnvram is dirty
<4>[ 3965.634435] flush kbox regions :end
<4>[ 3965.637900] flush kbox superblock :begin
<4>[ 3965.641796] supperblock dev=biosnvram is dirty
<4>[ 3965.646211] flush kbox superblock: end
<4>[ 3965.649935] close all redirect device :begin
<4>[ 3965.654175] close all redirect device :end
<4>[ 3965.658242] sync kbox :end
<4>[ 3965.660929] calling kbox_sync :end
<4>[ 3965.664394] do nothing after die!
<4>[ 3965.664395] now, die chain processing!
<4>[ 3965.667636] OS_SaveKernSegToNVRAM save 4510065 bytes
<4>[ 3965.673276] OS_SaveModSegToNVRAM the modules addr:ffffffff818250e0
<4>[ 3965.677342] OS_SaveModSegToNVRAM, no size left to content more ,exit...
<4>[ 3965.684146] OS_SaveVmallocToNVRAM
<4>[ 3965.684185] the count:511
<4>[ 3965.684187] now crash happen, set bsp watchdog to 5 min.
<5>[ 3965.684193] [992925][5400040602c1][INF][Begin To Set Watchdog Level:0x3.][BSP][SP2_Watc.merSet,700][sched_work]
<4>[ 3965.684259] CPU 1
<4>[ 3965.684260] Modules linked in: isd_iop(O) isd_xda(O) ivs_edft(O) ivs_xnet(O) ivs_emp(O) isd_rds(O) isd_idm(O) isd_dft(O) isd_base(O) sdm(O) isd_startwork(O) isd_fid(O) isd_cmm(O) isd_ibc(O) isd_lib(O) xve_hab(PO) xve_net(PO) xve_cls_msg_filter(PO) xve_dscp(PO) pagepool(PO) quota(PO) iod(O) cmm(PO) util(PO) intel_t10(PO) itest_nid(PO) dmi(PO) bsp_adapter(PO) iscsi_sw(PO) iscsi_prot(O) iscsi_seg(PO) iscsi_comm(PO) iscsi_initiator(PO) pciehp(PO) pcieaer(PO) pciecore(PO) quark(O) sal(O) foe(O) lfcoe(O) libfc(O) ib_uverbs(O) ibtgt(PO) ib_srpt(O) ib_cm(O) ib_sa(O) mlx4_ib(O) ib_umad(O) ib_mad(O) mlx4_core(O) ib_core(O) drvtom(O) cxgb4(O) drvtoecore(O) fcdrv(PO) unflowlevel_ioc(PO) unflowlevel(PO) unfcommon(O) fcportft(PO) mpa(O) drvmml(PO) scsi_transport_fc scsi_tgt memtest(PO) drv_iosubsys_ini(O) iocount(O) bsp_mml(PO) agetty_query(PO) cpufreq_powersave nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_limit xt_tcpudp xt_multiport nf_conntrack_ipv4 nf_defrag_ip!
v4 xt_st
ate nf_conntrack usr_cache(O) af_packet acpi_cpufreq sg mperf processor thermal_sys hwmon iptable_filter ip_tables x_tables ixgbe(O) igb(O) bonding(O) tg(O) 8021q e1000e(O) netmgmt(O) dal(PO) dca usb_storage(O) uhci_hcd ehci_hcd usbcore(O) usb_common sata_mml(O) ahci(O) libata(O) satahp(O) drvframe(PO) sd_mod crc_t10dif scsi_mod agetty_interface(PO) os_flush_mgt_ip_config(PO) ipmi_devintf ipmi_msghandler raid1 ext3 jbd mbcache nvram_printk(PO) os_feeddog(PO) os_debug(O) osp_proc(PO) os_rnvramdev(PO) vos(O) zlib_deflate os_die_handler(O) os_oom_handler(O) os_panic_handler(O) bsp(PO) biosnvram_driver(O) kbox(O)
<4>[ 3965.684382]
<4>[ 3965.684385] Pid: 318, comm: sched_work Tainted: P        W  O 3.4.24.19-0.11-default #1 Huawei Technologies Co., Ltd. Romley/STL2SPCAC.
<4>[ 3965.684391] RIP: 0010:[<ffffffff811a6722>]  [<ffffffff811a6722>] internal_create_group+0x202/0x230
<4>[ 3965.686224] RSP: 0018:ffff8801046efd20  EFLAGS: 00010246
<4>[ 3965.686227] RAX: 00000000fffffffe RBX: ffff880030aa7000 RCX: 000000000018b2fc
<4>[ 3965.686229] RDX: ffffffff81828f00 RSI: 0000000000000000 RDI: ffff880030aa7078
<4>[ 3965.686232] RBP: ffff8801046efd70 R08: 00000000000183d0 R09: ffff88010ae383d0
<4>[ 3965.686234] R10: ffffea0000fb2180 R11: ffffffff812d23b1 R12: ffff880030aa7000
<4>[ 3965.686237] R13: ffffffff81828f00 R14: 0000000000000000 R15: ffff880030aa7068
<4>[ 3965.686240] FS:  00007f89c61ef700(0000) GS:ffff88010ae20000(0000) knlGS:0000000000000000
<4>[ 3965.686243] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<4>[ 3965.686245] CR2: ffffffffff600400 CR3: 000000003bacc000 CR4: 00000000001407e0
<4>[ 3965.686248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[ 3965.686250] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[ 3965.686253] Process sched_work (pid: 318, threadinfo ffff8801046ee000, task ffff880066392e40)
<4>[ 3965.686670] Stack:
<4>[ 3965.686672]  ffff880030aa7350 ffff880030aa7078 ffff8801046efd40 00000000000000dd
<4>[ 3965.686677]  ffff8801046efd70 ffff880030aa7000 ffff880030aa7000 ffff88003cdb90b0
<4>[ 3965.686682]  ffff880030aa7068 ffff880030aa7068 ffff8801046efd80 ffffffff811a677e
<4>[ 3965.686686]  ffff8801046efd90 ffffffff810cfb04 ffff8801046efde0 ffffffff811fcabb
<4>[ 3965.686691]  ffff8801046efdf0 ffff8801046efdb0 ffff880030aa7078 ffff880030aa7000
<4>[ 3965.686695]  ffff88003cdb90b0 ffff880030aa700c ffff880030aa7078 ffff880030aa7068
<4>[ 3965.686699]  ffff8801046efe40 ffffffff812030bc 0fd0004072982900 ffff880030aa7000
<4>[ 3965.686703]  ffff880030aa7000 ffff880030aa7000 ffff8801046efe40 ffff880030aa7000
<4>[ 3965.686707]  ffff880072982900 0000000000000000 0000000000000282 0000000000000000
<4>[ 3965.686711]  ffff8801046efe90 ffffffffa36a67c6 ffff8801046efea8 ffff8801046efe60
<4>[ 3965.686715]  dead000000100100 0000000000000282 ffff8801046efe90 ffff880072982900
<4>[ 3965.686719]  ffffffffa36dbc20 ffffffffa36dbc60 ffff8801046eff00 ffffffffa36a6a55
<4>[ 3965.686723]  00000000000f2660 0000000000000282 ffff8801046efec0 ffffffffa0104000
<4>[ 3965.686727]  ffff8801046efef0 ffffffffa36dbc40 0000000000000000 ffffffffa36dbc98
<4>[ 3965.686731]  0000000000000001 ffffffffa36dbc90 0000000000000000 0000000000000000
<4>[ 3965.686735]  ffff8801046eff40 ffffffffa00dfb7e
<4>[ 3965.686738] Call Trace:
<4>[ 3965.686743]  [<ffffffff811a677e>] sysfs_create_group+0xe/0x10
<4>[ 3965.686748]  [<ffffffff810cfb04>] blk_trace_init_sysfs+0x14/0x20
<4>[ 3965.686753]  [<ffffffff811fcabb>] blk_register_queue+0x3b/0x120
<4>[ 3965.686756]  [<ffffffff812030bc>] add_disk+0x1cc/0x490
<4>[ 3965.686770]  [<ffffffffa36a67c6>] SDM_BLKAddBlkDisk+0x86/0x290 [sdm]
<4>[ 3965.686780]  [<ffffffffa36a6a55>] SDM_BLKRegisterThread+0x85/0x330 [sdm]
<4>[ 3965.686801]  [<ffffffffa00dfb7e>] LVOS_SchedWorkThread+0xde/0x190 [vos]
<4>[ 3965.686806]  [<ffffffff8144a8a4>] kernel_thread_helper+0x4/0x10
<4>[ 3965.686821]  [<ffffffffa00dfaa0>] ? LVOS_CreateSchedWorkThread+0xd0/0xd0 [vos]
<4>[ 3965.686825]  [<ffffffff8144a8a0>] ? gs_change+0x13/0x13



> 
> thanks,
> 
> greg k-h
> 
> .
> 


-- 
Thanks!
Yijing