2.6.25 md oops during boot.

2.6.25 md oops during boot.

[Dave Jones]

Hi Neil,
 Here's an odd one.
https://bugzilla.redhat.com/show_bug.cgi?id=442204

Slightly old (.25-rc8-git7), but I don't recall anything changing between
then and .25 final that could explain this.

	Dave

BUG: unable to handle kernel NULL pointer dereference at 00000020
IP: [<c04bbb65>] sysfs_addrm_start+0x21/0x7d
Oops: 0000 [#1] SMP 
Modules linked in: e1000(+) i2c_piix4 i2c_core sg ac button sr_mod cdrom
ata_piix libata dm_snapshot dm_zero dm_mirror dm_mod mptspi mptscsih mptbase
scsi_transport_spi sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd
[last unloaded: scsi_wait_scan]

Pid: 742, comm: mdadm Not tainted (2.6.25-0.218.rc8.git7.fc9.i686 #1)
EIP: 0060:[<c04bbb65>] EFLAGS: 00010246 CPU: 0
EIP is at sysfs_addrm_start+0x21/0x7d
EAX: c04bbc23 EBX: 00000000 ECX: 00000000 EDX: 00000062
ESI: f6ecdc84 EDI: f6ecdc94 EBP: f6ecdc78 ESP: f6ecdc6c
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process mdadm (pid: 742, ti=f6ecd000 task=f6e22e80 task.ti=f6ecd000)
Stack: f6ecdc84 f6e75bd0 fffffff4 f6ecdca0 c04bbfb7 00000000 00000000 00000000 
       00000000 00000000 f7853dc4 fffffffe f7af6824 f6ecdcb4 c04bc01c f6ecdcac 
       c04f08f2 f7853dc4 f6ecdcd0 c04f09ec f7af6824 f6ecdcd0 00000000 f7853dc4 
Call Trace:
 [<c04bbfb7>] ? create_dir+0x3a/0x72
 [<c04bc01c>] ? sysfs_create_dir+0x2d/0x41
 [<c04f08f2>] ? kobject_get+0x12/0x17
 [<c04f09ec>] ? kobject_add_internal+0xa4/0x145
 [<c04f0b21>] ? kobject_add_varg+0x35/0x41
 [<c04f0b92>] ? kobject_add+0x43/0x49
 [<c05948aa>] ? bind_rdev_to_array+0x124/0x1ab
 [<c04cf900>] ? task_has_capability+0x47/0x76
 [<c04f4f94>] ? copy_from_user+0x39/0x121
 [<c05994d1>] ? md_ioctl+0xf75/0x183b
 [<c04cfde2>] ? inode_has_perm+0x5b/0x65
 [<c0490ce5>] ? d_free+0x3b/0x4d
 [<c0492094>] ? dput+0x34/0xee
 [<c04efb51>] ? _atomic_dec_and_lock+0x29/0x44
 [<c04958b8>] ? mntput_no_expire+0x16/0x69
 [<c0488796>] ? path_put+0x20/0x23
 [<c048ab1b>] ? __link_path_walk+0xce7/0xcfc
 [<c04e831d>] ? blkdev_driver_ioctl+0x49/0x5b
 [<c04e8ab0>] ? blkdev_ioctl+0x781/0x79d
 [<c04d06a8>] ? selinux_file_free_security+0x14/0x16
 [<c04cfde2>] ? inode_has_perm+0x5b/0x65
 [<c04d01b9>] ? file_has_perm+0x7c/0x85
 [<c04a241b>] ? block_ioctl+0x16/0x1b
 [<c04a2405>] ? block_ioctl+0x0/0x1b
 [<c048c99e>] ? vfs_ioctl+0x22/0x69
 [<c048cc1e>] ? do_vfs_ioctl+0x239/0x24c
 [<c04d034f>] ? selinux_file_ioctl+0xa8/0xab
 [<c048cc71>] ? sys_ioctl+0x40/0x5b
 [<c0405cf6>] ? syscall_call+0x7/0xb
 =======================
Code: 89 31 8d 65 f4 5b 5e 5f 5d c3 55 b9 04 00 00 00 89 e5 57 89 c7 56 89 c6 53
31 c0 89 d3 f3 ab b8 34 fc 71 c0 89 16 e8 1f e2 16 00 <8b> 53 20 b9 f0 b7 4b c0
a1 88 f4 83 c0 53 e8 43 71 fd ff 5f 85 
EIP: [<c04bbb65>] sysfs_addrm_start+0x21/0x7d SS:ESP 0068:f6ecdc6c
---[ end trace ca143223eefdc828 ]---


-- 
http://www.codemonkey.org.uk

Re: 2.6.25 md oops during boot.

[Neil Brown]

On Wednesday June 4, davej@redhat.com wrote:
> Hi Neil,
>  Here's an odd one.
> https://bugzilla.redhat.com/show_bug.cgi?id=442204
> 
> Slightly old (.25-rc8-git7), but I don't recall anything changing between
> then and .25 final that could explain this.
> 
> 	Dave

Hi Dave.

Yes, Odd.

It appear that sysfs_addrm_start is being called with parent_sd == NULL.

That implies that sysfs_create_dir is being given a kobj where
->parent is non-NULL, and ->parent->sd is NULL.

So kobject_add is being given a parent with a NULL ->sd.

So in bind_rdev_to_array,  mddev->kobj.sd is NULL.

So in md_probe, either kobject_init_and_add is failing
to set up ->sd properly (which should result in an error message
   "md: cannot register md0/md - name in use"
) or alloc_disk is failing.


The most likely scenario is that alloc_disk is failing, so the
md_probe call in autorun_devices (line 3804 of md.c) fails.
The following mddev_find creates a new mddev which is not properly
initialised and gets used.

I wouldn't say this is a likely scenario as it requires (I think)
kmalloc failure very early in boot.  But I cannot see any other
possible cause.

I'll see about getting the error paths handled better.

NeilBrown

Re: 2.6.25 md oops during boot.

[Neil Brown]

On Thursday June 5, neilb@suse.de wrote:
> 
> I wouldn't say this is a likely scenario as it requires (I think)
> kmalloc failure very early in boot.  But I cannot see any other
> possible cause.

On closer inspection, I can see another possible cause.  I don't think
it is likely (yet) but it might be possible.

If two threads enter md_probe for the same mddev, then the second one
to get disks_mutex could exit before the first had called
kobject_init_and_add, so it could make available an mddev where
kobj.sd was NULL.

I cannot imagine how two threads could be doing that so early in boot,
but I cannot rule it out.

This (untested) patch should close both these possible problems.

NeilBrown


-----------------
Fix error paths if md_probe fails.

md_probe can fail (e.g. alloc_disk could fail) without
returning an error (as it alway returns NULL).
So when we call mddev_find immediately afterwards, we need
to check that md_probe actually succeeded.  This means checking
that mdev->gendisk is non-NULL.

Also there is a possible race - if two threads call md_probe
for the same device, then one could exit (having checked that
->gendisk exists) before the other has called kobject_init_and_add,
thus returning an incomplete kobj which is cause problems when
we try to add children to it.

So extend the range of protection of disks_mutex slightly to
avoid this possibility.

Cc: Dave Jones <davej@redhat.com>
Signed-off-by: Neil Brown <neilb@suse.de>

### Diffstat output
 ./drivers/md/md.c |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff .prev/drivers/md/md.c ./drivers/md/md.c
--- .prev/drivers/md/md.c	2008-06-03 16:35:41.000000000 +1000
+++ ./drivers/md/md.c	2008-06-05 09:19:56.000000000 +1000
@@ -3363,9 +3363,9 @@ static struct kobject *md_probe(dev_t de
 	disk->queue = mddev->queue;
 	add_disk(disk);
 	mddev->gendisk = disk;
-	mutex_unlock(&disks_mutex);
 	error = kobject_init_and_add(&mddev->kobj, &md_ktype, &disk->dev.kobj,
 				     "%s", "md");
+	mutex_unlock(&disks_mutex);
 	if (error)
 		printk(KERN_WARNING "md: cannot register %s/md - name in use\n",
 		       disk->disk_name);
@@ -3935,8 +3935,10 @@ static void autorun_devices(int part)
 
 		md_probe(dev, NULL, NULL);
 		mddev = mddev_find(dev);
-		if (!mddev) {
-			printk(KERN_ERR 
+		if (!mddev || !mddev->gendisk) {
+			if (mddev)
+				mddev_put(mddev);
+			printk(KERN_ERR
 				"md: cannot allocate memory for md drive.\n");
 			break;
 		}