oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

[Jiri Slaby]

Hi!

I got this oops on 2.6.19-rc6-mm2 when starting the system. It happened only
once -- when echo "raidautorun /dev/md0" | nash --quiet was executed. I don't
know why it happened, after reboot it started OK.

There is no md0, only md1, md2 and md3.

See camera-shots here:
http://www.fi.muni.cz/~xslaby/sklad/dm_oops1.png
http://www.fi.muni.cz/~xslaby/sklad/dm_oops2.png

regards,
-- 
http://www.fi.muni.cz/~xslaby/            Jiri Slaby
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Re: oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

[Jiri Kosina]

On Sat, 9 Dec 2006, Jiri Slaby wrote:

> I got this oops on 2.6.19-rc6-mm2 when starting the system. It happened 
> only once -- when echo "raidautorun /dev/md0" | nash --quiet was 
> executed. 

Hi,

this nash thing is exactly the command which triggers a bit different oops 
in my case. On my side, the oops is fully reproducible. If you manage to 
make your case also reproducible, could you please try to revert 
md-change-lifetime-rules-for-md-devices.patch? This made the oops vanish 
in my case. I think Neil is working on it.

-- 
Jiri Kosina

Re: oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

[Jiri Kosina]

On Sun, 10 Dec 2006, Jiri Kosina wrote:

> this nash thing is exactly the command which triggers a bit different 
> oops in my case. On my side, the oops is fully reproducible. If you 
> manage to make your case also reproducible, could you please try to 
> revert md-change-lifetime-rules-for-md-devices.patch? This made the oops 
> vanish in my case. I think Neil is working on it.

And I forgot to add reference to the original thread - 
http://lkml.org/lkml/2006/11/28/248

-- 
Jiri Kosina

Re: oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

[Neil Brown]

On Sunday December 10, jikos@jikos.cz wrote:
> On Sat, 9 Dec 2006, Jiri Slaby wrote:
> 
> > I got this oops on 2.6.19-rc6-mm2 when starting the system. It happened 
> > only once -- when echo "raidautorun /dev/md0" | nash --quiet was 
> > executed. 
> 
> Hi,
> 
> this nash thing is exactly the command which triggers a bit different oops 
> in my case. On my side, the oops is fully reproducible. If you manage to 
> make your case also reproducible, could you please try to revert 
> md-change-lifetime-rules-for-md-devices.patch? This made the oops vanish 
> in my case. I think Neil is working on it.

Trying to work on it - not making a lot of progress.   I find it hard
to see how anything in md can cause the inode for a block-device file
to disappear...

It is a bit of a long-shot, but this patch might change things.  It
changes the order in which things are de-allocated.

Jiri and Jiri: would either of both of you see if you can reproduce
the bug with this patch on 2.6.19-rc6-mm2 ???

Thanks,
NeilBrown


### Diffstat output
 ./drivers/md/md.c |   12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff .prev/drivers/md/md.c ./drivers/md/md.c
--- .prev/drivers/md/md.c	2006-12-11 13:04:23.000000000 +1100
+++ ./drivers/md/md.c	2006-12-11 13:07:28.000000000 +1100
@@ -224,16 +224,20 @@ static inline mddev_t *mddev_get(mddev_t
 
 static void mddev_put(mddev_t *mddev)
 {
+	request_queue_t *q;
+	struct gendisk *disk;
 	if (!atomic_dec_and_lock(&mddev->active, &all_mddevs_lock))
 		return;
 	list_del(&mddev->all_mddevs);
 	spin_unlock(&all_mddevs_lock);
 
-	del_gendisk(mddev->gendisk);
-	mddev->gendisk = NULL;
-	blk_cleanup_queue(mddev->queue);
-	mddev->queue = NULL;
+	disk = mddev->gendisk;
+	q = mddev->queue;
+
 	kobject_unregister(&mddev->kobj);
+	if (disk)
+		del_gendisk(disk);
+	blk_cleanup_queue(q);
 }
 
 static mddev_t * mddev_find(dev_t unit)

Re: oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

[Jiri Kosina]

On Mon, 11 Dec 2006, Neil Brown wrote:

> > this nash thing is exactly the command which triggers a bit different 
> > oops in my case. On my side, the oops is fully reproducible. If you 
> > manage to make your case also reproducible, could you please try to 
> > revert md-change-lifetime-rules-for-md-devices.patch? This made the 
> > oops vanish in my case. I think Neil is working on it.
> Trying to work on it - not making a lot of progress.  I find it hard to 
> see how anything in md can cause the inode for a block-device file to 
> disappear... It is a bit of a long-shot, but this patch might change 
> things.  It changes the order in which things are de-allocated. Jiri and 
> Jiri: would either of both of you see if you can reproduce the bug with 
> this patch on 2.6.19-rc6-mm2 ???

Hi Neil,

sorry to say that, but it's still there after applying your patch.

-- 
Jiri Kosina

Re: oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

[Neil Brown]

On Monday December 11, jikos@jikos.cz wrote:
> On Mon, 11 Dec 2006, Neil Brown wrote:
> 
> > > this nash thing is exactly the command which triggers a bit different 
> > > oops in my case. On my side, the oops is fully reproducible. If you 
> > > manage to make your case also reproducible, could you please try to 
> > > revert md-change-lifetime-rules-for-md-devices.patch? This made the 
> > > oops vanish in my case. I think Neil is working on it.
> > Trying to work on it - not making a lot of progress.  I find it hard to 
> > see how anything in md can cause the inode for a block-device file to 
> > disappear... It is a bit of a long-shot, but this patch might change 
> > things.  It changes the order in which things are de-allocated. Jiri and 
> > Jiri: would either of both of you see if you can reproduce the bug with 
> > this patch on 2.6.19-rc6-mm2 ???
> 
> Hi Neil,
> 
> sorry to say that, but it's still there after applying your patch.

Not a big surprise, but thanks a lot for testing.  I think I'm going
to have to try harder to duplicate it myself.

If I remember rightly you are using FC - which version exactly?  (I've
never installed FC before so this is going to be learning experience).

And you have no MD arrays at all - is that correct?

And you compile your own kernel.  Is it monolithic, or are you using
modules?  Do you boot with an initrd or just the kernel?

I'd like to duplicate your installation as closely as possible, so any
relevant details or recipes would be greatly appreciated.

Thanks,

NeilBrown

Re: oops on 2.6.19-rc6-mm2: deref of 0x28 at permission+0x7

[Jiri Slaby]

Neil Brown wrote:
> On Monday December 11, jikos@jikos.cz wrote:
>> On Mon, 11 Dec 2006, Neil Brown wrote:
>>
>>>> this nash thing is exactly the command which triggers a bit different 
>>>> oops in my case. On my side, the oops is fully reproducible. If you 
>>>> manage to make your case also reproducible, could you please try to 
>>>> revert md-change-lifetime-rules-for-md-devices.patch? This made the 
>>>> oops vanish in my case. I think Neil is working on it.
>>> Trying to work on it - not making a lot of progress.  I find it hard to 
>>> see how anything in md can cause the inode for a block-device file to 
>>> disappear... It is a bit of a long-shot, but this patch might change 
>>> things.  It changes the order in which things are de-allocated. Jiri and 
>>> Jiri: would either of both of you see if you can reproduce the bug with 
>>> this patch on 2.6.19-rc6-mm2 ???
>> Hi Neil,
>>
>> sorry to say that, but it's still there after applying your patch.
> 
> Not a big surprise, but thanks a lot for testing.  I think I'm going
> to have to try harder to duplicate it myself.

Away from that machine now, so that I can't test anything till thursday.

> If I remember rightly you are using FC - which version exactly?  (I've
> never installed FC before so this is going to be learning experience).

FC6 with latest updates.

> And you have no MD arrays at all - is that correct?

I do have. md1 md2 md3 -- raid0, 1, 0. But there is no md0 (removed in the past)
and 'raidautorun /dev/md0 | nash' causes the troubles.

> And you compile your own kernel.  Is it monolithic, or are you using
> modules?  Do you boot with an initrd or just the kernel?

Yup. Monolithic as much as possible (md is in the kernel and so dm is). / on the
lvm2 on the /dev/md1 with no initrd. All are sata disks, so sd_mod, sata_promise
and ata_piix are in the kernel.

> I'd like to duplicate your installation as closely as possible, so any
> relevant details or recipes would be greatly appreciated.

Hm, I have never seen FC6 installation process, so I can't say what special
option I have turned on -- it's 'yum upgrade'd from FC5, FC4...

regards,
-- 
http://www.fi.muni.cz/~xslaby/            Jiri Slaby
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E