[linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Jason Tackaberry]

I have discovered one or more severe bugs in any or all of lvm, md, and
reiserfs.  I'm not a kernel guy and I don't claim to know how these
things are supposed to interact at that level, so I'm not going to point
fingers.  Instead, here's what happened and I'll let you decide. :)

The problem surfaces when you create a volume group with lvm and include
a partition that happens to be mounted.  In my tests the cwd was this
mounted filesystem.  This may or may not be a necessary condition to
reproduce this problem.  (I didn't test otherwise.)

In my tests I have /dev/sda7 and /dev/sdb7.  I created an md raid0
device and assigned that to /dev/md3.  Then, here are the sequence of
commands I issued:

  mkreiserfs /dev/md3
  mount /dev/md3 /space
  cd /space
  pvcreate /dev/md3
  vgcreate vol01 /dev/md3

Obviously it's not normal to create a vg on a device that's mounted.  I
did this by accident, but the result was pretty ugly.  Immediately after
doing 'vgcreate' this popped up in the console:

--- snip ---------------------------------------------------------------
is_tree_node: node level 19784 does not match to the expected one -1
vs-5150: search_by_key: invalid format found in block 0. Fsck?
kernel BUG at namei.c:343!
invalid operand: 0000
CPU:    0
EIP:    0010:[<c017d965>]
EFLAGS: 00010286
eax: 0000001b   ebx: c2047e60   ecx: c2c0e000   edx: c02d88c4
esi: c28795e0   edi: c2047ea0   ebp: c2047e04   esp: c2047df8
ds: 0018   es: 0018   ss: 0018
Process bash (pid: 834, stackpage=c2047000)
Stack: c028ac46 c028acda 00000157 00000001 00000002 7f84dbff 000001f4 00000000
       00000003 c011ac84 00000001 c028ad8e c2047e60 c2047ea0 c23be0e0 c017da8b
       c28795e0 c23be13c 0000000a c2047e60 c2047ea0 00000000 00000001 c03275c0
Call Trace: [<c011ac84>] [<c017da8b>] [<c0139b70>] [<c013a582>] [<c0141999>] [<c0139c2f>] [<c013a341>]
       [<c013a94a>] [<c0137993>] [<c012fae3>] [<c0106ecb>]

Code: 0f 0b 83 c4 0c 8b 54 24 3c 52 8b 44 24 3c 50 57 55 e8 25 fc
--- snip ---------------------------------------------------------------

After realising that I'd tried to vgcreate a partition that was mounted,
I cd'ed out of /space and issued 'umount /space'.  Then I got this:

--- snip ---------------------------------------------------------------
journal-2332: Trying to log block 16, which is a log block
kernel BUG at prints.c:332!
invalid operand: 0000
CPU:    0
EIP:    0010:[<c01887a4>]
EFLAGS: 00010286
eax: 0000001c   ebx: c0335a80   ecx: cf41c000   edx: c02d88c4
esi: cc3c1e68   edi: c0335ebd   ebp: c0335a80   esp: cc3c1e2c
ds: 0018   es: 0018   ss: 0018
Process umount (pid: 881, stackpage=cc3c1000)
Stack: c028c6c6 c028c9c0 0000014c cc3c1e44 cc3c1e48 00000000 000007e0 00000001
       00000001 d0865024 00000000 d0865000 c0195e4e ce803800 c028fb80 00000010
       00000560 00000000 00000001 00000001 00000807 00000000 00000000 ce633ea0
Call Trace: [<c0195e4e>] [<c0193546>] [<c018684a>] [<c01935d1>] [<c018685a>] [<c0135174>] [<c0139b88>]
       [<c013559a>] [<c012fb83>] [<c0121ea2>] [<c01355dc>] [<c0106ecb>]

Code: 0f 0b 83 c4 0c 8b 54 24 28 85 d2 74 08 8b 44 24 28 83 48 28
Segmentation fault
--- snip ---------------------------------------------------------------

After umount segfaulted, I decided I ought to reboot.  ctrl-alt-del or
halt(8) appeared to have no effect, so I hit the reset button.

As my system was coming back up, the boot sequence continually said
"Can't open /etc/mtab: input/output error."  The boot and root
partitions are both md RAID1 devices with ext2.  The disks seemed to
fsck okay, and when I logged in, I indeed could not read /etc/mtab.  'ls
-l /etc/mtab' even yielded the same input/output error.  I rebooted with
the stock RH 7.1 kernel that has no lvm support and this error
persisted.  So this seems to be a problem with md?

I managed to fix it by booting the RedHat rescue cd, mounting the root
partition, and deleting /etc/mtab.  All seems okay.

Just to see if this problem was linked to md beneath LVM, I repeated the
above with /dev/sda7 instead of /dev/md3.  Same result.  Now, I also
repeated the above using ext2 instead of reiserfs.  ext2 complained:

EXT2-fs error (device sd(8,7)): ext2_write_inode: unable to read inode
block - inode=2, block=1899261259
attempt to access beyond end of device

But it did not cause any kernel errors.  This process failed gracefully
with ext2.

Now, if I had to blame LVM for something, I'd say it should at least not
let me vgcreate with a mounted partition, or one that has inodes in use.
mkraid, for exaple, would not let me start a raid device with a mounted
partition.  If vgcreate failed with an error saying "This partition is
mounted or has inodes in use" I would have realized the problem and
unmounted /space before vgcreate.  The kernel errors came from Reiserfs,
so there seems to be a problem there.  But like I said, I'm not a kernel
guy so I'll let you guys confirm this before I submit a bug report to
the Reiserfs team (or maybe one of the LVM hackers would be a better
candidate).  Finally, md did not handle this robustly at all.  The whole
/etc/mtab issue makes me extremely nervous since md is handling my boot
and root parition with RAID1.  I'm just in a testing stages,
fortunately, but I'm thinking md/lvm aren't suitable for this system
which will be in production in a month.

I'd be happy to repeat and experiments or give further information if
you need some.  This system is mine to beat on for a little while longer
before it needs to go live.

Cheers,
Jason.

--
Academic Computing Support Specialist         Assistant Section Editor
Algoma University College                     http://linux.com/develop
Sault Ste. Marie, Ontario                 
705-949-2301 x330                                   Personal Home Page
http://www.auc.ca                                     http://sault.org

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[AJ Lewis]

[-- Attachment #1: Type: text/plain, Size: 7425 bytes --]

On Thu, Jul 26, 2001 at 01:19:28PM -0400, Jason Tackaberry wrote:
> I have discovered one or more severe bugs in any or all of lvm, md, and
> reiserfs.  I'm not a kernel guy and I don't claim to know how these
> things are supposed to interact at that level, so I'm not going to point
> fingers.  Instead, here's what happened and I'll let you decide. :)
> 
> The problem surfaces when you create a volume group with lvm and include
> a partition that happens to be mounted.  In my tests the cwd was this
> mounted filesystem.  This may or may not be a necessary condition to
> reproduce this problem.  (I didn't test otherwise.)
> 
> In my tests I have /dev/sda7 and /dev/sdb7.  I created an md raid0
> device and assigned that to /dev/md3.  Then, here are the sequence of
> commands I issued:
> 
>   mkreiserfs /dev/md3
>   mount /dev/md3 /space
>   cd /space
>   pvcreate /dev/md3
>   vgcreate vol01 /dev/md3

Umm...try this one:

  mkreiserfs /dev/md3
  mount /dev/md3 /space
  cd /space
  dd if=/dev/zero of=/dev/md3 bs=1024k count=500

is that a bug?

I'm not sayign we shouldn't check for this if possible...I'm just saying
there *are* other things that do this...

> Obviously it's not normal to create a vg on a device that's mounted.  I
> did this by accident, but the result was pretty ugly.  Immediately after
> doing 'vgcreate' this popped up in the console:
> 
> --- snip ---------------------------------------------------------------
> is_tree_node: node level 19784 does not match to the expected one -1
> vs-5150: search_by_key: invalid format found in block 0. Fsck?
> kernel BUG at namei.c:343!
> invalid operand: 0000
> CPU:    0
> EIP:    0010:[<c017d965>]
> EFLAGS: 00010286
> eax: 0000001b   ebx: c2047e60   ecx: c2c0e000   edx: c02d88c4
> esi: c28795e0   edi: c2047ea0   ebp: c2047e04   esp: c2047df8
> ds: 0018   es: 0018   ss: 0018
> Process bash (pid: 834, stackpage=c2047000)
> Stack: c028ac46 c028acda 00000157 00000001 00000002 7f84dbff 000001f4 00000000
>        00000003 c011ac84 00000001 c028ad8e c2047e60 c2047ea0 c23be0e0 c017da8b
>        c28795e0 c23be13c 0000000a c2047e60 c2047ea0 00000000 00000001 c03275c0
> Call Trace: [<c011ac84>] [<c017da8b>] [<c0139b70>] [<c013a582>] [<c0141999>] [<c0139c2f>] [<c013a341>]
>        [<c013a94a>] [<c0137993>] [<c012fae3>] [<c0106ecb>]
> 
> Code: 0f 0b 83 c4 0c 8b 54 24 3c 52 8b 44 24 3c 50 57 55 e8 25 fc
> --- snip ---------------------------------------------------------------
> 
> After realising that I'd tried to vgcreate a partition that was mounted,
> I cd'ed out of /space and issued 'umount /space'.  Then I got this:
> 
> --- snip ---------------------------------------------------------------
> journal-2332: Trying to log block 16, which is a log block
> kernel BUG at prints.c:332!
> invalid operand: 0000
> CPU:    0
> EIP:    0010:[<c01887a4>]
> EFLAGS: 00010286
> eax: 0000001c   ebx: c0335a80   ecx: cf41c000   edx: c02d88c4
> esi: cc3c1e68   edi: c0335ebd   ebp: c0335a80   esp: cc3c1e2c
> ds: 0018   es: 0018   ss: 0018
> Process umount (pid: 881, stackpage=cc3c1000)
> Stack: c028c6c6 c028c9c0 0000014c cc3c1e44 cc3c1e48 00000000 000007e0 00000001
>        00000001 d0865024 00000000 d0865000 c0195e4e ce803800 c028fb80 00000010
>        00000560 00000000 00000001 00000001 00000807 00000000 00000000 ce633ea0
> Call Trace: [<c0195e4e>] [<c0193546>] [<c018684a>] [<c01935d1>] [<c018685a>] [<c0135174>] [<c0139b88>]
>        [<c013559a>] [<c012fb83>] [<c0121ea2>] [<c01355dc>] [<c0106ecb>]
> 
> Code: 0f 0b 83 c4 0c 8b 54 24 28 85 d2 74 08 8b 44 24 28 83 48 28
> Segmentation fault
> --- snip ---------------------------------------------------------------
> 
> After umount segfaulted, I decided I ought to reboot.  ctrl-alt-del or
> halt(8) appeared to have no effect, so I hit the reset button.
> 
> As my system was coming back up, the boot sequence continually said
> "Can't open /etc/mtab: input/output error."  The boot and root
> partitions are both md RAID1 devices with ext2.  The disks seemed to
> fsck okay, and when I logged in, I indeed could not read /etc/mtab.  'ls
> -l /etc/mtab' even yielded the same input/output error.  I rebooted with
> the stock RH 7.1 kernel that has no lvm support and this error
> persisted.  So this seems to be a problem with md?
> 
> I managed to fix it by booting the RedHat rescue cd, mounting the root
> partition, and deleting /etc/mtab.  All seems okay.
> 
> Just to see if this problem was linked to md beneath LVM, I repeated the
> above with /dev/sda7 instead of /dev/md3.  Same result.  Now, I also
> repeated the above using ext2 instead of reiserfs.  ext2 complained:
> 
> EXT2-fs error (device sd(8,7)): ext2_write_inode: unable to read inode
> block - inode=2, block=1899261259
> attempt to access beyond end of device
> 
> But it did not cause any kernel errors.  This process failed gracefully
> with ext2.
> 
> Now, if I had to blame LVM for something, I'd say it should at least not
> let me vgcreate with a mounted partition, or one that has inodes in use.
> mkraid, for exaple, would not let me start a raid device with a mounted
> partition.  If vgcreate failed with an error saying "This partition is
> mounted or has inodes in use" I would have realized the problem and
> unmounted /space before vgcreate.  The kernel errors came from Reiserfs,
> so there seems to be a problem there.  But like I said, I'm not a kernel
> guy so I'll let you guys confirm this before I submit a bug report to
> the Reiserfs team (or maybe one of the LVM hackers would be a better
> candidate).  Finally, md did not handle this robustly at all.  The whole
> /etc/mtab issue makes me extremely nervous since md is handling my boot
> and root parition with RAID1.  I'm just in a testing stages,
> fortunately, but I'm thinking md/lvm aren't suitable for this system
> which will be in production in a month.
> 
> I'd be happy to repeat and experiments or give further information if
> you need some.  This system is mine to beat on for a little while longer
> before it needs to go live.
> 
> Cheers,
> Jason.
> 
> --
> Academic Computing Support Specialist         Assistant Section Editor
> Algoma University College                     http://linux.com/develop
> Sault Ste. Marie, Ontario                 
> 705-949-2301 x330                                   Personal Home Page
> http://www.auc.ca                                     http://sault.org
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

-- 
AJ Lewis
Sistina Software Inc.                  Voice:  612-638-0500
1313 5th St SE, Suite 111              Fax:    612-638-0500
Minneapolis, MN 55414                  E-Mail: lewis@sistina.com
http://www.sistina.com

Current GPG fingerprint = 3B5F 6011 5216 76A5 2F6B  52A0 941E 1261 0029 2648
Get my key at: http://www.sistina.com/~lewis/gpgkey
 (Unfortunately, the PKS-type keyservers do not work with multiple sub-keys)

-----Begin Obligatory Humorous Quote----------------------------------------
"We succeed because our competitors are idiots" - anonymous
-----End Obligatory Humorous Quote------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Jason Tackaberry]

> Umm...try this one:
> 
>   mkreiserfs /dev/md3
>   mount /dev/md3 /space
>   cd /space
>   dd if=/dev/zero of=/dev/md3 bs=1024k count=500
> 
> is that a bug?

Good point.

Still, the bugs that surface in md and reiserfs make me a little
worried.  md has been around for a while now, hasn't it?  I get the
feeling it's a bit fragile.  Well, maybe it's just me.

Jason.

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[AJ Lewis]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

On Thu, Jul 26, 2001 at 04:07:01PM -0400, Jason Tackaberry wrote:
> > Umm...try this one:
> > 
> >   mkreiserfs /dev/md3
> >   mount /dev/md3 /space
> >   cd /space
> >   dd if=/dev/zero of=/dev/md3 bs=1024k count=500
> > 
> > is that a bug?
> 
> Good point.
> 
> Still, the bugs that surface in md and reiserfs make me a little
> worried.  md has been around for a while now, hasn't it?  I get the
> feeling it's a bit fragile.  Well, maybe it's just me.

I personally don't know a whole lot about MD, but there seem to be a lot
of people using it.  Maybe people on the list using MD and LVM together can
comment.

I'm just not sure if we can rely on something like /proc/mounts for pvcreate
and vgcreate...I will look into it though.  If you have any suggestions, or
if you want to send patches, I would appreciate it.

Regards,
-- 
AJ Lewis
Sistina Software Inc.                  Voice:  612-638-0500
1313 5th St SE, Suite 111              Fax:    612-638-0500
Minneapolis, MN 55414                  E-Mail: lewis@sistina.com
http://www.sistina.com

Current GPG fingerprint = 3B5F 6011 5216 76A5 2F6B  52A0 941E 1261 0029 2648
Get my key at: http://www.sistina.com/~lewis/gpgkey
 (Unfortunately, the PKS-type keyservers do not work with multiple sub-keys)

-----Begin Obligatory Humorous Quote----------------------------------------
Computers make very fast, very accurate mistakes.
-----End Obligatory Humorous Quote------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Jason Tackaberry]

> This was a stupid thing to do; and yes we should probably check to see

Indeed.  I didn't realize the partition was still mounted (I had left
the system and come back to it some time later).  Of course it should
fail, but not that bad. :)

> pvcreate would have stamped it's own metadata over md's metadata, so
> I'm not surprised md had trouble.  This is not a bug in md or
> reiserfs, just user error that LVM should have caught.

With respect to the problem I had with /etc/mtab: the root partition is
not handled by LVM at all.  It is strictly a md device.  I assume
/etc/mtab got hosed when umount segfaulted (because of the ReiserFS
bug).  But the root partition (and /etc on the root partition) is
md/raid1 with ext2.  This seems to me to be a real md bug.  What do you
think?

Thanks,
Jason.

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Joe Thornber]

On Thu, Jul 26, 2001 at 01:19:28PM -0400, Jason Tackaberry wrote:
> I have discovered one or more severe bugs in any or all of lvm, md, and
> reiserfs.  I'm not a kernel guy and I don't claim to know how these
> things are supposed to interact at that level, so I'm not going to point
> fingers.  Instead, here's what happened and I'll let you decide. :)
> 
> The problem surfaces when you create a volume group with lvm and include
> a partition that happens to be mounted.

This was a stupid thing to do; and yes we should probably check to see
if anyone else has the pv open when we do a pvcreate.  I will look into
it to see how much code needs to be changed.

pvcreate would have stamped it's own metadata over md's metadata, so
I'm not surprised md had trouble.  This is not a bug in md or
reiserfs, just user error that LVM should have caught.

- Joe

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Werner John]

AJ Lewis writes:
> [...]
> > 
> > Still, the bugs that surface in md and reiserfs make me a little
> > worried.  md has been around for a while now, hasn't it?  I get the
> > feeling it's a bit fragile.  Well, maybe it's just me.
> 
> I personally don't know a whole lot about MD, but there seem to be a lot
> of people using it.  Maybe people on the list using MD and LVM together can
> comment.
> [...]

I cannot comment on reiserfs, because I only use it on my equipment for
test reasons, but I have LVM on top of a RAID-5 and a RAID-0 up and running
for more than 140 days on one of our machines. No problems so far, although
it is still 0.9.1-beta5 and a "rather old" kernel ;-). Maybe it is time to
upgrade this one, too.

I use RAID since kernel 2.2.14 came out, one machine here is still running
2.2.18 with a RAID-5 (no LVM) and is up ever since.

Regards,

-- 
   Werner John                         Tel: +49 (0)69 829 757-0
   Offenbacher Stanzwerkzeugfabrik     Fax: +49 (0)69 829 757-25
   Ludwigstr. 30                       E-Mail: john at oswf dot de
   63067 Offenbach, Germany

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Jason Tackaberry]

> I cannot comment on reiserfs, because I only use it on my equipment for
> test reasons, but I have LVM on top of a RAID-5 and a RAID-0 up and running

I guess you mean software RAID using md? 

> for more than 140 days on one of our machines. No problems so far, although
> it is still 0.9.1-beta5 and a "rather old" kernel ;-). Maybe it is time to
> upgrade this one, too.

I guess this is the concern: md clearly works when all is going well,
but when somethings goes awry (like, for example, in my case where a
kernel bug in reiserfs caused umount to segfault while it had a lock on
/etc/mtab), is it robust enough?

Are there others on this list who can share their md/lvm experiences use
with me?

Thanks,
Jason.

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Kevin M Corry]

> > pvcreate would have stamped it's own metadata over md's metadata, so
> > I'm not surprised md had trouble.  This is not a bug in md or
> > reiserfs, just user error that LVM should have caught.

Given his example, pvcreate would have overwritten Reiser's metadata, not
md's, so md shouldn't have really cared.

>
> With respect to the problem I had with /etc/mtab: the root partition is
> not handled by LVM at all.  It is strictly a md device.  I assume
> /etc/mtab got hosed when umount segfaulted (because of the ReiserFS
> bug).  But the root partition (and /etc on the root partition) is
> md/raid1 with ext2.  This seems to me to be a real md bug.  What do you
> think?

Don't go blaming md yet. I was playing around with some new code on my
test box and crashed the system. When it came back, I had the exact same
problem with I/O errors to /etc/mtab. The system simply would not mount
any drives besides root. I actually had to fsck the mounted root fs to
correct the /etc/mtab problem. My root is ext2 on LVM. No md or Reiser.
But I don't believe the problem had anything to do with LVM. I believe
it was just due to really bad filesystem corruption.

-Kevin

-----------------
corryk@us.ibm.com
http://www.sf.net/project/evms/

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Jason Tackaberry]

> any drives besides root. I actually had to fsck the mounted root fs to
> correct the /etc/mtab problem. My root is ext2 on LVM. No md or Reiser.
> But I don't believe the problem had anything to do with LVM. I believe
> it was just due to really bad filesystem corruption.

Interesting.  This indeed seems to vindicate md.  Maybe I'll have
another serious look at it. :)

Cheers,
Jason.

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Werner John]

Jason Tackaberry writes:
> [...]
> 
> I guess you mean software RAID using md? 

yes.

> > for more than 140 days on one of our machines. No problems so far, although
> > it is still 0.9.1-beta5 and a "rather old" kernel ;-). Maybe it is time to
> > upgrade this one, too.
> 
> I guess this is the concern: md clearly works when all is going well,
> but when somethings goes awry (like, for example, in my case where a
> kernel bug in reiserfs caused umount to segfault while it had a lock on
> /etc/mtab), is it robust enough?

don't mix md with the file system. The FS doesn't care about anything that
goes on below its level. You could have an ext2 on a ramdisk, or you could
have an ext2 on a SW RAID-0 on top of a ramdisk and a floppy drive.

> Are there others on this list who can share their md/lvm experiences use
> with me?

I have "tested" the SW RAID and LVM by simply pulling the plug while
writing a 500MB file to my disks. Well, I know, this should not be called a
real test, I consider it a brute force attack ;-) Just one time, the FS was
damaged so badly that I could not repair it.

Regards,

Werner

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Ralph Jennings]

On Mon, Jul 30, 2001 at 09:46:16AM +0200, Werner John wrote:
> I have "tested" the SW RAID and LVM by simply pulling the plug while
> writing a 500MB file to my disks. Well, I know, this should not be called a
> real test, I consider it a brute force attack ;-) Just one time, the FS was
> damaged so badly that I could not repair it.

Ever try it with ReiserFS?  It *shouldn't* have any problems, other
than the 500MB file will be <500MB. (shouldn't being the opperative
word here)

Re: [linux-lvm] Nasty bug in lvm and/or md and/or reiserfs

[Werner John]

Ralph Jennings writes:
> On Mon, Jul 30, 2001 at 09:46:16AM +0200, Werner John wrote:
> > I have "tested" the SW RAID and LVM by simply pulling the plug while
> > writing a 500MB file to my disks. Well, I know, this should not be called a
> > real test, I consider it a brute force attack ;-) Just one time, the FS was
> > damaged so badly that I could not repair it.
> 
> Ever try it with ReiserFS?  It *shouldn't* have any problems, other
> than the 500MB file will be <500MB. (shouldn't being the opperative
> word here)

no, sorry.

Werner