[linux-lvm] reiserfs panic

[linux-lvm] reiserfs panic

[Patrick Caulfield]

I'm posting this here partly because I hope Chris Mason is reading(!) and partly
because it may be of some use to others on the list.

There seems to be a problem when reading from a deactivated snapshot of a
reiserfs volume that causes a panic.

The problem is caused by the fact that when a snapshot becomes full, LVM
returns I/O errors for the volume and when reiserfs hits an I/O error it panics.

The solution is, of course, to make your snapshots large enough that this never
happens :-) but just in case this is not possible the following patch may be of
use:


diff -u fs/reiserfs/namei.c.orig fs/reiserfs/namei.c
--- fs/reiserfs/namei.c.orig    Thu Nov 29 16:13:28 2001
+++ fs/reiserfs/namei.c Thu Nov 29 16:34:49 2001
@@ -320,9 +320,10 @@
 
     while (1) {
        retval = search_by_entry_key (dir->i_sb, &key_to_search, path_to_entry, de);
-       if (retval == IO_ERROR)
+       if (retval == IO_ERROR) 
+           return NAME_NOT_FOUND;
            // FIXME: still has to be dealt with
-           reiserfs_panic (dir->i_sb, "zam-7001: io error in " __FUNCTION__ "\n");
+           // reiserfs_panic (dir->i_sb, "zam-7001: io error in " __FUNCTION__ "\n");
 
        /* compare names for all entries having given hash value */
        retval = linear_search_in_dir_item (&key_to_search, de, name, namelen);



patrick

Re: [linux-lvm] reiserfs panic

[Chris Mason]

On Friday, November 30, 2001 10:24:23 AM +0000 Patrick Caulfield
<patrick@tykepenguin.com> wrote:

> I'm posting this here partly because I hope Chris Mason is reading(!) and
> partly because it may be of some use to others on the list.

;-)

> 
> There seems to be a problem when reading from a deactivated snapshot of a
> reiserfs volume that causes a panic.
> 
> The problem is caused by the fact that when a snapshot becomes full, LVM
> returns I/O errors for the volume and when reiserfs hits an I/O error it
> panics.
> 
> The solution is, of course, to make your snapshots large enough that this
> never happens :-) but just in case this is not possible the following patch
> may be of use:
> 
> 
> diff -u fs/reiserfs/namei.c.orig fs/reiserfs/namei.c
> --- fs/reiserfs/namei.c.orig    Thu Nov 29 16:13:28 2001
> +++ fs/reiserfs/namei.c Thu Nov 29 16:34:49 2001
> @@ -320,9 +320,10 @@
>  
>      while (1) {
>         retval = search_by_entry_key (dir->i_sb, &key_to_search,
> path_to_entry, de); -       if (retval == IO_ERROR)
> +       if (retval == IO_ERROR) 
> +           return NAME_NOT_FOUND;

The problem here is that NAME_NOT_FOUND does not mean error.  It means it is
safe to insert something with that name, which isn't what we want.  For
snapshots though, it will work fine.

Now that 2.5.x is out I expect more of these to get cleaned up, and I'm
workin on patches to turn reiserfs_panic into a forced readonly mount.

-chris

Re: [linux-lvm] reiserfs panic

[Patrick Caulfield]

On Fri, Nov 30, 2001 at 08:57:12AM -0500, Chris Mason wrote:
> 
> 
> The problem here is that NAME_NOT_FOUND does not mean error.  It means it is
> safe to insert something with that name, which isn't what we want.  For
> snapshots though, it will work fine.

Oh granted. I didn't intend it to be a fix as such - sorry if some read it as
that - merely a (rather crude) way to prevent a kernel panic in the event a
snapshot gets full.

 
> Now that 2.5.x is out I expect more of these to get cleaned up, and I'm
> workin on patches to turn reiserfs_panic into a forced readonly mount.

That would be nice - but maybe a no-op in the case of an LVM snapshot :-)

patrick

Re: [linux-lvm] reiserfs panic

[Andreas Dilger]

On Nov 30, 2001  08:57 -0500, Chris Mason wrote:
> Now that 2.5.x is out I expect more of these to get cleaned up, and I'm
> workin on patches to turn reiserfs_panic into a forced readonly mount.

Just as an FYI, ext2 has a distinction between "ext2_panic", "ext2_error",
and "ext2_warning".  The panic is just what it sounds like (and can only
be triggered by something like memory corruption, it is almost unused).
The error has a configurable behaviour - panic also, ro-mount, warning.
In all cases, it marks the filesystem superblock with an error.  The
default is ro-mounting the filesystem.  The warning is just a warning
(unusual condition, but a 100% recoverable situation).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

[linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Chris Worley]

If you remove an lv, and the minor number of the removed lv is less than
the minor number of your root partition, then the vgscan during initrd
will run correctly, but assign a minor device number to the root
partition that lilo doesn't expect (and you won't be able to mount the
root partition).

This caused a few tense moments with root booting LVM this morning ;)...

With a new big hard drive in hand, I started upgrading my server.  Added
the new disk as it's own logical volume, reiser format, root initrd'd,
and copied all my other disks to it, then started upgrading to SuSE 7.3.

During the upgrade, I was running out of disk space , so I decommisioned
one of the older lv's with lvremove, then lvextended it onto the new lv,
and had reiser expand into the new space.  Quite easy and nifty; that's
what LVM's all about.

I even ran lilo before the next reboot, but it wouldn't have mattered...

Upon the next reboot, the initrd ran fine... found and activated the
lv's... but the kernel said "no root partition" and hung.  I could
rescue boot the system from the SuSE CD -- everything was there... I
finally added a "vgdisplay -v" to the initrd's linuxrc, and that showed
me the problem.

The file system it was trying to mount was, to lilo, the device "3a:03"
(major device number 58, minor 3), but, once I'd removed the old lv for
extra disk space, any new vgscan during initrd assigned my new root lv
to minor device number 2.  "root=/dev/..." doesn't help, for lvm (it
doesn't like the name).  There was no longer an lvm minor device 3 to
mount.

Once I saw the problem, I recue booted and chrooted to my new lvm, and
manually changed the minor number before running lilo.  

I'm guessing vgscan can be run on an active system to realign the minor
numbers, before running lilo, but I'm not sure... 

_BUT_, I think the faq's ought to add a little note concerning this side
effect to root booted lvm file systems to the "Removing a logical
volume" section, and save the next person some "tense moments ;)".

If this info is already there, then disregard this sugestion (I haven't
seen it, but I do tend to skim the manuals).

Chris

RE: [linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Ben Holness]

> Upon the next reboot, the initrd ran fine... found and activated the
> lv's... but the kernel said "no root partition" and hung.  I could
> rescue boot the system from the SuSE CD -- everything was there... I
> finally added a "vgdisplay -v" to the initrd's linuxrc, and
> that showed me the problem.
>
> The file system it was trying to mount was, to lilo, the device "3a:03"
> (major device number 58, minor 3), but, once I'd removed the old lv for
> extra disk space, any new vgscan during initrd assigned my new root lv
> to minor device number 2.  "root=/dev/..." doesn't help, for lvm (it
> doesn't like the name).  There was no longer an lvm minor device 3 to
> mount.

This looks like the same problem that I am having, although I haven't
removed any logical volumes!

I am not 100% sure how to find the device number, but common sense tells me
that it is the "Block #" at the bottom of the logical volume information, in
which case my root LV is Block # 58:1

> Once I saw the problem, I recue booted and chrooted to my new lvm, and
> manually changed the minor number before running lilo.

Please can you tell me how you changed this?

Cheers,

Ben

RE: [linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Steven Lembark]

-- Ben Holness <ben@bens-house.org.uk>

>> Upon the next reboot, the initrd ran fine... found and activated the
>> lv's... but the kernel said "no root partition" and hung.  I could
>> rescue boot the system from the SuSE CD -- everything was there... I
>> finally added a "vgdisplay -v" to the initrd's linuxrc, and
>> that showed me the problem.

We can add it under the general heading of why using
LVM for boot volumes doesn't work on i386 platforms.

Giving /usr its own mount point genrally levaes the
root voume small, static and perfectly happy on a
small partition at the top of your disk. Using a
static link for LVM and installing it in the root
leaves the whole setup quite a bit more dependable.

--
Steven Lembark                               2930 W. Palmer
Workhorse Computing                       Chicago, IL 60647
                                            +1 800 762 1582

Re: [linux-lvm] reiserfs panic

[Ed Tomlinson]

Wonder if this is the result of something not getting into cvs?  Mid October
Chris fixed a bunch of snapshot problems.  One was with full snapshots...
He posted a fix here, some one else posted an alternate.  Bet neither has 
been put into cvs.  Here is what I am using with 'rc4ish'.  Does it help?

Ed Tomlinson

------------------ (From Chris Mason)
Hi guys,

Here's a repost of the patch that fixes oopsen when snapshots
get full.  Please comment and/or include ;-)

Against 1.0.1rc4, the bug should affect all kernel versions.

-chris

--- 0.21/drivers/md/lvm.c Sun, 07 Oct 2001 22:15:54 -0400 
+++ 0.21(w)/drivers/md/lvm.c Mon, 08 Oct 2001 15:54:42 -0400 
@@ -1142,7 +1142,8 @@
 
 	/* we must redo lvm_snapshot_remap_block in order to avoid a
 	   race condition in the gap where no lock was held */
-	if (!lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv) &&
+	if (lv->lv_block_exception && 
+	    !lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv) &&
 	    !lvm_snapshot_COW(rdev, rsector, pe_start, rsector, vg, lv))
 		lvm_write_COW_table_block(vg, lv);
 
@@ -1151,11 +1152,12 @@
 
 static inline void _remap_snapshot(kdev_t rdev, ulong rsector,
 				   ulong pe_start, lv_t *lv, vg_t *vg) {
-	int r;
+	int r = 0;
 
 	/* check to see if this chunk is already in the snapshot */
 	down_read(&lv->lv_lock);
-	r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv);
+	if (lv->lv_block_exception)
+		r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv);
 	up_read(&lv->lv_lock);
 
 	if (!r)
Index: 0.21/drivers/md/lvm-snap.c
--- 0.21/drivers/md/lvm-snap.c Sat, 06 Oct 2001 00:07:22 -0400 root (linux/i/c/38_lvm-snap.c 1.1.2.1.2.1 644)
+++ 0.21(w)/drivers/md/lvm-snap.c Mon, 08 Oct 2001 15:13:10 -0400 root (linux/i/c/38_lvm-snap.c 1.1.2.1.2.1 644)
@@ -140,6 +140,8 @@
 	unsigned long mask = lv->lv_snapshot_hash_mask;
 	int chunk_size = lv->lv_chunk_size;
 
+	if (!hash_table)
+		BUG() ;
 	hash_table = &hash_table[hashfn(org_dev, org_start, mask, chunk_size)];
 	list_add(&exception->hash, hash_table);
 }


_______________________________________________
linux-lvm mailing list
linux-lvm@sistina.com
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
------------------

On November 30, 2001 05:24 am, Patrick Caulfield wrote:
> I'm posting this here partly because I hope Chris Mason is reading(!) and
> partly because it may be of some use to others on the list.
>
> There seems to be a problem when reading from a deactivated snapshot of a
> reiserfs volume that causes a panic.
>
> The problem is caused by the fact that when a snapshot becomes full, LVM
> returns I/O errors for the volume and when reiserfs hits an I/O error it
> panics.
>
> The solution is, of course, to make your snapshots large enough that this
> never happens :-) but just in case this is not possible the following patch
> may be of use:
>
>
> diff -u fs/reiserfs/namei.c.orig fs/reiserfs/namei.c
> --- fs/reiserfs/namei.c.orig    Thu Nov 29 16:13:28 2001
> +++ fs/reiserfs/namei.c Thu Nov 29 16:34:49 2001
> @@ -320,9 +320,10 @@
>
>      while (1) {
>         retval = search_by_entry_key (dir->i_sb, &key_to_search,
> path_to_entry, de); -       if (retval == IO_ERROR)
> +       if (retval == IO_ERROR)
> +           return NAME_NOT_FOUND;
>             // FIXME: still has to be dealt with
> -           reiserfs_panic (dir->i_sb, "zam-7001: io error in "
> __FUNCTION__ "\n"); +           // reiserfs_panic (dir->i_sb, "zam-7001: io
> error in " __FUNCTION__ "\n");
>
>         /* compare names for all entries having given hash value */
>         retval = linear_search_in_dir_item (&key_to_search, de, name,
> namelen);
>
>
>
> patrick
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] reiserfs panic

[Andreas Dilger]

On Nov 30, 2001  07:44 -0500, Ed Tomlinson wrote:
> Wonder if this is the result of something not getting into cvs?  Mid October
> Chris fixed a bunch of snapshot problems.  One was with full snapshots...
> He posted a fix here, some one else posted an alternate.  Bet neither has 
> been put into cvs.  Here is what I am using with 'rc4ish'.  Does it help?

Hmm, I thought I had put this into CVS.  Note that my version put the
checks into lvm_snapshot_remap_block() rather than requiring the caller
to check it.  It looks like it is there in my CVS.

> @@ -1142,7 +1142,8 @@
>  
>  	/* we must redo lvm_snapshot_remap_block in order to avoid a
>  	   race condition in the gap where no lock was held */
> -	if (!lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv) &&
> +	if (lv->lv_block_exception && 
> +	    !lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv) &&
>  	    !lvm_snapshot_COW(rdev, rsector, pe_start, rsector, vg, lv))
>  		lvm_write_COW_table_block(vg, lv);

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

[linux-lvm] Fun little horror story -- please add to FAQ if it isn't already documented

[Chris Worley]

Sorry for the late reply... I don't subscribe to the list.

You've got to change the device minor number associated with the lvm 
partition.  I.E. if you partition name is /dev/foo/system, then you've 
got to get it to have the same minor device number vgscan is going to 
give during initrd.

If you're booted from some sort of rescue system that has a different 
root, "mount" your lvm root file system, "cd" to the directory where 
mounted, "chroot ." in that directory, manually delete and recreate 
(using "mknod") the /dev/foo/system file, with the correct minor 
number (the one vgscan is going to assign when initrd is running), 
then re-run lilo.

Note that this assumes that you've got a different root file system 
than the one you want, and the one you want won't boot because you 
deleted a lv with a lower minor number than your root partition.

<rant>I still think this should be documented under lvremove</rant>

Chris

Ben Holness writes:

>> Upon the next reboot, the initrd ran fine... found and activated the
>> lv's... but the kernel said "no root partition" and hung.  I could
>> rescue boot the system from the SuSE CD -- everything was there... I
>> finally added a "vgdisplay -v" to the initrd's linuxrc, and
>> that showed me the problem.
>>
>> The file system it was trying to mount was, to lilo, the device "3a:03"
>> (major device number 58, minor 3), but, once I'd removed the old lv for
>> extra disk space, any new vgscan during initrd assigned my new root lv
>> to minor device number 2.  "root=/dev/..." doesn't help, for lvm (it
>> doesn't like the name).  There was no longer an lvm minor device 3 to
>> mount.
> 
> This looks like the same problem that I am having, although I haven't
> removed any logical volumes!
> 
> I am not 100% sure how to find the device number, but common sense tells me
> that it is the "Block #" at the bottom of the logical volume information, in
> which case my root LV is Block # 58:1
> 
>> Once I saw the problem, I recue booted and chrooted to my new lvm, and
>> manually changed the minor number before running lilo.
> 
> Please can you tell me how you changed this?