[linux-lvm] Corrupted LV after resize2fs crashed when adding new disk to LV

[linux-lvm] Corrupted LV after resize2fs crashed when adding new disk to LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 2512 bytes --]

I've written several extensive posts about this problem to the fedora forum
(software) with no real results -and FWIW, I am getting very desparate. I
was referred here, or possibly post bug report.

The thread at Fedora Forums is >>click
here<<<http://forums.fedoraforum.org/showthread.php?t=231481>
.

Basically, this is the problem:

I have an ext4 filesystem in VolGroup3W-LogVol3W. Originally, it was 331G,
and was working fine. e2fsck ran through without finding any problems.

Then I tried to add another disk (111G) to the LV. I created the PV
(pvcreate), added it to the VG (vgextend). All this was okay. I then added
this to the LV (lvextend) and then tried to resize the filesystem
(resize2fs).

The last command - resize2fs - crashed before completing.

After that:

- vgdisplay showed a total 442G for VolGroup3W (the original size plus the
new PV) - that's okay
- lvs showed a total of 331G for LogVol3W (the original size, without the
new PV)
- mount failed, with "EXT4-fs: bad geometry: block count 102432768 exceeds
size of device (86773760 blocks)" in dmesg
- e2fsck for the LV reports basically the same thing.

I removed the new diisk from the VG. I couldn't do lvreduce, since the LV
didn't see the new PV space.

When I run e2fsck and answer 'n' to abort, I get 2648 Group checksum errors,
and after answering 'y' to fix them, I get this:

"/dev/VolGroup3W/LogVol3W contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error reading block 86802432 (Invalid argument) while getting next inode
from scan.  Ignore error? no

Error while scanning inodes (21700608): Can't read next
inode
e2fsck: aborted"

FWIW, I did try running e2fsck with other superblocks, with this result:

"[root@Elmer ~]# e2fsck -b 32768 /dev/VolGroup3W/LogVol3W
e2fsck 1.41.4 (27-Jan-2009)
/dev/VolGroup3W/LogVol3W: recovering journal
e2fsck: unable to set superblock flags on /dev/VolGroup3W/LogVol3W"


So it seems to me, IMHO, that the LVM is looking at one set of data, while
the other utilities (eg, mount, e2fsck) are looking at another. I am
assuming that the original LV data is still intact, just somewhere the file
size/block count is wrong.(one place says the original size, the other sees
the size with the new disk (attempted to be) added.

Is there ANY way possible to rectify this? PLEASE!!!

As I said, I am rather desparate to recover the data on the original LV
(yes, I do know the value of backing up data :-( ). So any ideas will be
greatly appreciated.

ken

[-- Attachment #2: Type: text/html, Size: 2799 bytes --]

Re: [linux-lvm] Corrupted LV after resize2fs crashed when adding new disk to LV

[Stuart D. Gathman]

On Thu, 8 Oct 2009, Ken Bass wrote:

> The last command - resize2fs - crashed before completing.

Did you let resize2fs default to the device size?

> After that:
> 
> - vgdisplay showed a total 442G for VolGroup3W (the original size plus the
> new PV) - that's okay
> - lvs showed a total of 331G for LogVol3W (the original size, without the
> new PV)

Assuming the previous answer was yes (if 'no', then that is your problem),
looks like the LVM metadata got rolled back.  (Effectively undoing the
lvextend.)  When that happened is when resize2fs crashed.

> When I run e2fsck and answer 'n' to abort, I get 2648 Group checksum errors,
> and after answering 'y' to fix them, I get this:

Bad move, when things go south, you want to leave things unmodified until
you figure it out.

> So it seems to me, IMHO, that the LVM is looking at one set of data, while
> the other utilities (eg, mount, e2fsck) are looking at another. I am

No, the mount,e2fsck are looking at a filesystem on an LV that got reduced (mid
resize, apparently).  Did you do 'lvs' before starting the resize?

> assuming that the original LV data is still intact, just somewhere the file
> size/block count is wrong.(one place says the original size, the other sees
> the size with the new disk (attempted to be) added.
> 
> Is there ANY way possible to rectify this? PLEASE!!!

Look at your metadata backups, and see if you can find the version with
the larger LV.  Look at the timestamp on the revert to get a clue on
how it got reverted.

First, I would make a copy of what you got (it will fit on a cheap
external USB disk) before making any more mods.  Then restore the metadata
to the extended LV before retrying e2fsck.

Disclaimer, I am not an LVM expert.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Corrupted LV after resize2fs crashed when adding new disk to LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 2304 bytes --]

The gods of Linux must be watching over me . I shall never blaspheme against
Linux again!

I had removed the new drive from the VG, then restored the VG (vgrestore) to
an archived version created before I had added the drive to the PV. I then
added the drive back to the VG (vgextend), extended the LV partition to the
max (lvextend), then resized the LV (resize2fs). Oh, before I could do the
resize, I had to run e2fsck, which praise the gods, ran successfully. As did
the resize.

I re-ran e2fsck (with -f) and ir ran 100% successful and clean.

And then the piece de resistance - I mounted the LV and everything was as I
orginally wanted. The old data was still there and valid, and the new size
of the LV was the original + the new disk!!!!!

I don't know what I did to make this work. I all seems magical to me now. As
with Stuart's below:

Disclaimer, I am (in no way, shape or form) an LV expert.

Thanks all.

ken

PS:

"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

This is from Mozart's unfinished Requiem. If you watch the movie Amadeus, it
is at the end right before he dies. I don't know Latin very well, but I
think it has something to do with the burning flames of h*ll :-).


On Thu, Oct 8, 2009 at 12:37 PM, Stuart D. Gathman <stuart@bmsi.com> wrote:

>
> On Thu, 8 Oct 2009, Ken Bass wrote:
>
> Look at your metadata backups, and see if you can find the version with
> the larger LV.  Look at the timestamp on the revert to get a clue on
> how it got reverted.
>
> First, I would make a copy of what you got (it will fit on a cheap
> external USB disk) before making any more mods.  Then restore the metadata
> to the extended LV before retrying e2fsck.
>
> Disclaimer, I am not an LVM expert.
>
> --
>              Stuart D. Gathman <stuart@bmsi.com>
>    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?" commercial.
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 3022 bytes --]