[linux-lvm] Rebuilding ext4 filesystem on an LV

[linux-lvm] Rebuilding ext4 filesystem on an LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]

In short:

- originally created LV with several physical drives (5)
- formatted LV for ext4
- one physical drive got many sectors trashed. Those included the partition
and metadata.
- did the following to recover the LV (VolGroupX-LogVolX):
---pvcreate --uuid <original uuid>  /dev/sdX  (successfully created)
---vgcfgrestore VolGroupX (restsored volume group VolGroupX)
---vgscan  (found volume group "VolGroupX" using metadata type lvm2)
--- vgchange -ay VolGroupX ( note response:
" device-mapper: reload ioctl failed: Invalid argument.
 1 logical volume(s) in volume group "VolGroupX" now active")

At this pont, /dev/mapper/VolGroupX-LogVolX exists, but I can't run any
utilities on it, since it can't find any superblock. For example:
________________
[root@Elmer ken] # e2fsck  /dev/mapper/VolGroupX-LogVolX
e2fsck 1.41.9 (22-Aug-2009)
e2fsck: Invalid argument while trying to open /dev/mapper/VolGroupX-LogVolX

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

[root@Elmer ken] #
_________________________
I really want to recover at least the data on the 4 other drives, and any
data left from this drive as well.

Any help will be greatly appreciated.

TIA

ken

[-- Attachment #2: Type: text/html, Size: 1633 bytes --]

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Bryan Whitehead]

Dig deep into your memory (or just try different combos) of running
mkfs.ext4 -n /dev/yourVG/yourLV

-n is like a dry run so you can look for the alternate superblock locations.

Using the above info, run fsck.ext4 -b <alt location> /dev/yourVG/yourLV

Even if it can find an alternate superblock, I'm not confident the
fsck would be able to help much�

That said, I think you might be out of luck. Go ahead and give it a
shot, I'm curious how much data you'll get.

On Sun, Jul 4, 2010 at 6:30 PM, Ken Bass <daytooner@gmail.com> wrote:
> In short:
>
> - originally created LV with several physical drives (5)
> - formatted LV for ext4
> - one physical drive got many sectors trashed. Those included the partition
> and metadata.
> - did the following to recover the LV (VolGroupX-LogVolX):
> ---pvcreate --uuid <original uuid>� /dev/sdX� (successfully created)
> ---vgcfgrestore VolGroupX (restsored volume group VolGroupX)
> ---vgscan� (found volume group "VolGroupX" using metadata type lvm2)
> --- vgchange -ay VolGroupX ( note response:
> " device-mapper: reload ioctl failed: Invalid argument.
> �1 logical volume(s) in volume group "VolGroupX" now active")
>
> At this pont, /dev/mapper/VolGroupX-LogVolX exists, but I can't run any
> utilities on it, since it can't find any superblock. For example:
> ________________
> [root@Elmer ken] # e2fsck� /dev/mapper/VolGroupX-LogVolX
> e2fsck 1.41.9 (22-Aug-2009)
> e2fsck: Invalid argument while trying to open /dev/mapper/VolGroupX-LogVolX
>
> The superblock could not be read or does not describe a correct ext2
> filesystem.� If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
> ��� e2fsck -b 8193 <device>
>
> [root@Elmer ken] #
> _________________________
> I really want to recover at least the data on the 4 other drives, and any
> data left from this drive as well.
>
> Any help will be greatly appreciated.
>
> TIA
>
> ken
>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Sun, Jul 4, 2010 at 10:54 PM, Bryan Whitehead <driver@megahappy.net>wrote:

> Dig deep into your memory (or just try different combos) of running
> mkfs.ext4 -n /dev/yourVG/yourLV
>
> -n is like a dry run so you can look for the alternate superblock
> locations.
>
> Using the above info, run fsck.ext4 -b <alt location> /dev/yourVG/yourLV
>
>
Tried all of that, with many different offsets. No luck.


> Even if it can find an alternate superblock, I'm not confident the
> fsck would be able to help much…
>

Do you mean in general, or just for recovery of the trashed drive? I didn't
have much hope for the trashed drive, but I was hoping to recover what was
on the other drives.


> That said, I think you might be out of luck. Go ahead and give it a
> shot, I'm curious how much data you'll get.
>

Me too :-).

But what bothers me is this message I get when I run e2fsck (which runs
fsck.ext4):
"e2fsck: Invalid argument while trying to open
/dev/mapper/VolGroupX-LogVolX"

Why am I getting that?

thx for the response.

ken

[-- Attachment #2: Type: text/html, Size: 1739 bytes --]

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Bryan Whitehead]

> But what bothers me is this message I get when I run e2fsck (which runs
> fsck.ext4):
> "e2fsck: Invalid argument while trying to open
> /dev/mapper/VolGroupX-LogVolX"
>
> Why am I getting that?
> thx for the response.

The lv is available right?

It would be great to see the output from lvdisplay and vgdisplay.

-Bryan

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

On Mon, Jul 5, 2010 at 3:06 AM, Bryan Whitehead <driver@megahappy.net>wrote:

>
> The lv is available right?
>
> It would be great to see the output from lvdisplay and vgdisplay.
>
> -Bryan
>
>
___________________________________
[root@Elmer ken]# lvdisplay /dev/VolGroupX/LogVolX
  --- Logical volume ---
  LV Name                /dev/VolGroupX/LogVolX
  VG Name                VolGroupW
  LV UUID                eBQgqM-RHra-ruaK-CcEt-stnX-U9yR-Uz4UH5
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                442.80 GB
  Current LE             113358
  Segments               5
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:11

[root@Elmer ken]# vgdisplay VolGroupX
  --- Volume group ---
  VG Name               VolGroupX
  System ID
  Format                lvm2
  Metadata Areas        5
  Metadata Sequence No  28
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                5
  Act PV                5
  VG Size               442.80 GB
  PE Size               4.00 MB
  Total PE              113358
  Alloc PE / Size       113358 / 442.80 GB
  Free  PE / Size       0 / 0
  VG UUID               2CXUUY-j6nu-07qP-kKd1-ZkBJ-hNwf-iunkwK

___________________________________

I would think I should be able to find a backup superblock among the
remaining drives. I tried running mke2fs -n to get a list of possible
superblocks, but when I ran e2fsck with those values, I got the ioctl error,
and then the no superblock found message.

Is there any way to scan each of the individual drives for data, outside of
LVM? Just another desparate thought.

thx again.

ken

[-- Attachment #2: Type: text/html, Size: 2425 bytes --]

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Stuart D. Gathman]

On Mon, 5 Jul 2010, Ken Bass wrote:

> But what bothers me is this message I get when I run e2fsck (which runs
> fsck.ext4):
> "e2fsck: Invalid argument while trying to open
> /dev/mapper/VolGroupX-LogVolX"
> 
> Why am I getting that?

Not an expert, just a user, but I suspect that the ioerr is because the
metadata for that LV is still pointing to the missing PV.  (I know you replaced
it, but that doesn't update the LV.)  I also noticed that the lvdisplay
you posted says LogVolX belongs to VolGroupW, not VolGroupX.  Perhaps you
sanitized the names and that is a typo - otherwise, there is something screwy.

I suspect recovering the remaining data will require skipping the (huge)
chuck of LV that was on the missing PV.  Or maybe the experts here
will recommend editing the metadata to move the extent from the missing
PV to the new PV.  The huge chunk will now have random data from the 
new drive.  I would have initialized the new drive to zero (using the
security erase) before using (maybe you did).

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

On Mon, Jul 5, 2010 at 11:27 AM, Stuart D. Gathman <stuart@bmsi.com> wrote:

> On Mon, 5 Jul 2010, Ken Bass wrote:
>
> > But what bothers me is this message I get when I run e2fsck (which runs
> > fsck.ext4):
> > "e2fsck: Invalid argument while trying to open
> > /dev/mapper/VolGroupX-LogVolX"
> >
> > Why am I getting that?
>
> Not an expert, just a user, but I suspect that the ioerr is because the
> metadata for that LV is still pointing to the missing PV.  (I know you
> replaced
> it, but that doesn't update the LV.)  I also noticed that the lvdisplay
> you posted says LogVolX belongs to VolGroupW, not VolGroupX.  Perhaps you
> sanitized the names and that is a typo - otherwise, there is something
> screwy.
>
>
Yes, I guess my sanitizing wasn't 100% sterile :-(.


> I suspect recovering the remaining data will require skipping the (huge)
> chuck of LV that was on the missing PV.  Or maybe the experts here
> will recommend editing the metadata to move the extent from the missing
> PV to the new PV.
>

That is what I am really trying to do. As I said early on, I was hoping to
find a backup superblock on the remaining part of the LV that might help
restore some of the data.. I have also looked at the metadata files, and was
wondering if I could manually rearrange the PVs. Any expert opinion on that?


FWIW: I found some utilities that supposedly will scan a drive with an
ext2/3 filesystem and recover files from it - although it only runs on
windows. It's free, so maybe worth a try.

thx to all for the help.

ken

[-- Attachment #2: Type: text/html, Size: 2116 bytes --]

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Malahal Naineni]

Stuart D. Gathman [stuart@bmsi.com] wrote:
> On Mon, 5 Jul 2010, Ken Bass wrote:
> 
> > But what bothers me is this message I get when I run e2fsck (which runs
> > fsck.ext4):
> > "e2fsck: Invalid argument while trying to open
> > /dev/mapper/VolGroupX-LogVolX"
> > 
> > Why am I getting that?
> 
> Not an expert, just a user, but I suspect that the ioerr is because
> the metadata for that LV is still pointing to the missing PV.  (I know
> you replaced it, but that doesn't update the LV.)

If everything went right, he should have the new PV in the LV, but looks
like something failed while activating the LV. Run 'dmsetup table' to
find out if the LV currently has new PV or old PV. You need
device-mapper knowledge for this.

> I suspect recovering the remaining data will require skipping the (huge)
> chuck of LV that was on the missing PV.  Or maybe the experts here
> will recommend editing the metadata to move the extent from the missing
> PV to the new PV.  The huge chunk will now have random data from the 
> new drive.  I would have initialized the new drive to zero (using the
> security erase) before using (maybe you did).

The best course of action would have been 'reading what ever data
available' from old disk to new disk. If the disk is absolutely
thrashed, then zero'ing the new PV may be better.

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

On Tue, Jul 6, 2010 at 9:17 AM, Malahal Naineni <malahal@us.ibm.com> wrote:


> If everything went right, he should have the new PV in the LV, but looks
> like something failed while activating the LV. Run 'dmsetup table' to
> find out if the LV currently has new PV or old PV. You need
> device-mapper knowledge for this.
>
> I will have to try that out. Will let you know.


>
> The best course of action would have been 'reading what ever data
> available' from old disk to new disk. If the disk is absolutely
> thrashed, then zero'ing the new PV may be better.
>
>
I did try to read anything I could out of the old disk. I do believe there
is still some data on it that wasn't trashed. The problem is how to find it
and reconstruct it without any of the LVM or ext4 filesystem  information
(as I said, I couldn't find any superblocks). Are there any utilities to do
something like that?

Then there are the remaining drives that weren't affected. Unfortunately
without the first drive, I can't get to anything on the others. Or can I?
I've tried several things, but I haven't found anything that works. Are
there any utilities for that?

I found an app that will scan a disk looking for lost partitions, and then
looking for various filesystems within them - and it does recognize ext2/3.
It will then recover files at that point. But it runs on windows (won't work
with wine), and it is taking several days to scan the entire drive (100G).
Also, there could be several hundred files, and trying to identify all of
those (possibly) recovered files would be excrutiating :-( .

As always, thx everyone for your help.

ken

[-- Attachment #2: Type: text/html, Size: 2261 bytes --]

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Malahal Naineni]

Ken Bass [daytooner@gmail.com] wrote:
>    I did try to read anything I could out of the old disk. I do believe there
>    is still some data on it that wasn't trashed. The problem is how to find
>    it and reconstruct it without any of the LVM or ext4 filesystem�
>    information (as I said, I couldn't find any superblocks). Are there any
>    utilities to do something like that?

I don't know much about ext2/3/4 file systems but I imagine debugfs may
help you. I never used it myself though.

Thanks, Malahal.

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Stuart D. Gathman]

On Wed, 7 Jul 2010, Ken Bass wrote:

> > The best course of action would have been 'reading what ever data
> > available' from old disk to new disk. If the disk is absolutely
> > thrashed, then zero'ing the new PV may be better.
> >
> >
> I did try to read anything I could out of the old disk. I do believe there
> is still some data on it that wasn't trashed. The problem is how to find it
> and reconstruct it without any of the LVM or ext4 filesystem  information
> (as I said, I couldn't find any superblocks). Are there any utilities to do
> something like that?

Method 1:
If you can, have *all* the drives connected, the failed drive, and the new
drive.  Restore LVM metadata to include the failed drive again.  Now add the
new drive and use pvmove to migrate data that can be read to the new
drive.  (I've never used pvmove for this purpose - does it handle IO
errors and continue?)

Method 2:
With both new and failed drive connected, use dd_rescue or equivalent to
copy (most/some of) the failed drive to the new drive.  Remove failed
drive and bring up VG.  If UUID was not readable on failed drive, use
pvcreate to restore it on new drive.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Stuart D. Gathman]

On Thu, 8 Jul 2010, Stuart D. Gathman wrote:

> Method 2:
> With both new and failed drive connected, use dd_rescue or equivalent to
> copy (most/some of) the failed drive to the new drive.  Remove failed
> drive and bring up VG.  If UUID was not readable on failed drive, use
> pvcreate to restore it on new drive.

I should mention that an external case-free USB adapter is the best way
to attempt recovery of failed drives.  You can even run the IDE/SATA/Power
cable to a box in the freezer (and the USB adapter can sit on top of the
freezer and a long USB cable go to your computer) - which often helps recover
data from failing drives.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Rebuilding ext4 filesystem on an LV

[Ken Bass]

[-- Attachment #1: Type: text/plain, Size: 2778 bytes --]

On Thu, Jul 8, 2010 at 12:13 PM, Stuart D. Gathman <stuart@bmsi.com> wrote:

> On Thu, 8 Jul 2010, Stuart D. Gathman wrote:
>
> > Method 2:
> > With both new and failed drive connected, use dd_rescue or equivalent to
> > copy (most/some of) the failed drive to the new drive.  Remove failed
> > drive and bring up VG.  If UUID was not readable on failed drive, use
> > pvcreate to restore it on new drive.
>
>
(Question: does new drive need to be same physical size? can Or can I just
create a new partition with the old size - new drive is bigger.)

The real problem seems to be that the LV appears to the LVM as not
formatted. I am guessing this is because not only the partition got trashed,
but also the superblock(s)  data. I have tried to find backup superblocks,
but I keep getting the ioctl error, even from mke2fs. I will try dd_rescue -
I have to install it, since it is not by default.

I added a new drive to the LV, and tried pvmove, but, I got the ioctl error
again with that.

I should mention that an external case-free USB adapter is the best way
> to attempt recovery of failed drives.  You can even run the IDE/SATA/Power
> cable to a box in the freezer (and the USB adapter can sit on top of the
> freezer and a long USB cable go to your computer) - which often helps
> recover
> data from failing drives.
>
>
If the mountain won't come to Mohammed... I can always just get a large bag
of ice and (carefully wrapped up in waterproof cover) put the drive into it.
Will try that also :-).

FWIW: I did run dmsetup _table, and from /var/log/messages:

____________
Jul 12 09:17:25 Elmer kernel: device-mapper: ioctl: error adding target to
table
Jul 12 09:24:16 Elmer kernel: device-mapper: table: device 253:3 too small
for target
____________

Couple of thoughts:
* Using dd_rescue, is there a utility that can read through the physical
blocks on the drive, searching for something that looks like a superblock?

* Or, is it possible to physically disconnect the bad drive, bring up the
remainder of the LV with the hope of recovering whatever data was on those
drives. Att this point, I am resigned to the fact that the first drive is
dead or on its last gasp (SMARTD says something like: "Danger (Will
Robinson!) drive failure imminent. Back up data immediately.)" But the other
drives in the LV should have something on them still. Right?

As always thanks for the help.

             Stuart D. Gathman <stuart@bmsi.com>
>    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?" commercial.
>
>
"All consuming fires of hell" - very loose translation from Mozart's
unfinished Mass for the Dead :-)

ken

[-- Attachment #2: Type: text/html, Size: 3731 bytes --]