[linux-lvm] Restoring data after losing a drive

[linux-lvm] Restoring data after losing a drive

[Saad Shakhshir]

[-- Attachment #1: Type: text/plain, Size: 4106 bytes --]

I lost one of my drives that was a physical volume in my lvm.  The volume
group spanned across all the drives and so did the logical volume.  I know
that the data on the good drives is still intact I just don't know how to
get the lvm up and running again to be able to read the data.

I tried running 'vgreduce --removemissing fileserver' and that managed to
restore my other small logical volume that was located entirely on one of
the (still good) physical volumes.  The other large logical volume is still
not showing up.

I'm going to attach the last good configuration from /etc/lvm/archive  The
drive that died was pv0.

I really hope that someone can help with this.  It's terrible losing so much
data...

# Generated by LVM2: Fri May 11 19:48:00 2007

contents = "Text Format Volume Group"
version = 1

description = "Created *before* executing 'vgreduce --removemissing
fileserver'"

creation_host = "veritas"    # Linux veritas 2.6.20-15-generic #2 SMP Sun
Apr 15 07:36:31 UTC 2007 i686
creation_time = 1178927280    # Fri May 11 19:48:00 2007

fileserver {
    id = "9vePH6-W5lO-S1HD-aQyC-jOCY-R0Re-nIBPcJ"
    seqno = 12
    status = ["RESIZEABLE", "PARTIAL", "READ"]
    extent_size = 8192        # 4 Megabytes
    max_lv = 0
    max_pv = 0

    physical_volumes {

        pv0 {
            id = "V2yZV5-qx19-7c98-5WLY-9lTF-Wb4c-70Q0na"
            device = "unknown device"    # Hint only

            status = ["ALLOCATABLE"]
            pe_start = 384
            pe_count = 76310    # 298.086 Gigabytes
        }

        pv1 {
            id = "MZrncO-HYwT-6YEB-w3hV-2YqP-HZRY-u6OMyf"
            device = "/dev/mapper/hdb1"    # Hint only

            status = ["ALLOCATABLE"]
            pe_start = 384
            pe_count = 59618    # 232.883 Gigabytes
        }

        pv2 {
            id = "52vIhv-lOnI-RDQ3-Z5KT-4Ifc-thL0-uLZQVo"
            device = "/dev/mapper/sdb1"    # Hint only

            status = ["ALLOCATABLE"]
            pe_start = 384
            pe_count = 59618    # 232.883 Gigabytes
        }

        pv3 {
            id = "0q8rC2-9Nxi-YriB-7oa6-0wp4-frBa-GT9aQt"
            device = "/dev/mapper/hda3"    # Hint only

            status = ["ALLOCATABLE"]
            pe_start = 384
            pe_count = 26611    # 103.949 Gigabytes
        }
    }

    logical_volumes {

        data {
            id = "O0WU1C-0f0H-ZZvL-AwIc-vl8x-seFe-J3uE2X"
            status = ["READ", "WRITE", "VISIBLE"]
            segment_count = 4

            segment1 {
                start_extent = 0
                extent_count = 76310    # 298.086 Gigabytes

                type = "striped"
                stripe_count = 1    # linear

                stripes = [
                    "pv0", 0
                ]
            }
            segment2 {
                start_extent = 76310
                extent_count = 59618    # 232.883 Gigabytes

                type = "striped"
                stripe_count = 1    # linear

                stripes = [
                    "pv1", 0
                ]
            }
            segment3 {
                start_extent = 135928
                extent_count = 51937    # 202.879 Gigabytes

                type = "striped"
                stripe_count = 1    # linear

                stripes = [
                    "pv2", 0
                ]
            }
            segment4 {
                start_extent = 187865
                extent_count = 26611    # 103.949 Gigabytes

                type = "striped"
                stripe_count = 1    # linear

                stripes = [
                    "pv3", 0
                ]
            }
        }

        home {
            id = "13Bra6-cE4i-MRgA-CZhC-RC0f-qtEo-R2Y0x2"
            status = ["READ", "WRITE", "VISIBLE"]
            segment_count = 1

            segment1 {
                start_extent = 0
                extent_count = 7681    # 30.0039 Gigabytes

                type = "striped"
                stripe_count = 1    # linear

                stripes = [
                    "pv2", 51937
                ]
            }
        }
    }
}

[-- Attachment #2: Type: text/html, Size: 9615 bytes --]

Re: [linux-lvm] Restoring data after losing a drive

[Ian Kent]

On Fri, 2007-05-11 at 23:29 -0400, Saad Shakhshir wrote:
> I lost one of my drives that was a physical volume in my lvm.  The
> volume group spanned across all the drives and so did the logical
> volume.  I know that the data on the good drives is still intact I
> just don't know how to get the lvm up and running again to be able to
> read the data. 
> 
> I tried running 'vgreduce --removemissing fileserver' and that managed
> to restore my other small logical volume that was located entirely on
> one of the (still good) physical volumes.  The other large logical
> volume is still not showing up. 
> 
> I'm going to attach the last good configuration from /etc/lvm/archive
> The drive that died was pv0.  
> 
> I really hope that someone can help with this.  It's terrible losing
> so much data... 

I don't understand the question.
How will LVM know what was contained in the blocks on the failed disk?

I think you'll need to restore the damaged volume from backup.

Ian

Re: [linux-lvm] Restoring data after losing a drive

[Saad Shakhshir]

[-- Attachment #1: Type: text/plain, Size: 1820 bytes --]

Hi Ian, thanks for your response.  Let me clarify.

The data on the damaged disk is not recoverable at this point.  However
there is data on the other remaining good disks that was part of that one
large logical volume.  At this point I want to get back the remaining data
that was in that logical volume and on the remaining good physical volumes
without the data that was on the physical volume I lost.  I know that the
data is still intact on those drives... I just need to know how to get my
system to recognise that it's still there.

On 5/12/07, Ian Kent <ikent@redhat.com> wrote:
>
> On Fri, 2007-05-11 at 23:29 -0400, Saad Shakhshir wrote:
> > I lost one of my drives that was a physical volume in my lvm.  The
> > volume group spanned across all the drives and so did the logical
> > volume.  I know that the data on the good drives is still intact I
> > just don't know how to get the lvm up and running again to be able to
> > read the data.
> >
> > I tried running 'vgreduce --removemissing fileserver' and that managed
> > to restore my other small logical volume that was located entirely on
> > one of the (still good) physical volumes.  The other large logical
> > volume is still not showing up.
> >
> > I'm going to attach the last good configuration from /etc/lvm/archive
> > The drive that died was pv0.
> >
> > I really hope that someone can help with this.  It's terrible losing
> > so much data...
>
> I don't understand the question.
> How will LVM know what was contained in the blocks on the failed disk?
>
> I think you'll need to restore the damaged volume from backup.
>
> Ian
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2431 bytes --]

Re: [linux-lvm] Restoring data after losing a drive

[Stuart D. Gathman]

On Sat, 12 May 2007, Saad Shakhshir wrote:

> The data on the damaged disk is not recoverable at this point.  However
> there is data on the other remaining good disks that was part of that one
> large logical volume.  At this point I want to get back the remaining data
> that was in that logical volume and on the remaining good physical volumes
> without the data that was on the physical volume I lost.  I know that the
> data is still intact on those drives... I just need to know how to get my
> system to recognise that it's still there.

The solution involves restoring the metadata to memory or a replacement hard
drive from /etc/lvm (if it is intact) or a backup.  I'll let the experts
talk about details.

However, the LVM should be able to handle losing a PV and still bring LVs
for that PV online.  Any attempted IO would result in errors, of course.
But the metadata for a PV should be automatically loadable even with
the PV missing.  When the missing PV blows a hole in your large LV,
it would simplify recovering the pieces if the missing data got I/O errors.
If you replace the drive and restore the metadata, the missing data will
have whatever is on the drive.  It might help recovery to write a pattern
to the replacement drive to help recognize the missing data.

You will also need to be a filesystem wizard to navigate with a huge hole
like that.  If you have one large file with a regular record format, it
might be simplest to scan all blocks for records - then paste any missing

I used to run into such problems a lot, and developed a filesystem where
each block is tagged with the inode of the file it belonged to.  The
recovery program can recover each and every readable block into the
proper file in the proper location regardless of how much of the filesystem
is missing or garbled due to horrendous errors.  There is no inode table
either - any block can be an inode (so blowing away the first part of the
filesystem doesn't lose all your files).  There are drawbacks to
this approach, of course.  E.g. block size is reduced by the header
present on every block (and is therefore not a power of 2).

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Restoring data after losing a drive

[Saad Shakhshir]

[-- Attachment #1: Type: text/plain, Size: 3477 bytes --]

Right now after running 'vgreduce --removemissing' it resized the volume
group to include only the available drives.  I got back my one small logical
volume (/dev/fileserver/home) and the data is all intact.  The other large
logical volume which spanned the rest of the drives (including the one that
went bad) isn't there anymore.  So it thinks that the majority of the volume
group is unused space, however there is data on there.  If I now create a
logical volume on the remaining space in the volume group, will it
automatically see the data that is there?

Where does the filesystem information get stored in terms of inode
locations?  Will that get overwritten now if I create a new logical volume?

On 5/12/07, Stuart D. Gathman <stuart@bmsi.com> wrote:
>
> On Sat, 12 May 2007, Saad Shakhshir wrote:
>
> > The data on the damaged disk is not recoverable at this point.  However
> > there is data on the other remaining good disks that was part of that
> one
> > large logical volume.  At this point I want to get back the remaining
> data
> > that was in that logical volume and on the remaining good physical
> volumes
> > without the data that was on the physical volume I lost.  I know that
> the
> > data is still intact on those drives... I just need to know how to get
> my
> > system to recognise that it's still there.
>
> The solution involves restoring the metadata to memory or a replacement
> hard
> drive from /etc/lvm (if it is intact) or a backup.  I'll let the experts
> talk about details.
>
> However, the LVM should be able to handle losing a PV and still bring LVs
> for that PV online.  Any attempted IO would result in errors, of course.
> But the metadata for a PV should be automatically loadable even with
> the PV missing.  When the missing PV blows a hole in your large LV,
> it would simplify recovering the pieces if the missing data got I/O
> errors.
> If you replace the drive and restore the metadata, the missing data will
> have whatever is on the drive.  It might help recovery to write a pattern
> to the replacement drive to help recognize the missing data.
>
> You will also need to be a filesystem wizard to navigate with a huge hole
> like that.  If you have one large file with a regular record format, it
> might be simplest to scan all blocks for records - then paste any missing
>
> I used to run into such problems a lot, and developed a filesystem where
> each block is tagged with the inode of the file it belonged to.  The
> recovery program can recover each and every readable block into the
> proper file in the proper location regardless of how much of the
> filesystem
> is missing or garbled due to horrendous errors.  There is no inode table
> either - any block can be an inode (so blowing away the first part of the
> filesystem doesn't lose all your files).  There are drawbacks to
> this approach, of course.  E.g. block size is reduced by the header
> present on every block (and is therefore not a power of 2).
>
> --
>               Stuart D. Gathman <stuart@bmsi.com>
>     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703
> 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?" commercial.
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 4308 bytes --]

Re: [linux-lvm] Restoring data after losing a drive

[Saad Shakhshir]

[-- Attachment #1: Type: text/plain, Size: 3783 bytes --]

Does anyone have ideas on this?  I would really appreciate the help.

On 5/13/07, Saad Shakhshir <saads@alum.mit.edu> wrote:
>
> Right now after running 'vgreduce --removemissing' it resized the volume
> group to include only the available drives.  I got back my one small logical
> volume (/dev/fileserver/home) and the data is all intact.  The other large
> logical volume which spanned the rest of the drives (including the one that
> went bad) isn't there anymore.  So it thinks that the majority of the volume
> group is unused space, however there is data on there.  If I now create a
> logical volume on the remaining space in the volume group, will it
> automatically see the data that is there?
>
> Where does the filesystem information get stored in terms of inode
> locations?  Will that get overwritten now if I create a new logical volume?
>
> On 5/12/07, Stuart D. Gathman <stuart@bmsi.com> wrote:
> >
> > On Sat, 12 May 2007, Saad Shakhshir wrote:
> >
> > > The data on the damaged disk is not recoverable at this
> > point.  However
> > > there is data on the other remaining good disks that was part of that
> > one
> > > large logical volume.  At this point I want to get back the remaining
> > data
> > > that was in that logical volume and on the remaining good physical
> > volumes
> > > without the data that was on the physical volume I lost.  I know that
> > the
> > > data is still intact on those drives... I just need to know how to get
> > my
> > > system to recognise that it's still there.
> >
> > The solution involves restoring the metadata to memory or a replacement
> > hard
> > drive from /etc/lvm (if it is intact) or a backup.  I'll let the experts
> > talk about details.
> >
> > However, the LVM should be able to handle losing a PV and still bring
> > LVs
> > for that PV online.  Any attempted IO would result in errors, of course.
> > But the metadata for a PV should be automatically loadable even with
> > the PV missing.  When the missing PV blows a hole in your large LV,
> > it would simplify recovering the pieces if the missing data got I/O
> > errors.
> > If you replace the drive and restore the metadata, the missing data will
> >
> > have whatever is on the drive.  It might help recovery to write a
> > pattern
> > to the replacement drive to help recognize the missing data.
> >
> > You will also need to be a filesystem wizard to navigate with a huge
> > hole
> > like that.  If you have one large file with a regular record format, it
> > might be simplest to scan all blocks for records - then paste any
> > missing
> >
> > I used to run into such problems a lot, and developed a filesystem where
> >
> > each block is tagged with the inode of the file it belonged to.  The
> > recovery program can recover each and every readable block into the
> > proper file in the proper location regardless of how much of the
> > filesystem
> > is missing or garbled due to horrendous errors.  There is no inode table
> > either - any block can be an inode (so blowing away the first part of
> > the
> > filesystem doesn't lose all your files).  There are drawbacks to
> > this approach, of course.  E.g. block size is reduced by the header
> > present on every block (and is therefore not a power of 2).
> >
> > --
> >               Stuart D. Gathman <stuart@bmsi.com >
> >     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703
> > 591-6154
> > "Confutatis maledictis, flammis acribus addictis" - background song for
> > a Microsoft sponsored "Where do you want to go from here?" commercial.
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >
>
>

[-- Attachment #2: Type: text/html, Size: 5133 bytes --]