[linux-lvm] accidently formatted a physical volume...

[linux-lvm] accidently formatted a physical volume...

[Adam McDaniel]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

Hi.. I've had a simple implementation of LVM running on two 10gig
harddrives.. Everything was working fine until one day a co-worker
decided he wanted to run `mkfs.ext2 /dev/hdb`

Needless to say, the VG was screwed and I couldn't get any data out.
My question is that I'm wondering if there is any way I can salvage
the other PV and alteast get SOME data out.

Or, is there any way that I can fix/restore the formatted drive?

For some background info, I had one VG, two PVs, and one LV with ext2
running on it. I *believe* I had it configured to concatinate the
data, but im not 100% if it was setup in stripe mode. I'm running LVM
1.0.0-3 (debian/testing)

Sorry if this question has been asked before, I tried searching
through the mailing-list/howto for an answer but found nothing
useful.

Thanks in advance :)

-- 
Adam McDaniel
Array Networks
Calgary, AB, Canada

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] accidently formatted a physical volume...

[Steven Lembark]

> Or, is there any way that I can fix/restore the formatted drive?

Not at the *NIX level. If the data were SERIOUSLY valuable
it can be recovered by physical means. Short of that I'd
say it's time to reach for your tapes...


--
Steven Lembark                               2930 W. Palmer
Workhorse Computing                       Chicago, IL 60647
                                            +1 800 762 1582

Re: [linux-lvm] accidently formatted a physical volume...

[Anders Widman]

>> Or, is there any way that I can fix/restore the formatted drive?

> Not at the *NIX level. If the data were SERIOUSLY valuable
> it can be recovered by physical means. Short of that I'd
> say it's time to reach for your tapes...

Are you sure? The data should be there in the co-worker did not make a
"full" (zeroing blocks) format. Is it not possible to restore the LVM
headers (or what it is called) on that drive?

- Anders

> --
> Steven Lembark                               2930 W. Palmer
> Workhorse Computing                       Chicago, IL 60647
>                                             +1 800 762 1582

> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

Re: [linux-lvm] accidently formatted a physical volume...

[Adam McDaniel]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

On Mon, Aug 19, 2002 at 10:41:24PM +0200, Anders Widman wrote:
> 
> >> Or, is there any way that I can fix/restore the formatted drive?
> 
> > Not at the *NIX level. If the data were SERIOUSLY valuable
> > it can be recovered by physical means. Short of that I'd
> > say it's time to reach for your tapes...

The unfortunate thing was that timing couldn't have been worse. The
last backup I made before the incident would only contain a fraction
of the info that I actually need. Partly my fault, I know, but being
able to restore the drive would be the goal.

> Are you sure? The data should be there in the co-worker did not make a
> "full" (zeroing blocks) format. Is it not possible to restore the LVM
> headers (or what it is called) on that drive?

I do know that it was a straight simple format, literally
'mkfs.ext2 /dev/hdb', so I know the info is still there. My experience
in this is rather limited, so I'm not sure of the best way to fish it
out.

-- 
Adam McDaniel
Array Networks
Calgary, AB, Canada

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] accidently formatted a physical volume...

[Kirby C. Bohling]

Start by using dd or something to get an original image of the drive 
before anything else bad is done to it.  Copy it off to another drive, 
copy to tape.  Personally, I'd copy it to a big filesystem (if it 
supports bigger then 2 GB), and see if you can mount it loopback.  I 
know I read on the LKML a guy mounted several filesytems that way that 
were considerablely larger then 2GB.

With the backup, when something goes wrong you can always start over 
fresh without worring about the undoing your original attempts.

I'd imagine that doing a vgcfgrestore to the device, followed by 
mounting the filesystem without fsck'ing it will get you some data. 
Running fsck on it afterwards and see what else you can get.  Do *_NOT_* 
try that without a backup of the original media.  Your big problem is 
that you're going to have superblocks and inode tables written right in 
the middle of your drive other then that, everything should be on the 
disk right where you left it.

I had a friend who just lost an ext2 drive due to a power surge the 
other day, fscking it was destorying the filesystem, just mounting it 
and running tar got us 90% of the data.  Fortuantely the other 10% came 
off a RH CD so it was no problem.

	Thanks,
		Kirby

Adam McDaniel wrote:
> On Mon, Aug 19, 2002 at 10:41:24PM +0200, Anders Widman wrote:
> 
>>>>Or, is there any way that I can fix/restore the formatted drive?
>>>
>>>Not at the *NIX level. If the data were SERIOUSLY valuable
>>>it can be recovered by physical means. Short of that I'd
>>>say it's time to reach for your tapes...
>>
> 
> The unfortunate thing was that timing couldn't have been worse. The
> last backup I made before the incident would only contain a fraction
> of the info that I actually need. Partly my fault, I know, but being
> able to restore the drive would be the goal.
> 
> 
>>Are you sure? The data should be there in the co-worker did not make a
>>"full" (zeroing blocks) format. Is it not possible to restore the LVM
>>headers (or what it is called) on that drive?
> 
> 
> I do know that it was a straight simple format, literally
> 'mkfs.ext2 /dev/hdb', so I know the info is still there. My experience
> in this is rather limited, so I'm not sure of the best way to fish it
> out.
> 

Re: [linux-lvm] accidently formatted a physical volume...

[Adam McDaniel]

[-- Attachment #1: Type: text/plain, Size: 2732 bytes --]

On Mon, Aug 19, 2002 at 03:49:46PM -0500, Kirby C. Bohling wrote:
> I'd imagine that doing a vgcfgrestore to the device, followed by 
> mounting the filesystem without fsck'ing it will get you some data. 

Unfortunatly, I don't have any data for vgcfgrestore to build upon.
Here's the current status:

/dev/loop0 is pointing to /root/lvm/hde1
/dev/loop1 is pointing to /root/lvm/hdf1
(built via 'dd if=/dev/hd?1 of=/root/lvm/hd?1')

/dev/loop0 is the good image
/dev/loop1 is the fubar'd one.

pvdisplay /dev/loop0:

---
PV Name               /dev/loop0
VG Name               data
PV Size               14.31 GB [30009357 secs] / NOT usable 1.01 MB [LVM: 135 KB]
PV#                   2
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       4096
Total PE              3663
Free PE               0
Allocated PE          3663
PV UUID               3IRYim-Lpgb-2N1Y-6TPu-C8mZ-58hy-exlmBm
---

pvdisplay /dev/loop1:

---
PV Name               /dev/loop1
VG Name               data
PV Size               9.44 GB [19792017 secs] / NOT usable 4.07 MB [LVM: 130 KB]
PV#                   1
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       4096
Total PE              2415
Free PE               0
Allocated PE          2415
PV UUID               P7xAxa-J36q-2C8i-2vKc-5KJO-2tQ6-H3eGiz
---

First I tried running pvscan:

---
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- inactive PV "/dev/loop0"  is associated to unknown VG "data" (run vgscan)
pvscan -- inactive PV "/dev/loop1"  is associated to unknown VG "data" (run vgscan)
pvscan -- total: 2 [23.75 GB] / in use: 2 [23.75 GB] / in no VG: 0 [0]
---

fair enough.. so I run vgscan -v:

---
vgscan -- removing "/etc/lvmtab" and "/etc/lvmtab.d"
vgscan -- creating empty "/etc/lvmtab" and "/etc/lvmtab.d"
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- scanning for all active volume group(s) first
vgscan -- reading data of volume group "data" from physical volume(s)
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of volume group "data" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume group
---

ofcourse /etc/lvmtab and /etc/lvmtab.d/* are all blank/empty.

If anything, I'm just about at the point where the dead drive is a
writeoff (/dev/loop1).. but I would like to try to salvage the data on
/dev/loop0

Any ideas?

Thanks.

-- 
Adam McDaniel
Array Networks
Calgary, AB, Canada

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] accidently formatted a physical volume...

[Kirby C. Bohling]

	Okay, now I'm completely out of my league, and I'm making this up as I 
go.  The only way I'd now to salvage anything is to try and figure out 
the layout of the LVM and the physical extents, read them off into a big 
flat file and mount it loop on the loop back as extended two.

	Is there anyways to get pvscan to be incredibly verbose and list the 
starting offset, and ending offset on the physical disk for each 
physical extent?  Essentially, write fsck.lvm/debug_lvm the parameters 
would be the lv you want to salvage, and the target location you want to 
write it to.  The big trick will be identifying corrupt LVM metadata, 
and avoiding those blocks like the plauge.

	In the end, LVM is an abstraction for a block device.  So I'd create a 
file and connect a loopback that was the same size as the LV.  So pull a 
physical extent, figure out if it is associated with the lv you're 
recovering.  Take the lv extent offset it is, and seek the extent size * 
logical extent number and write it to the loopback device.  Throwing 
away the LVM metadata.  Now, you can take this big pile of bits to an 
ext2/3 guru, and ask them how to glean as much data as possible out of 
it.  If the LV was completely on one disk, it should just fsck and 
mount.  Assuming you didn't specify the sizes, and do a lot of resizing 
of the LV's that should be the case.  Possibly asking the ext2/3 guru 
what the holes should be filled in with, and put that there. I'd guess 
zero's.  This trick should work on both sets of disks, but it'll be 
easier to pull off with known good meta-data.

	It'd be a big pile of work, but if the data is worth it you might try it. 
  It's entirely possible this program exists and I'm unaware of it, but 
if it doesn't, somebody should take the time to write it for the current 
code.  I don't know LVM and the metadata, and I'm too busy to figure it 
out... Sorry.  Just think if you do it, you'll be famous...

	Thanks,
		Kirby





Adam McDaniel wrote:
> On Mon, Aug 19, 2002 at 03:49:46PM -0500, Kirby C. Bohling wrote:
> 
>>I'd imagine that doing a vgcfgrestore to the device, followed by 
>>mounting the filesystem without fsck'ing it will get you some data. 
> 
> 
> Unfortunatly, I don't have any data for vgcfgrestore to build upon.
> Here's the current status:
> 
> /dev/loop0 is pointing to /root/lvm/hde1
> /dev/loop1 is pointing to /root/lvm/hdf1
> (built via 'dd if=/dev/hd?1 of=/root/lvm/hd?1')
> 
> /dev/loop0 is the good image
> /dev/loop1 is the fubar'd one.
> 
> pvdisplay /dev/loop0:
> 
> ---
> PV Name               /dev/loop0
> VG Name               data
> PV Size               14.31 GB [30009357 secs] / NOT usable 1.01 MB [LVM: 135 KB]
> PV#                   2
> PV Status             available
> Allocatable           yes (but full)
> Cur LV                1
> PE Size (KByte)       4096
> Total PE              3663
> Free PE               0
> Allocated PE          3663
> PV UUID               3IRYim-Lpgb-2N1Y-6TPu-C8mZ-58hy-exlmBm
> ---
> 
> pvdisplay /dev/loop1:
> 
> ---
> PV Name               /dev/loop1
> VG Name               data
> PV Size               9.44 GB [19792017 secs] / NOT usable 4.07 MB [LVM: 130 KB]
> PV#                   1
> PV Status             available
> Allocatable           yes (but full)
> Cur LV                1
> PE Size (KByte)       4096
> Total PE              2415
> Free PE               0
> Allocated PE          2415
> PV UUID               P7xAxa-J36q-2C8i-2vKc-5KJO-2tQ6-H3eGiz
> ---
> 
> First I tried running pvscan:
> 
> ---
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- inactive PV "/dev/loop0"  is associated to unknown VG "data" (run vgscan)
> pvscan -- inactive PV "/dev/loop1"  is associated to unknown VG "data" (run vgscan)
> pvscan -- total: 2 [23.75 GB] / in use: 2 [23.75 GB] / in no VG: 0 [0]
> ---
> 
> fair enough.. so I run vgscan -v:
> 
> ---
> vgscan -- removing "/etc/lvmtab" and "/etc/lvmtab.d"
> vgscan -- creating empty "/etc/lvmtab" and "/etc/lvmtab.d"
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- scanning for all active volume group(s) first
> vgscan -- reading data of volume group "data" from physical volume(s)
> vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of volume group "data" from physical volume(s)
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume group
> ---
> 
> ofcourse /etc/lvmtab and /etc/lvmtab.d/* are all blank/empty.
> 
> If anything, I'm just about at the point where the dead drive is a
> writeoff (/dev/loop1).. but I would like to try to salvage the data on
> /dev/loop0
> 
> Any ideas?
> 
> Thanks.
> 

Re: [linux-lvm] accidently formatted a physical volume...

[Andreas Dilger]

On Aug 20, 2002  13:20 -0500, Kirby C. Bohling wrote:
> In the end, LVM is an abstraction for a block device.  So I'd create 
> a file and connect a loopback that was the same size as the LV.  So pull a 
> physical extent, figure out if it is associated with the lv you're 
> recovering.  Take the lv extent offset it is, and seek the extent size * 
> logical extent number and write it to the loopback device.  Throwing 
> away the LVM metadata.  Now, you can take this big pile of bits to an 
> ext2/3 guru, and ask them how to glean as much data as possible out of 
> it.  If the LV was completely on one disk, it should just fsck and 
> mount.  Assuming you didn't specify the sizes, and do a lot of resizing 
> of the LV's that should be the case.  Possibly asking the ext2/3 guru 
> what the holes should be filled in with, and put that there. I'd guess 
> zero's.  This trick should work on both sets of disks, but it'll be 
> easier to pull off with known good meta-data.

Run something like "gpart" or build the findsuper program in
e2fsprogs/misc, which should give you a pretty good idea of where things
are put on the disk.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/