[linux-lvm] I broke my PVs!

[linux-lvm] I broke my PVs!

[Austin Gonyou]

I have 3 disks, one 9GB and two 18GB scsi disks. I just re-installed my
system with RH 8.0, and it didn't like that the two 18's weren't
initialized, I guess I screwed up, cause I didn't think I was
initializing them(since I told the installer not to), and instead ran
fdisk and then saved without making partitions. 

I can see my data still on the disks if I do strings /dev/sdb or
dev/sdc. I have another system that also has 3 scsi disks using two of
them with LVM. 

I was curious, if I could take say the first 512 bytes, or whatever
space is required to write the pv information on a disk, from those
disks, and then write them to the first 512 bytes, or whatever, to the
disks on my system, then do pv scan and see if they get activated, or
found to be in a vg that the other disks were in.

I've got about 30GB of data on this VG, and I am really hoping I don't
have to lose it. A lot of email there, etc. Please advise. 

Austin

Re: [linux-lvm] I broke my PVs!

[Luca Berra]

On Sat, Oct 19, 2002 at 11:14:02PM -0500, Austin Gonyou wrote:
>I have 3 disks, one 9GB and two 18GB scsi disks. I just re-installed my
>system with RH 8.0, and it didn't like that the two 18's weren't
>initialized, I guess I screwed up, cause I didn't think I was
>initializing them(since I told the installer not to), and instead ran
>fdisk and then saved without making partitions. 

what do you mean with initialized?
mke2fs? in this case you will find lovely ext2 superblock copies in the
middle of your data, but then you could salvage most of your data.

lost partition table? just recreate it, it is not destructive?

lost lvm metadata?
i s'pose you don't have a metadata backup handy,
how were your lv's laid out in your vg?

i never tried this on linux, but on hp-ux pvcreate, vgcreate and lvcreate weren't
destructive, it only touched disk metadata. so you could try pvcreate,
vgcreate and lvcreate.
DO A BACKUP OF YOUR RAW DISK DATA FIRST

L.
-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: [linux-lvm] I broke my PVs!

[Austin Gonyou]

On Sun, 2002-10-20 at 05:49, Luca Berra wrote:
> On Sat, Oct 19, 2002 at 11:14:02PM -0500, Austin Gonyou wrote:
> >I have 3 disks, one 9GB and two 18GB scsi disks. I just re-installed my
> >system with RH 8.0, and it didn't like that the two 18's weren't
> >initialized, I guess I screwed up, cause I didn't think I was
> >initializing them(since I told the installer not to), and instead ran
> >fdisk and then saved without making partitions. 


> what do you mean with initialized?
> mke2fs? in this case you will find lovely ext2 superblock copies in the
> middle of your data, but then you could salvage most of your data.

I ran fdisk, then just saved. So I guess you could say I just lost the
partition table.
 
> lost partition table? just recreate it, it is not destructive?

I didn't have any partitions just whole disks. 

> lost lvm metadata?
> i s'pose you don't have a metadata backup handy,
> how were your lv's laid out in your vg?

one LV, one VG, and two PVs in the VG, striped. -i2 -I64
 
> i never tried this on linux, but on hp-ux pvcreate, vgcreate and
> lvcreate weren't
> destructive, it only touched disk metadata. so you could try pvcreate,
> vgcreate and lvcreate.
> DO A BACKUP OF YOUR RAW DISK DATA FIRST

what about vgcfgrestore from another set of smaller disks that are laid
out the same way? could that work?

Wish I could, but I can't. Don't have enough disk space. :(

> L.
> -- 
> Luca Berra -- bluca@comedia.it
>         Communication Media & Services S.r.l.
>  /"\
>  \ /     ASCII RIBBON CAMPAIGN
>   X        AGAINST HTML MAIL
>  / \
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] I broke my PVs!

[Luca Berra]

On Sun, Oct 20, 2002 at 03:16:34PM -0500, Austin Gonyou wrote:
>I ran fdisk, then just saved. So I guess you could say I just lost the
>partition table.

ok, you lost lvm metadata...

>> destructive, it only touched disk metadata. so you could try pvcreate,
>> vgcreate and lvcreate.
>> DO A BACKUP OF YOUR RAW DISK DATA FIRST
>
>what about vgcfgrestore from another set of smaller disks that are laid
>out the same way? could that work?
>
>Wish I could, but I can't. Don't have enough disk space. :(

i believe it should be the same...

Heinz? someone?

l.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: [linux-lvm] I broke my PVs!

[Austin Gonyou]

On Sun, 2002-10-20 at 16:34, Luca Berra wrote:
> On Sun, Oct 20, 2002 at 03:16:34PM -0500, Austin Gonyou wrote:
> >I ran fdisk, then just saved. So I guess you could say I just lost the
> >partition table.
> 
> ok, you lost lvm metadata...

If I lost metadata, what is the best procudure to put it back? I
thought, on linux anyway, that pvcreate was descructive. Can someone
tell me if I'm wrong there?

 > >> destructive, it only touched disk metadata. so you could try
> pvcreate,
> >> vgcreate and lvcreate.
> >> DO A BACKUP OF YOUR RAW DISK DATA FIRST
> >
> >what about vgcfgrestore from another set of smaller disks that are laid
> >out the same way? could that work?
> >
> >Wish I could, but I can't. Don't have enough disk space. :(
> 
> i believe it should be the same...

If anyone can give me a definitive answer, I'd truly appreciate it!

> Heinz? someone?
> 
> l.
> 
> -- 
> Luca Berra -- bluca@comedia.it
>         Communication Media & Services S.r.l.
>  /"\
>  \ /     ASCII RIBBON CAMPAIGN
>   X        AGAINST HTML MAIL
>  / \
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] I broke my PVs!

[Heinz J . Mauelshagen]

On Sun, Oct 20, 2002 at 03:16:34PM -0500, Austin Gonyou wrote:
> On Sun, 2002-10-20 at 05:49, Luca Berra wrote:
> > On Sat, Oct 19, 2002 at 11:14:02PM -0500, Austin Gonyou wrote:
> > >I have 3 disks, one 9GB and two 18GB scsi disks. I just re-installed my
> > >system with RH 8.0, and it didn't like that the two 18's weren't
> > >initialized, I guess I screwed up, cause I didn't think I was
> > >initializing them(since I told the installer not to), and instead ran
> > >fdisk and then saved without making partitions. 
> 
> 
> > what do you mean with initialized?
> > mke2fs? in this case you will find lovely ext2 superblock copies in the
> > middle of your data, but then you could salvage most of your data.
> 
> I ran fdisk, then just saved. So I guess you could say I just lost the
> partition table.
>  
> > lost partition table? just recreate it, it is not destructive?
> 
> I didn't have any partitions just whole disks. 

Austin,
so your 2 18Gb drives have been in one VG as whole disk PVs
and you wrote a partition table onto those. which toasted the PV
structures on both forcing vgscan to fail finding the VG.

Before you write anything else onto the disks, you should save the rest
of your LVM metadata for later help.
Let's assume your PVs are /dev/sdb and /dev/sdc and your VG name
is "vg00", then you want to run:

# dd if=/dev/sdb of=sdb.vgda bs=1k count=4k
# dd if=/dev/sdc of=sdc.vgda bs=1k count=4k
# tar jcf vg00.vgda.tar.bz2 sd[bc].vgda
# rm sd[bc].vgda

We need vg00.vgda.tar.bz2 if it comes to option 3 below :)


The 3 options are:

- follow Luca's advice to recreate everything with the _very_ same parameters
  (pvcreate, vgcreate, lvcreate -Zn). LVM doesn't destroy the data then;
  this presumes that you just ran those commands giving you a simple layout
  in the first place.
  YOU NEED TO RUN "lvcreate -Zn ..." IN ORDER TO AVOID ZEROING THE FIRST
  BLOCK OF THE LV DATA!!!

- vgcfgrestore your metadata if you still have a backup:
  
  # dd if=/dev/zero of=/dev/sdb count=1 bs=512
  # dd if=/dev/zero of=/dev/sdc count=1 bs=512
  # pvcreate -ff /dev/sd[bc]
  # vgcfgrestore -n vg00 -f /etc/lvmconf/vg00.conf /dev/sdb
  # vgcfgrestore -n vg00 -f /etc/lvmconf/vg00.conf /dev/sdc
  # vgscan
  # vgchange -ay

- send vg00.tar.bz2 to me for a hack to get your PV structures back;
  you can restore the hacked metadata to the devices using dd


If your layout is _that_ simple and you have the exact parameters at
hand (PE size, stripe size), you want to prefer option one above.
Again: run "lvcreate -Zn ..." because otherwise your FS will be in trouble.

Regards,
Heinz    -- The LVM Guy --


> 
> > lost lvm metadata?
> > i s'pose you don't have a metadata backup handy,
> > how were your lv's laid out in your vg?
> 
> one LV, one VG, and two PVs in the VG, striped. -i2 -I64
>  
> > i never tried this on linux, but on hp-ux pvcreate, vgcreate and
> > lvcreate weren't
> > destructive, it only touched disk metadata. so you could try pvcreate,
> > vgcreate and lvcreate.
> > DO A BACKUP OF YOUR RAW DISK DATA FIRST
> 
> what about vgcfgrestore from another set of smaller disks that are laid
> out the same way? could that work?
> 
> Wish I could, but I can't. Don't have enough disk space. :(
> 
> > L.
> > -- 
> > Luca Berra -- bluca@comedia.it
> >         Communication Media & Services S.r.l.
> >  /"\
> >  \ /     ASCII RIBBON CAMPAIGN
> >   X        AGAINST HTML MAIL
> >  / \
> > 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] I broke my PVs!

[Austin Gonyou]

You're On Mon, 2002-10-21 at 04:52, Heinz J . Mauelshagen wrote:
> On Sun, Oct 20, 2002 at 03:16:34PM -0500, Austin Gonyou wrote:
> > On Sun, 2002-10-20 at 05:49, Luca Berra wrote:
> > > On Sat, Oct 19, 2002 at 11:14:02PM -0500, Austin Gonyou wrote:
> > > >I have 3 disks, one 9GB and two 18GB scsi disks. I just
> re-installed my
> > > >system with RH 8.0, and it didn't like that the two 18's weren't
> > > >initialized, I guess I screwed up, cause I didn't think I was
> > > >initializing them(since I told the installer not to), and instead
> ran
> > > >fdisk and then saved without making partitions. 
> > 
> > 
> > > what do you mean with initialized?
> > > mke2fs? in this case you will find lovely ext2 superblock copies in
> the
> > > middle of your data, but then you could salvage most of your data.
> > 
> > I ran fdisk, then just saved. So I guess you could say I just lost the
> > partition table.
> >  
> > > lost partition table? just recreate it, it is not destructive?
> > 
> > I didn't have any partitions just whole disks. 
> 
> Austin,
> so your 2 18Gb drives have been in one VG as whole disk PVs
> and you wrote a partition table onto those. which toasted the PV
> structures on both forcing vgscan to fail finding the VG.
> 
> Before you write anything else onto the disks, you should save the rest
> of your LVM metadata for later help.
> Let's assume your PVs are /dev/sdb and /dev/sdc and your VG name
> is "vg00", then you want to run:
>
Gahh...:( I was talking to pjc, and I think I've got it working, but
wont' know until after I create a new kernel and get xfs utilities.

> # dd if=/dev/sdb of=sdb.vgda bs=1k count=4k
> # dd if=/dev/sdc of=sdc.vgda bs=1k count=4k
> # tar jcf vg00.vgda.tar.bz2 sd[bc].vgda
> # rm sd[bc].vgda
> 
> We need vg00.vgda.tar.bz2 if it comes to option 3 below :)
> 

We ended up making some loopback devices, using losetup on them,
pvcreating the loops with --size, vgcreating with them, then lvcreating
using that vg. 

vgchange -an on that vg, then losetup -d on the used loops, then
vgcfgrestore from that config for each pv, tweaking as needed based on
size. 


> The 3 options are:
> 
> - follow Luca's advice to recreate everything with the _very_ same
> parameters
>   (pvcreate, vgcreate, lvcreate -Zn). LVM doesn't destroy the data then;
>   this presumes that you just ran those commands giving you a simple
> layout
>   in the first place.
>   YOU NEED TO RUN "lvcreate -Zn ..." IN ORDER TO AVOID ZEROING THE FIRST
>   BLOCK OF THE LV DATA!!!
> 
> - vgcfgrestore your metadata if you still have a backup:

Yeah..I blew it away, before getting the backup, which happened on a
stupid hunch in the first place.

> - send vg00.tar.bz2 to me for a hack to get your PV structures back;
>   you can restore the hacked metadata to the devices using dd

:( pvcreate already run on the disks. :(

 
> 
> If your layout is _that_ simple and you have the exact parameters at
> hand (PE size, stripe size), you want to prefer option one above.
> Again: run "lvcreate -Zn ..." because otherwise your FS will be in
> trouble.

Understood. It is that simple. I will see what will happen when I get an
xfs enabled kernel and updated utilities. 

I will let you know what happens, but thanks for all the help. :)

--
Austin Gonyou
Coremetrics, Inc.