[linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[Sebastian Beneke]

Hi there,

i've got a really big problem. My VG "storage" consists of two PVs:
/dev/hdc (120GB full) and /dev/hde (160GB, 40 GB free). Now /dev/hde
was destroyed by"dd if=/dev/zero of=/dev/diskname bs=1k count=1"

I did these steps until now:

1.) copy back my old files: /etc/lvmconf/ /etc/lvmtab
/etc/lvmtab.d/

2.) take a new hdd (same type and size as my old /dev/hde)

3.) pvcreate /dev/hde

4.) vgcfgrestore -f /etc/lvmconf/storage.conf -n storage /dev/hde

===> This gave me following error: vgcfgrestore -- ERROR
"vg_cfgrestore(): pv_check_consistency" restoring volume group
"storage"

So just replacing the old disk doesn't work for me. I gave it up. But
there are still 120GB of data on /dev/hdc, is there any way I can
rescue this data? It hurts enough to lose 100GB, but I couldn't take
it to lose another 120GB!

I hope anybody can help me out here!


Best Regards
Sebastian

Re: [linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[Heinz J . Mauelshagen]

Sebastian,

you either want to use LVM2/device-mapper and try the "vgchange -P ..."
option to activate partially available VGs

-or-

a quick hack to get LVM1 going:

mkdir /etc/lvmtab.d
cp /etc/lvmconf/storage.conf /etc/lvmtab.d/storage
echo -ne "storage\0" > /etc/lvmtab
vgchange -ay storage

If you have those contents in /etc/lvmtab and /etc/lvmtab.d/storage restored
anyway, "vgchange -ay storage" is all you need to do.

Either way presumes, that enough logical extends are accessable fine.
Backup your data and recreate the VG afterwards.

Regards,
Heinz    -- The LVM Guy --


On Sat, Nov 01, 2003 at 02:25:38PM +0100, Sebastian Beneke wrote:
> Hi there,
> 
> i've got a really big problem. My VG "storage" consists of two PVs:
> /dev/hdc (120GB full) and /dev/hde (160GB, 40 GB free). Now /dev/hde
> was destroyed by"dd if=/dev/zero of=/dev/diskname bs=1k count=1"
> 
> I did these steps until now:
> 
> 1.) copy back my old files: /etc/lvmconf/ /etc/lvmtab
> /etc/lvmtab.d/
> 
> 2.) take a new hdd (same type and size as my old /dev/hde)
> 
> 3.) pvcreate /dev/hde
> 
> 4.) vgcfgrestore -f /etc/lvmconf/storage.conf -n storage /dev/hde
> 
> ===> This gave me following error: vgcfgrestore -- ERROR
> "vg_cfgrestore(): pv_check_consistency" restoring volume group
> "storage"
> 
> So just replacing the old disk doesn't work for me. I gave it up. But
> there are still 120GB of data on /dev/hdc, is there any way I can
> rescue this data? It hurts enough to lose 100GB, but I couldn't take
> it to lose another 120GB!
> 
> I hope anybody can help me out here!
> 
> 
> Best Regards
> Sebastian
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[Holger Parplies]

> On Sat, Nov 01, 2003 at 02:25:38PM +0100, Sebastian Beneke wrote:
> > Now /dev/hde
> > was destroyed by"dd if=/dev/zero of=/dev/diskname bs=1k count=1"

Do I understand correctly that only the first 1K of the disk was
overwritten? Of the PV, of the disk (/dev/hde), of something else ... ?

If it's the first 1K of /dev/hde, you've probably only lost the
partition table, which you could re-create with fdisk, presuming you
remember exactly how your disk was partitioned (i.e. where your PV was).
Remember to set the type of the partition correctly (8e - Linux LVM).

I'm not sure what the beginning of a PV contains and whether there's a
simple way to restore that ...

Hope that helps,

Holger

Re: [linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[Sebastian Beneke]

On Wed, 5 Nov 2003 15:16:44 +0100
Holger Parplies <wopp@planungsteam-eb.de> wrote:

> > On Sat, Nov 01, 2003 at 02:25:38PM +0100, Sebastian Beneke wrote:
> > > Now /dev/hde
> > > was destroyed by"dd if=/dev/zero of=/dev/diskname bs=1k count=1"
> 
> Do I understand correctly that only the first 1K of the disk was
> overwritten? Of the PV, of the disk (/dev/hde), of something else
> ... ?
> 
> If it's the first 1K of /dev/hde, you've probably only lost the
> partition table, which you could re-create with fdisk, presuming you
> remember exactly how your disk was partitioned (i.e. where your PV
> was). Remember to set the type of the partition correctly (8e -
> Linux LVM).

Yes, absolutely. I thought about it. Both harddisks are ok. Only the
PT of the second disk must be destroyed. The best: there weren't any
partitions on it, I just createt a LVM-PV and then created an ext3
filesystem. I think all the data should be there, but I don't know how
to repair it. On the second disk I created a new VG called "Backup"
(thinking it was another disk). What I have to do is: Get the old PT
with the old LVM informations on it back. But how? I am an absolute
LVM-Newbie. This information should not be more than some bytes,
right?

Anyone any ideas here?

Re: [linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[Holger Parplies]

Sebastian Beneke wrote:
> > > > Now /dev/hde
> > > > was destroyed by"dd if=/dev/zero of=/dev/diskname bs=1k count=1"
> > 
> > If it's the first 1K of /dev/hde, [...]
> 
> Yes, absolutely. I thought about it. Both harddisks are ok. Only the
> PT of the second disk must be destroyed. The best: there weren't any
> partitions on it, I just createt a LVM-PV and then created an ext3
> filesystem.

Hmm, if /dev/hde was a PV, then there's no partition table on it. You've
zeroed the first 1K of the PV, which I'd guess contains either [part of]
the LVM metadata or random FS data (in case the LVM metadata is located
at the end of a PV - I'm not sure). In either case, you've completely
overwritten your LVM metadata for that PV:

> On the second disk I created a new VG called "Backup"
> (thinking it was another disk).

I'm not sure whether everything you need is in the metadata backups
(under /etc/lvmconf/). Presumably it is, and you'll need to find the
correct backup to restore. See the man pages for 'vgcfgrestore' and
'pvcreate' (-ff).
I see you've read them already, and vgcfgrestore to a different disk
didn't work. Hmm. It will be of more use with the original disk anyway
if you want to regain access to the data on it.
- Do you get the same error with the original disk? You'll need to use
  'pvcreate' on it first if LVM thinks it currently belongs to VG Backup,
  but I believe it will tell you so.
- The disk was originally connected to your computer as /dev/hde. Is it
  connected as /dev/hde again now? If not, see the '-o' switch of
  'vgcfgrestore'.
- Did you change the configuration of VG storage after doing the 'dd'?
  Check the file dates of the backups in /etc/lvmconf to be sure. Select
  the correct backup to restore with '-b' or '-f'. Trying out different
  ones until you find one that works can't really hurt much, since the
  LVM metadata on your disk is messed up anyway. Just don't select the
  wrong target disk ;-). Selecting a too old backup will (obviously)
  undo the change causing creation of the backup, meaning LVs may be
  missing or have the wrong size etc.
- Err, the VG "Backup" you created was left empty, wasn't it? You didn't
  create LVs and FSes and populate them, did you?

Presuming you can regain access to your VG storage and the LV(s) in it,
you should check the integrity of the FS(es) (in case the 'dd' affected
part of the FS metadata). If an LV happened to start with the wiped out
blocks (probably meaning you can't mount or check it), you might have a
look at 'e2fsck -b ...' or 'mke2fs -S' - just to give you some pointers.
Read the warnings in the manual pages before destroying the data on your
hard disks ;-).

Thinking about it, if LVM metadata WERE at the end of a PV, you couldn't
have created VG "Backup" on your disk, so your FS is probably safe.

Well, good luck.

Regards,

Holger

Re: [linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[Sebastian Beneke]

On Mon, 3 Nov 2003 11:05:55 +0100
"Heinz J . Mauelshagen" <mauelshagen@sistina.com> wrote:

> 
> Sebastian,
> 
> you either want to use LVM2/device-mapper and try the "vgchange -P
> ..." option to activate partially available VGs

Heinz,

thanks a lot for your respond. Because I can't compile device-mapper
on the fileserver I decided to move the specific disk to another
machine. Here I compiled a 2.4.21 Kernel with device-mapper and lvm2
and connected the disk as /dev/hdc (same as before). Then I copyed all
the files from /etc/lvmconf and /etc/lvmtab.d to the other machine.

Next I loaded dm-mod (compiled device-mapper as a module).

pvscan found the pv:

***
scratchy:/home/basti# pvscan
  1 PV(s) found for VG storage: expected 2
  Logical volume (stor1) contains an incomplete mapping table.
  PV /dev/hdc    is in exported VG storage [114,44 GB / 0    free]
  Total: 1 [0   ] / in use: 1 [0   ] / in no VG: 0 [0   ]
***

so far so good, but then vgchange gave me these errors:

***
scratchy:/dev/mapper# vgchange -P -ay storage
  Partial mode. Incomplete volume groups will be activated read-only.
  1 PV(s) found for VG storage: expected 2
  Logical volume (stor1) contains an incomplete mapping table.
  1 PV(s) found for VG storage: expected 2
  Logical volume (stor1) contains an incomplete mapping table.
  device-mapper ioctl cmd 9 failed: Das Argument ist ungültig   <==== 
  Couldn't load device 'storage-stor1'.  1 logical volume(s) in volume
group "storage" now active
***

in /dev/mapper I found these files / devices:

***
scratchy:/dev/mapper# l
insgesamt 28
drwxr-xr-x    2 root     root         4096  7. Nov 19:30 ./
drwxr-xr-x   11 root     root        24576  7. Nov 18:49 ../
crw-------    1 root     root      10,  63  7. Nov 18:02 control
brw-------    1 root     root     254,   0  7. Nov 19:30 storage-stor1
***

as you can see storage-stor1 is 0 bytes - so it didn't work.

As well I tried your LVM-1 workaround on the other machine, didn't
work as well :-(

Here is some more info on the PV:

***
scratchy:/dev/mapper# pvdisplay --verbose
    Scanning for physical volume names
  1 PV(s) found for VG storage: expected 2
  Logical volume (stor1) contains an incomplete mapping table.
  Physical volume "/dev/hdc" of volume group "storage" is exported
  --- Physical volume ---
  PV Name               /dev/hdc
  VG Name               storage (exported)
  PV Size               114,50 GB / not usable 62,94 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              3662
  Free PE               0
  Allocated PE          3662
  PV UUID               woPtjg-SME7-mnSd-GaZd-f3C7-RwtW-4I4WKs
***

As I told the list in another mail, I killed the missing second PV
with a 1 byte (?) dd and created a new VG on it. So I think the data
is still there. Heinz, do you see any way for recreating the original
header(?)-lvm information with a hex editor or something on the
second PV? Maybe this could help me.

I hope anyone can help me out here :-(


Best Regards
Sebastian

Re: [linux-lvm] One PV destroyed - now the whole VG isn't working anymore

[Heinz J . Mauelshagen]

On Fri, Nov 07, 2003 at 07:39:38PM +0100, Sebastian Beneke wrote:
> On Mon, 3 Nov 2003 11:05:55 +0100
> "Heinz J . Mauelshagen" <mauelshagen@sistina.com> wrote:
> 
> > 
> > Sebastian,
> > 
> > you either want to use LVM2/device-mapper and try the "vgchange -P
> > ..." option to activate partially available VGs
> 
> Heinz,
> 
> thanks a lot for your respond. Because I can't compile device-mapper
> on the fileserver I decided to move the specific disk to another
> machine. Here I compiled a 2.4.21 Kernel with device-mapper and lvm2
> and connected the disk as /dev/hdc (same as before). Then I copyed all
> the files from /etc/lvmconf and /etc/lvmtab.d to the other machine.
> 
> Next I loaded dm-mod (compiled device-mapper as a module).
> 
> pvscan found the pv:
> 
> ***
> scratchy:/home/basti# pvscan
>   1 PV(s) found for VG storage: expected 2
>   Logical volume (stor1) contains an incomplete mapping table.
>   PV /dev/hdc    is in exported VG storage [114,44 GB / 0    free]
>   Total: 1 [0   ] / in use: 1 [0   ] / in no VG: 0 [0   ]
> ***
> 
> so far so good, but then vgchange gave me these errors:
> 
> ***
> scratchy:/dev/mapper# vgchange -P -ay storage
>   Partial mode. Incomplete volume groups will be activated read-only.
>   1 PV(s) found for VG storage: expected 2
>   Logical volume (stor1) contains an incomplete mapping table.
>   1 PV(s) found for VG storage: expected 2
>   Logical volume (stor1) contains an incomplete mapping table.
>   device-mapper ioctl cmd 9 failed: Das Argument ist ungültig   <==== 
>   Couldn't load device 'storage-stor1'.  1 logical volume(s) in volume
> group "storage" now active
> ***
> 
> in /dev/mapper I found these files / devices:
> 
> ***
> scratchy:/dev/mapper# l
> insgesamt 28
> drwxr-xr-x    2 root     root         4096  7. Nov 19:30 ./
> drwxr-xr-x   11 root     root        24576  7. Nov 18:49 ../
> crw-------    1 root     root      10,  63  7. Nov 18:02 control
> brw-------    1 root     root     254,   0  7. Nov 19:30 storage-stor1
> ***
> 
> as you can see storage-stor1 is 0 bytes - so it didn't work.

This is a missinterpreation.
storage-stor1 is a device special file with major 254 and minor 0.

Just use it and try if you can copy data off the partially accessible
logical volume.
Try to mount the filesystem in /dev/storage/stor1 (this symbolic link
should exist and point to /dev/mapper/storage-stor1) readonly.


> 
> As well I tried your LVM-1 workaround on the other machine, didn't
> work as well :-(
> 
> Here is some more info on the PV:
> 
> ***
> scratchy:/dev/mapper# pvdisplay --verbose
>     Scanning for physical volume names
>   1 PV(s) found for VG storage: expected 2
>   Logical volume (stor1) contains an incomplete mapping table.
>   Physical volume "/dev/hdc" of volume group "storage" is exported
>   --- Physical volume ---
>   PV Name               /dev/hdc
>   VG Name               storage (exported)
>   PV Size               114,50 GB / not usable 62,94 MB
>   Allocatable           yes (but full)
>   PE Size (KByte)       32768
>   Total PE              3662
>   Free PE               0
>   Allocated PE          3662
>   PV UUID               woPtjg-SME7-mnSd-GaZd-f3C7-RwtW-4I4WKs
> ***
> 
> As I told the list in another mail, I killed the missing second PV
> with a 1 byte (?) dd and created a new VG on it. So I think the data
> is still there. Heinz, do you see any way for recreating the original
> header(?)-lvm information with a hex editor or something on the
> second PV? Maybe this could help me.

Presumably you have the metadata archives in /etc/lvmconf/ preserved,
pvcreate the PV and vgcfgrestore the metadata to it.

> 
> I hope anyone can help me out here :-(
> 
> 
> Best Regards
> Sebastian
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Regards,
Heinz    -- The LVM Guy --


*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-