[linux-lvm] LVM label lost / system does not boot

[linux-lvm] LVM label lost / system does not boot

[Andreas Schild]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]

Hi
The first part might sound like I am in the wrong group, but bear with me...
(I probably am, but I googled up and down RAID and LVM lists and I am still
 stuck):
I have a software RAID 5 with 4 disks and LVM on top. I had one volume group
with two logical volumes (for root and data).
I wanted to upgrade capacity and started by failing a drive, replacing it
with a bigger one and let the RAID resync. Worked fine for the first disk.
The second disk apparently worked (resynced, all looked good), but after a
reboot the system hung.
After some back and forth with superblocks (on the devices, never on the
array) I was able to re-assemble the array clean.
The system still does not reboot though: "Volume group "cmain" not found".

I booted a live cd, assembled the array and did a pvck on the array
(/dev/md0):
"Could not find LVM label on /dev/md0"
pvdisplay /dev/md0 results in:
  No physical volume label read from /dev/md0
  Failed to read physical volume "/dev/md0"

I do not have a backup of my /etc/ and therefore no details regarding the
configuration of the LVM setup (yes, I know...)
All I have of the broken system is the /boot partition with its content

Several questions arise:
- Is it possible to "reconstitute" the LVM with what I have?
- Is the RAID array really ok, or is it possibly corrupt to begin with (and
the reason no LVM labels are around)?
- Should I try to reconstruct with pvcreate/vgcreate? (I shied away from any
*create commands to not make things worse.)
- If all is lost, what did I do wrong and what would I need to backup for a
next time?

Any ideas on how I could get the data back would greatly be appreciated. I
am in way over my head, so if somebody knowledgeable tells me: "you lost,
move on" would be bad, but at least would save me some time...

Thanks,
Andreas

[-- Attachment #2: Type: text/html, Size: 2212 bytes --]

Re: [linux-lvm] LVM label lost / system does not boot

[Ger Apeldoorn]

On 06/05/2011 02:47 PM, Andreas Schild wrote:
> Hi
> The first part might sound like I am in the wrong group, but bear with 
> me...
> (I probably am, but I googled up and down RAID and LVM lists and I am 
> still  stuck):
> I have a software RAID 5 with 4 disks and LVM on top. I had one volume 
> group with two logical volumes (for root and data).
> I wanted to upgrade capacity and started by failing a drive, replacing 
> it with a bigger one and let the RAID resync. Worked fine for the 
> first disk. The second disk apparently worked (resynced, all looked 
> good), but after a reboot the system hung.
> After some back and forth with superblocks (on the devices, never on 
> the array) I was able to re-assemble the array clean.
> The system still does not reboot though: "Volume group "cmain" not found".
>
> I booted a live cd, assembled the array and did a pvck on the array 
> (/dev/md0):
> "Could not find LVM label on /dev/md0"
> pvdisplay /dev/md0 results in:
>   No physical volume label read from /dev/md0
>   Failed to read physical volume "/dev/md0"
>
> I do not have a backup of my /etc/ and therefore no details regarding 
> the configuration of the LVM setup (yes, I know...)
> All I have of the broken system is the /boot partition with its content
>
> Several questions arise:
> - Is it possible to "reconstitute" the LVM with what I have?
> - Is the RAID array really ok, or is it possibly corrupt to begin with 
> (and the reason no LVM labels are around)?
> - Should I try to reconstruct with pvcreate/vgcreate? (I shied away 
> from any *create commands to not make things worse.)
> - If all is lost, what did I do wrong and what would I need to backup 
> for a next time?
>
> Any ideas on how I could get the data back would greatly be 
> appreciated. I am in way over my head, so if somebody knowledgeable 
> tells me: "you lost, move on" would be bad, but at least would save me 
> some time...
>
> Thanks,
> Andreas
>
Have you tried 'vgscan -vvv' and 'pvscan -vvv' when started from a 
usb-stick or bootable dvd-rom? It might give you some more info. (post 
it here)

Ger.

Re: [linux-lvm] LVM label lost / system does not boot

[Anthony Nelson]

[-- Attachment #1: Type: text/plain, Size: 3088 bytes --]

Don't think I'm much of an expert in this area, but I had a similar 
issue that I worked through last week. I wasn't using mdadm, but I don't 
really think this is a situation that is really relevant to that.

I posted the question on serverfault and got enough help to get me 
through it. You can see it here:
http://serverfault.com/questions/275679/rescue-disk-is-unable-to-see-the-lvm-physical-volumes

The suggestion that got me on the right track seems to be very similar 
to your situation. How to recover your lvm configuration with mdadm.
http://www.howtoforge.com/recover_data_from_raid_lvm_partitions

The steps I went through were basically:
* Recover config using dd per steps in that article
* Recreate the physical volume using the same device
* Do a pvcfgrestore on the recovered config from step 1

That got me back up and running. I hope it helps.

Anthony Nelson
Affinity Global Solutions
701.223.3565 Ext: 13


On 6/5/2011 7:47 AM, Andreas Schild wrote:
> Hi
> The first part might sound like I am in the wrong group, but bear with 
> me...
> (I probably am, but I googled up and down RAID and LVM lists and I am 
> still  stuck):
> I have a software RAID 5 with 4 disks and LVM on top. I had one volume 
> group with two logical volumes (for root and data).
> I wanted to upgrade capacity and started by failing a drive, replacing 
> it with a bigger one and let the RAID resync. Worked fine for the 
> first disk. The second disk apparently worked (resynced, all looked 
> good), but after a reboot the system hung.
> After some back and forth with superblocks (on the devices, never on 
> the array) I was able to re-assemble the array clean.
> The system still does not reboot though: "Volume group "cmain" not found".
>
> I booted a live cd, assembled the array and did a pvck on the array 
> (/dev/md0):
> "Could not find LVM label on /dev/md0"
> pvdisplay /dev/md0 results in:
>   No physical volume label read from /dev/md0
>   Failed to read physical volume "/dev/md0"
>
> I do not have a backup of my /etc/ and therefore no details regarding 
> the configuration of the LVM setup (yes, I know...)
> All I have of the broken system is the /boot partition with its content
>
> Several questions arise:
> - Is it possible to "reconstitute" the LVM with what I have?
> - Is the RAID array really ok, or is it possibly corrupt to begin with 
> (and the reason no LVM labels are around)?
> - Should I try to reconstruct with pvcreate/vgcreate? (I shied away 
> from any *create commands to not make things worse.)
> - If all is lost, what did I do wrong and what would I need to backup 
> for a next time?
>
> Any ideas on how I could get the data back would greatly be 
> appreciated. I am in way over my head, so if somebody knowledgeable 
> tells me: "you lost, move on" would be bad, but at least would save me 
> some time...
>
> Thanks,
> Andreas
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

[-- Attachment #2: Type: text/html, Size: 4733 bytes --]

Re: [linux-lvm] LVM label lost / system does not boot

[Andreas Schild]

[-- Attachment #1: Type: text/plain, Size: 3530 bytes --]

Thanks to a link from Ger (and a lot of googling) I am a couple of steps
ahead:
I was able to recover the Volume Group Configuration and use it to do a
"vgcfgrestore", which worked (!)
next was a vgchange:
vgchange -a y
  device-mapper: resume ioctl failed: Invalid argument
  Unable to resume cmain-data (252:4)
  5 logical volume(s) in volume group "cmain" now active

dmesg showed:
device-mapper: table: 252:4: md0 too small for target: start=2390753664,
len=527835136, dev_size=2918584320

I tried to mount the other volumes, but this did not work either: classical
"specify file system type" (xfs for me) and then the error about bad
superblock etc. xfs_check did not help either.

I went one step back and looked at the superblocks of the two old harddisks
(which are not longer part of the array). I realized one difference: The
Chunk Size for the "old" RAID array was 64K, for the new array this is 512K.

Could this be a reason lvm is off and looks at the wrong place?

Current situation is this: 2 old disks and 2 new disks constitute a clean(?)
RAID 5 array with chunk size 512K with the problems described.
I do have the two old drives, but the file system content has changed during
the step up. My guess is I cannot just hook together everything as it was
and be happy. (I would probably have to zero the superblocks of the two old
HDs (the ones that were in the new array) but would probably end up with the
same problems).

Any ideas? or is this the point where I am in the wrong mailing list?

Thanks so far, I already learned a lot.
Andreas



On Sun, Jun 5, 2011 at 14:47, Andreas Schild <andreas@soulboarder.net>wrote:

> Hi
> The first part might sound like I am in the wrong group, but bear with
> me...
> (I probably am, but I googled up and down RAID and LVM lists and I am still
>  stuck):
> I have a software RAID 5 with 4 disks and LVM on top. I had one volume
> group with two logical volumes (for root and data).
> I wanted to upgrade capacity and started by failing a drive, replacing it
> with a bigger one and let the RAID resync. Worked fine for the first disk.
> The second disk apparently worked (resynced, all looked good), but after a
> reboot the system hung.
> After some back and forth with superblocks (on the devices, never on the
> array) I was able to re-assemble the array clean.
> The system still does not reboot though: "Volume group "cmain" not found".
>
> I booted a live cd, assembled the array and did a pvck on the array
> (/dev/md0):
> "Could not find LVM label on /dev/md0"
> pvdisplay /dev/md0 results in:
>   No physical volume label read from /dev/md0
>   Failed to read physical volume "/dev/md0"
>
> I do not have a backup of my /etc/ and therefore no details regarding the
> configuration of the LVM setup (yes, I know...)
> All I have of the broken system is the /boot partition with its content
>
> Several questions arise:
> - Is it possible to "reconstitute" the LVM with what I have?
> - Is the RAID array really ok, or is it possibly corrupt to begin with (and
> the reason no LVM labels are around)?
> - Should I try to reconstruct with pvcreate/vgcreate? (I shied away from
> any *create commands to not make things worse.)
> - If all is lost, what did I do wrong and what would I need to backup for a
> next time?
>
> Any ideas on how I could get the data back would greatly be appreciated. I
> am in way over my head, so if somebody knowledgeable tells me: "you lost,
> move on" would be bad, but at least would save me some time...
>
> Thanks,
> Andreas
>
>

[-- Attachment #2: Type: text/html, Size: 4538 bytes --]