[linux-lvm] vg disappeared after replacing disc in raid10

[linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

\x10Hello, 

After replacing a disc in my raid10 system (ubuntu 12.04), my volume group (that contained all my data and my system) was gone.

Problem description is here:

http://ubuntuforums.org/showthread.php?t=2128504

I managed to retrieve the metadata of the volume group and the logical volumes underneath, but I did not succeed using

vgcfgrestore 

successfully in order to regain access to my data.

It seems I might have to try dmsetup, but I am afraid I might destroy my data if I use it. I have no experience with that program.


If any of you has any idea or any experience with restoring lvm volumes or using dsetup, please help me!

[linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

\x10Hello, 

After replacing a disc in my raid10 system (ubuntu 12.04), my volume group (that contained all my data and my system) was gone.

Problem description is here:

http://ubuntuforums.org/showthread.php?t=2128504

I managed to retrieve the metadata of the volume group and the logical volumes underneath, but I did not succeed using

vgcfgrestore 

successfully in order to regain access to my data.

It seems I might have to try dmsetup, but I am afraid I might destroy my data if I use it. I have no experience with that program.


If any of you has any idea or any experience with restoring lvm volumes or using dsetup, please help me!

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

Long ago, Nostradamus foresaw that on 03/25/2013 10:04 AM, Bj�rn
Nadrowski would write:
> After replacing a disc in my raid10 system (ubuntu 12.04), my volume group (that contained all my data and my system) was gone.
>
> Problem description is here:
>
> http://ubuntuforums.org/showthread.php?t=2128504
>
> I managed to retrieve the metadata of the volume group and the logical volumes underneath, but I did not succeed using
>
> vgcfgrestore 
>
> successfully in order to regain access to my data.
>
> It seems I might have to try dmsetup, but I am afraid I might destroy my data if I use it. I have no experience with that program.
>
No immediate help, but I would have been more paranoid under knoppix,
and checked for the volume group (vgscan) *before* failing the bad disk,
and *before* adding the replacement.  Running pvs at those points would
be advisable also.

I just installed a customer system with raid10, so I am *very*
interested in what went wrong.  I have only used raid1 to this point,
and have been *very* impressed with the improved performance of 4 disks
with raid10 vs. 2 with raid1. 

Here is a theory to toss out:  raid10 on 3 disks can tolerate only 1
disk failure.  Maybe there were 2?  Maybe replaced the wrong drive? 

Does pvs show the raid10 drive as a PV?  There is no point trying to use
vgcfgrestore until you have some PVs to restore it to.
Where did you find the metadata?  From a backup?  From the beginning of
the raid10 drive?

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

Thanks for the answer. 
I am sorry I will not be able to point you to the exact problem that caused the disruption of the volume group. 
I did not make any checks before I started disassembling and reassembling the array. 

I did recreate the physical volume and the volume group using pvcreate and vgcreate, please see the description of the problem 
on ubuntuforums for further details.

I now have a volume group vol0, but still no logical volumes, and vgcfgrestore still does not work.
If you have any hints as to what to do now, I would be most delighted.

 
On Mar 25, 2013, at 20:00 , Stuart D Gathman <stuart@bmsi.com> wrote:

> Long ago, Nostradamus foresaw that on 03/25/2013 10:04 AM, Bj�rn
> Nadrowski would write:
>> After replacing a disc in my raid10 system (ubuntu 12.04), my volume group (that contained all my data and my system) was gone.
>> 
>> Problem description is here:
>> 
>> http://ubuntuforums.org/showthread.php?t=2128504
>> 
>> I managed to retrieve the metadata of the volume group and the logical volumes underneath, but I did not succeed using
>> 
>> vgcfgrestore 
>> 
>> successfully in order to regain access to my data.
>> 
>> It seems I might have to try dmsetup, but I am afraid I might destroy my data if I use it. I have no experience with that program.
>> 
> No immediate help, but I would have been more paranoid under knoppix,
> and checked for the volume group (vgscan) *before* failing the bad disk,
> and *before* adding the replacement.  Running pvs at those points would
> be advisable also.
> 
> I just installed a customer system with raid10, so I am *very*
> interested in what went wrong.  I have only used raid1 to this point,
> and have been *very* impressed with the improved performance of 4 disks
> with raid10 vs. 2 with raid1. 
> 
> Here is a theory to toss out:  raid10 on 3 disks can tolerate only 1
> disk failure.  Maybe there were 2?  Maybe replaced the wrong drive? 
> 
> Does pvs show the raid10 drive as a PV?  There is no point trying to use
> vgcfgrestore until you have some PVs to restore it to.
> Where did you find the metadata?  From a backup?  From the beginning of
> the raid10 drive?
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

Long ago, Nostradamus foresaw that on 03/25/2013 04:25 PM, Bj�rn
Nadrowski would write:
>
> I did recreate the physical volume and the volume group using pvcreate and vgcreate, please see the description of the problem 
> on ubuntuforums for further details.
>
That was not a good idea.  You've now written over the major clue to
what happened.   Examining the beginning of the PV for the metadata -
perhaps offset or scrambled because of some inadvertent change to the
raid10 parameters - should have been your major priority!

Theory 2: the md driver on the knoppix CD is an earlier version that
puts metadata at the end of the volume, instead of at the beginning. 
This would change the offsets of your metadata and extents.  It would
not see the newer md raid metadata, and create a new drive.  I'm not
sure if this theory is consistent with your history.

I *strongly* advise not doing any more writing to this device until you
know *exactly* what happened.

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

You are certainly right. 
However, I have no clue what to do. 
The pvcreate was a hint given on the lvm pages at redhat. It should only have overwritten the UUID in the metadata, nothing else. 
Recreation of the volume group seemed  a good idea as I only had one volume group on the pv and therefore I did not see how this could have destroyed anything.
The (human-readable) data at the beginning of the device /dev/md127 is still there and does display the same information. Please see my posting
on ubuntuforums for more details. 
Did you look that up? Do you have any idea what I can do?

I am anyways prepared to give up in the foreseeable future. This would be a major blow to me, this computer contained a lot of
old programs and information - but as it goes, I never looked up most of it.
There are some programs on which I was working at the moment, loss of those would be my major inconvenience at the moment. 
But as my data recovery attempts seem to go nowhere, I will have to stop trying some day.




On Mar 25, 2013, at 21:36 , Stuart D Gathman <stuart@bmsi.com> wrote:

> Long ago, Nostradamus foresaw that on 03/25/2013 04:25 PM, Bj�rn
> Nadrowski would write:
>> 
>> I did recreate the physical volume and the volume group using pvcreate and vgcreate, please see the description of the problem 
>> on ubuntuforums for further details.
>> 
> That was not a good idea.  You've now written over the major clue to
> what happened.   Examining the beginning of the PV for the metadata -
> perhaps offset or scrambled because of some inadvertent change to the
> raid10 parameters - should have been your major priority!
> 
> Theory 2: the md driver on the knoppix CD is an earlier version that
> puts metadata at the end of the volume, instead of at the beginning. 
> This would change the offsets of your metadata and extents.  It would
> not see the newer md raid metadata, and create a new drive.  I'm not
> sure if this theory is consistent with your history.
> 
> I *strongly* advise not doing any more writing to this device until you
> know *exactly* what happened.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

Long ago, Nostradamus foresaw that on 03/25/2013 04:46 PM, Bj�rn
Nadrowski would write:
> Recreation of the volume group seemed  a good idea as I only had one volume group on the pv and therefore I did not see how this could have destroyed anything.
> The (human-readable) data at the beginning of the device /dev/md127 is still there and does display the same information. Please see my posting
Recreating the vg overwrites the human-readable data at the beginning of
the device.  You claim that it is "still there" and is the "same
information".  But this cannot be true because originally that
information had your LV data, and now (apparently) it does not.  Do you
see your LVs when looking at the vg metadata with an editor??   What do
you mean by "displays the same information"?  Does lvs list your LVs?

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

[-- Attachment #1: Type: text/plain, Size: 2703 bytes --]

You were right,
the data has been overwritten.
It now reads thus (or rather, the first instance of human-readable volume
group description reads thus; my old volume group description is still
there, but further down...):

""""""""""""""""""""""""""""""""""""""""
vol0 {
id = "8kEHPL-yDuF-7ffg-GeGH-dEg6-D0Qn-lUrKoh"
seqno = 1
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0

physical_volumes {

pv0 {
id = "WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26"
device = "/dev/md127"

status = ["ALLOCATABLE"]
flags = []
dev_size = 5820316928
pe_start = 384
pe_count = 710487
}
}

}
# Generated by LVM2 version 2.02.95(2) (2012-03-06): Sun Mar 24 23:39:50
2013

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "Microknoppix"    # Linux Microknoppix 3.6.11 #12 SMP
PREEMPT Thu Dec 20 04:04:10 CET 2012 i686
creation_time = 1364168390    # Sun Mar 24 23:39:50 2013
"""""""""""""""""""""""""""""""""""""""""'''

No wonder I do not see any logical volumes (well, I did not see any logical
volumes before creating the volume neither...)

So what should I do?

lvs does not see anything, of course.

I could recreate the physical volume using

sudo pvcreate --uuid WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26 --restorefile
vol0.t
xt /dev/md127

(using my old volume group description in vol0.txt, of course).

Is this what I should do? Or rather remove the physical volume first?



On Mon, Mar 25, 2013 at 9:30 PM, Stuart D Gathman <stuart@bmsi.com> wrote:

> Long ago, Nostradamus foresaw that on 03/25/2013 04:46 PM, Björn
> Nadrowski would write:
> > Recreation of the volume group seemed  a good idea as I only had one
> volume group on the pv and therefore I did not see how this could have
> destroyed anything.
> > The (human-readable) data at the beginning of the device /dev/md127 is
> still there and does display the same information. Please see my posting
> Recreating the vg overwrites the human-readable data at the beginning of
> the device.  You claim that it is "still there" and is the "same
> information".  But this cannot be true because originally that
> information had your LV data, and now (apparently) it does not.  Do you
> see your LVs when looking at the vg metadata with an editor??   What do
> you mean by "displays the same information"?  Does lvs list your LVs?
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 3912 bytes --]

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

Well, I see I would have to remove the volume group first, at the very
least.
Should I just issue

sudo vgremove vol0

followed by

sudo pvremove /dev/127

and followed by

sudo pvcreate --uuid WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26 --restorefile
vol0.t
xt /dev/md127

?

But I would just be back in the middle of my problem description on
ubuntuforums, wouldn' t I?
Still no volume group, still no logical volumes...


On Mon, Mar 25, 2013 at 9:30 PM, Stuart D Gathman <stuart@bmsi.com> wrote:

> Long ago, Nostradamus foresaw that on 03/25/2013 04:46 PM, Björn
> Nadrowski would write:
> > Recreation of the volume group seemed  a good idea as I only had one
> volume group on the pv and therefore I did not see how this could have
> destroyed anything.
> > The (human-readable) data at the beginning of the device /dev/md127 is
> still there and does display the same information. Please see my posting
> Recreating the vg overwrites the human-readable data at the beginning of
> the device.  You claim that it is "still there" and is the "same
> information".  But this cannot be true because originally that
> information had your LV data, and now (apparently) it does not.  Do you
> see your LVs when looking at the vg metadata with an editor??   What do
> you mean by "displays the same information"?  Does lvs list your LVs?
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

[-- Attachment #2: Type: text/html, Size: 2184 bytes --]

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

Long ago, Nostradamus foresaw that on 03/25/2013 05:45 PM, Bj�rn
Nadrowski would write:
> You were right,
> the data has been overwritten.
> It now reads thus (or rather, the first instance of human-readable
> volume group description reads thus; my old volume group description
> is still there, but further down...):
Now you are getting somewhere.  The offset of your PV has shifted. 
First, copy that 2nd human readable part to removable media somewhere. 
If possible, backup your drives (e.g. to USB drives) so you can recover
from further mistakes.

Why has the offset shifted?  Possibilities:

1) different md driver version on knoppix (although using an older
version would move the md superblock from end of drive to beginning of
drive - shifting the opposite way).  Or different options.  BUT mdadm
seemed to recognize your RAID, so let's
discount this theory.

2) alignment options to pvcreate - but it still should have been seen by
lvs!

3) the offset hasn't shifted, but you've written over the first few
sectors with the empty VG.  Is the 2nd human readable part, by any
chance, missing the first part?  You could post both parts for us to
look at.

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

[-- Attachment #1: Type: text/plain, Size: 7507 bytes --]


On Mar 26, 2013, at 14:41 , Stuart D Gathman <stuart@bmsi.com> wrote:

> Long ago, Nostradamus foresaw that on 03/25/2013 05:45 PM, Björn
> Nadrowski would write:
>> You were right,
>> the data has been overwritten.
>> It now reads thus (or rather, the first instance of human-readable
>> volume group description reads thus; my old volume group description
>> is still there, but further down...):
> Now you are getting somewhere.  The offset of your PV has shifted. 
> First, copy that 2nd human readable part to removable media somewhere. 
> If possible, backup your drives (e.g. to USB drives) so you can recover
> from further mistakes.
> 
> Why has the offset shifted?  Possibilities:
> 
> 1) different md driver version on knoppix (although using an older
> version would move the md superblock from end of drive to beginning of
> drive - shifting the opposite way).  Or different options.  BUT mdadm
> seemed to recognize your RAID, so let's
> discount this theory.
> 
> 2) alignment options to pvcreate - but it still should have been seen by
> lvs!
> 
> 3) the offset hasn't shifted, but you've written over the first few
> sectors with the empty VG.  Is the 2nd human readable part, by any
> chance, missing the first part?  You could post both parts for us to
> look at.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Thanks for all these hints. 
I can actually just restore the first bytes of the device as I saved the output of the original dd command to a file. 

sudo dd if=/dev/md127 bs=512 count=255 skip=1 of=md0.txt

So I could just do 
sudo dd if=md0.txt bs=512 count=255 skip=1 of=/dev/md127

Might this be the appropriate action?

I doubt that the entire array has shifted, as md0.txt contained two complete human-readable descriptions of my volume groups and my 
logical volumes. This is the human-readable part of md0.txt (result of the dd command right after disc replacement):

------------------------------
vol0 {
id = "2daJft-nq2X-zJen-2VO1-c2Od-HREv-zyjKTz"
seqno = 7
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 8192
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = "WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26"
device = "/dev/md0"

status = ["ALLOCATABLE"]
dev_size = 5820316928
pe_start = 384
pe_count = 710487
}
}

logical_volumes {

root {
id = "iO0y32-L53R-msRI-3JiE-t7j1-Ija2-3SOnpC"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 11920

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 0
]
}
}

home {
id = "1Vk6QQ-c3NF-6z6h-zw7E-fkFU-mRWU-DKLM6t"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 674725

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 11920
]
}
}
}
}
# Generated by LVM2 version 2.02.39 (2008-06-27): Wed Jun 10 13:21:55 2009

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "cn-b204-4"	# Linux cn-b204-4 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64
creation_time = 1244640115	# Wed Jun 10 13:21:55 2009






vol0 {
id = "2daJft-nq2X-zJen-2VO1-c2Od-HREv-zyjKTz"
seqno = 8
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 8192
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = "WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26"
device = "/dev/md0"

status = ["ALLOCATABLE"]
dev_size = 5820316928
pe_start = 384
pe_count = 710487
}
}

logical_volumes {

root {
id = "iO0y32-L53R-msRI-3JiE-t7j1-Ija2-3SOnpC"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 11920

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 0
]
}
}

home {
id = "1Vk6QQ-c3NF-6z6h-zw7E-fkFU-mRWU-DKLM6t"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 674725

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 11920
]
}
}

empty {
id = "YToY9o-223w-nMrv-X5qU-0S8t-oUbG-Zk6ZmV"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 23842

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 686645
]
}
}
}
}
# Generated by LVM2 version 2.02.39 (2008-06-27): Wed Jun 10 13:22:04 2009

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "cn-b204-4"	# Linux cn-b204-4 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64
creation_time = 1244640124	# Wed Jun 10 13:22:04 2009

---------------------------

This is the human-readable part of the beginning of the device after my vgcreate action:

---------------------------

vol0 {
id = "8kEHPL-yDuF-7ffg-GeGH-dEg6-D0Qn-lUrKoh"
seqno = 1
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0

physical_volumes {

pv0 {
id = "WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26"
device = "/dev/md127"

status = ["ALLOCATABLE"]
flags = []
dev_size = 5820316928
pe_start = 384
pe_count = 710487
}
}

}
# Generated by LVM2 version 2.02.95(2) (2012-03-06): Sun Mar 24 23:39:50 2013

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "Microknoppix"	# Linux Microknoppix 3.6.11 #12 SMP PREEMPT Thu Dec 20 04:04:10 CET 2012 i686
creation_time = 1364168390	# Sun Mar 24 23:39:50 2013




vol0 {
id = "2daJft-nq2X-zJen-2VO1-c2Od-HREv-zyjKTz"
seqno = 8
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 8192
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = "WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26"
device = "/dev/md0"

status = ["ALLOCATABLE"]
dev_size = 5820316928
pe_start = 384
pe_count = 710487
}
}

logical_volumes {

root {
id = "iO0y32-L53R-msRI-3JiE-t7j1-Ija2-3SOnpC"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 11920

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 0
]
}
}

home {
id = "1Vk6QQ-c3NF-6z6h-zw7E-fkFU-mRWU-DKLM6t"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 674725

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 11920
]
}
}

empty {
id = "YToY9o-223w-nMrv-X5qU-0S8t-oUbG-Zk6ZmV"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 23842

type = "striped"
stripe_count = 1	# linear

stripes = [
"pv0", 686645
]
}
}
}
}
# Generated by LVM2 version 2.02.39 (2008-06-27): Wed Jun 10 13:22:04 2009

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "cn-b204-4"	# Linux cn-b204-4 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64
creation_time = 1244640124	# Wed Jun 10 13:22:04 2009


---------------------------------

It seems, in both cases I have the description of the two last configurations of my volume group. The first one is the actual one. 
Both are complete.

So, should I rewrite the beginning of the device with the dd command?

What then?

Thanks a lot for all your help!





[-- Attachment #2: Type: text/html, Size: 12964 bytes --]

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

On Mar 26, Stuart D Gathman transmitted in part:

> Why has the offset shifted?  Possibilities:
>
> 1) different md driver version on knoppix (although using an older
> version would move the md superblock from end of drive to beginning of
> drive - shifting the opposite way).  Or different options.  BUT mdadm
> seemed to recognize your RAID, so let's
> discount this theory.
>
> 2) alignment options to pvcreate - but it still should have been seen by
> lvs!
>
> 3) the offset hasn't shifted, but you've written over the first few
> sectors with the empty VG.  Is the 2nd human readable part, by any
> chance, missing the first part?  You could post both parts for us to
> look at.

4) the shape of the md array changed.  Say, you somehow didn't 
actually remove the failed drive before adding the replacement.  What
does md raid10 do if you add a 4th disk to a 3 drive array?  Add
it as a spare?  Reshape, shuffling all your data?

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

As I say, 
I do not think (maybe that is just a hope) that the offset did shift. 
I believe that because

* there are two complete vg descriptions in the same part at the beginning of the device, before and after my vgcreate action
* the older vg description has been replaced by the newer, actual one after the vgcreate action

I made absolutely sure that the failing device had been flagged as failed and removed via mdadm --remove. 
When I added the new drive, the raid10 only ran on three disks.

What do you think I should do?
restore the beginning of the device using the file md0.txt?
Then recreate the physical volume again using the -uuid switch?
But what then?


On Mar 27, 2013, at 5:29 , Stuart D Gathman <stuart@bmsi.com> wrote:

> On Mar 26, Stuart D Gathman transmitted in part:
> 
>> Why has the offset shifted?  Possibilities:
>> 
>> 1) different md driver version on knoppix (although using an older
>> version would move the md superblock from end of drive to beginning of
>> drive - shifting the opposite way).  Or different options.  BUT mdadm
>> seemed to recognize your RAID, so let's
>> discount this theory.
>> 
>> 2) alignment options to pvcreate - but it still should have been seen by
>> lvs!
>> 
>> 3) the offset hasn't shifted, but you've written over the first few
>> sectors with the empty VG.  Is the 2nd human readable part, by any
>> chance, missing the first part?  You could post both parts for us to
>> look at.
> 
> 4) the shape of the md array changed.  Say, you somehow didn't actually remove the failed drive before adding the replacement.  What
> does md raid10 do if you add a 4th disk to a 3 drive array?  Add
> it as a spare?  Reshape, shuffling all your data?
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 951 bytes --]

On Mar 27, Bjï¿œrn Nadrowski transmitted in part:

> What do you think I should do?
> restore the beginning of the device using the file md0.txt?

At this point, since you know where the metadata is going (and has
already been wiped out), and you have a good metadata (that you can
look at with an editor and see your LVs), I would use vgcfgrestore 
to restore your good metadata backup.  (If not produced with
vgcfgbackup, compare with one that is).

Be careful not to modify any LVs until you have checked that they
are not scrambled.  Any mounts should be read-only, any fscks should
be with -n.

It the device is already listed in pvs, it is already a PV.  It must be,
because you claim that you see the new (empty) VG.

You will need to deactivate the new VG before using vgcfgrestore,
e.g. vgchange -an emptyvg.

If there is any way to backup stuff before attempted recovery, do so.

After vgcfgrestore, do vgscan and lvs.

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On 03/26/2013 02:13 PM, Bj�rn Nadrowski expounded in part:
>
> So, should I rewrite the beginning of the device with the dd command?
>
> What then?
>
> Thanks a lot for all your help!
>
You could try the dd, if it has a good chance of restoring the device to 
before the pvcreate.  I don't know offhand how much of a PV pvcreate 
overwrites.  Hopefully, an LVM guru can chime in. Before doing pvcreate, 
things like vg_scan -v should give a clue as to whether knoppix is even 
looking at md127.  You might need to tweak the device filter in lvm.conf 
for knoppix.

[-- Attachment #2: Type: text/html, Size: 1400 bytes --]

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

On 03/25/2013 05:50 PM, Bj�rn Nadrowski expounded in part:
> Well, I see I would have to remove the volume group first, at the very 
> least.
I wouldn't.  Just think of your vgcreate as removing all LVs and 
renaming the VG.  Now you want to undo all that
with a vgcfgrestore.
> But I would just be back in the middle of my problem description on 
> ubuntuforums, wouldn' t I?
> Still no volume group, still no logical volumes...
Does knoppix have an older lvm version that doesn't recognize your 
metadata?  I *hope* it doesn't have lvm1, which used binary metadata....

If you see the human readable metadata, and knoppix doesn't, then there 
is a problem with knoppix.  Either ancient lvm, or the device filtered 
out for vgscan in lvm.conf.

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Björn Nadrowski]

[-- Attachment #1: Type: text/plain, Size: 3512 bytes --]

I recovered my data. 
Thanks for all your help. 
The problem was in fact knoppix, but not because it had an older version of lvm2 or even lvm1.

The problem was that the human-readable part I extracted from the beginning of the device contained invisible characters in front of the 
'vol0' label in the file 'volt.txt' .
These characters were not visible in terminal using 'cat', 'more', and they also did not appear in standard error as a consequence of the 'vgcfgrestore' command, as witnessed by the first line of the error:

'data/vol0.txt' does not contain volume group 'vol0'.

The reason why I could not observe this problem under knoppix is, that I did not use any other programs to display text than 'cat', 'more', standard error/standard out, and libreoffice. So I could not see that there were these invisible characters in front of 'vol0'.
Under knoppix, I did not have xemacs, which I usually use, so I could not locate the problem, thinking that my restore file vol0.txt was ok.
This belief was reinforced by the fact that I could use the file without problems to recreate the physical volume using pvcreate with the uuid switch.

You helped me finding the problem because of your comments on possible lvm2-version differences between ubuntu and knoppix, so I redid the whole thing with a ubuntu live cd, where I could install xemacs, and I noticed the characters. 
I could then remove the bogey volume group that I had created from scratch using vgremove, and after that, the
vgcfgrestore command with the corrected file vol0.txt worked without problems. 

Right now I am doing my backup, before switching the faulty drive to the new replacement… This is not gonna happen again.

Bottom-line: 
Something during the replacement happened that did something with the metadata for the physical volume. I do not know what that was.  
The human-readable metadata at the beginning of the raid10 device was still there and intact.
I copied the data from the beginning of the raid10 device using 

sudo dd if=/dev/md127 bs=512 count=255 skip=1 of=md0.txt

Then copied the volume group description from md0.txt to a file called vol0.txt.

I could then recreate the physical volume using 

sudo pvcreate --uuid WFo1On-anFb-2av8-DuRq-vKee-nJEt-ZLnu26 --restorefile vol0.txt /dev/md127

and I could recover the volume group and all its logical volumes using

sudo vgcfgrestore -f vol0.txt vol0

Thanks for helping!



On Mar 27, 2013, at 21:08 , Stuart D Gathman <stuart@bmsi.com> wrote:

> On 03/25/2013 05:50 PM, Björn Nadrowski expounded in part:
>> Well, I see I would have to remove the volume group first, at the very least.
> I wouldn't.  Just think of your vgcreate as removing all LVs and renaming the VG.  Now you want to undo all that
> with a vgcfgrestore.
>> But I would just be back in the middle of my problem description on ubuntuforums, wouldn' t I?
>> Still no volume group, still no logical volumes...
> Does knoppix have an older lvm version that doesn't recognize your metadata?  I *hope* it doesn't have lvm1, which used binary metadata....
> 
> If you see the human readable metadata, and knoppix doesn't, then there is a problem with knoppix.  Either ancient lvm, or the device filtered out for vgscan in lvm.conf.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


[-- Attachment #2: Type: text/html, Size: 5503 bytes --]

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On 03/27/2013 07:39 PM, Bj�rn Nadrowski expounded in part:
> I recovered my data.
> Thanks for all your help.
> The problem was in fact knoppix, but not because it had an older 
> version of lvm2 or even lvm1.
>
> The problem was that the human-readable part I extracted from the 
> beginning of the device contained invisible characters in front of the
> 'vol0' label in the file 'volt.txt' .
> These characters were not visible in terminal using 'cat', 'more', and 
> they also did not appear in standard error as a consequence of the 
> 'vgcfgrestore' command, as witnessed by the first line of the error:
>
And we owe some pizzas/beers/rawjuicedrinks to the LVM designer who 
insisted on human readable metadata.  It has been invaluable in many 
data recovery scenarios.

[-- Attachment #2: Type: text/html, Size: 1400 bytes --]

Re: [linux-lvm] vg disappeared after replacing disc in raid10

[Stuart D Gathman]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

On 03/27/2013 07:39 PM, Bj�rn Nadrowski expounded in part:
> I recovered my data.
> Thanks for all your help.
> The problem was in fact knoppix, but not because it had an older 
> version of lvm2 or even lvm1.
>
> The problem was that the human-readable part I extracted from the 
> beginning of the device contained invisible characters in front of the
> 'vol0' label in the file 'volt.txt' .
> These characters were not visible in terminal using 'cat', 'more', and 
> they also did not appear in standard error as a consequence of the 
> 'vgcfgrestore' command, as witnessed by the first line of the error:
>
Learn to use 'vim', at least for rudimentary use.  It is the code editor 
on live CDs because its minimal version is very small.  The enhanced 
version adds a macro language with a huge library.

[-- Attachment #2: Type: text/html, Size: 1436 bytes --]