[linux-lvm] LVM2 recovery on IP-SAN after OS re-install

[linux-lvm] LVM2 recovery on IP-SAN after OS re-install

[tariq wali]

[-- Attachment #1: Type: text/plain, Size: 2471 bytes --]

Hi,
 We had an OS crash on x86_64 CentOS , LVM2 and it had 6TB LUN  via  IP
SCSI SAN mounted to it and after the OS reinstall on a crash we had hard
time to mount the original SAN volume as you can imagine 6TB
of marketing data with 3 LVM partitions as data data1 and data2 . After
troubleshooting almost for a day I was able to get the LVM metadata using
dd from the SAN LUN /dev/sdd1 which I used to recreate and restore all LVM
partitions back to original state , we sure became the raving fans of LVM
today ! however after all the recovery I see this annoying warning each
time I execute any of the lvm commands ..

lvs
*  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using
/dev/sdd1 not/dev/sdc1
*
  LV    VG   Attr   LSize Origin Snap%  Move Log Copy%  Convert
  data  vg0  -wi-ao 2.00T
  data1 vg0  -wi-ao 2.00T
  data2 vg0  -wi-ao 1.44T


I believe previously we had  created PV using /dev/sdd1 and /dev/sdc1
although from the same LUN but now I was able to restore everything using
just /dev/sdd1 and everything looks intact (until I hear marketing whine
tom:)

vgs -o +devices
  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not
/dev/sdc1
  VG   #PV #LV #SN Attr   VSize VFree Devices
  vg0    1   3   0 wz--n- 5.45T 8.06G /dev/sdd1(0)
  vg0    1   3   0 wz--n- 5.45T 8.06G /dev/sdd1(524288)
  vg0    1   3   0 wz--n- 5.45T 8.06G /dev/sdd1(1048576)


 lvs -P -a -o +devices
  Partial mode. Incomplete logical volumes will be processed.
  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not
/dev/sdc1
  LV    VG   Attr   LSize Origin Snap%  Move Log Copy%  Convert Devices

  data  vg0  -wi-ao 2.00T
/dev/sdd1(0)
  data1 vg0  -wi-ao 2.00T
/dev/sdd1(524288)
  data2 vg0  -wi-ao 1.44T
/dev/sdd1(1048576)

I tried to fix this by but it just won't go..
vgreduce --removemissing vg0
  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not
/dev/sdc1
  Volume group "vg0" is already consistent

why does LVM think that it's still using PV ( /dev/sdc1 ) when it no more
exists ? and how do i get rid of it ?

I also see some of these in syslog messages

hfs: unable to find HFS+ superblock
hfs: unable to find HFS+ superblock
VFS: Can't find ext3 filesystem on dev sdc1.
VFS: Can't find ext3 filesystem on dev sdc1.
hfs: unable to find HFS+ superblock
VFS: Can't find ext3 filesystem on dev sdc1.
VFS: Can't find ext3 filesystem on dev sdc1.
VFS: Can't find ext3 filesystem on dev sdd1.

thx,
Tariq

[-- Attachment #2: Type: text/html, Size: 3487 bytes --]

Re: [linux-lvm] LVM2 recovery on IP-SAN after OS re-install

[James Hawtin]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

On 28/05/12 19:40, tariq wali wrote:
> Hi,
>  We had an OS crash on x86_64 CentOS , LVM2 and it had 6TB LUN  via 
>  IP SCSI SAN mounted to it and after the OS reinstall on a crash we 
> had hard time to mount the original SAN volume as you can imagine 6TB 
> of marketing data with 3 LVM partitions as data data1 and data2 . 
> After troubleshooting almost for a day I was able to get the LVM 
> metadata using dd from the SAN LUN /dev/sdd1 which I used to recreate 
> and restore all LVM partitions back to original state , we sure became 
> the raving fans of LVM today ! however after all the recovery I see 
> this annoying warning each time I execute any of the lvm commands ..
>
> lvs
> *  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using 
> /dev/sdd1 not/dev/sdc1*
>   LV    VG   Attr   LSize Origin Snap%  Move Log Copy%  Convert
>   data  vg0  -wi-ao 2.00T
>   data1 vg0  -wi-ao 2.00T
>   data2 vg0  -wi-ao 1.44T
>

1) The the same lun presented to the computer more than once? If this is 
a deliberate thing for redundancy then your need to setup multipath to 
take advantage use the devices in /dev/mapper of this and mask off the 
raw /dev/sd? devices in your /etc/lvm/lvm.conf

2) Did you copy all the data from one disk to another? if so is the old 
disk still available to the system?

3) Check you lun masking.

James

[-- Attachment #2: Type: text/html, Size: 2698 bytes --]

Re: [linux-lvm] LVM2 recovery on IP-SAN after OS re-install

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/28/2012 07:40 PM, tariq wali wrote:
> I tried to fix this by but it just won't go.. vgreduce
> --removemissing vg0 Found duplicate PV
> XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 
> Volume group "vg0" is already consistent
> 
> why does LVM think that it's still using PV ( /dev/sdc1 ) when it
> no more exists ? and how do i get rid of it ?

Although you have removed the PV from the volume group it still has a
PV label on it which has the same UUID value as some other valid
device that is part of the volume group.

You should verify which device is correct and then remove the PV label
from the other device (or re-run pvcreate to give it a new random UUID):

# pvremove /dev/sdc1

Once the label is removed you should stop seeing those messages.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/gq5EACgkQ6YSQoMYUY97zTACdEn2l2OkwM/4naoqzZwJFDSiO
WpYAnjSN8Uppia//lZK28FnhXvyDwK+Y
=v7ST
-----END PGP SIGNATURE-----

Re: [linux-lvm] LVM2 recovery on IP-SAN after OS re-install

[tariq wali]

[-- Attachment #1: Type: text/plain, Size: 6711 bytes --]

*Bryn,*
*
*
*
*
*Thanks for the mail but I might have just created a havoc by force
removing the PV*
Here is a series of commands and output ..

[root@datanalytics4 ~]# pvs
  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not *
/dev/sdc1*
  PV           VG  Fmt  Attr    PSize     PFree
  /dev/sda5  vg1  lvm2 a--     162.99G     0
  /dev/sdb5  vg2  lvm2 a--     230.97G   20.97G
  /dev/PV    VG   Fmt   Attr  PSize       PFree
[root@datanalytics4 ~]#

[root@datanalytics4 ~]# pvdisplay
  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not
/dev/sdc1
  --- Physical volume ---
  PV Name               /dev/sdd1
  VG Name               vg0
  PV Size               5.45 TB / not usable 1.14 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              1428127
  Free PE               2063
  Allocated PE          1426064
  PV UUID               XtGmeF-OiCT-WpLB-ckKZ-KlGt-YKPV-dBgVJn

  --- Physical volume ---
  PV Name               /dev/sdb5
  VG Name               vg2
  PV Size               230.98 GB / not usable 3.41 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              59129
  Free PE               5369
  Allocated PE          53760
  PV UUID               UP9bH6-Huaf-02ua-tsL0-lfcU-axc3-CIkSvS

  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               vg1
  PV Size               162.99 GB / not usable 1.83 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              41725
  Free PE               0
  Allocated PE          41725
  PV UUID               9DSyjC-fvzX-wMTa-7NNG-JOzZ-Tu3F-utmpFT

[root@datanalytics4 ~]#


[root@datanalytics4 ~]# pvremove --force --force  /dev/sdc1
  Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not
/dev/sdc1
Really WIPE LABELS from physical volume "/dev/sdc1" of volume group "vg0"
[y/n]? y
  WARNING: Wiping physical volume label from /dev/sdc1 of volume group "vg0"
  Labels on physical volume "/dev/sdc1" successfully wiped

NOW

[root@datanalytics4 ~]# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda5  vg1  lvm2 a--  162.99G     0
  /dev/sdb5  vg2  lvm2 a--  230.97G 20.97G
[root@datanalytics4 ~]#

I see the the major chunk vg0 is gone ...however the file system still
exists in df -h

Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/vg2-data3          07G  188M  197G   1% /data3
/dev/mapper/vg1-data--sys     61G  188M  153G   1% /data-sys
*/dev/mapper/vg0-data             2.0T  1.4T  561G  71% /data*
*/dev/mapper/vg0-data1           2.0T 1001G  914G  53% /data1*
*/dev/mapper/vg0-data2           1.5T  1.2T  184G  87% /data2*

[root@datanalytics4 ~]# dmsetup ls
vg0-data2 (253, 4)
vg0-data1 (253, 2)
vg1-data--sys (253, 1)
vg0-data (253, 3)
vg2-data3 (253, 0)
[root@datanalytics4 ~]#


[root@datanalytics4 ~]# dmsetup -vvv status
dm version   OF   [16384]
dm names   OF   [16384]
dm status vg0-data2  OF   [16384]
Name:              vg0-data2
State:             ACTIVE
vg0-data2: read ahead is 256
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-nvyZYGRy1X5SmKvKhqdpOeb4DZVxsdFgUX0dltiRbAVlZEuS2onnkTcEcDguv4IN

0 3092381696 linear

dm status vg0-data1  OF   [16384]
Name:              vg0-data1
State:             ACTIVE
vg0-data1: read ahead is 256
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 2
Number of targets: 1
UUID: LVM-nvyZYGRy1X5SmKvKhqdpOeb4DZVxsdFgUWO0MJ5zoLA1Tgpa1wor3aAV35LItt46

0 4294967296 linear

dm status vg1-data--sys  OF   [16384]
Name:              vg1-data--sys
State:             ACTIVE
vg1-data--sys: read ahead is 256
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-gIeIufCQkSPf4iElhYM7c2Ja1OJZvycI55wsnecjG3VANNHAMz1BmThmiEIKh02b

0 341811200 linear

dm status vg0-data  OF   [16384]
Name:              vg0-data
State:             ACTIVE
vg0-data: read ahead is 256
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: LVM-nvyZYGRy1X5SmKvKhqdpOeb4DZVxsdFgQe2Q2DWZcxbwsPEaD2OsF7ZmgUsnCVfE

0 4294967296 linear

dm status vg2-data3  OF   [16384]
Name:              vg2-data3
State:             ACTIVE
vg2-data3: read ahead is 256
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-kCeA3BTSB2jeagki4MtPzswpZMmS2nuHXCMnuXWN9YdPlDuiLo03ebvuCMZTkzBr

0 440401920 linear

[root@datanalytics4 ~]#


as you can see above the file systems are there but i dont see the vg and
lv in the system now

[root@datanalytics4 ~]# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda5  vg1  lvm2 a--  162.99G     0
  /dev/sdb5  vg2  lvm2 a--  230.97G 20.97G
[root@datanalytics4 ~]# vgs -a
  VG   #PV #LV #SN Attr   VSize   VFree
  vg1    1   1   0 wz--n- 162.99G     0
  vg2    1   1   0 wz--n- 230.97G 20.97G
[root@datanalytics4 ~]# lvs -a
  LV       VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  data-sys vg1  -wi-ao 162.99G
  data3    vg2  -wi-ao 210.00G
[root@datanalytics4 ~]#


Did I screw up ?? is there a way to fix ? why does lvm not show the vg0 its
LV's but they exist in the system >?


On Tue, Jun 19, 2012 at 10:10 PM, Bryn M. Reeves <bmr@redhat.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05/28/2012 07:40 PM, tariq wali wrote:
> > I tried to fix this by but it just won't go.. vgreduce
> > --removemissing vg0 Found duplicate PV
> > XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1
> > Volume group "vg0" is already consistent
> >
> > why does LVM think that it's still using PV ( /dev/sdc1 ) when it
> > no more exists ? and how do i get rid of it ?
>
> Although you have removed the PV from the volume group it still has a
> PV label on it which has the same UUID value as some other valid
> device that is part of the volume group.
>
> You should verify which device is correct and then remove the PV label
> from the other device (or re-run pvcreate to give it a new random UUID):
>
> # pvremove /dev/sdc1
>
> Once the label is removed you should stop seeing those messages.
>
> Regards,
> Bryn.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk/gq5EACgkQ6YSQoMYUY97zTACdEn2l2OkwM/4naoqzZwJFDSiO
> WpYAnjSN8Uppia//lZK28FnhXvyDwK+Y
> =v7ST
> -----END PGP SIGNATURE-----
>



-- 
*Tariq Wali.*

[-- Attachment #2: Type: text/html, Size: 9926 bytes --]