* [linux-lvm] LVM2 recovery on IP-SAN after OS re-install @ 2012-05-28 18:40 tariq wali 2012-05-29 9:14 ` James Hawtin 2012-06-19 16:40 ` Bryn M. Reeves 0 siblings, 2 replies; 4+ messages in thread From: tariq wali @ 2012-05-28 18:40 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1: Type: text/plain, Size: 2471 bytes --] Hi, We had an OS crash on x86_64 CentOS , LVM2 and it had 6TB LUN via IP SCSI SAN mounted to it and after the OS reinstall on a crash we had hard time to mount the original SAN volume as you can imagine 6TB of marketing data with 3 LVM partitions as data data1 and data2 . After troubleshooting almost for a day I was able to get the LVM metadata using dd from the SAN LUN /dev/sdd1 which I used to recreate and restore all LVM partitions back to original state , we sure became the raving fans of LVM today ! however after all the recovery I see this annoying warning each time I execute any of the lvm commands .. lvs * Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not/dev/sdc1 * LV VG Attr LSize Origin Snap% Move Log Copy% Convert data vg0 -wi-ao 2.00T data1 vg0 -wi-ao 2.00T data2 vg0 -wi-ao 1.44T I believe previously we had created PV using /dev/sdd1 and /dev/sdc1 although from the same LUN but now I was able to restore everything using just /dev/sdd1 and everything looks intact (until I hear marketing whine tom:) vgs -o +devices Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 VG #PV #LV #SN Attr VSize VFree Devices vg0 1 3 0 wz--n- 5.45T 8.06G /dev/sdd1(0) vg0 1 3 0 wz--n- 5.45T 8.06G /dev/sdd1(524288) vg0 1 3 0 wz--n- 5.45T 8.06G /dev/sdd1(1048576) lvs -P -a -o +devices Partial mode. Incomplete logical volumes will be processed. Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices data vg0 -wi-ao 2.00T /dev/sdd1(0) data1 vg0 -wi-ao 2.00T /dev/sdd1(524288) data2 vg0 -wi-ao 1.44T /dev/sdd1(1048576) I tried to fix this by but it just won't go.. vgreduce --removemissing vg0 Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 Volume group "vg0" is already consistent why does LVM think that it's still using PV ( /dev/sdc1 ) when it no more exists ? and how do i get rid of it ? I also see some of these in syslog messages hfs: unable to find HFS+ superblock hfs: unable to find HFS+ superblock VFS: Can't find ext3 filesystem on dev sdc1. VFS: Can't find ext3 filesystem on dev sdc1. hfs: unable to find HFS+ superblock VFS: Can't find ext3 filesystem on dev sdc1. VFS: Can't find ext3 filesystem on dev sdc1. VFS: Can't find ext3 filesystem on dev sdd1. thx, Tariq [-- Attachment #2: Type: text/html, Size: 3487 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [linux-lvm] LVM2 recovery on IP-SAN after OS re-install 2012-05-28 18:40 [linux-lvm] LVM2 recovery on IP-SAN after OS re-install tariq wali @ 2012-05-29 9:14 ` James Hawtin 2012-06-19 16:40 ` Bryn M. Reeves 1 sibling, 0 replies; 4+ messages in thread From: James Hawtin @ 2012-05-29 9:14 UTC (permalink / raw) To: LVM general discussion and development [-- Attachment #1: Type: text/plain, Size: 1345 bytes --] On 28/05/12 19:40, tariq wali wrote: > Hi, > We had an OS crash on x86_64 CentOS , LVM2 and it had 6TB LUN via > IP SCSI SAN mounted to it and after the OS reinstall on a crash we > had hard time to mount the original SAN volume as you can imagine 6TB > of marketing data with 3 LVM partitions as data data1 and data2 . > After troubleshooting almost for a day I was able to get the LVM > metadata using dd from the SAN LUN /dev/sdd1 which I used to recreate > and restore all LVM partitions back to original state , we sure became > the raving fans of LVM today ! however after all the recovery I see > this annoying warning each time I execute any of the lvm commands .. > > lvs > * Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using > /dev/sdd1 not/dev/sdc1* > LV VG Attr LSize Origin Snap% Move Log Copy% Convert > data vg0 -wi-ao 2.00T > data1 vg0 -wi-ao 2.00T > data2 vg0 -wi-ao 1.44T > 1) The the same lun presented to the computer more than once? If this is a deliberate thing for redundancy then your need to setup multipath to take advantage use the devices in /dev/mapper of this and mask off the raw /dev/sd? devices in your /etc/lvm/lvm.conf 2) Did you copy all the data from one disk to another? if so is the old disk still available to the system? 3) Check you lun masking. James [-- Attachment #2: Type: text/html, Size: 2698 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [linux-lvm] LVM2 recovery on IP-SAN after OS re-install 2012-05-28 18:40 [linux-lvm] LVM2 recovery on IP-SAN after OS re-install tariq wali 2012-05-29 9:14 ` James Hawtin @ 2012-06-19 16:40 ` Bryn M. Reeves 2012-06-20 10:06 ` tariq wali 1 sibling, 1 reply; 4+ messages in thread From: Bryn M. Reeves @ 2012-06-19 16:40 UTC (permalink / raw) To: LVM general discussion and development; +Cc: tariq wali -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/28/2012 07:40 PM, tariq wali wrote: > I tried to fix this by but it just won't go.. vgreduce > --removemissing vg0 Found duplicate PV > XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 > Volume group "vg0" is already consistent > > why does LVM think that it's still using PV ( /dev/sdc1 ) when it > no more exists ? and how do i get rid of it ? Although you have removed the PV from the volume group it still has a PV label on it which has the same UUID value as some other valid device that is part of the volume group. You should verify which device is correct and then remove the PV label from the other device (or re-run pvcreate to give it a new random UUID): # pvremove /dev/sdc1 Once the label is removed you should stop seeing those messages. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/gq5EACgkQ6YSQoMYUY97zTACdEn2l2OkwM/4naoqzZwJFDSiO WpYAnjSN8Uppia//lZK28FnhXvyDwK+Y =v7ST -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [linux-lvm] LVM2 recovery on IP-SAN after OS re-install 2012-06-19 16:40 ` Bryn M. Reeves @ 2012-06-20 10:06 ` tariq wali 0 siblings, 0 replies; 4+ messages in thread From: tariq wali @ 2012-06-20 10:06 UTC (permalink / raw) To: Bryn M. Reeves; +Cc: LVM general discussion and development [-- Attachment #1: Type: text/plain, Size: 6711 bytes --] *Bryn,* * * * * *Thanks for the mail but I might have just created a havoc by force removing the PV* Here is a series of commands and output .. [root@datanalytics4 ~]# pvs Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not * /dev/sdc1* PV VG Fmt Attr PSize PFree /dev/sda5 vg1 lvm2 a-- 162.99G 0 /dev/sdb5 vg2 lvm2 a-- 230.97G 20.97G /dev/PV VG Fmt Attr PSize PFree [root@datanalytics4 ~]# [root@datanalytics4 ~]# pvdisplay Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 --- Physical volume --- PV Name /dev/sdd1 VG Name vg0 PV Size 5.45 TB / not usable 1.14 MB Allocatable yes PE Size (KByte) 4096 Total PE 1428127 Free PE 2063 Allocated PE 1426064 PV UUID XtGmeF-OiCT-WpLB-ckKZ-KlGt-YKPV-dBgVJn --- Physical volume --- PV Name /dev/sdb5 VG Name vg2 PV Size 230.98 GB / not usable 3.41 MB Allocatable yes PE Size (KByte) 4096 Total PE 59129 Free PE 5369 Allocated PE 53760 PV UUID UP9bH6-Huaf-02ua-tsL0-lfcU-axc3-CIkSvS --- Physical volume --- PV Name /dev/sda5 VG Name vg1 PV Size 162.99 GB / not usable 1.83 MB Allocatable yes (but full) PE Size (KByte) 4096 Total PE 41725 Free PE 0 Allocated PE 41725 PV UUID 9DSyjC-fvzX-wMTa-7NNG-JOzZ-Tu3F-utmpFT [root@datanalytics4 ~]# [root@datanalytics4 ~]# pvremove --force --force /dev/sdc1 Found duplicate PV XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 Really WIPE LABELS from physical volume "/dev/sdc1" of volume group "vg0" [y/n]? y WARNING: Wiping physical volume label from /dev/sdc1 of volume group "vg0" Labels on physical volume "/dev/sdc1" successfully wiped NOW [root@datanalytics4 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda5 vg1 lvm2 a-- 162.99G 0 /dev/sdb5 vg2 lvm2 a-- 230.97G 20.97G [root@datanalytics4 ~]# I see the the major chunk vg0 is gone ...however the file system still exists in df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg2-data3 07G 188M 197G 1% /data3 /dev/mapper/vg1-data--sys 61G 188M 153G 1% /data-sys */dev/mapper/vg0-data 2.0T 1.4T 561G 71% /data* */dev/mapper/vg0-data1 2.0T 1001G 914G 53% /data1* */dev/mapper/vg0-data2 1.5T 1.2T 184G 87% /data2* [root@datanalytics4 ~]# dmsetup ls vg0-data2 (253, 4) vg0-data1 (253, 2) vg1-data--sys (253, 1) vg0-data (253, 3) vg2-data3 (253, 0) [root@datanalytics4 ~]# [root@datanalytics4 ~]# dmsetup -vvv status dm version OF [16384] dm names OF [16384] dm status vg0-data2 OF [16384] Name: vg0-data2 State: ACTIVE vg0-data2: read ahead is 256 Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 4 Number of targets: 1 UUID: LVM-nvyZYGRy1X5SmKvKhqdpOeb4DZVxsdFgUX0dltiRbAVlZEuS2onnkTcEcDguv4IN 0 3092381696 linear dm status vg0-data1 OF [16384] Name: vg0-data1 State: ACTIVE vg0-data1: read ahead is 256 Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 2 Number of targets: 1 UUID: LVM-nvyZYGRy1X5SmKvKhqdpOeb4DZVxsdFgUWO0MJ5zoLA1Tgpa1wor3aAV35LItt46 0 4294967296 linear dm status vg1-data--sys OF [16384] Name: vg1-data--sys State: ACTIVE vg1-data--sys: read ahead is 256 Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 1 Number of targets: 1 UUID: LVM-gIeIufCQkSPf4iElhYM7c2Ja1OJZvycI55wsnecjG3VANNHAMz1BmThmiEIKh02b 0 341811200 linear dm status vg0-data OF [16384] Name: vg0-data State: ACTIVE vg0-data: read ahead is 256 Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 3 Number of targets: 1 UUID: LVM-nvyZYGRy1X5SmKvKhqdpOeb4DZVxsdFgQe2Q2DWZcxbwsPEaD2OsF7ZmgUsnCVfE 0 4294967296 linear dm status vg2-data3 OF [16384] Name: vg2-data3 State: ACTIVE vg2-data3: read ahead is 256 Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 0 Number of targets: 1 UUID: LVM-kCeA3BTSB2jeagki4MtPzswpZMmS2nuHXCMnuXWN9YdPlDuiLo03ebvuCMZTkzBr 0 440401920 linear [root@datanalytics4 ~]# as you can see above the file systems are there but i dont see the vg and lv in the system now [root@datanalytics4 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda5 vg1 lvm2 a-- 162.99G 0 /dev/sdb5 vg2 lvm2 a-- 230.97G 20.97G [root@datanalytics4 ~]# vgs -a VG #PV #LV #SN Attr VSize VFree vg1 1 1 0 wz--n- 162.99G 0 vg2 1 1 0 wz--n- 230.97G 20.97G [root@datanalytics4 ~]# lvs -a LV VG Attr LSize Origin Snap% Move Log Copy% Convert data-sys vg1 -wi-ao 162.99G data3 vg2 -wi-ao 210.00G [root@datanalytics4 ~]# Did I screw up ?? is there a way to fix ? why does lvm not show the vg0 its LV's but they exist in the system >? On Tue, Jun 19, 2012 at 10:10 PM, Bryn M. Reeves <bmr@redhat.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 05/28/2012 07:40 PM, tariq wali wrote: > > I tried to fix this by but it just won't go.. vgreduce > > --removemissing vg0 Found duplicate PV > > XtGmeFOiCTWpLBckKZKlGtYKPVdBgVJn: using /dev/sdd1 not /dev/sdc1 > > Volume group "vg0" is already consistent > > > > why does LVM think that it's still using PV ( /dev/sdc1 ) when it > > no more exists ? and how do i get rid of it ? > > Although you have removed the PV from the volume group it still has a > PV label on it which has the same UUID value as some other valid > device that is part of the volume group. > > You should verify which device is correct and then remove the PV label > from the other device (or re-run pvcreate to give it a new random UUID): > > # pvremove /dev/sdc1 > > Once the label is removed you should stop seeing those messages. > > Regards, > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk/gq5EACgkQ6YSQoMYUY97zTACdEn2l2OkwM/4naoqzZwJFDSiO > WpYAnjSN8Uppia//lZK28FnhXvyDwK+Y > =v7ST > -----END PGP SIGNATURE----- > -- *Tariq Wali.* [-- Attachment #2: Type: text/html, Size: 9926 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-06-20 10:06 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-05-28 18:40 [linux-lvm] LVM2 recovery on IP-SAN after OS re-install tariq wali 2012-05-29 9:14 ` James Hawtin 2012-06-19 16:40 ` Bryn M. Reeves 2012-06-20 10:06 ` tariq wali
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).