[linux-lvm] how to swap a bad drive on a non-standard mirror

[linux-lvm] how to swap a bad drive on a non-standard mirror

[Randy Schultz]

Hey all,

I have a system that somebody set up, reportedly during the RHEL install, to have the 2 on-board drives
mirrored.  We lost the second drive.  I know how to do deal with this on a standard setup with 2 PV's, but it
looks like there is only 1 PV and I cannot find any documentation on how to deal with this.

The system has 2 drives partitioned this way:
      ? fdisk -l

      Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
      255 heads, 63 sectors/track, 121601 cylinders
      Units = cylinders of 16065 * 512 = 8225280 bytes

         Device Boot      Start         End      Blocks   Id  System
      /dev/sda1   *           1          13      104391   83  Linux
      /dev/sda2              14      121600   976647577+  8e  Linux LVM

      Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
      255 heads, 63 sectors/track, 121601 cylinders
      Units = cylinders of 16065 * 512 = 8225280 bytes

         Device Boot      Start         End      Blocks   Id  System
      /dev/sdb1   *           1          13      104391   83  Linux
      /dev/sdb2              14      121600   976647577+  8e  Linux LVM


The fstab mounts things as:
      ? cat /etc/fstab
      /dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
      /dev/VolGroup00/LogVol02 /var                    ext3    defaults        1 2
      /dev/VolGroup00/LogVol03 /opt                    ext3    defaults        1 2
      LABEL=/boot             /boot                   ext3    defaults        1 2
      tmpfs                   /dev/shm                tmpfs   defaults        0 0
      devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
      sysfs                   /sys                    sysfs   defaults        0 0
      proc                    /proc                   proc    defaults        0 0
      /dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0


giving these mountpoints:
      ? df -h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VolGroup00-LogVol00
                             24G  9.1G   14G  41% /
      /dev/mapper/VolGroup00-LogVol02
                             24G  555M   22G   3% /var
      /dev/mapper/VolGroup00-LogVol03
                            770G   45G  686G   7% /opt
      /dev/mapper/isw_ccebcbejfi_Volume0p1
                             99M   26M   69M  28% /boot
      tmpfs                  14G     0   14G   0% /dev/shm

There are no md devices:
      ? cat /proc/mdstat
      Personalities :
      unused devices: <none>


DMsetup shows:
      ? dmsetup status
      isw_ccebcbejfi_Volume0p2: 0 1953295155 linear
      isw_ccebcbejfi_Volume0p1: 0 208782 linear
      VolGroup00-LogVol03: 0 1666580480 linear
      VolGroup00-LogVol02: 0 51183616 linear
      VolGroup00-LogVol01: 0 184287232 linear
      VolGroup00-LogVol00: 0 51183616 linear
      isw_ccebcbejfi_Volume0: 0 1953519352 mirror 2 8:0 8:16 14905/14905 1 AR 1 core


Here's where things get weird.  Pvscan shows only 1 device:
      ? pvscan -v
          Wiping cache of LVM-capable devices
          Wiping internal VG cache
          Walking through all physical volumes
        PV /dev/mapper/isw_ccebcbejfi_Volume0p2   VG VolGroup00   lvm2 [931.38 GB / 0    free]
        Total: 1 [931.38 GB] / in use: 1 [931.38 GB] / in no VG: 0 [0   ]


So, if I understand this correctly, sda2 and sdb2 were tied together somehow as a mirror, then this was
presented to lvm as the PV.  Cool, but how do I swap out sdb?!

--
  Randy    (schulra@earlham.edu)      765.983.1283         <*>

nosce te ipsum

Re: [linux-lvm] how to swap a bad drive on a non-standard mirror

[Scott Merrilees]

[-- Attachment #1: Type: text/plain, Size: 4143 bytes --]

Looks like a dmraid setup.  Googling Linux isw found me a rebuild guide.
On 06/11/2012 9:05 AM, "Randy Schultz" <schulra@earlham.edu> wrote:

> Hey all,
>
> I have a system that somebody set up, reportedly during the RHEL install,
> to have the 2 on-board drives
> mirrored.  We lost the second drive.  I know how to do deal with this on a
> standard setup with 2 PV's, but it
> looks like there is only 1 PV and I cannot find any documentation on how
> to deal with this.
>
> The system has 2 drives partitioned this way:
>      ? fdisk -l
>
>      Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
>      255 heads, 63 sectors/track, 121601 cylinders
>      Units = cylinders of 16065 * 512 = 8225280 bytes
>
>         Device Boot      Start         End      Blocks   Id  System
>      /dev/sda1   *           1          13      104391   83  Linux
>      /dev/sda2              14      121600   976647577+  8e  Linux LVM
>
>      Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
>      255 heads, 63 sectors/track, 121601 cylinders
>      Units = cylinders of 16065 * 512 = 8225280 bytes
>
>         Device Boot      Start         End      Blocks   Id  System
>      /dev/sdb1   *           1          13      104391   83  Linux
>      /dev/sdb2              14      121600   976647577+  8e  Linux LVM
>
>
> The fstab mounts things as:
>      ? cat /etc/fstab
>      /dev/VolGroup00/LogVol00 /                       ext3    defaults
>    1 1
>      /dev/VolGroup00/LogVol02 /var                    ext3    defaults
>    1 2
>      /dev/VolGroup00/LogVol03 /opt                    ext3    defaults
>    1 2
>      LABEL=/boot             /boot                   ext3    defaults
>    1 2
>      tmpfs                   /dev/shm                tmpfs   defaults
>    0 0
>      devpts                  /dev/pts                devpts
>  gid=5,mode=620  0 0
>      sysfs                   /sys                    sysfs   defaults
>    0 0
>      proc                    /proc                   proc    defaults
>    0 0
>      /dev/VolGroup00/LogVol01 swap                    swap    defaults
>    0 0
>
>
> giving these mountpoints:
>      ? df -h
>      Filesystem            Size  Used Avail Use% Mounted on
>      /dev/mapper/VolGroup00-**LogVol00
>                             24G  9.1G   14G  41% /
>      /dev/mapper/VolGroup00-**LogVol02
>                             24G  555M   22G   3% /var
>      /dev/mapper/VolGroup00-**LogVol03
>                            770G   45G  686G   7% /opt
>      /dev/mapper/isw_ccebcbejfi_**Volume0p1
>                             99M   26M   69M  28% /boot
>      tmpfs                  14G     0   14G   0% /dev/shm
>
> There are no md devices:
>      ? cat /proc/mdstat
>      Personalities :
>      unused devices: <none>
>
>
> DMsetup shows:
>      ? dmsetup status
>      isw_ccebcbejfi_Volume0p2: 0 1953295155 linear
>      isw_ccebcbejfi_Volume0p1: 0 208782 linear
>      VolGroup00-LogVol03: 0 1666580480 linear
>      VolGroup00-LogVol02: 0 51183616 linear
>      VolGroup00-LogVol01: 0 184287232 linear
>      VolGroup00-LogVol00: 0 51183616 linear
>      isw_ccebcbejfi_Volume0: 0 1953519352 mirror 2 8:0 8:16 14905/14905 1
> AR 1 core
>
>
> Here's where things get weird.  Pvscan shows only 1 device:
>      ? pvscan -v
>          Wiping cache of LVM-capable devices
>          Wiping internal VG cache
>          Walking through all physical volumes
>        PV /dev/mapper/isw_ccebcbejfi_**Volume0p2   VG VolGroup00   lvm2
> [931.38 GB / 0    free]
>        Total: 1 [931.38 GB] / in use: 1 [931.38 GB] / in no VG: 0 [0   ]
>
>
> So, if I understand this correctly, sda2 and sdb2 were tied together
> somehow as a mirror, then this was
> presented to lvm as the PV.  Cool, but how do I swap out sdb?!
>
> --
>  Randy    (schulra@earlham.edu)      765.983.1283         <*>
>
> nosce te ipsum
>
> ______________________________**_________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/**mailman/listinfo/linux-lvm<https://www.redhat.com/mailman/listinfo/linux-lvm>
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-**HOWTO/<http://tldp.org/HOWTO/LVM-HOWTO/>
>

[-- Attachment #2: Type: text/html, Size: 4916 bytes --]

Re: [linux-lvm] how to swap a bad drive on a non-standard mirror

[Stuart D Gathman]

Long ago, Nostradamus foresaw that on 11/05/2012 04:59 PM, Randy Schultz
would write:
> Hey all,
>
> I have a system that somebody set up, reportedly during the RHEL
> install, to have the 2 on-board drives
> mirrored.  We lost the second drive.  I know how to do deal with this
> on a standard setup with 2 PV's, but it
> looks like there is only 1 PV and I cannot find any documentation on
> how to deal with this.
>
>
> DMsetup shows:
>      ? dmsetup status
>      isw_ccebcbejfi_Volume0p2: 0 1953295155 linear
>      isw_ccebcbejfi_Volume0p1: 0 208782 linear
>      VolGroup00-LogVol03: 0 1666580480 linear
>      VolGroup00-LogVol02: 0 51183616 linear
>      VolGroup00-LogVol01: 0 184287232 linear
>      VolGroup00-LogVol00: 0 51183616 linear
>      isw_ccebcbejfi_Volume0: 0 1953519352 mirror 2 8:0 8:16
> 14905/14905 1 AR 1 core
>
>
> Here's where things get weird.  Pvscan shows only 1 device:
>      ? pvscan -v
>          Wiping cache of LVM-capable devices
>          Wiping internal VG cache
>          Walking through all physical volumes
>        PV /dev/mapper/isw_ccebcbejfi_Volume0p2   VG VolGroup00   lvm2
> [931.38 GB / 0    free]
>        Total: 1 [931.38 GB] / in use: 1 [931.38 GB] / in no VG: 0 [0   ]
>
>
> So, if I understand this correctly, sda2 and sdb2 were tied together
> somehow as a mirror, then this was
> presented to lvm as the PV.  Cool, but how do I swap out sdb?!
This has nothing to do with LVM, but google is your friend ("isw
mirror").  This is Intel Raid, and dmsetup cannot rebuild it.  There
might be some Intel tool to do so, but I would highly recommend
converting to standard software raid (mdadm).

The steps would be (untested, of course):

1) shut down system and boot from live media - you can't snapshot a VG
2) attach the new disk in some way that the Intel raid won't be using
it.  Maybe via USB.
3) Create software RAID partitions on the new disk.  They will need to
be slightly larger to hold the md metadata.
4) Start mdadm for the new partitions.  Check that the logical size of
the md devices is >= your isw partitions.
5) copy the PV (sda2) to new md device.   Copy the boot partition (sda1)
also.
6) remove sda and replace with sdb (keep sda safe in case you need to
start over), boot from SuperGrub or live media to install grub
7) Boot from converted system.  You might need to chroot from live media
and rebuild initramfs to include md driver.  The VG will be recognized
by UUID, even though it is now on a different device.
8) When You are sure everything is working well, add old sda as second
disk and add partitions as mirrors for md raid.  It might be safer to
just buy another disk if you haven't figured out how to disable the
Intel raid.

Re: [linux-lvm] how to swap a bad drive on a non-standard mirror

[Stuart D Gathman]

Long ago, Nostradamus foresaw that on 11/05/2012 04:59 PM, Randy Schultz
would write:
> Hey all,
>
> I have a system that somebody set up, reportedly during the RHEL
> install, to have the 2 on-board drives
> mirrored.  We lost the second drive.  I know how to do deal with this
> on a standard setup with 2 PV's, but it
> looks like there is only 1 PV and I cannot find any documentation on
> how to deal with this.
>
In case you decide to stick with the Intel raid, this is someone who
figured out how to rebuild it:


http://techie.org/Blog/2010/09/03/how-to-rebuild-intel-raid-isw-on-linux/

Re: [linux-lvm] how to swap a bad drive on a non-standard mirror

[Randy Schultz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 391 bytes --]

On Tue, 6 Nov 2012, Scott Merrilees spaketh thusly:

-}
-}Looks like a dmraid setup.  Googling Linux isw found me a rebuild guide.

Tnx much Scott and Stuart.  It seems being told they were set with LVM, I
fixated on it and didn't think to look deeper (even though the evidence was
smacking me in the face ;).

--
 Randy    (schulra@earlham.edu)      765.983.1283         <*>

nosce te ipsum

[-- Attachment #2: Type: TEXT/PLAIN, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/