Create software RAID from active partition

Create software RAID from active partition

[Michael Guyver]

Hi there,

I've got a question about creating a RAID-1 array on a remote server -
ie: if the operation fails, it's going to be very expensive. The
server has two 200 GB drives and during a hurried re-install of CentOS
5.2 the creation of software RAID partitions was omitted. This means
that the array would include the currently active partition on which
the kernel is installed. So my first question is as to the feasibility
of this operation, and its safety: any comments?

The following may give an insight into the current setup should you
need it to answer my question more accurately.

-------------------------------------------------------------
# fdisk -l
Disk /dev/sda: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14       24773   198884700   8e  Linux LVM

Disk /dev/sdb: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   83  Linux
/dev/sdb2              14       24773   198884700   8e  Linux LVM
-------------------------------------------------------------
# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
-------------------------------------------------------------
# pvdisplay
  Found duplicate PV g7ZWtzNQcHx2PMQghP0NBHDXuYcaYqAt: using /dev/sdb2
not /dev/sda2
  --- Physical volume ---
  PV Name               /dev/sdb2
  VG Name               VolGroup00
  PV Size               189.67 GB / not usable 15.34 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              6069
  Free PE               0
  Allocated PE          6069
  PV UUID               g7ZWtz-NQcH-x2PM-QghP-0NBH-DXuY-caYqAt
-------------------------------------------------------------
# lvdisplay
  Found duplicate PV g7ZWtzNQcHx2PMQghP0NBHDXuYcaYqAt: using /dev/sdb2
not /dev/sda2
  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol00
  VG Name                VolGroup00
  LV UUID                rvPZJS-6Z7a-kXzk-aLcM-vv13-eRCK-kjg6I1
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                187.72 GB
  Current LE             6007
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol01
  VG Name                VolGroup00
  LV UUID                zvxDsa-MZXn-akSA-DlzC-49IX-65Fo-HPBuyJ
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.94 GB
  Current LE             62
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

-------------------------------------------------------------

Judging from the "Found duplicate " messages produced by pvdisplay and
lvdisplay, as well as the mount output, it seems that the root
partition is being loaded from /dev/sdb2. What /dev/sda2 is doing
right now is, I guess, completely sweet FA.

Can anyone point me to the way of finding out a file's physical
location on disc so that I can verify this is the case? So, for
example, I would like to check that my latest edit to ~/somefile.txt
is in fact on /dev/sdb1 at location xyz and that can be verified by
using dd to copy those bytes to a file in /tmp.

Having started reading the docs related to creating a RAID device, it
seems likely that the order of the listed devices is significant when
the array is initialised. However, I haven't yet been able to confirm
that were I to write

mdadm -C /dev/md0 --level raid1 --raid-disks 2 /dev/sdb1 /dev/sda1

that it would start to copy data from sdb1 to sda1 - or have I
misunderstood the initialisation process?

These questions may not seem very well framed, but some initial
guidance while I'm still reading into the problem would be
appreciated.

Best wishes

Michael

Re: Create software RAID from active partition

[Steve Fairbairn]

Michael Guyver wrote:

> I've got a question about creating a RAID-1 array on a remote server -
> ie: if the operation fails, it's going to be very expensive. The
> server has two 200 GB drives and during a hurried re-install of CentOS
> 5.2 the creation of software RAID partitions was omitted. This means
> that the array would include the currently active partition on which
> the kernel is installed. So my first question is as to the feasibility
> of this operation, and its safety: any comments?
> 

That would imply that one of the disks is currently doing nothing, which 
would make it feasible as far as I can see.

> # pvdisplay
>   Found duplicate PV g7ZWtzNQcHx2PMQghP0NBHDXuYcaYqAt: using /dev/sdb2
> not /dev/sda2
> 

This would seem to say to me that it's using sdb for all data currently, 
but...

> 
> Can anyone point me to the way of finding out a file's physical
> location on disc so that I can verify this is the case? So, for
> example, I would like to check that my latest edit to ~/somefile.txt
> is in fact on /dev/sdb1 at location xyz and that can be verified by
> using dd to copy those bytes to a file in /tmp.
> 

I can't help you here as I never bother with LVM, so I've not idea how 
to work out which physical device the mounted LVM is on.

> Having started reading the docs related to creating a RAID device, it
> seems likely that the order of the listed devices is significant when
> the array is initialised. However, I haven't yet been able to confirm
> that were I to write
> 
> mdadm -C /dev/md0 --level raid1 --raid-disks 2 /dev/sdb1 /dev/sda1
> 
> that it would start to copy data from sdb1 to sda1 - or have I
> misunderstood the initialisation process?
>

Please accept my standard disclaimer of 'I'm no expert, and I may be 
wrong.'...

I don't believe you'd want to do this.  What I think you'd want to do 
instead is create a degraded RAID 1 array using just the currently 
unused disk, then install LVM and a filesystem on that array, then copy 
all your data across.

Make sure you install a boot loader in the boot block of the disk you've 
made part of the array, and do whatever else you can to ensure the 
system next boots off the new md device.

Reboot, and then ensure you really are using the md device for your 
mounted filesystems...

Once you are certain, add the now unused drive into your RAID 1 array, 
and the replication should start.

> These questions may not seem very well framed, but some initial
> guidance while I'm still reading into the problem would be
> appreciated.
> 

Others will be able to give you more specific answers.  I unfortunately 
don't have linux in front of me, so can't check out the required mdadm 
incantations.


Hope this helps a little,

Steve.

Re: Create software RAID from active partition

[Alan Jenkins]

Michael Guyver wrote:
> Hi there,
>
> I've got a question about creating a RAID-1 array on a remote server -
> ie: if the operation fails, it's going to be very expensive. The
> server has two 200 GB drives and during a hurried re-install of CentOS
> 5.2 the creation of software RAID partitions was omitted. This means
> that the array would include the currently active partition on which
> the kernel is installed. So my first question is as to the feasibility
> of this operation, and its safety: any comments?
>
> The following may give an insight into the current setup should you
> need it to answer my question more accurately.
>
> -------------------------------------------------------------
> # fdisk -l
> Disk /dev/sda: 203.9 GB, 203928109056 bytes
> 255 heads, 63 sectors/track, 24792 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
>
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sda1   *           1          13      104391   83  Linux
> /dev/sda2              14       24773   198884700   8e  Linux LVM
>
> Disk /dev/sdb: 203.9 GB, 203928109056 bytes
> 255 heads, 63 sectors/track, 24792 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
>
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sdb1   *           1          13      104391   83  Linux
> /dev/sdb2              14       24773   198884700   8e  Linux LVM
> -------------------------------------------------------------
> # mount
> /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> /dev/sda1 on /boot type ext3 (rw)
> tmpfs on /dev/shm type tmpfs (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> -------------------------------------------------------------
> # pvdisplay
>   Found duplicate PV g7ZWtzNQcHx2PMQghP0NBHDXuYcaYqAt: using /dev/sdb2
> not /dev/sda2
>   --- Physical volume ---
>   PV Name               /dev/sdb2
>   VG Name               VolGroup00
>   PV Size               189.67 GB / not usable 15.34 MB
>   Allocatable           yes (but full)
>   PE Size (KByte)       32768
>   Total PE              6069
>   Free PE               0
>   Allocated PE          6069
>   PV UUID               g7ZWtz-NQcH-x2PM-QghP-0NBH-DXuY-caYqAt
> -------------------------------------------------------------
> # lvdisplay
>   Found duplicate PV g7ZWtzNQcHx2PMQghP0NBHDXuYcaYqAt: using /dev/sdb2
> not /dev/sda2
>   --- Logical volume ---
>   LV Name                /dev/VolGroup00/LogVol00
>   VG Name                VolGroup00
>   LV UUID                rvPZJS-6Z7a-kXzk-aLcM-vv13-eRCK-kjg6I1
>   LV Write Access        read/write
>   LV Status              available
>   # open                 1
>   LV Size                187.72 GB
>   Current LE             6007
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     auto
>   - currently set to     256
>   Block device           253:0
>
>   --- Logical volume ---
>   LV Name                /dev/VolGroup00/LogVol01
>   VG Name                VolGroup00
>   LV UUID                zvxDsa-MZXn-akSA-DlzC-49IX-65Fo-HPBuyJ
>   LV Write Access        read/write
>   LV Status              available
>   # open                 1
>   LV Size                1.94 GB
>   Current LE             62
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     auto
>   - currently set to     256
>   Block device           253:1
>
> -------------------------------------------------------------
>
> Judging from the "Found duplicate " messages produced by pvdisplay and
> lvdisplay, as well as the mount output, it seems that the root
> partition is being loaded from /dev/sdb2. What /dev/sda2 is doing
> right now is, I guess, completely sweet FA.
>   

Hmm.  It looks like it was set up as some sort of RAID1, hence the
duplicate PVs, but it is no longer using it.

> Can anyone point me to the way of finding out a file's physical
> location on disc so that I can verify this is the case? So, for
> example, I would like to check that my latest edit to ~/somefile.txt
> is in fact on /dev/sdb1 at location xyz and that can be verified by
> using dd to copy those bytes to a file in /tmp.
>
>   

swap_offset program from the uswsusp package?  I'd trust pvdisplay that
you're only using sdb2 though, and not sda2.

> Having started reading the docs related to creating a RAID device, it
> seems likely that the order of the listed devices is significant when
> the array is initialised. However, I haven't yet been able to confirm
> that were I to write
>
> mdadm -C /dev/md0 --level raid1 --raid-disks 2 /dev/sdb1 /dev/sda1
>
> that it would start to copy data from sdb1 to sda1 - or have I
> misunderstood the initialisation process?
>
> These questions may not seem very well framed, but some initial
> guidance while I'm still reading into the problem would be
> appreciated.
>
> Best wishes
>
> Michael
>   
a) For safety, the trick would be to do it in stages.  I've done this
locally to add RAID to my existing desktop.

Create a 1-device RAID1 (you can do this, though you have to force it)
on the *unused* drive.  Copy your data into the RAID device (e.g. using
dd).  Get it to the point where it's bootable on it's own.  Boot into it
and check it works (Use the BIOS boot drive selection.  You do have
remote BIOS access, right?).  Then grow the RAID device by adding the
other disk - overwriting the old contents.

That's one more copy than strictly needed, but it's worth it for peace
of mind.

b) Also, it sounds like you're missing a piece of knowledge.  (Sane)
software RAID requires a RAID superblock at the end of the device for
identification.  That means you can't take a non-RAID disk and turn it
into a RAID disk while leaving the drive unchanged.  You would have to
shrink the LVM physical volume.  With the stepwise approach you may be
able to avoid shrinking in-place - which could be risky.  E.g. shrink
the  boot partition slightly instead before copying.

c) Thinking of the boot partition, you also need to make that RAID1 in
the same way.  Unlike LVM, RAID1 is transparent to GRUB (because GRUB
doesn't write to the filesystem).  But it's recommended you do the
individual partitions separately, and don't try to do whole-disk RAID.

Don't forget you've got /boot on sda1 - that's going to be a confusing
problem whatever you do.



Alan

Re: Create software RAID from active partition

[David Greaves]

Michael Guyver wrote:
> Hi there,
> 
> I've got a question about creating a RAID-1 array on a remote server -
> ie: if the operation fails, it's going to be very expensive. The
> server has two 200 GB drives and during a hurried re-install of CentOS
> 5.2 the creation of software RAID partitions was omitted. This means
> that the array would include the currently active partition on which
> the kernel is installed. So my first question is as to the feasibility
> of this operation, and its safety: any comments?

It looks to me like you have a system that is (almost) setup to use lvm
mirroring, not md (raid) mirroring.

Are you sure you want to swap to using md mirroring? Or do you want to restore
the lvm mirror?

> # mount
> /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> /dev/sda1 on /boot type ext3 (rw)
> tmpfs on /dev/shm type tmpfs (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

So there are no issues mirroring the / fs.
Simply make the /dev/mapper/VolGroup00-LogVol00 mirrored
lvconvert -m or something.

Bear in mind that /boot and possibly the mbr won't be replicated so if a disk
fails you may not lose your rootfs but you won't be booting either...

> Judging from the "Found duplicate " messages produced by pvdisplay and
> lvdisplay, as well as the mount output, it seems that the root
> partition is being loaded from /dev/sdb2. What /dev/sda2 is doing
> right now is, I guess, completely sweet FA.
> 
> Can anyone point me to the way of finding out a file's physical
> location on disc so that I can verify this is the case? So, for
> example, I would like to check that my latest edit to ~/somefile.txt
> is in fact on /dev/sdb1 at location xyz and that can be verified by
> using dd to copy those bytes to a file in /tmp.
This is a red herring.
Unless you are into block level data recovery in which case go to the
smartmontools pages - they have a solution for this for ext3. Once you start to
introduce raid/lvm/other fs then you are in a world of confusion.


> Having started reading the docs related to creating a RAID device, it
> seems likely that the order of the listed devices is significant when
> the array is initialised. However, I haven't yet been able to confirm
> that were I to write
> 
> mdadm -C /dev/md0 --level raid1 --raid-disks 2 /dev/sdb1 /dev/sda1
> 
> that it would start to copy data from sdb1 to sda1 - or have I
> misunderstood the initialisation process?
Yes.
Initialising will likely destroy data unless you are careful about superblock
locations.
Even then you'd need to create the array degraded and grow it.

I'd create the array in degraded mode using the blank disk and then copy the
data and test boot (retaining the ability to boot the old system)
Then, once it worked I'd wipe add the old disk in.

David

Re: Create software RAID from active partition

[Michael Guyver]

2008/9/1 David Greaves <david@dgreaves.com>:
> Michael Guyver wrote:
>> Hi there,
>>
>> I've got a question about creating a RAID-1 array on a remote server -
>> ie: if the operation fails, it's going to be very expensive. The
>> server has two 200 GB drives and during a hurried re-install of CentOS
>> 5.2 the creation of software RAID partitions was omitted. This means
>> that the array would include the currently active partition on which
>> the kernel is installed. So my first question is as to the feasibility
>> of this operation, and its safety: any comments?
>
> It looks to me like you have a system that is (almost) setup to use lvm
> mirroring, not md (raid) mirroring.
>
> Are you sure you want to swap to using md mirroring? Or do you want to restore
> the lvm mirror?
>

Well, I'm happy to take the path of least resistance but which would
also leave me with redundancy in case of a disk failure.

> So there are no issues mirroring the / fs.
> Simply make the /dev/mapper/VolGroup00-LogVol00 mirrored
> lvconvert -m or something.

I'll have a read of the lvconvert docs and

> Bear in mind that /boot and possibly the mbr won't be replicated so if a disk
> fails you may not lose your rootfs but you won't be booting either...

Thanks very much David, Steve and Allen for your helpful comments.
I'll do some reading about LVM mirroring and if it's as easy as it
appears may go down that route as it sounds slightly less hairy than
setting up software RAID on a server to which I have no physical
access. The costs of having someone at the data-centre restore the
server are prohibitive and I'm terrified of bricking the server (again
- long story ;).

Cheers

Michael

Re: Create software RAID from active partition

[Michal Soltys]

What do:

mdadm -E /dev/sda1
mdadm -E /dev/sdb1
mdadm -E /dev/sda2
mdadm -E /dev/sdb2

show, if anything ?

If they report they belonged once to an array with superblock positioned 
at the end (so either 0.9 or 1.0), you could create RAID1 array missing 
one component. In your case, something like:

umount /dev/mapper/VolGroup00-LogVol00
umount /dev/mapper/VolGroup00-LogVol01
vgchange -an VolGroup00

mdadm  -C /dev/md/1 -l1 -n2 -e0 /dev/sdb2 missing

-e0 assuming it was v0.9 superblock
-e1.0 assuming it was v1.0

and then:

mdadm /dev/md/1 --add /dev/sda2

..wait for resync, then:

vgscan --mknodes
vgchange -ay

..should use /dev/md/1 - but  doublecheck /etc/lvm/lvm.conf as well.


Of course it's potentially risky, so doublecheck everything and if it's 
possible somehow (considering remote location), backup /dev/sdb2 first 
(actually, you could use /dev/sda2 for that, as it's unused at this 
moment anyway).


/dev/sd{a,b}1 case will be difficult w/o physical access, if they were 
part of another raid1 array.

Re: Create software RAID from active partition

[Michael Guyver]

2008/9/1 Michal Soltys <soltys@ziu.info>:
> What do:
>
> mdadm -E /dev/sda1
> mdadm -E /dev/sdb1
> mdadm -E /dev/sda2
> mdadm -E /dev/sdb2
>
> show, if anything ?
>

Hi Michal,

Each device reports the same as /dev/sdb1:

# mdadm -E /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.

I think there might be a more subtle problem - the disks /dev/sda2 and
/dev/sdb2 appear to have the same UUID. I'm not sure of the
significance at present and am going to have to do a bit more
investigation, but it is enough to prevent

pvcreate /dev/sda2

from working - unless I specify the -ff flag as an additional
parameter, which I am loathe to try without understanding what might
happen. I was working towards creating an lvm-mirror following some
earlier advice in this thread, and found this problem when using
system-config-lvm. The GUI shows /dev/sda2 as uninitialised and the
error occurs when trying to initialise it. Any further advice would be
gratefully received (although I appreciate that this is the
linux-raid, not linux-lvm group)!

Best wishes and thanks for your suggestions.

Michael