Enlarging device of linear array again (Thank you Stan!)

Enlarging device of linear array again (Thank you Stan!)

[Ramon Hofer]

Hi all

In 2012 this list - especially Stan Hoeppner - helped me setting up a
linear RAID containing four RAID5. Thank you so much again! It is still
working amazingly well :-D

Now I am at the point where I want to add more storage but have no more
slots. As discussed in 2013 I will replace four discs with partitions
from a larger HDD and add the remaining space to a new RAID5.

md2 should be replaced:

$ sudo mdadm -D /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Sun Jun 17 20:08:48 2012
     Raid Level : raid5
     Array Size : 4395412224 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 1465137408 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue Sep 20 09:12:16 2016
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           Name : media-server:2  (local to host media-server)
           UUID : 1c74447b:33070712:cfcfa5af:cbfea660
         Events : 2449

    Number   Major   Minor   RaidDevice State
       0       8       64        0      active sync   /dev/sde
       1       8       80        1      active sync   /dev/sdf
       2       8       96        2      active sync   /dev/sdg
       4       8      112        3      active sync   /dev/sdh


$ sudo fdisk -l /dev/sd[efgh]

Disk /dev/sde: 1.4 TiB, 1500301910016 bytes, 2930277168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00097014


Disk /dev/sdf: 1.4 TiB, 1500301910016 bytes, 2930277168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0007694c


Disk /dev/sdg: 1.4 TiB, 1500301910016 bytes, 2930277168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000da169


Disk /dev/sdh: 1.4 TiB, 1500301910016 bytes, 2930277168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000beebc


$ sudo mdadm --examine /dev/sd[efgh]
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c74447b:33070712:cfcfa5af:cbfea660
           Name : media-server:2  (local to host media-server)
  Creation Time : Sun Jun 17 20:08:48 2012
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 4395412224 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274816 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=304 sectors
          State : clean
    Device UUID : 270f29c9:0a36cd7a:27324b70:7f4e929b

    Update Time : Tue Sep 20 09:12:16 2016
       Checksum : dc24915c - correct
         Events : 2449

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c74447b:33070712:cfcfa5af:cbfea660
           Name : media-server:2  (local to host media-server)
  Creation Time : Sun Jun 17 20:08:48 2012
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 4395412224 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274816 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=304 sectors
          State : clean
    Device UUID : fe2fffdc:6d072a9d:87757913:ae7365db

    Update Time : Tue Sep 20 09:12:16 2016
       Checksum : f558ec26 - correct
         Events : 2449

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c74447b:33070712:cfcfa5af:cbfea660
           Name : media-server:2  (local to host media-server)
  Creation Time : Sun Jun 17 20:08:48 2012
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 4395412224 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274816 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=304 sectors
          State : clean
    Device UUID : f27f55c2:f66f5fe9:02943932:2cf47cca

    Update Time : Tue Sep 20 09:12:16 2016
       Checksum : 34bc439e - correct
         Events : 2449

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c74447b:33070712:cfcfa5af:cbfea660
           Name : media-server:2  (local to host media-server)
  Creation Time : Sun Jun 17 20:08:48 2012
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275120 (1397.26 GiB 1500.30 GB)
     Array Size : 4395412224 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 2930274816 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=304 sectors
          State : clean
    Device UUID : 642511df:7f5d1022:a1b40e7b:6ebd37c6

    Update Time : Tue Sep 20 09:12:16 2016
       Checksum : ceb8c19b - correct
         Events : 2449

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)




I hope to simply do the following steps:

## Replace old disk with new partitions

# Step 1: Create partitions on disk
$ sudo fdisk /dev/sde
create 1465137408 (1397.26 GiB 1500.30 GB) partition and a second on
with the rest

# Step 2: Replace old disk with new partition
$ sudo mdadm --manage /dev/md4 --add /dev/sde1

# Step 3: Wait until rebuilt

# Step 4: Repeat steps 1-3 for /dev/sd[fgh]


## Add new partitions

# Step 5: Create new Raid5
sudo mdadm -C /dev/md5 -c 128 -n4 -l5 /dev/sd[efgh]2

# Step 6: Add new Raid5 to linear
sudo mdadm --grow /dev/md0 --add /dev/md5

# Step 7: Grow filesystem
sudo xfs_growfs /mnt/media-raid



I am using 4 TB WD red to replace the 1.5 TB disks.

Do you think this could work?
Are there any pitfalls?
Should I unmount the array to perform all these steps? It might be
safer since there is no redundancy during the replacement?

I just wanted to check first before making any mistakes. Instead of
maybe afterwards asking for help recovering my data :-P


Best regards,
Ramon

Re: Enlarging device of linear array again (Thank you Stan!)

[Wols Lists]

On 20/09/16 20:34, Ramon Hofer wrote:
> I am using 4 TB WD red to replace the 1.5 TB disks.
> 
> Do you think this could work?
> Are there any pitfalls?
> Should I unmount the array to perform all these steps? It might be
> safer since there is no redundancy during the replacement?
> 
> I just wanted to check first before making any mistakes. Instead of
> maybe afterwards asking for help recovering my data :-P

I don't quite understand what you are doing, but ...

Do you have four existing drives (sde, sdf, sdg, sdh) and you are
planning on replacing them with four new, larger drives? (It comes over
that you are planning to create four partitions on one drive to replace
the four drives, and that just isn't right).

If you are planning to use four new drives, then the obvious thing is to
get either an add-in pci card to give you more SATA connections, or a
USB adaptor (caveat, I've never done anything like this) so you can
temporarily add in an extra drive.

If you've got an add-in PCI card, stick a new 4TB drive in, use "mdadm
--replace" to move one of the old drives across, and remove the old
drive. Rinse and repeat until all four drives are done.

If you got a USB case, put a new drive in the computer, and the old
drive in the case. Again, use "mdadm --replace" to move the old drive
across, and remove the old drive from the case. Rinse and repeat until
all four drives are done. I believe that it's a lot faster reading from
than writing to USB, hence me saying move the small drive to USB then
copy from it.

That way, at no point is your raid at risk while you're replacing a drive.

The other way to do it is just shut the machine down, remove all drives
except one and put a new big drive in. Boot from a rescue CD, partition
the new drive and dd the old partitions to the new drive. Again rinse
and repeat for all four drives, and then put all four new drives in the
system and reboot. At which point you can expand all the
filesystems/raids to use all the space available. This *shouldn't* be a
risky operation. Indeed, if you're going to shut the array down, this is
probably the simplest and safest option.

Cheers,
Wol

Re: Enlarging device of linear array again (Thank you Stan!)

[Ramon Hofer]

Thank you very much for your answer, Wol!


On Tue, 20 Sep 2016 22:52:00 +0100
Wols Lists <antlists@youngman.org.uk> wrote:

> On 20/09/16 20:34, Ramon Hofer wrote:
> > I am using 4 TB WD red to replace the 1.5 TB disks.
> > 
> > Do you think this could work?
> > Are there any pitfalls?
> > Should I unmount the array to perform all these steps? It might be
> > safer since there is no redundancy during the replacement?
> > 
> > I just wanted to check first before making any mistakes. Instead of
> > maybe afterwards asking for help recovering my data :-P  
> 
> I don't quite understand what you are doing, but ...

Sorry I forgot to mention some things.

I have a linear md0 containing four raid5 (md[1234]) to which XFS is
stripe aligned due to the performance. [1], [2]

The case for my home media server is a Norco [3] with 20 slots. Four are
dedicated to MythTV.
4 x 4 are in md0.

I want to replace the four 1.5 TB disks with 4 TB to have more
space.

On the four 4 TB disks I create two partitions (1.5 TB and 2.5 TB).
Each old 1.5 TB disk gets replaced with the partition on the 4 TB
disks. From the 2.5 TB of each of the four 4TB disks I create a new
RAID5 (md5) and add this to the linear md0. At the end I expand the
file system. 


> The other way to do it is just shut the machine down, remove all
> drives except one and put a new big drive in. Boot from a rescue CD,
> partition the new drive and dd the old partitions to the new drive.
> Again rinse and repeat for all four drives, and then put all four new
> drives in the system and reboot. At which point you can expand all the
> filesystems/raids to use all the space available. This *shouldn't* be
> a risky operation. Indeed, if you're going to shut the array down,
> this is probably the simplest and safest option.

So you suggest to dd the 1.5 TB disks to the 1.5 TB
partition on the 4 TB disks instead of just let mdadm rebuild the
array?
I was thinking about this as well because the dd process might be
faster because the CPU load would be significantly less than the RAID5
rebuild process. And only one disk is read during the copy process -
instead of all the other three for each of the four rebuilds.

Can I just do:

sudo dd if=/dev/sde of=/dev/sdf1

Assuming sde is the old 1.5 TB disk and sdf1 is the 1.5 TB partition on
the new 4 TB disk.
Can I be sure that mdadm recognizes the partitions correctly as the
replacement for the old disks?


Best regards,
Ramon


[1] http://marc.info/?l=linux-raid&m=137077874619605&w=2
[2] http://marc.info/?l=linux-raid&m=134031891621599&w=2
[3] http://www.norcotek.com/?s=RPC-4020

Re: Enlarging device of linear array again (Thank you Stan!)

[Adam Goryachev]

On 21/09/16 09:06, Ramon Hofer wrote:
> Thank you very much for your answer, Wol!
>
>
> On Tue, 20 Sep 2016 22:52:00 +0100
> Wols Lists <antlists@youngman.org.uk> wrote:
>
>> On 20/09/16 20:34, Ramon Hofer wrote:
>>> I am using 4 TB WD red to replace the 1.5 TB disks.
>>>
>>> Do you think this could work?
>>> Are there any pitfalls?
>>> Should I unmount the array to perform all these steps? It might be
>>> safer since there is no redundancy during the replacement?
>>>
>>> I just wanted to check first before making any mistakes. Instead of
>>> maybe afterwards asking for help recovering my data :-P
>> I don't quite understand what you are doing, but ...
> Sorry I forgot to mention some things.
>
> I have a linear md0 containing four raid5 (md[1234]) to which XFS is
> stripe aligned due to the performance. [1], [2]
>
> The case for my home media server is a Norco [3] with 20 slots. Four are
> dedicated to MythTV.
> 4 x 4 are in md0.
>
> I want to replace the four 1.5 TB disks with 4 TB to have more
> space.
>
> On the four 4 TB disks I create two partitions (1.5 TB and 2.5 TB).
> Each old 1.5 TB disk gets replaced with the partition on the 4 TB
> disks. From the 2.5 TB of each of the four 4TB disks I create a new
> RAID5 (md5) and add this to the linear md0. At the end I expand the
> file system.
Assuming that you are replacing the four disks that are in the RAID5 
array which is at the "end" of your linear raid0, then you don't 
actually need to create two partitions at all. Just replace the four 
drives (dd from source to destination while offline). When complete, (if 
they are partitioned, delete/re-create the single partition to fill the 
drive), grow the RAID5 array to fill the new drives, then grow the md0 
to take the rest of the RAID5 space onto the end of the drive, then grow 
your filesystem/etc.

>> The other way to do it is just shut the machine down, remove all
>> drives except one and put a new big drive in. Boot from a rescue CD,
>> partition the new drive and dd the old partitions to the new drive.
>> Again rinse and repeat for all four drives, and then put all four new
>> drives in the system and reboot. At which point you can expand all the
>> filesystems/raids to use all the space available. This *shouldn't* be
>> a risky operation. Indeed, if you're going to shut the array down,
>> this is probably the simplest and safest option.
> So you suggest to dd the 1.5 TB disks to the 1.5 TB
> partition on the 4 TB disks instead of just let mdadm rebuild the
> array?
Agreed, possibly faster, definitely safer if you value your data.
> Can I just do:
>
> sudo dd if=/dev/sde of=/dev/sdf1
Depends, is the existing sde partitioned? Make extra sure you pair up 
each source/destination drive, etc...
>
> Assuming sde is the old 1.5 TB disk and sdf1 is the 1.5 TB partition on
> the new 4 TB disk.
> Can I be sure that mdadm recognizes the partitions correctly as the
> replacement for the old disks?
mdadm will use the content of the drive/partition to work out what 
belongs in/to what array, so it should be fine. However, its a great 
idea to keep the four old drives aside until you are sure everything is 
working properly (another advantage to avoid replacing one at a time...)

Note that this advice might mess with your "performance tuning", I don't 
know enough about that side of things to comment further.

Regards,
Adam



-- 
Adam Goryachev Website Managers www.websitemanagers.com.au