My MD is too big to resize ext4.

My MD is too big to resize ext4.

[Ram Ramesh]

Hi,

I asked in other linux forums and did not get enough info. So I come 
here even though this is not mdadm/RAID issue.

I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB. 
I had 32bit version of ext4 on it which can only be grown to 16TB. Any 
one had this issue before and any _/inplace/_ good solution to grow ext4 
to full 24TB? It is unlikely that I will be able to back up and recreate 
file system.

My filesystem is on md0 drive so I guess partitioning and making into 2x 
12TB ext4 will not work. I am not even sure if we can partition md like 
any other disk.

On the web, I only found one solution that required upgrading kernel to 
some very recent one (not in my distro) and getting the bleeding edge 
resize2fs. This makes me nervous. Is there a solution that avoids this.

Ramesh

Re: My MD is too big to resize ext4.

[Andreas Klauer]

On Fri, Jul 07, 2017 at 07:41:13PM -0500, Ram Ramesh wrote:
> On the web, I only found one solution that required upgrading kernel to 
> some very recent one (not in my distro) and getting the bleeding edge 
> resize2fs. This makes me nervous. Is there a solution that avoids this.

Well, if that works nowadays, then just go for it (LiveCD). 
For details you should ask the ext4 mailing list I guess.

You can assemble your RAID in read-only mode and then use 
overlays for a non-destructive test run. To make really extra sure 
you could even export it via NBD/KVM to see if the old kernel 
is able to mount it.

IIRC, btrfs also has a migration path (ext4 to btrfs converter) 
and then you could grow that. But then you're stuck with btrfs.

> My filesystem is on md0 drive so I guess partitioning and making into 2x 
> 12TB ext4 will not work. I am not even sure if we can partition md like 
> any other disk.

Adding a partition table or LVM header would shift the start position.
So it would involve moving/shifting all your data in-place or using 
a conversion to LVM. (We just had that in the thread above yours, 
subject "linear device of two arrays")

If your RAID has a large enough data offset, you could shift that 
to add some extra space at the start of the RAID, but it has to 
be done with great care (must be aligned to whatever your RAID 
layout is so data will be intact and not garbled).

All in-place operations are kind of hackish/dangerous.

I actually prefer using several smaller filesystems over one gigantic one.
It's all great until your one and only filesystem goes corrupt.

Regards
Andreas Klauer

Re: My MD is too big to resize ext4.

[Brad Campbell]

On 08/07/17 08:41, Ram Ramesh wrote:
> Hi,
>
> I asked in other linux forums and did not get enough info. So I come
> here even though this is not mdadm/RAID issue.
>
> I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB.
> I had 32bit version of ext4 on it which can only be grown to 16TB. Any
> one had this issue before and any _/inplace/_ good solution to grow ext4
> to full 24TB? It is unlikely that I will be able to back up and recreate
> file system.

Ignore this first suggestion if you are absolutely reliant on data 
integrity and don't have backups of any critical data.

Take 2 of the 6TB drives, create a 12TB RAID0 or Linear and copy all 
your data to that.

Create the 6 drive RAID6 with 2 disks missing, copy all the data back, 
then add the disks you used in the first copy back to the array.


I had this issue a few years ago and decided buying some extra disks to 
use as scratch was a better move for me. In your shoes I would have 
found 6 extra SATA ports on a machine (any machine), created the new 
filesystem and rsynced the data across, then swapped the disks into the 
production machine.

In fact I've just done that migrating from 14x2TB drives to 8x6TB 
drives, but I keep a staging machine around for testing that has 15 
drive bays for just this sort of migration.

This is the 4th time I've done it in 13 years. 15x250GB -> 8x1TB -> 
14x2TB -> 8x6TB. The advantage is you never lose data integrity or 
redundancy, and you can keep the old drives around in a box until you 
are sure the new build is reliable.

Alternatively, find a machine with 6 slots, build a RAID6 from the 
drives you already have. Copy the data across, re-format the new array 
and copy the data back. A bit of double handling, but no loss of 
redundancy or integrity. Run md5 or something over all the files to make 
sure it's all good, or use rsync with the "always checksum" option, 
although this is slow.

Brad

Re: My MD is too big to resize ext4.

[Reindl Harald]

Am 08.07.2017 um 09:34 schrieb Brad Campbell:
> Alternatively, find a machine with 6 slots, build a RAID6 from the 
> drives you already have. Copy the data across, re-format the new array 
> and copy the data back. A bit of double handling, but no loss of 
> redundancy or integrity. Run md5 or something over all the files to make 
> sure it's all good, or use rsync with the "always checksum" option, 
> although this is slow

--checksum
If the file size and time match, it will do a checksum at both ends to 
see if the files are really identical

that won't do much when the target is empty

Re: My MD is too big to resize ext4.

[Brad Campbell]

On 08/07/17 15:40, Reindl Harald wrote:
>
>
> Am 08.07.2017 um 09:34 schrieb Brad Campbell:
>> Alternatively, find a machine with 6 slots, build a RAID6 from the
>> drives you already have. Copy the data across, re-format the new array
>> and copy the data back. A bit of double handling, but no loss of
>> redundancy or integrity. Run md5 or something over all the files to
>> make sure it's all good, or use rsync with the "always checksum"
>> option, although this is slow
>
> --checksum
> If the file size and time match, it will do a checksum at both ends to
> see if the files are really identical
>
> that won't do much when the target is empty
>

Sorry, if you read the man page I thought it'd be pretty obvious you 
need to run it twice and I didn't need to explicitly spell it out.

Brad

Re: My MD is too big to resize ext4.

[Wols Lists]

On 08/07/17 01:41, Ram Ramesh wrote:
> Hi,
> 
> I asked in other linux forums and did not get enough info. So I come
> here even though this is not mdadm/RAID issue.
> 
> I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB.
> I had 32bit version of ext4 on it which can only be grown to 16TB. Any
> one had this issue before and any _/inplace/_ good solution to grow ext4
> to full 24TB? It is unlikely that I will be able to back up and recreate
> file system.
> 
> My filesystem is on md0 drive so I guess partitioning and making into 2x
> 12TB ext4 will not work. I am not even sure if we can partition md like
> any other disk.

You can partition an array and create partitions in it - caveat I
haven't done it, and I've never seen anyone here mention that they've
done it - lvm seems to be the more popular version.
> 
> On the web, I only found one solution that required upgrading kernel to
> some very recent one (not in my distro) and getting the bleeding edge
> resize2fs. This makes me nervous. Is there a solution that avoids this.
> 
resize2fs is bleeding edge? I suspect it's v0.99 quality, ie nobody has
the nerve to upgrade it to v1, despite it being rock solid. I know I've
used it without trouble. Most of these utilities are pretty solid
(unless the underlying filesystem itself is experimental ...)

Cheers,
Wol

Re: My MD is too big to resize ext4.

[Roman Mamedov]

On Fri, 7 Jul 2017 19:41:13 -0500
Ram Ramesh <rramesh2400@gmail.com> wrote:

> On the web, I only found one solution that required upgrading kernel to 
> some very recent one (not in my distro) and getting the bleeding edge 
> resize2fs. This makes me nervous. Is there a solution that avoids this.

Considering the other possible options that have been mentioned, using the
Ext4 built-in larger devices support (which has been implemented recently)
seems to be your best bet.
https://askubuntu.com/questions/779754/how-do-i-resize-an-ext4-partition-beyond-the-16tb-limit

The required e2fsprogs version 1.43 is included in Ubuntu 16.10 by now, so you
don't even need to build it from the source.
https://packages.ubuntu.com/yakkety/e2fsprogs

But really, if you have everything on one array with a single huge filesystem
and no backups, that's just asking for trouble and a complete data loss.

-- 
With respect,
Roman

Re: My MD is too big to resize ext4.

[Ram Ramesh]

On 07/08/2017 01:50 AM, Andreas Klauer wrote:
> On Fri, Jul 07, 2017 at 07:41:13PM -0500, Ram Ramesh wrote:
>> On the web, I only found one solution that required upgrading kernel to
>> some very recent one (not in my distro) and getting the bleeding edge
>> resize2fs. This makes me nervous. Is there a solution that avoids this.
> Well, if that works nowadays, then just go for it (LiveCD).
> For details you should ask the ext4 mailing list I guess.
>
> You can assemble your RAID in read-only mode and then use
> overlays for a non-destructive test run. To make really extra sure
> you could even export it via NBD/KVM to see if the old kernel
> is able to mount it.
>
> IIRC, btrfs also has a migration path (ext4 to btrfs converter)
> and then you could grow that. But then you're stuck with btrfs.
>
>> My filesystem is on md0 drive so I guess partitioning and making into 2x
>> 12TB ext4 will not work. I am not even sure if we can partition md like
>> any other disk.
> Adding a partition table or LVM header would shift the start position.
> So it would involve moving/shifting all your data in-place or using
> a conversion to LVM. (We just had that in the thread above yours,
> subject "linear device of two arrays")
>
> If your RAID has a large enough data offset, you could shift that
> to add some extra space at the start of the RAID, but it has to
> be done with great care (must be aligned to whatever your RAID
> layout is so data will be intact and not garbled).
>
> All in-place operations are kind of hackish/dangerous.
>
> I actually prefer using several smaller filesystems over one gigantic one.
> It's all great until your one and only filesystem goes corrupt.
>
> Regards
> Andreas Klauer

Thanks. After thinking a bit more, I also like several smaller ext4/mds 
than large one. Since this is a mythtv machine, it can share multiple 
mount point and distribute data smartly (in its own sense).

In order to create several md devices out of the 6x6TB drives I have, I 
have to do the following. Let me know if this sounds possible.

    1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So 
I can create partitions on the drive instead of on the MD.
    2. This means I need to shrink my current md device to smaller size 
(say 12TB-14TB) - need to check my current
          active ext4 data size. It is definitely less than 16TB.
    3. Repartition the disks to create sd{b-g}2 for the reamining unused 
6xnTB area.
    4. Created md1 with sd{b-g]2 to get md1.
    5. Mount and use md1.

Do you see every step here reasonably safe to try on a 
unmounted/readonly md disks.

Ramesh

Re: My MD is too big to resize ext4.

[Ram Ramesh]

On 07/08/2017 02:34 AM, Brad Campbell wrote:
> On 08/07/17 08:41, Ram Ramesh wrote:
>> Hi,
>>
>> I asked in other linux forums and did not get enough info. So I come
>> here even though this is not mdadm/RAID issue.
>>
>> I replaced 3TB disks in my 6-disk RAID6 with 6TB and now my md0 is 24TB.
>> I had 32bit version of ext4 on it which can only be grown to 16TB. Any
>> one had this issue before and any _/inplace/_ good solution to grow ext4
>> to full 24TB? It is unlikely that I will be able to back up and recreate
>> file system.
>
> Ignore this first suggestion if you are absolutely reliant on data 
> integrity and don't have backups of any critical data.
>
> Take 2 of the 6TB drives, create a 12TB RAID0 or Linear and copy all 
> your data to that.
>
> Create the 6 drive RAID6 with 2 disks missing, copy all the data back, 
> then add the disks you used in the first copy back to the array.
>
>
> I had this issue a few years ago and decided buying some extra disks 
> to use as scratch was a better move for me. In your shoes I would have 
> found 6 extra SATA ports on a machine (any machine), created the new 
> filesystem and rsynced the data across, then swapped the disks into 
> the production machine.
>
> In fact I've just done that migrating from 14x2TB drives to 8x6TB 
> drives, but I keep a staging machine around for testing that has 15 
> drive bays for just this sort of migration.
>
> This is the 4th time I've done it in 13 years. 15x250GB -> 8x1TB -> 
> 14x2TB -> 8x6TB. The advantage is you never lose data integrity or 
> redundancy, and you can keep the old drives around in a box until you 
> are sure the new build is reliable.
>
> Alternatively, find a machine with 6 slots, build a RAID6 from the 
> drives you already have. Copy the data across, re-format the new array 
> and copy the data back. A bit of double handling, but no loss of 
> redundancy or integrity. Run md5 or something over all the files to 
> make sure it's all good, or use rsync with the "always checksum" 
> option, although this is slow.
>
> Brad
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks for the details. I think I have 2x4TB and 2x3TB spares. This is 
enough for the data I have and I have scripted a way to distribute files 
across those to make a mock backup/snapshot. They are old drives and I 
do not trust their shelf life. So, I consider that there is no backup. 
However, for short/week exercise, it may be considered as a safe data. 
So, I am not that paranoid about doing inplace changes. I just don't 
like to choose known risky paths - a more tested approach is preferred 
even if it has nonzero propability of failure. That is all.

Ramesh

Re: My MD is too big to resize ext4.

[Ram Ramesh]

On 07/08/2017 03:30 AM, Wols Lists wrote:resize2fs is bleeding edge? I 
suspect it's v0.99 quality, ie nobody has
> the nerve to upgrade it to v1, despite it being rock solid. I know I've
> used it without trouble. Most of these utilities are pretty solid
> (unless the underlying filesystem itself is experimental ...)
>
> Cheers,
> Wol
>
The web solution talks about not yet released resize2fs source and 
compiling in latest kernel etc. That is bleeding edge to me. I like to 
stick to stable versions of distro.

My days of young and adventurous are gone. I have been with Linus/linux 
when he released some mock alpha kernlel that would only print A/B to 
show context switching. I remember the lengthy conversation/argument 
that he had with Prof. Tannenbaum (sp?) on monolithic vs. micro kernel. 
I enjoyed compiling kernel and reading code to figure out why some thing 
did not work. However that is a long time ago. I just have aged, become 
less current and rusty with tricks. I use stable distros and stick the 
trusted kernel/packages in them now. So, in that context, would you not 
say resize2fs approach is bleeding edge :-)

Ramesh

Re: My MD is too big to resize ext4.

[Ram Ramesh]

On 07/08/2017 06:15 AM, Roman Mamedov wrote:
> On Fri, 7 Jul 2017 19:41:13 -0500
> Ram Ramesh <rramesh2400@gmail.com> wrote:
>
>> On the web, I only found one solution that required upgrading kernel to
>> some very recent one (not in my distro) and getting the bleeding edge
>> resize2fs. This makes me nervous. Is there a solution that avoids this.
> Considering the other possible options that have been mentioned, using the
> Ext4 built-in larger devices support (which has been implemented recently)
> seems to be your best bet.
> https://askubuntu.com/questions/779754/how-do-i-resize-an-ext4-partition-beyond-the-16tb-limit
>
> The required e2fsprogs version 1.43 is included in Ubuntu 16.10 by now, so you
> don't even need to build it from the source.
> https://packages.ubuntu.com/yakkety/e2fsprogs
>
> But really, if you have everything on one array with a single huge filesystem
> and no backups, that's just asking for trouble and a complete data loss.
>
Thanks. This is exactly the one that I was talking about as a web 
solution. Did not know it is available in 16.10.

Besides the above, I hear your last caution well. I do have backup by 
distributing files using rsync over multiple (old) disks (4+4+3+3TB). I 
just don't trust my back up and thus pretend that I do not have one. 
However, for a short one week trial, I think, I am ok.

Going forward, I will not be able to backup 24TB volume (if I succeed in 
building one big MD) as I will not have enough drives to distribute 
data. So, there is going to be a problem. So, I am thinking seriously 
building several smaller MD devices out of my 6x6TB drives. Look at my 
reply to Andreas. Let me know your thoughts on that. If I build smaller 
MDs each can be temporarily backed up for updates etc. So, I will still 
be ok with data as long as I work only on one MD at a time that fits my 
set of backup disks.

Ramesh

Re: My MD is too big to resize ext4.

[Ram Ramesh]

>> I had this issue a few years ago and decided buying some extra disks 
>> to use as scratch was a better move for me. In your shoes I would 
>> have found 6 extra SATA ports on a machine (any machine), created the 
>> new filesystem and rsynced the data across, then swapped the disks 
>> into the production machine.

Brad,

   I try to keep disks from different production batch in my MD as 
proabilistically speaking, this is safe. This means I buy one disk every 
year and replace the oldest one. This time it is special case that My 
last 3TB got replaced with new 6TB and now I can actually expand the 
array.  So, the above method may not always work for me. However, I see 
your point and therefore see merit to it. In any case, I am taking 
Andreas (and few others)  advice and try to build several smaller MD 
devices instead of one large one. I responded to him on this idea. Let 
me see if any one sees issues with that.

Ramesh

Re: My MD is too big to resize ext4.

[Andreas Klauer]

On Sat, Jul 08, 2017 at 01:12:11PM -0500, Ram Ramesh wrote:
>     1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So 
> I can create partitions on the drive instead of on the MD.
>     2. This means I need to shrink my current md device to smaller size 
> (say 12TB-14TB) - need to check my current
>           active ext4 data size. It is definitely less than 16TB.
>     3. Repartition the disks to create sd{b-g}2 for the reamining unused 
> 6xnTB area.
>     4. Created md1 with sd{b-g]2 to get md1.
>     5. Mount and use md1.

Should work; you have to shrink the filesystem first, (or stick to 
whatever size it has now), then the md, then the partition.

What makes this risky is that you have to pick the correct sizes. 
When growing you can't go wrong. When shrinking, you have to be 
careful not to shrink too much. Filesystems don't like it at all 
if their end is missing, and md doesn't like it if the block device 
is smaller than what it says in the metadata.

So you have to determine the exact filesystem size (tune2fs -l), 
and take mdadm data offsets into account.

Not necessary if you know what you're doing but if in doubt,
you can leave a safety margin with each of these steps.

Regards
Andreas Klauer

Re: My MD is too big to resize ext4.

[Ram Ramesh]

On 07/08/2017 02:44 PM, Andreas Klauer wrote:
> On Sat, Jul 08, 2017 at 01:12:11PM -0500, Ram Ramesh wrote:
>>      1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So
>> I can create partitions on the drive instead of on the MD.
>>      2. This means I need to shrink my current md device to smaller size
>> (say 12TB-14TB) - need to check my current
>>            active ext4 data size. It is definitely less than 16TB.
>>      3. Repartition the disks to create sd{b-g}2 for the reamining unused
>> 6xnTB area.
>>      4. Created md1 with sd{b-g]2 to get md1.
>>      5. Mount and use md1.
> Should work; you have to shrink the filesystem first, (or stick to
> whatever size it has now), then the md, then the partition.
>
> What makes this risky is that you have to pick the correct sizes.
> When growing you can't go wrong. When shrinking, you have to be
> careful not to shrink too much. Filesystems don't like it at all
> if their end is missing, and md doesn't like it if the block device
> is smaller than what it says in the metadata.
>
> So you have to determine the exact filesystem size (tune2fs -l),
> and take mdadm data offsets into account.
>
> Not necessary if you know what you're doing but if in doubt,
> you can leave a safety margin with each of these steps.
>
> Regards
> Andreas Klauer
My df shows that I use 10431258240 blocks (1K) in my ext4 on the md. 
This is less than 10T. So, resize2fs to 11T should be fine. I will 
shrink md to 11.5T and partition to 12T. I will use 2^30 to mean TB. Is 
0.5T enough margin in each case?

I can also stretch the limit to 15TB for ext4 and 15.5 for md and 16TB 
for partiton. Is this better because it maxes out the 32bit
ext4? Somehow something tells me pushing the limit is always a bad idea 
as mistakes generally happen on checking  boundary conditions.

Alternatively, I can do resize2fs with -M to minimal size, but the man 
says calculations are not perfect for 1/2K block sizes. However, I see 
that dumpe2fs says I have 4K blocks.

Ramesh

Re: My MD is too big to resize ext4.

[Ram Ramesh]

On 07/08/2017 02:44 PM, Andreas Klauer wrote:
> On Sat, Jul 08, 2017 at 01:12:11PM -0500, Ram Ramesh wrote:
>>      1. My MD uses partitions sd{b,c,d,e,f,g}1 instead of full disks. So
>> I can create partitions on the drive instead of on the MD.
>>      2. This means I need to shrink my current md device to smaller size
>> (say 12TB-14TB) - need to check my current
>>            active ext4 data size. It is definitely less than 16TB.
>>      3. Repartition the disks to create sd{b-g}2 for the reamining unused
>> 6xnTB area.
>>      4. Created md1 with sd{b-g]2 to get md1.
>>      5. Mount and use md1.
> Should work; you have to shrink the filesystem first, (or stick to
> whatever size it has now), then the md, then the partition.
>
> What makes this risky is that you have to pick the correct sizes.
> When growing you can't go wrong. When shrinking, you have to be
> careful not to shrink too much. Filesystems don't like it at all
> if their end is missing, and md doesn't like it if the block device
> is smaller than what it says in the metadata.
>
> So you have to determine the exact filesystem size (tune2fs -l),
> and take mdadm data offsets into account.
>
> Not necessary if you know what you're doing but if in doubt,
> you can leave a safety margin with each of these steps.
>
> Regards
> Andreas Klauer
I already shrunk ext4 and md as they can be done without having to boot 
to a rescue disk. I like to do the last step of partitioning disk using 
a rescue disk.

Here is my partition table on one disk (they all are very similar or 
identical - each is a 6TB enterprise disk)
> zym [rramesh] 431 > sudo gdisk -l /dev/sdb
> GPT fdisk (gdisk) version 0.8.8
>
>
> Partition table scan:
>   MBR: protective
>   BSD: not present
>   APM: not present
>   GPT: present
>
> Found valid GPT with protective MBR; using GPT.
> Disk /dev/sdb: 11721045168 sectors, 5.5 TiB
> Logical sector size: 512 bytes
> Disk identifier (GUID): D5C9B768-D2E5-4DEE-8D89-73A7B631FE28
> Partition table holds up to 128 entries
> First usable sector is 34, last usable sector is 11721045134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 2014 sectors (1007.0 KiB)
>
> Number  Start (sector)    End (sector)  Size       Code  Name
>    1            2048     11721045134   5.5 TiB     FD00  Linux RAID
>
Here is mdadm -E on the same disk
> zym [rramesh] 430 > sudo mdadm -E /dev/sdb1
> /dev/sdb1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : 0e9f76b5:4a89171a:a930bccd:78749144
>            Name : zym:0  (local to host zym)
>   Creation Time : Mon Apr 22 00:08:12 2013
>      Raid Level : raid6
>    Raid Devices : 6
>
>  Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
>      Array Size : 12348030976 (11776.00 GiB 12644.38 GB)
>   Used Dev Size : 6174015488 (2944.00 GiB 3161.10 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>           State : active
>     Device UUID : 2a844ae2:4b028cc3:36095185:7b09f7cc
>
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Sun Jul  9 17:55:51 2017
>        Checksum : 74d5464 - correct
>          Events : 290897
>
>          Layout : left-symmetric
>      Chunk Size : 64K
>
>    Device Role : Active device 4
>    Array State : AAAAAA ('A' == active, '.' == missing)

I am guessing anything more than 6174015488+2048 as the end sector for 
/dev/sdb1 should be ok with mdadm. Let me know if I am not calculating 
correctly.

I plan to set /dev/sd{b,c,d,e,f,g}1 to 3TB even (start = 2048, end = 
6442452991 size = 6442450944 sectors = 3072GiB)
to get a 12TB md0.

Ramesh