Online resize issue with 3.13.5 & 3.15.6

Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

G'day all,

Machine was running 3.13.5. x86_64.

I had a 12 device (2TB) RAID-6 formatted ext4. I added 2 drives to its 
underlying md and restriped it (no issues). After the restripe I 
attempted an online resize using ext2progs 1.42.5 (Debian stable). This 
failed with a message about the size not fitting into 32 bits so I 
compiled 1.42.11 and tried again.

This resulted in a message I no longer have access to that indicated 
that something went wrong. I attempted it a couple more times (how dumb 
am I?) The resulting parts of dmesg are :

Jul 20 17:20:13 srv kernel: [11893469.381692] EXT4-fs (md0): resizing 
filesystem from 4883458240 to 5860149888 blocks
Jul 20 17:20:23 srv kernel: [11893479.597505] EXT4-fs (md0): resized to 
5128585216 blocks

Jul 20 17:20:43 srv kernel: [11893499.681961] EXT4-fs (md0): resized to 
5525995520 blocks
Jul 20 17:20:53 srv kernel: [11893509.762719] EXT4-fs (md0): resized to 
5641863168 blocks
Jul 20 17:21:02 srv kernel: [11893517.869988] EXT4-fs warning (device 
md0): verify_reserved_gdb:705: reserved GDT 2769 missing grp 177147 
(5804755665)
Jul 20 17:21:02 srv kernel: [11893517.906663] EXT4-fs (md0): resized 
filesystem to 5860149888
Jul 20 17:21:08 srv kernel: [11893523.795964] EXT4-fs warning (device 
md0): ext4_group_extend:1712: can't shrink FS - resize aborted
Jul 20 17:21:17 srv kernel: [11893533.224440] EXT4-fs (md0): resizing 
filesystem from 5804916736 to 5860149888 blocks
Jul 20 17:21:17 srv kernel: [11893533.261982] EXT4-fs warning (device 
md0): verify_reserved_gdb:705: reserved GDT 2769 missing grp 177147 
(5804755665)
Jul 20 17:21:17 srv kernel: [11893533.300352] EXT4-fs (md0): resized 
filesystem to 5860149888
Jul 20 17:21:17 srv kernel: [11893533.636745] EXT4-fs warning (device 
md0): ext4_group_extend:1712: can't shrink FS - resize aborted

Jul 20 17:23:11 srv kernel: [11893647.253580] EXT4-fs (md0): resizing 
filesystem from 5804916736 to 5860149888 blocks
Jul 20 17:23:11 srv kernel: [11893647.291562] EXT4-fs warning (device 
md0): verify_reserved_gdb:705: reserved GDT 2769 missing grp 177147 
(5804755665)
Jul 20 17:23:11 srv kernel: [11893647.330267] EXT4-fs (md0): resized 
filesystem to 5860149888
Jul 20 17:23:12 srv kernel: [11893647.675745] EXT4-fs warning (device 
md0): ext4_group_extend:1712: can't shrink FS - resize aborted


At this point I thought it best to reboot the machine, so I upgraded to 
3.15.6 and brought it up in single user mode. The filesystem passed fsck 
with a message about an uninitialised block group and no other errors. 
I've since repeated the fsck several times and it is clean.

The issue is it locks up resize2fs hard (just spins on one core). Once 
it starts spinning there is no strace, so it's chasing its tail.

This is the current state of the fs.

root@srv:/s# dumpe2fs -h /dev/md0
dumpe2fs 1.42.11 (09-Jul-2014)
Filesystem volume name:   <none>
Last mounted on:          /s/src
Filesystem UUID:          99566e8e-e66d-4351-9675-0b3a549e2ba5
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index 
filetype extent 64bit flex_bg sparse_super large_file huge_file 
uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              362807296
Block count:              5804916736
Reserved block count:     0
Free blocks:              1407676872
Free inodes:              358800089
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      585
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         2048
Inode blocks per group:   128
RAID stride:              32
RAID stripe width:        320
Flex block group size:    16
Filesystem created:       Wed Jul 31 15:02:47 2013
Last mount time:          Sun Jul 20 17:41:16 2014
Last write time:          Sun Jul 20 18:48:00 2014
Mount count:              0
Maximum mount count:      -1
Last checked:             Sun Jul 20 18:48:00 2014
Check interval:           0 (<none>)
Lifetime writes:          4088 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      c08e3b0a-2c23-4b0f-b2d6-9bb8f26e0b48
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke journal_64bit
Journal size:             128M
Journal length:           32768
Journal sequence:         0x00229921
Journal start:            0

root@srv:/s# mdadm --detail /dev/md0
/dev/md0:
         Version : 1.2
   Creation Time : Wed Jul 31 15:02:11 2013
      Raid Level : raid6
      Array Size : 23440599552 (22354.70 GiB 24003.17 GB)
   Used Dev Size : 1953383296 (1862.89 GiB 2000.26 GB)
    Raid Devices : 14
   Total Devices : 14
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Sun Jul 20 18:54:56 2014
           State : active
  Active Devices : 14
Working Devices : 14
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-symmetric
      Chunk Size : 128K

            Name : srv:0  (local to host srv)
            UUID : a66b7f8a:dcf6b939:c14a87af:b21fcedf
          Events : 303231

     Number   Major   Minor   RaidDevice State
        0       8       64        0      active sync   /dev/sde
        1       8      144        1      active sync   /dev/sdj
        2       8      160        2      active sync   /dev/sdk
       14       8      176        3      active sync   /dev/sdl
        4       8      192        4      active sync   /dev/sdm
        5       8      224        5      active sync   /dev/sdo
        6       8      208        6      active sync   /dev/sdn
        7      65        0        7      active sync   /dev/sdq
        8      65       16        8      active sync   /dev/sdr
        9      65       48        9      active sync   /dev/sdt
       13      65      112       10      active sync   /dev/sdx
       12       8       32       11      active sync   /dev/sdc
       16      65       32       12      active sync   /dev/sds
       15       8      240       13      active sync   /dev/sdp

The filesystem looks clean, everything is accessible and though this is 
a production box, no business critical elements are on this array so we 
can live without it mounted if someone can give me some stuff to try.

Regards,
Brad

Re: Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

On 20/07/14 19:26, Brad Campbell wrote:
> G'day all,
>
> Machine was running 3.13.5. x86_64.
>
> I had a 12 device (2TB) RAID-6 formatted ext4. I added 2 drives to its
> underlying md and restriped it (no issues). After the restripe I
> attempted an online resize using ext2progs 1.42.5 (Debian stable). This
> failed with a message about the size not fitting into 32 bits so I
> compiled 1.42.11 and tried again.


More info:
I discovered the debug flags, so this is resize2fs -f 255 /dev/md0

last_start just keeps incrementing for as long as I care to leave it run.

fs has 4007207 inodes, 1957 groups required.
fs requires 4374122900 data blocks.
With 1957 group(s), we have 63820826 blocks available.
Added 131540 extra group(s), blks_needed 4374122900, data_blocks 
62023030, last_start 4356599580
Added 131595 extra group(s), blks_needed 4374122900, data_blocks 
73483100, last_start 5781212288
Added 131246 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781244926
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781277564
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781310202
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781342840
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781375478
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781408116
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781440754
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781473392
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781506030
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781538668
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781571306
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781603944
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781636582
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781669220
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781701858
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781734496
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781767134
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781799772
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781832410
Added 131072 extra group(s), blks_needed 4374122900, data_blocks 
79184732, last_start 5781865048

Re: Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

On 21/07/14 09:03, Brad Campbell wrote:
> On 20/07/14 19:26, Brad Campbell wrote:
>> G'day all,
>>
>> Machine was running 3.13.5. x86_64.
>>
>> I had a 12 device (2TB) RAID-6 formatted ext4. I added 2 drives to its
>> underlying md and restriped it (no issues). After the restripe I
>> attempted an online resize using ext2progs 1.42.5 (Debian stable). This
>> failed with a message about the size not fitting into 32 bits so I
>> compiled 1.42.11 and tried again.
>
>
> More info:
> I discovered the debug flags, so this is resize2fs -f 255 /dev/md0
>
> last_start just keeps incrementing for as long as I care to leave it run.
>
> fs has 4007207 inodes, 1957 groups required.
> fs requires 4374122900 data blocks.
> With 1957 group(s), we have 63820826 blocks available.

Ping? Working filesystem that locks up resize2fs 1.42.11 and 
consequently fails to fill the entire device. Passes fsck cleanly but 
will not resize either online or off.

Is there any other data I can provide or pointers on how I might 
un-break things?

Regards,
Brad

Re: Online resize issue with 3.13.5 & 3.15.6

[Azat Khuzhin]

On Mon, Jul 21, 2014 at 09:03:22AM +0800, Brad Campbell wrote:
> On 20/07/14 19:26, Brad Campbell wrote:
> >G'day all,
> >
> >Machine was running 3.13.5. x86_64.
> >
> >I had a 12 device (2TB) RAID-6 formatted ext4. I added 2 drives to its
> >underlying md and restriped it (no issues). After the restripe I
> >attempted an online resize using ext2progs 1.42.5 (Debian stable). This
> >failed with a message about the size not fitting into 32 bits so I
> >compiled 1.42.11 and tried again.
> 
> 
> More info:
> I discovered the debug flags, so this is resize2fs -f 255 /dev/md0
> 
> last_start just keeps incrementing for as long as I care to leave it run.
> 
> fs has 4007207 inodes, 1957 groups required.
> fs requires 4374122900 data blocks.
> With 1957 group(s), we have 63820826 blocks available.
> Added 131540 extra group(s), blks_needed 4374122900, data_blocks 62023030,
> last_start 4356599580
> Added 131595 extra group(s), blks_needed 4374122900, data_blocks 73483100,
> last_start 5781212288
> Added 131246 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781244926
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781277564
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781310202
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781342840
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781375478
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781408116
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781440754
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781473392
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781506030
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781538668
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781571306
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781603944
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781636582
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781669220
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781701858
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781734496
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781767134
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781799772
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781832410
> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
> last_start 5781865048

blocks_per_group=32768 (u32)
extra_groups=131072 (u32)
b*e=4294967296 # overflow

Could you try attached patched? (I will add appropriate message letter)
It is totally untested and shouldn't break anything, *but* I strongly
recommend you to have a backup before running it.
You could also wait until somebody else review it.

diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index a8af969..98ce10a 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -2479,7 +2479,8 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs, int flags)
                extra_grps = ext2fs_div64_ceil(remainder,
                                               EXT2_BLOCKS_PER_GROUP(fs->super));
 
-               data_blocks += extra_grps * EXT2_BLOCKS_PER_GROUP(fs->super);
+               data_blocks += (unsigned long long)extra_grps *
+                       EXT2_BLOCKS_PER_GROUP(fs->super);
 
                /* ok we have to account for the last group */
                overhead = calc_group_overhead(fs, groups-1, old_desc_blocks);


-- 
Respectfully
Azat Khuzhin

Re: Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

On 25/07/14 16:13, Azat Khuzhin wrote:
> On Mon, Jul 21, 2014 at 09:03:22AM +0800, Brad Campbell wrote:
>> On 20/07/14 19:26, Brad Campbell wrote:
>> Added 131072 extra group(s), blks_needed 4374122900, data_blocks 79184732,
>> last_start 5781865048
>
> blocks_per_group=32768 (u32)
> extra_groups=131072 (u32)
> b*e=4294967296 # overflow
>
> Could you try attached patched? (I will add appropriate message letter)
> It is totally untested and shouldn't break anything, *but* I strongly
> recommend you to have a backup before running it.
> You could also wait until somebody else review it.
>
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index a8af969..98ce10a 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -2479,7 +2479,8 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs, int flags)
>                  extra_grps = ext2fs_div64_ceil(remainder,
>                                                 EXT2_BLOCKS_PER_GROUP(fs->super));
>
> -               data_blocks += extra_grps * EXT2_BLOCKS_PER_GROUP(fs->super);
> +               data_blocks += (unsigned long long)extra_grps *
> +                       EXT2_BLOCKS_PER_GROUP(fs->super);
>
>                  /* ok we have to account for the last group */
>                  overhead = calc_group_overhead(fs, groups-1, old_desc_blocks);
>
>

G'day Azat,

Appreciate you taking a look at this and I see where you are going with it.

While I have a backup of _all_ critical data on that array, I don't have 
20TB of spare space to back up the remainder so if you don't mind I'd 
like to wait for a review prior to letting it loose on the array.

Regards,
Brad

Re: Online resize issue with 3.13.5 & 3.15.6

[Theodore Ts'o]

On Fri, Jul 25, 2014 at 07:44:07PM +0800, Brad Campbell wrote:
> 
> Appreciate you taking a look at this and I see where you are going with it.

This patch looks good to me.  If you could give it a try, I would
appreciate it.

This looks like the same bug which was reported in Ubuntu:

https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1321958

							- Ted

Re: Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

On 25/07/14 22:07, Theodore Ts'o wrote:
> On Fri, Jul 25, 2014 at 07:44:07PM +0800, Brad Campbell wrote:
>>
>> Appreciate you taking a look at this and I see where you are going with it.
>
> This patch looks good to me.  If you could give it a try, I would
> appreciate it.
>
> This looks like the same bug which was reported in Ubuntu:
>
> https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1321958


The patch would not apply automatically to 1.42.11 but was easy enough 
to apply by hand.

root@srv:~/src/e2fsprogs/e2fsprogs-1.42.11/resize# ./resize2fs -d 255 
/dev/md0
resize2fs 1.42.11 (09-Jul-2014)
fs has 4007207 inodes, 1957 groups required.
fs requires 4374122900 data blocks.
With 1957 group(s), we have 63820826 blocks available.
Added 131540 extra group(s), blks_needed 4374122900, data_blocks 
4356990326, last_start 4356599580
Added 523 extra group(s), blks_needed 4374122900, data_blocks 
4374059350, last_start 4373440788
Added 2 extra group(s), blks_needed 4374122900, data_blocks 4374124886, 
last_start 4373473426
Last group's overhead is 1430
Need 649474 data blocks in last group
Final size of last group is 650904
Estimated blocks needed: 4391437088
Extents safety margin: 2826959
Filesystem at /dev/md0 is mounted on /server; on-line resizing required
old_desc_blocks = 2768, new_desc_blocks = 2795
./resize2fs: Invalid argument While checking for on-line resizing support

[489412.650430] EXT4-fs (md0): resizing filesystem from 5804916736 to 
5860149888 blocks
[489412.700282] EXT4-fs warning (device md0): verify_reserved_gdb:713: 
reserved GDT 2769 missing grp 177147 (5804755665)
[489412.739676] EXT4-fs (md0): resized filesystem to 5860149888
[489413.215230] EXT4-fs warning (device md0): ext4_group_extend:1720: 
can't shrink FS - resize aborted

Re: Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

On 26/07/14 11:31, Brad Campbell wrote:
> On 25/07/14 22:07, Theodore Ts'o wrote:
>> On Fri, Jul 25, 2014 at 07:44:07PM +0800, Brad Campbell wrote:
>>>
>>> Appreciate you taking a look at this and I see where you are going
>>> with it.
>>
>> This patch looks good to me.  If you could give it a try, I would
>> appreciate it.
>>
>> This looks like the same bug which was reported in Ubuntu:
>>
>> https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1321958
>
>
> The patch would not apply automatically to 1.42.11 but was easy enough
> to apply by hand.
>
> root@srv:~/src/e2fsprogs/e2fsprogs-1.42.11/resize# ./resize2fs -d 255
> /dev/md0
> resize2fs 1.42.11 (09-Jul-2014)
> fs has 4007207 inodes, 1957 groups required.
> fs requires 4374122900 data blocks.
> With 1957 group(s), we have 63820826 blocks available.
> Added 131540 extra group(s), blks_needed 4374122900, data_blocks
> 4356990326, last_start 4356599580
> Added 523 extra group(s), blks_needed 4374122900, data_blocks
> 4374059350, last_start 4373440788
> Added 2 extra group(s), blks_needed 4374122900, data_blocks 4374124886,
> last_start 4373473426
> Last group's overhead is 1430
> Need 649474 data blocks in last group
> Final size of last group is 650904
> Estimated blocks needed: 4391437088
> Extents safety margin: 2826959
> Filesystem at /dev/md0 is mounted on /server; on-line resizing required
> old_desc_blocks = 2768, new_desc_blocks = 2795
> ./resize2fs: Invalid argument While checking for on-line resizing support
>
> [489412.650430] EXT4-fs (md0): resizing filesystem from 5804916736 to
> 5860149888 blocks
> [489412.700282] EXT4-fs warning (device md0): verify_reserved_gdb:713:
> reserved GDT 2769 missing grp 177147 (5804755665)
> [489412.739676] EXT4-fs (md0): resized filesystem to 5860149888
> [489413.215230] EXT4-fs warning (device md0): ext4_group_extend:1720:
> can't shrink FS - resize aborted

Rebooted, did an fsck -f (was clean) and ran it again without the fs 
mounted.

root@srv:~/src/e2fsprogs/e2fsprogs-1.42.11/resize# ./resize2fs -d 255 
/dev/md0
resize2fs 1.42.11 (09-Jul-2014)
fs has 4007359 inodes, 1957 groups required.
fs requires 4655464570 data blocks.
With 1957 group(s), we have 63820826 blocks available.
Added 140126 extra group(s), blks_needed 4655464570, data_blocks 
4637219414, last_start 4636829448
Added 557 extra group(s), blks_needed 4655464570, data_blocks 
4655398390, last_start 4654584520
Added 3 extra group(s), blks_needed 4655464570, data_blocks 4655496694, 
last_start 4654617158
Last group's overhead is 1820
Need 847412 data blocks in last group
Final size of last group is 849232
Estimated blocks needed: 4674027936
Extents safety margin: 2261777
Resizing the filesystem on /dev/md0 to 5860149888 (4k) blocks.
read_bitmaps: Memory used: 132k/763980k (49k/84k), time: 113.93/ 1.26/ 4.45
read_bitmaps: I/O read: 648MB, write: 0MB, rate: 5.69MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/763980k (49k/84k), time: 
7.61/ 7.91/ 0.00
adjust_superblock: Memory used: 132k/18014398507538268k (52k/81k), time: 
  0.49/ 0.57/ 0.00
fix_uninit_block_bitmaps 2: Memory used: 132k/18014398507538268k 
(52k/81k), time:  8.10/ 8.17/ 0.00
blocks_to_move: Memory used: 132k/18014398508253624k (52k/81k), time: 
0.74/ 0.50/ 0.05
Number of free blocks: 1126335202/7242495991050, Needed: 0
block_mover: Memory used: 132k/18014398508254140k (57k/76k), time: 
192.51/192.28/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 0.01MB/s
inode_scan_and_fix: Memory used: 132k/18014398508254140k (57k/76k), 
time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 132k/18014398508254140k (57k/76k), time: 
0.00/ 0.00/ 0.00
move_itables: Memory used: 132k/18014398508254140k (57k/76k), time: 
0.01/ 0.00/ 0.00
calculate_summary_stats: Memory used: 132k/18014398508254140k (57k/76k), 
time: 168.74/168.43/ 0.01
fix_resize_inode: Memory used: 132k/18014398508254140k (61k/72k), time: 
  0.00/ 0.00/ 0.00
fix_resize_inode: I/O read: 0MB, write: 3MB, rate: 1843.88MB/s
fix_sb_journal_backup: Memory used: 132k/18014398508254140k (61k/72k), 
time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 132k/18014398508254140k (61k/72k), time: 
492.61/379.13/ 4.52
overall resize2fs: I/O read: 648MB, write: 13MB, rate: 1.34MB/s
The filesystem on /dev/md0 is now 5860149888 blocks long.

So it would appear to have succeeded when run offline.

Re: Online resize issue with 3.13.5 & 3.15.6

[Azat Khuzhin]

On Sat, Jul 26, 2014 at 12:12:11PM +0800, Brad Campbell wrote:
> On 26/07/14 11:31, Brad Campbell wrote:
> >On 25/07/14 22:07, Theodore Ts'o wrote:
> >>On Fri, Jul 25, 2014 at 07:44:07PM +0800, Brad Campbell wrote:
> >>>
> >>>Appreciate you taking a look at this and I see where you are going
> >>>with it.
> >>
> >>This patch looks good to me.  If you could give it a try, I would
> >>appreciate it.
> >>
> >>This looks like the same bug which was reported in Ubuntu:
> >>
> >>https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1321958
> >
> >
> >The patch would not apply automatically to 1.42.11 but was easy enough
> >to apply by hand.
> >
> >root@srv:~/src/e2fsprogs/e2fsprogs-1.42.11/resize# ./resize2fs -d 255
> >/dev/md0
> >resize2fs 1.42.11 (09-Jul-2014)
> >fs has 4007207 inodes, 1957 groups required.
> >fs requires 4374122900 data blocks.
> >With 1957 group(s), we have 63820826 blocks available.
> >Added 131540 extra group(s), blks_needed 4374122900, data_blocks
> >4356990326, last_start 4356599580
> >Added 523 extra group(s), blks_needed 4374122900, data_blocks
> >4374059350, last_start 4373440788
> >Added 2 extra group(s), blks_needed 4374122900, data_blocks 4374124886,
> >last_start 4373473426
> >Last group's overhead is 1430
> >Need 649474 data blocks in last group
> >Final size of last group is 650904
> >Estimated blocks needed: 4391437088
> >Extents safety margin: 2826959
> >Filesystem at /dev/md0 is mounted on /server; on-line resizing required
> >old_desc_blocks = 2768, new_desc_blocks = 2795
> >./resize2fs: Invalid argument While checking for on-line resizing support
> >
> >[489412.650430] EXT4-fs (md0): resizing filesystem from 5804916736 to
> >5860149888 blocks
> >[489412.700282] EXT4-fs warning (device md0): verify_reserved_gdb:713:
> >reserved GDT 2769 missing grp 177147 (5804755665)
> >[489412.739676] EXT4-fs (md0): resized filesystem to 5860149888
> >[489413.215230] EXT4-fs warning (device md0): ext4_group_extend:1720:
> >can't shrink FS - resize aborted

Online resizing needs support in kernel, so it is not strange that it
doesn't fixed with this patch.
I will take a look into online resizing code, but don't promise
anything.

> 
> Rebooted, did an fsck -f (was clean) and ran it again without the fs
> mounted.
> 
> root@srv:~/src/e2fsprogs/e2fsprogs-1.42.11/resize# ./resize2fs -d 255
> /dev/md0
> resize2fs 1.42.11 (09-Jul-2014)
> fs has 4007359 inodes, 1957 groups required.
> fs requires 4655464570 data blocks.
> With 1957 group(s), we have 63820826 blocks available.
> Added 140126 extra group(s), blks_needed 4655464570, data_blocks 4637219414,
> last_start 4636829448
> Added 557 extra group(s), blks_needed 4655464570, data_blocks 4655398390,
> last_start 4654584520
> Added 3 extra group(s), blks_needed 4655464570, data_blocks 4655496694,
> last_start 4654617158
> Last group's overhead is 1820
> Need 847412 data blocks in last group
> Final size of last group is 849232
> Estimated blocks needed: 4674027936
> Extents safety margin: 2261777
> Resizing the filesystem on /dev/md0 to 5860149888 (4k) blocks.
> read_bitmaps: Memory used: 132k/763980k (49k/84k), time: 113.93/ 1.26/ 4.45
> read_bitmaps: I/O read: 648MB, write: 0MB, rate: 5.69MB/s
> fix_uninit_block_bitmaps 1: Memory used: 132k/763980k (49k/84k), time: 7.61/
> 7.91/ 0.00
> adjust_superblock: Memory used: 132k/18014398507538268k (52k/81k), time:
> 0.49/ 0.57/ 0.00
> fix_uninit_block_bitmaps 2: Memory used: 132k/18014398507538268k (52k/81k),
> time:  8.10/ 8.17/ 0.00
> blocks_to_move: Memory used: 132k/18014398508253624k (52k/81k), time: 0.74/
> 0.50/ 0.05
> Number of free blocks: 1126335202/7242495991050, Needed: 0
> block_mover: Memory used: 132k/18014398508254140k (57k/76k), time:
> 192.51/192.28/ 0.00
> block_mover: I/O read: 1MB, write: 0MB, rate: 0.01MB/s
> inode_scan_and_fix: Memory used: 132k/18014398508254140k (57k/76k), time:
> 0.00/ 0.00/ 0.00
> inode_ref_fix: Memory used: 132k/18014398508254140k (57k/76k), time: 0.00/
> 0.00/ 0.00
> move_itables: Memory used: 132k/18014398508254140k (57k/76k), time: 0.01/
> 0.00/ 0.00
> calculate_summary_stats: Memory used: 132k/18014398508254140k (57k/76k),
> time: 168.74/168.43/ 0.01
> fix_resize_inode: Memory used: 132k/18014398508254140k (61k/72k), time:
> 0.00/ 0.00/ 0.00
> fix_resize_inode: I/O read: 0MB, write: 3MB, rate: 1843.88MB/s
> fix_sb_journal_backup: Memory used: 132k/18014398508254140k (61k/72k), time:
> 0.00/ 0.00/ 0.00
> overall resize2fs: Memory used: 132k/18014398508254140k (61k/72k), time:
> 492.61/379.13/ 4.52
> overall resize2fs: I/O read: 648MB, write: 13MB, rate: 1.34MB/s
> The filesystem on /dev/md0 is now 5860149888 blocks long.
> 
> So it would appear to have succeeded when run offline.

Brad, many thanks for testing, I will resend patch later.
(with your tested-by if you don't mind)

Thanks.
Azat.

Re: Online resize issue with 3.13.5 & 3.15.6

[Azat Khuzhin]

On Sat, Jul 26, 2014 at 11:31:02AM +0800, Brad Campbell wrote:
> On 25/07/14 22:07, Theodore Ts'o wrote:
> >On Fri, Jul 25, 2014 at 07:44:07PM +0800, Brad Campbell wrote:
> >>
> >>Appreciate you taking a look at this and I see where you are going with it.
> >
> >This patch looks good to me.  If you could give it a try, I would
> >appreciate it.
> >
> >This looks like the same bug which was reported in Ubuntu:
> >
> >https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1321958
> 
> 
> The patch would not apply automatically to 1.42.11 but was easy enough to
> apply by hand.
> 
> root@srv:~/src/e2fsprogs/e2fsprogs-1.42.11/resize# ./resize2fs -d 255
> /dev/md0
> resize2fs 1.42.11 (09-Jul-2014)
> fs has 4007207 inodes, 1957 groups required.
> fs requires 4374122900 data blocks.
> With 1957 group(s), we have 63820826 blocks available.
> Added 131540 extra group(s), blks_needed 4374122900, data_blocks 4356990326,
> last_start 4356599580
> Added 523 extra group(s), blks_needed 4374122900, data_blocks 4374059350,
> last_start 4373440788
> Added 2 extra group(s), blks_needed 4374122900, data_blocks 4374124886,
> last_start 4373473426
> Last group's overhead is 1430
> Need 649474 data blocks in last group
> Final size of last group is 650904
> Estimated blocks needed: 4391437088
> Extents safety margin: 2826959
> Filesystem at /dev/md0 is mounted on /server; on-line resizing required
> old_desc_blocks = 2768, new_desc_blocks = 2795
> ./resize2fs: Invalid argument While checking for on-line resizing support
> 
> [489412.650430] EXT4-fs (md0): resizing filesystem from 5804916736 to
> 5860149888 blocks
> [489412.700282] EXT4-fs warning (device md0): verify_reserved_gdb:713:
> reserved GDT 2769 missing grp 177147 (5804755665)
> [489412.739676] EXT4-fs (md0): resized filesystem to 5860149888
> [489413.215230] EXT4-fs warning (device md0): ext4_group_extend:1720: can't
> shrink FS - resize aborted

And I guess that attached patch can fix the online resizing issue.
Ted, could you take a look?

Thanks.
Azat.

diff --git a/fs/ext4/resize.c b/fs/ext4/resize.c
index bb0e80f..38f7ced 100644
--- a/fs/ext4/resize.c
+++ b/fs/ext4/resize.c
@@ -1928,7 +1928,8 @@ retry:
                        n_desc_blocks = o_desc_blocks +
                                le16_to_cpu(es->s_reserved_gdt_blocks);
                        n_group = n_desc_blocks * EXT4_DESC_PER_BLOCK(sb);
-                       n_blocks_count = n_group * EXT4_BLOCKS_PER_GROUP(sb);
+                       n_blocks_count = (ext4_fsblk_t)n_group *
+                               EXT4_BLOCKS_PER_GROUP(sb);
                        n_group--; /* set to last group number */
                }

Re: Online resize issue with 3.13.5 & 3.15.6

[Theodore Ts'o]

OK, it looks like the e2fsprogs patch got you through the first
hurdle, but the failure is something that made no sense at first:

> [489412.650430] EXT4-fs (md0): resizing filesystem from 5804916736 to
> 5860149888 blocks
> [489412.700282] EXT4-fs warning (device md0): verify_reserved_gdb:713:
> reserved GDT 2769 missing grp 177147 (5804755665)

The code path which emitted the above warning something that should
ever be entered for file systems greater than 16TB.  But then I took a
look at the first message that you sent on this thread, and I think
see what's going wrong.  From your dumpe2fs -h output:

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype
extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
extra_isize
Block count:              5804916736
Reserved GDT blocks:      585

If the block count is greater than 2**32 (4294967296), resize_inode
must not be set, and reserved GDT blocks should be zero.  So this is
definitely not right.

I'm going to guess that this file system was originally a smaller size
(and probably smaller than 16T), and then was resized to 22TB,
probably using an earlier version of the kernel and/or e2fsprogs.  Is
my guess correct?  If so, do you remember the history of what size the
file system was, and in what steps it was resized, and what version of
the e2fsprogs and the kernel that was used at each stage, starting
from the original mke2fs and each successive resize?

I'm guessing what happened is that an earlier version of the kernel or
the e2fsprogs left the file system in an unexpected state.  It's
apparently one that e2fsck didn't complain about, but it definitely
wasn't normal.

In any case, the way I would suggest proceeding is the following.

1)  Unmount the file system

2)  Run "e2fsck -f" on the file system to make sure things look sane.

3) Run "debugfs -R 'stat <7>' /dev/md0" and "dumpe2fs -h /dev/md0" and
   send me the outputs just because I'm curious exactly what state the
   resize_inode --- which really shouldn't be there --- was actually in.

4)  Run "tune2fs -O ^resize_inode" on the file system

5) Run e2fsck -fy" on the file system.  The "errors" that fixed are
the result of clearing the resize_inode; don't be alarmed by them.

6) Remount the file system

7) Retry the resize2fs, making sure you are using your 3.15.6 kernel

That should hopefully allow things to work correctly.  In the future,
I will want to make the kernel and e2fsprogs more robust against this
sort of "should never happen" state, which is why I'm interested in
knowing the history of how the file system had been created and
resized in the past, and what the state was of the resize inode before
we blow it away.

Cheers,

							- Ted

Re: Online resize issue with 3.13.5 & 3.15.6

[Azat Khuzhin]

On Sat, Jul 26, 2014 at 08:45:57AM -0400, Theodore Ts'o wrote:
> Filesystem features:      has_journal ext_attr resize_inode dir_index filetype
> extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
> extra_isize
> Block count:              5804916736
> Reserved GDT blocks:      585
> 
> If the block count is greater than 2**32 (4294967296), resize_inode
> must not be set, and reserved GDT blocks should be zero.  So this is
> definitely not right.

I've just tried to reproduce this on 24T->48T raid0 array (resizing from
24T to 48T), without any success.
And that's why: I don't have resize_inode in filesystem features.
Thanks for pointing on this, Ted.

Cheers,
Azat.

Re: Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

On 26/07/14 20:45, Theodore Ts'o wrote:
> OK, it looks like the e2fsprogs patch got you through the first
> hurdle, but the failure is something that made no sense at first:
>
>> [489412.650430] EXT4-fs (md0): resizing filesystem from 5804916736 to
>> 5860149888 blocks
>> [489412.700282] EXT4-fs warning (device md0): verify_reserved_gdb:713:
>> reserved GDT 2769 missing grp 177147 (5804755665)
> The code path which emitted the above warning something that should
> ever be entered for file systems greater than 16TB.  But then I took a
> look at the first message that you sent on this thread, and I think
> see what's going wrong.  From your dumpe2fs -h output:
>
> Filesystem features:      has_journal ext_attr resize_inode dir_index filetype
> extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
> extra_isize
> Block count:              5804916736
> Reserved GDT blocks:      585
>
> If the block count is greater than 2**32 (4294967296), resize_inode
> must not be set, and reserved GDT blocks should be zero.  So this is
> definitely not right.
>
> I'm going to guess that this file system was originally a smaller size
> (and probably smaller than 16T), and then was resized to 22TB,
> probably using an earlier version of the kernel and/or e2fsprogs.  Is
> my guess correct?  If so, do you remember the history of what size the
> file system was, and in what steps it was resized, and what version of
> the e2fsprogs and the kernel that was used at each stage, starting
> from the original mke2fs and each successive resize?
>
This was the first resize of this FS. Initially this array was about 
15T. About 12 months ago I attempted to resize it up to 19T and bumped 
up against the fact I had not created the initial filesystem with 64 bit 
support, so I cobbled together some storage and did a 
backup/create/restore. At that point I would probably have specified 
resize_inode manually as an option (as reading the man page it looked 
like a good idea as I always had plans to expand in future) to mke2fs 
along with 64bit. Fast forward 12 months and I've added 2 drives to the 
array and bumped up against this issue. So it was initially 4883458240 
blocks. It would have been created with e2fsprogs from Debian Stable (so 
1.42.5).

I can't test this to verify my memory however as I don't seem to be able 
to create a sparse file large enough to create a filesystem in. I appear 
to be bumping up against a 2T filesize limit.

-- 
Dolphins are so intelligent that within a few weeks they can
train Americans to stand at the edge of the pool and throw them
fish.

Re: Online resize issue with 3.13.5 & 3.15.6

[Theodore Ts'o]

On Sat, Jul 26, 2014 at 09:46:20PM +0800, Brad Campbell wrote:
> This was the first resize of this FS. Initially this array was about 15T.
> About 12 months ago I attempted to resize it up to 19T and bumped up against
> the fact I had not created the initial filesystem with 64 bit support, so I
> cobbled together some storage and did a backup/create/restore. At that point
> I would probably have specified resize_inode manually as an option (as
> reading the man page it looked like a good idea as I always had plans to
> expand in future) to mke2fs along with 64bit. Fast forward 12 months and
> I've added 2 drives to the array and bumped up against this issue. So it was
> initially 4883458240 blocks. It would have been created with e2fsprogs from
> Debian Stable (so 1.42.5).

So mke2fs 1.42.11 does the right thing (although it really should just
tell you that there's no point using the resize_inode).

% mke2fs -Fq -t ext4 -O resize_inode,64bit /mnt/foo.img 19T
/mnt/foo.img contains a ext4 file system
	     created on Sat Jul 26 09:54:30 2014

% dumpe2fs -h /mnt/foo.img  | grep "Reserved GDT"
dumpe2fs 1.42.11 (09-Jul-2014)

% debugfs -R "stat <7>" /mnt/foo.img
debugfs 1.42.11 (09-Jul-2014)
Inode: 7   Type: bad type    Mode:  0000   Flags: 0x0
Generation: 0    Version: 0x00000000
User:     0   Group:     0   Size: 0
File ACL: 0    Directory ACL: 0
Links: 0   Blockcount: 0
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x00000000 -- Wed Dec 31 19:00:00 1969
atime: 0x00000000 -- Wed Dec 31 19:00:00 1969
mtime: 0x00000000 -- Wed Dec 31 19:00:00 1969
Size of extra inode fields: 0
BLOCKS:

> I can't test this to verify my memory however as I don't seem to be able to
> create a sparse file large enough to create a filesystem in. I appear to be
> bumping up against a 2T filesize limit.

Yep, when I do this testing I create a loopback mounted (sparse) xfs
file system, so I can create the sparse files needed to do this sort
of testing.

My guess is that 1.42.5 is not doing the right thing, although I
haven't had a chance to test it yet.

					- Ted

Re: Online resize issue with 3.13.5 & 3.15.6

[Theodore Ts'o]

On Sat, Jul 26, 2014 at 09:46:20PM +0800, Brad Campbell wrote:
> Fast forward 12 months and
> I've added 2 drives to the array and bumped up against this issue. So it was
> initially 4883458240 blocks. It would have been created with e2fsprogs from
> Debian Stable (so 1.42.5).

Confirmed; debian stable's mke2fs version 1.42.5 does the wrong thing
if you do:

   mke2fs -t ext4 -O 64bit,resize_inode /mnt/foo.img  19T

It creates a file system which is has a resize_inode which --- well,
is in a format that the kernel isn't capable of handling and which is
not something that was originally intended.  The interesting thing is
that resize2fs 1.42.11 is apparently able to handle resizing the above
file system created with mke2fs 1.42.5.

Mke2fs 1.42.12 will avoid creating a resize_inode for file systems >
16T, so this situation doesn't arise.

I need to take a closer look to see if there's some way I can teach
the kernel to be able to deal with this file system, but for now, you
can work around it by using tune2fs to remove the resize_inode, and
after running e2fsck and remounting the file system, the on-line
resize will work correctly.

Cheers,

					- Ted

Re: Online resize issue with 3.13.5 & 3.15.6

[Brad Campbell]

On 29/07/14 10:46, Theodore Ts'o wrote:
> On Sat, Jul 26, 2014 at 09:46:20PM +0800, Brad Campbell wrote:
>> Fast forward 12 months and
>> I've added 2 drives to the array and bumped up against this issue. So it was
>> initially 4883458240 blocks. It would have been created with e2fsprogs from
>> Debian Stable (so 1.42.5).
>
> Confirmed; debian stable's mke2fs version 1.42.5 does the wrong thing
> if you do:
>
>     mke2fs -t ext4 -O 64bit,resize_inode /mnt/foo.img  19T
>
> It creates a file system which is has a resize_inode which --- well,
> is in a format that the kernel isn't capable of handling and which is
> not something that was originally intended.  The interesting thing is
> that resize2fs 1.42.11 is apparently able to handle resizing the above
> file system created with mke2fs 1.42.5.

And it did just fine once patched for the 32bit overflow. The off-line 
resize went without a hitch. Oddly enough the Kernel took it out to 
5641863168 blocks ok in the first online attempt.

> Mke2fs 1.42.12 will avoid creating a resize_inode for file systems >
> 16T, so this situation doesn't arise.
>
> I need to take a closer look to see if there's some way I can teach
> the kernel to be able to deal with this file system, but for now, you
> can work around it by using tune2fs to remove the resize_inode, and
> after running e2fsck and remounting the file system, the on-line
> resize will work correctly.

Would it not be easier just to put a test in resize2fs which says "I 
can't do that, please fix your filesystem first" ?

1.42.5 won't try (32 bit limit).
1.42.11 will spin (32 bit overflow)
So just make 1.42.12 abort with an error.

Is it worth having e2fsck check for mis-applied features and flags?