Re: resize2fs: Should never happen: resize inode corrupt! - lost key inodes

[Johan Harvyl <johan@harvyl.se>]

On 2015-08-13 15:27, Theodore Ts'o wrote:
> On Thu, Aug 13, 2015 at 12:00:50AM +0200, Johan Harvyl wrote:
>
>>> I'm not aware of any offline resize with 1.42.13, but it sounds like
>>> you were originally using mke2fs and resize2fs 1.42.10, which did have
>>> some bugs, and so the question is what sort of might it might have
>>> left things.
>> What kind of bugs are we talking about, mke2fs? resize2fs? e2fsck? Any
>> specific commits of interest?
> I suspect it was caused by a bug in resize2fs 1.42.10.  The problem is
> that off-line resize2fs is much more powerful; it can handle moving
> file system metadata blocks around, so it can grow file systems in
> cases which aren't supported by online resize --- and it can shrink
> file systems when online resize doesn't support any kind of file
> system shrink.  As such, the code is a lot more complicated, whereas
> the online resize code is much simpler, and ultimately, much more
> robust.
Understood, so would it have been possible to move from my 20 TB -> 24 
TB fs with
online resize? I am confused by the threads I see on the net with 
regards to this.
>> Can you think of why it would zero out the first thousands of
>> inodes, like the root inode, lost+found and so on? I am thinking
>> that would help me assess the potential damage to the files. Could I
>> perhaps expect the same kind of zeroed out blocks at regular
>> intervals all over the device?
> I didn't realize that the first thousands of inodes had been zeroed;
> either you didn't mention this earier or I had missed that from your
> e-mail.  I suspect the resize inode before the resize was pretty
> terribly corrupted, but in a way that e2fsck didn't complain.

Hi,

I may not have been clear on that it was not just the first handful of 
inodes.

When I manually sampled some inodes with debugfs and a disk editor, the 
first group
I found valid inodes in was:
  Group 48: block bitmap at 1572864, inode bitmap at 1572880, inode 
table at 1572896

With 512 inodes per group that would mean at least some 24k inodes are 
blanked out,
but I did not check them all, I just sampled groups manually so there 
could be some
valid in some of the groups below group 48 or a lot more invalid afterwards.

> I'll have to try to reproduce the problem based how you originally
> created and grew the file system and see if I can somehow reproduce
> the problem.  Obviously e2fsck and resize2fs should be changed to make
> this operation much more robust.  If you can tell me the exact
> original size (just under 16TB is probably good enough, but if you
> know the exact starting size, that might be helpful), and then steps
> by which the file system was grown, and which version of e2fsprogs was
> installed at the time, that would be quite helpful.
>
> Thanks,
>
> 						- Ted

Cool, I will try to go through its history in some detail below.

If you have ideas on what I could look for, like ideas on if there is a 
particular periodicity
to the corruption I can write some python to explore such theories.


The filesystem was originally created with e2fsprogs 1.42.10-1 and most 
likely linux-image
3.14 from Debian.

# mkfs.ext4 /dev/md0 -i 262144 -m 0 -O 64bit
mke2fs 1.42.10 (18-May-2014)
Creating filesystem with 3906887168 4k blocks and 61045248 inodes
Filesystem UUID: 13c2eb37-e951-4ad1-b194-21f0880556db
Superblock backups stored on blocks:
         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 
2654208,
         4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
         102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
         2560000000, 3855122432

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
#

It was expanded with 4 TB (another 976721792 4k blocks). Best I can tell 
from my logs this
was done with either e2fsprogs:amd64 1.42.12-1 or 1.42.12-1.1 (debian 
packages) and
Linux 3.16. Everything was running fine after this.
NOTE #1: It does *not* look like this filesystem was ever touched by 
resize2fs 1.42.10.
NOTE #2: The diff between debian packages 1.42.12-1 and 1.42.12-1.1 
appear to be this:
49d0fe2 libext2fs: fix potential buffer overflow in closefs()

Then for the final 4 TB for a total of 5860330752 4k blocks which was 
done with
e2fsprogs:amd64 1.42.13-1 and Linux 4.0. This is where the:
"Should never happen: resize inode corrupt"
was seen.

In both cases the same offline resize was done, with no exotic options:
# umount /dev/md0
# fsck.ext4 -f /dev/md0
# resize2fs /dev/md0

thanks,
-johan