Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation

[Ferry Toth <fntoth@gmail.com>]

Op 20-10-2019 om 16:24 schreef Ferry Toth:
> Op 20-10-2019 om 16:11 schreef Qu Wenruo:
>>
>>
>> On 2019/10/20 下午9:29, Ferry Toth wrote:
>>> Op 20-10-2019 om 15:15 schreef Qu WenRuo:
>>>>
>>>>
>>>> On 2019/10/20 下午9:04, Ferry Toth wrote:
>>>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>>>>>
>>>>>>
>>>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>>>>>> We have at least two user reports about bad inode generation 
>>>>>>>>>>> makes
>>>>>>>>>>> kernel reject the fs.
>>>>>>>>>>
>>>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 
>>>>>>>>>> 19.10 so
>>>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>>>>>
>>>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>>>>>> on /@
>>>>>>>>>>
>>>>>>>>>> In initramfs I can try to mount but get something like
>>>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>>>>>> failed
>>>>>>>>>>
>>>>>>>>>> Booting old kernel works just as before, no errors.
>>>>>>>>>>
>>>>>>>>>>> According to the creation time, the inode is created by some 
>>>>>>>>>>> 2014
>>>>>>>>>>> kernel.
>>>>>>>>>>
>>>>>>>>>> How do I get the creation time?
>>>>>>>>>
>>>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your
>>>>>>>>> device>
>>>>>>>>
>>>>>>>> I just went back to the office to reboot to 5.3 and check the
>>>>>>>> creation
>>>>>>>> times and found they were 2013 - 2014.
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> And the generation member of INODE_ITEM is not updated 
>>>>>>>>>>> (unlike the
>>>>>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>>>>>> detects.
>>>>>>>>>>>
>>>>>>>>>>> Even the situation can be fixed by reverting back to older 
>>>>>>>>>>> kernel
>>>>>>>>>>> and
>>>>>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>>>>>> offending
>>>>>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>>>>>
>>>>>>>>>> How to find the offending dir/file from the command line 
>>>>>>>>>> manually?
>>>>>>>>>
>>>>>>>>> # find <mount point> -inum <inode number>
>>>>>>>>
>>>>>>>> This works, thanks.
>>>>>>>>
>>>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4
>>>>>>>> more,
>>>>>>>> then 16, then I gave up.
>>>>>>
>>>>>> Another solution is use "find" to locate the files with creation time
>>>>>> before 2015, and copy them to a new file, then replace the old file
>>>>>> with
>>>>>> the new file.
>>>>>
>>>>> Hmm. But how do I "find" by creation time (otime)? Do you have a
>>>>> suggestion for this?
>>>>
>>>> $ touch -t 201501010000 /tmp/sample
>>>> $ find <mnt> -not -cnewer /tmp/sample
>>>
>>> AFAIK this compares file modified date with status changed date. So, no
>>> search for creation date.
>>>
>>> And stat /tmp/sample (sorry dutch lang output):
>>>
>>> ferry@ferry-quad:~$ stat /tmp/sample
>>>    Bestand: /tmp/sample
>>>    Grootte: 0            Blokken: 0            IO-blok: 4096   leeg
>>> normaal bestand
>>> Apparaat: 1bh/27d   Inode: 62005381     Koppelingen: 1
>>> Toegang: (0664/-rw-rw-r--)   UID: ( 1001/   ferry)   GID: ( 1001/   
>>> ferry)
>>> Toegang:   2015-01-01 00:00:00.000000000 +0100
>>> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100
>>> Veranderd: 2019-10-20 15:20:50.366163766 +0200
>>> Ontstaan:  -
>>
>> My bad, always got confused by o/a/c/mtime, as c really looks like *c*
>> reation, so I always got confused between ctime and otime.
>>
>> Then considering not all fs supports otime, find doesn't support that.
>> I guess it's only possible by other tools....
> 
> New stat will support it, but not in Ubuntu 19.10. We did find this:
> https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.c
> 
> and are trying to work out a script that will gzip all files with 
> creation < 2015. Then we can unzip again and overwrite.

So we combined find + test-statx + awk to produce a list of files with 
creation date 1970 that are then fed to tar. Finally we untar that to 
overwrite the original files.

I found the following:
- the problematic files are both ordinary and symlinks
- they all seem to have the same inode generation
- all had no creation date (so were set to jan 1970)
- not all files without creation date were problematic (only ~10%)
- modified dates were 2013/2014 afaikt

The procedure keeps the modified date, but sets creation date to today, 
creates a new inode and fixes the error.

>> BTW, did you find any patterns in those existing offending inodes?
>> I guess it would be faster than finding a tool supporting otime search.
> 
> I didn't see any logic. A mix of logs, cached files, journal files etc. 
> + a .kde directory in the /
> 
>> Thanks,
>> Qu
>>
>>>
>>>
>>>> If you want, you can add -exec to that find, but I'd only add that 
>>>> after
>>>> confirming the execution load is verified.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>
>>>>>> It would be much safer than btrfs check --repair.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>>>>>> with a
>>>>>>>>>>> simple test image.
>>>>>>>>>>>
>>>>>>>>>>> Qu Wenruo (3):
>>>>>>>>>>>        btrfs-progs: check/lowmem: Add check and repair for 
>>>>>>>>>>> invalid
>>>>>>>>>>> inode
>>>>>>>>>>>          generation
>>>>>>>>>>>        btrfs-progs: check/original: Add check and repair for
>>>>>>>>>>> invalid inode
>>>>>>>>>>>          generation
>>>>>>>>>>>        btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>>>>>> generation
>>>>>>>>>>>          repair
>>>>>>>>>>>
>>>>>>>>>>>       check/main.c                                  |  50
>>>>>>>>>>> +++++++++++-
>>>>>>>>>>>       check/mode-lowmem.c                           |  76
>>>>>>>>>>> ++++++++++++++++++
>>>>>>>>>>>       check/mode-original.h                         |   1 +
>>>>>>>>>>>       .../.lowmem_repairable                        |   0
>>>>>>>>>>>       .../bad_inode_geneartion.img.xz               | Bin 0 
>>>>>>>>>>> -> 2012
>>>>>>>>>>> bytes
>>>>>>>>>>>       5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>>>>>       create mode 100644
>>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>>>>>       create mode 100644
>>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz 
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>>>>>> errors).
>>>>>>>
>>>>>>> It's really not recommended to run btrfs check, especially repair
>>>>>>> on the
>>>>>>> mounted fs, unless it's RO.
>>>>>>>
>>>>>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>>>>>
>>>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted
>>>>>>>> distorts
>>>>>>>> the results? Else I'm going to need a live usb with a v5.3 
>>>>>>>> kernel and
>>>>>>>> v5.3 btrfs-progs.
>>>>>>>>
>>>>>>>> If you like I can share the log. Let me know.
>>>>>>>>
>>>>>>>> This issue can potentially cause a lot of grief. Our company server
>>>>>>>> runs
>>>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs 
>>>>>>>> with
>>>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on
>>>>>>>> each
>>>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>>>>>> ~5.6)?
>>>>>>>
>>>>>>> Yes.
>>>>>>>
>>>>>>>>
>>>>>>>> Do I understand correctly that this FTB is caused by more strict
>>>>>>>> checking of the fs by the kernel, while the tools to fix the 
>>>>>>>> detected
>>>>>>>> corruptions are not yet released?
>>>>>>>
>>>>>>> Yes.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>