Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation

[Ferry Toth <fntoth@gmail.com>]

Op 20-10-2019 om 16:11 schreef Qu Wenruo:
> 
> 
> On 2019/10/20 下午9:29, Ferry Toth wrote:
>> Op 20-10-2019 om 15:15 schreef Qu WenRuo:
>>>
>>>
>>> On 2019/10/20 下午9:04, Ferry Toth wrote:
>>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>>>>
>>>>>
>>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>>>>
>>>>>>
>>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>>>>> We have at least two user reports about bad inode generation makes
>>>>>>>>>> kernel reject the fs.
>>>>>>>>>
>>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>>>>
>>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>>>>> on /@
>>>>>>>>>
>>>>>>>>> In initramfs I can try to mount but get something like
>>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>>>>> failed
>>>>>>>>>
>>>>>>>>> Booting old kernel works just as before, no errors.
>>>>>>>>>
>>>>>>>>>> According to the creation time, the inode is created by some 2014
>>>>>>>>>> kernel.
>>>>>>>>>
>>>>>>>>> How do I get the creation time?
>>>>>>>>
>>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your
>>>>>>>> device>
>>>>>>>
>>>>>>> I just went back to the office to reboot to 5.3 and check the
>>>>>>> creation
>>>>>>> times and found they were 2013 - 2014.
>>>>>>>
>>>>>>>>>
>>>>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>>>>> detects.
>>>>>>>>>>
>>>>>>>>>> Even the situation can be fixed by reverting back to older kernel
>>>>>>>>>> and
>>>>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>>>>> offending
>>>>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>>>>
>>>>>>>>> How to find the offending dir/file from the command line manually?
>>>>>>>>
>>>>>>>> # find <mount point> -inum <inode number>
>>>>>>>
>>>>>>> This works, thanks.
>>>>>>>
>>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4
>>>>>>> more,
>>>>>>> then 16, then I gave up.
>>>>>
>>>>> Another solution is use "find" to locate the files with creation time
>>>>> before 2015, and copy them to a new file, then replace the old file
>>>>> with
>>>>> the new file.
>>>>
>>>> Hmm. But how do I "find" by creation time (otime)? Do you have a
>>>> suggestion for this?
>>>
>>> $ touch -t 201501010000 /tmp/sample
>>> $ find <mnt> -not -cnewer /tmp/sample
>>
>> AFAIK this compares file modified date with status changed date. So, no
>> search for creation date.
>>
>> And stat /tmp/sample (sorry dutch lang output):
>>
>> ferry@ferry-quad:~$ stat /tmp/sample
>>    Bestand: /tmp/sample
>>    Grootte: 0            Blokken: 0            IO-blok: 4096   leeg
>> normaal bestand
>> Apparaat: 1bh/27d   Inode: 62005381     Koppelingen: 1
>> Toegang: (0664/-rw-rw-r--)   UID: ( 1001/   ferry)   GID: ( 1001/   ferry)
>> Toegang:   2015-01-01 00:00:00.000000000 +0100
>> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100
>> Veranderd: 2019-10-20 15:20:50.366163766 +0200
>> Ontstaan:  -
> 
> My bad, always got confused by o/a/c/mtime, as c really looks like *c*
> reation, so I always got confused between ctime and otime.
> 
> Then considering not all fs supports otime, find doesn't support that.
> I guess it's only possible by other tools....

New stat will support it, but not in Ubuntu 19.10. We did find this:
https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.c

and are trying to work out a script that will gzip all files with 
creation < 2015. Then we can unzip again and overwrite.

> BTW, did you find any patterns in those existing offending inodes?
> I guess it would be faster than finding a tool supporting otime search.

I didn't see any logic. A mix of logs, cached files, journal files etc. 
+ a .kde directory in the /

> Thanks,
> Qu
> 
>>
>>
>>> If you want, you can add -exec to that find, but I'd only add that after
>>> confirming the execution load is verified.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>>> It would be much safer than btrfs check --repair.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>>>>> with a
>>>>>>>>>> simple test image.
>>>>>>>>>>
>>>>>>>>>> Qu Wenruo (3):
>>>>>>>>>>        btrfs-progs: check/lowmem: Add check and repair for invalid
>>>>>>>>>> inode
>>>>>>>>>>          generation
>>>>>>>>>>        btrfs-progs: check/original: Add check and repair for
>>>>>>>>>> invalid inode
>>>>>>>>>>          generation
>>>>>>>>>>        btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>>>>> generation
>>>>>>>>>>          repair
>>>>>>>>>>
>>>>>>>>>>       check/main.c                                  |  50
>>>>>>>>>> +++++++++++-
>>>>>>>>>>       check/mode-lowmem.c                           |  76
>>>>>>>>>> ++++++++++++++++++
>>>>>>>>>>       check/mode-original.h                         |   1 +
>>>>>>>>>>       .../.lowmem_repairable                        |   0
>>>>>>>>>>       .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012
>>>>>>>>>> bytes
>>>>>>>>>>       5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>>>>       create mode 100644
>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>>>>       create mode 100644
>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>>>>> errors).
>>>>>>
>>>>>> It's really not recommended to run btrfs check, especially repair
>>>>>> on the
>>>>>> mounted fs, unless it's RO.
>>>>>>
>>>>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>>>>
>>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted
>>>>>>> distorts
>>>>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>>>>>> v5.3 btrfs-progs.
>>>>>>>
>>>>>>> If you like I can share the log. Let me know.
>>>>>>>
>>>>>>> This issue can potentially cause a lot of grief. Our company server
>>>>>>> runs
>>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on
>>>>>>> each
>>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>>>>> ~5.6)?
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>>>
>>>>>>> Do I understand correctly that this FTB is caused by more strict
>>>>>>> checking of the fs by the kernel, while the tools to fix the detected
>>>>>>> corruptions are not yet released?
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>
>>>>
>>
>