Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation

[Qu WenRuo <wqu@suse.com>]

On 2019/10/20 下午9:04, Ferry Toth wrote:
> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>
>>
>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>
>>>
>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>> Hi,
>>>>
>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>
>>>>>
>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>> We have at least two user reports about bad inode generation makes
>>>>>>> kernel reject the fs.
>>>>>>
>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>
>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>> on /@
>>>>>>
>>>>>> In initramfs I can try to mount but get something like
>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>> failed
>>>>>>
>>>>>> Booting old kernel works just as before, no errors.
>>>>>>
>>>>>>> According to the creation time, the inode is created by some 2014
>>>>>>> kernel.
>>>>>>
>>>>>> How do I get the creation time?
>>>>>
>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
>>>>
>>>> I just went back to the office to reboot to 5.3 and check the creation
>>>> times and found they were 2013 - 2014.
>>>>
>>>>>>
>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>> detects.
>>>>>>>
>>>>>>> Even the situation can be fixed by reverting back to older kernel
>>>>>>> and
>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>> offending
>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>
>>>>>> How to find the offending dir/file from the command line manually?
>>>>>
>>>>> # find <mount point> -inum <inode number>
>>>>
>>>> This works, thanks.
>>>>
>>>> But appears unpractical. After fix 2 files and reboot, I found 4 more,
>>>> then 16, then I gave up.
>>
>> Another solution is use "find" to locate the files with creation time
>> before 2015, and copy them to a new file, then replace the old file with
>> the new file.
> 
> Hmm. But how do I "find" by creation time (otime)? Do you have a
> suggestion for this?

$ touch -t 201501010000 /tmp/sample
$ find <mnt> -not -cnewer /tmp/sample

If you want, you can add -exec to that find, but I'd only add that after
confirming the execution load is verified.

Thanks,
Qu

> 
>> It would be much safer than btrfs check --repair.
>>
>> Thanks,
>> Qu
>>
>>
>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>
>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>> with a
>>>>>>> simple test image.
>>>>>>>
>>>>>>> Qu Wenruo (3):
>>>>>>>      btrfs-progs: check/lowmem: Add check and repair for invalid
>>>>>>> inode
>>>>>>>        generation
>>>>>>>      btrfs-progs: check/original: Add check and repair for
>>>>>>> invalid inode
>>>>>>>        generation
>>>>>>>      btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>> generation
>>>>>>>        repair
>>>>>>>
>>>>>>>     check/main.c                                  |  50 +++++++++++-
>>>>>>>     check/mode-lowmem.c                           |  76
>>>>>>> ++++++++++++++++++
>>>>>>>     check/mode-original.h                         |   1 +
>>>>>>>     .../.lowmem_repairable                        |   0
>>>>>>>     .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012
>>>>>>> bytes
>>>>>>>     5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>     create mode 100644
>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>     create mode 100644
>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>> errors).
>>>
>>> It's really not recommended to run btrfs check, especially repair on the
>>> mounted fs, unless it's RO.
>>>
>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>
>>>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts
>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>>> v5.3 btrfs-progs.
>>>>
>>>> If you like I can share the log. Let me know.
>>>>
>>>> This issue can potentially cause a lot of grief. Our company server
>>>> runs
>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on each
>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>> ~5.6)?
>>>
>>> Yes.
>>>
>>>>
>>>> Do I understand correctly that this FTB is caused by more strict
>>>> checking of the fs by the kernel, while the tools to fix the detected
>>>> corruptions are not yet released?
>>>
>>> Yes.
>>>
>>> Thanks,
>>> Qu
>>>
>>
>