Re: [PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation

Qu Wenruo
2019-10-20T14:12:08Z

[-- Attachment #1.1: Type: text/plain, Size: 6844 bytes --]

On 2019/10/20 下午9:29, Ferry Toth wrote:
> Op 20-10-2019 om 15:15 schreef Qu WenRuo:
>>
>>
>> On 2019/10/20 下午9:04, Ferry Toth wrote:
>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>>>
>>>>
>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>>>
>>>>>>>
>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>>>> We have at least two user reports about bad inode generation makes
>>>>>>>>> kernel reject the fs.
>>>>>>>>
>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>>>
>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>>>> on /@
>>>>>>>>
>>>>>>>> In initramfs I can try to mount but get something like
>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>>>> failed
>>>>>>>>
>>>>>>>> Booting old kernel works just as before, no errors.
>>>>>>>>
>>>>>>>>> According to the creation time, the inode is created by some 2014
>>>>>>>>> kernel.
>>>>>>>>
>>>>>>>> How do I get the creation time?
>>>>>>>
>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your
>>>>>>> device>
>>>>>>
>>>>>> I just went back to the office to reboot to 5.3 and check the
>>>>>> creation
>>>>>> times and found they were 2013 - 2014.
>>>>>>
>>>>>>>>
>>>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>>>> detects.
>>>>>>>>>
>>>>>>>>> Even the situation can be fixed by reverting back to older kernel
>>>>>>>>> and
>>>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>>>> offending
>>>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>>>
>>>>>>>> How to find the offending dir/file from the command line manually?
>>>>>>>
>>>>>>> # find <mount point> -inum <inode number>
>>>>>>
>>>>>> This works, thanks.
>>>>>>
>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4
>>>>>> more,
>>>>>> then 16, then I gave up.
>>>>
>>>> Another solution is use "find" to locate the files with creation time
>>>> before 2015, and copy them to a new file, then replace the old file
>>>> with
>>>> the new file.
>>>
>>> Hmm. But how do I "find" by creation time (otime)? Do you have a
>>> suggestion for this?
>>
>> $ touch -t 201501010000 /tmp/sample
>> $ find <mnt> -not -cnewer /tmp/sample
>
> AFAIK this compares file modified date with status changed date. So, no
> search for creation date.
>
> And stat /tmp/sample (sorry dutch lang output):
>
> ferry@ferry-quad:~$ stat /tmp/sample
>   Bestand: /tmp/sample
>   Grootte: 0            Blokken: 0            IO-blok: 4096   leeg
> normaal bestand
> Apparaat: 1bh/27d   Inode: 62005381     Koppelingen: 1
> Toegang: (0664/-rw-rw-r--)   UID: ( 1001/   ferry)   GID: ( 1001/   ferry)
> Toegang:   2015-01-01 00:00:00.000000000 +0100
> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100
> Veranderd: 2019-10-20 15:20:50.366163766 +0200
> Ontstaan:  -

My bad, always got confused by o/a/c/mtime, as c really looks like *c*
reation, so I always got confused between ctime and otime.

Then considering not all fs supports otime, find doesn't support that.
I guess it's only possible by other tools....

BTW, did you find any patterns in those existing offending inodes?
I guess it would be faster than finding a tool supporting otime search.

Thanks,
Qu

>
>
>> If you want, you can add -exec to that find, but I'd only add that after
>> confirming the execution load is verified.
>>
>> Thanks,
>> Qu
>>
>>>
>>>> It would be much safer than btrfs check --repair.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>>
>>>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>>>> with a
>>>>>>>>> simple test image.
>>>>>>>>>
>>>>>>>>> Qu Wenruo (3):
>>>>>>>>>       btrfs-progs: check/lowmem: Add check and repair for invalid
>>>>>>>>> inode
>>>>>>>>>         generation
>>>>>>>>>       btrfs-progs: check/original: Add check and repair for
>>>>>>>>> invalid inode
>>>>>>>>>         generation
>>>>>>>>>       btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>>>> generation
>>>>>>>>>         repair
>>>>>>>>>
>>>>>>>>>      check/main.c                                  |  50
>>>>>>>>> +++++++++++-
>>>>>>>>>      check/mode-lowmem.c                           |  76
>>>>>>>>> ++++++++++++++++++
>>>>>>>>>      check/mode-original.h                         |   1 +
>>>>>>>>>      .../.lowmem_repairable                        |   0
>>>>>>>>>      .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012
>>>>>>>>> bytes
>>>>>>>>>      5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>>>      create mode 100644
>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>>>      create mode 100644
>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>>>> errors).
>>>>>
>>>>> It's really not recommended to run btrfs check, especially repair
>>>>> on the
>>>>> mounted fs, unless it's RO.
>>>>>
>>>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>>>
>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted
>>>>>> distorts
>>>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>>>>> v5.3 btrfs-progs.
>>>>>>
>>>>>> If you like I can share the log. Let me know.
>>>>>>
>>>>>> This issue can potentially cause a lot of grief. Our company server
>>>>>> runs
>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on
>>>>>> each
>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>>>> ~5.6)?
>>>>>
>>>>> Yes.
>>>>>
>>>>>>
>>>>>> Do I understand correctly that this FTB is caused by more strict
>>>>>> checking of the fs by the kernel, while the tools to fix the detected
>>>>>> corruptions are not yet released?
>>>>>
>>>>> Yes.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>
>>>
>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]