Re: Scrub aborts due to corrupt leaf

[Larkin Lowrey <llowrey@nuclearwinter.com>]

On 9/11/2018 11:23 AM, Larkin Lowrey wrote:
> On 8/29/2018 1:32 AM, Qu Wenruo wrote:
>>
>> On 2018/8/28 下午9:56, Chris Murphy wrote:
>>> On Tue, Aug 28, 2018 at 7:42 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> 
>>> wrote:
>>>>
>>>> On 2018/8/28 下午9:29, Larkin Lowrey wrote:
>>>>> On 8/27/2018 10:12 PM, Larkin Lowrey wrote:
>>>>>> On 8/27/2018 12:46 AM, Qu Wenruo wrote:
>>>>>>>> The system uses ECC memory and edac-util has not reported any 
>>>>>>>> errors.
>>>>>>>> However, I will run a memtest anyway.
>>>>>>> So it should not be the memory problem.
>>>>>>>
>>>>>>> BTW, what's the current generation of the fs?
>>>>>>>
>>>>>>> # btrfs inspect dump-super <device> | grep generation
>>>>>>>
>>>>>>> The corrupted leaf has generation 2862, I'm not sure how recent 
>>>>>>> did the
>>>>>>> corruption happen.
>>>>>> generation              358392
>>>>>> chunk_root_generation   357256
>>>>>> cache_generation        358392
>>>>>> uuid_tree_generation    358392
>>>>>> dev_item.generation     0
>>>>>>
>>>>>> I don't recall the last time I ran a scrub but I doubt it has been
>>>>>> more than a year.
>>>>>>
>>>>>> I am running 'btrfs check --init-csum-tree' now. Hopefully that 
>>>>>> clears
>>>>>> everything up.
>>>>> No such luck:
>>>>>
>>>>> Creating a new CRC tree
>>>>> Checking filesystem on /dev/Cached/Backups
>>>>> UUID: acff5096-1128-4b24-a15e-4ba04261edc3
>>>>> Reinitialize checksum tree
>>>>> csum result is 0 for block 2412149436416
>>>>> extent-tree.c:2764: alloc_tree_block: BUG_ON `ret` triggered, 
>>>>> value -28
>>>> It's ENOSPC, meaning btrfs can't find enough space for the new csum 
>>>> tree
>>>> blocks.
>>> Seems bogus, there's >4TiB unallocated.
>> What a shame.
>> Btrfs won't try to allocate new chunk if we're allocating new tree
>> blocks for metadata trees (extent, csum, etc).
>>
>> One quick (and dirty) way to avoid such limitation is to use the
>> following patch
>
>> <<patch removed>>
>
> No luck.
>
> # ./btrfs check --init-csum-tree /dev/Cached/Backups
> Creating a new CRC tree
> Opening filesystem to check...
> Checking filesystem on /dev/Cached/Backups
> UUID: acff5096-1128-4b24-a15e-4ba04261edc3
> Reinitialize checksum tree
> Segmentation fault (core dumped)
>
>  btrfs[16575]: segfault at 7ffc4f74ef60 ip 000000000040d4c3 sp 
> 00007ffc4f74ef50 error 6 in btrfs[400000+bf000]
>
> # ./btrfs --version
> btrfs-progs v4.17.1
>
> I cloned  btrfs-progs from git and applied your patch.
>
> BTW, I've been having tons of trouble with two hosts after updating 
> from kernel 4.17.12 to 4.17.14 and beyond. The fs will become 
> unresponsive and all processes will end up stuck waiting on io. The 
> system will end up totally idle but unable perform any io on the 
> filesystem. So far things have been stable after reverting back to 
> 4.17.12. It looks like there was a btrfs change in 4.17.13. Could that 
> be related to this csum tree corruption?

About once a week, or so, I'm running into the above situation where FS 
seems to deadlock. All IO to the FS blocks, there is no IO activity at 
all. I have to hard reboot the system to recover. There are no error 
indications except for the following which occurs well before the FS 
freezes up:

BTRFS warning (device dm-3): block group 78691883286528 has wrong amount 
of free space
BTRFS warning (device dm-3): failed to load free space cache for block 
group 78691883286528, rebuilding it now

Do I have any options other the nuking the FS and starting over?

--Larkin