Re: [PATCH DRAFT] btrfs: RAID56J journal on-disk format draft

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/6/6 16:16, Qu Wenruo wrote:
>
>
[...]
>>>
>>> Hello Qu,
>>>
>>> If you don't care about the write-hole, you can also use a dirty bitmap
>>> like mdraid 5/6 does. There, one bit in the bitmap represents for
>>> example one gigabyte of the disk that _may_ be dirty, and the bit is
>>> left
>>> dirty for a while and doesn't need to be set for each write. Or you
>>> could do a per-block-group dirty bit.
>>
>> That would be a pretty good way for auto scrub after dirty close.
>>
>> Currently we have quite some different ideas, but some are pretty
>> similar but at different side of a spectrum:
>>
>>      Easier to implement        ..     Harder to implement
>> |<- More on mount time scrub   ..     More on journal ->|
>> |                    |    |    \- Full journal
>> |                    |    \--- Per bg dirty bitmap
>> |                    \----------- Per bg dirty flag
>> \--------------------------------------------------- Per sb dirty flag
>
> In fact, recently I'm checking the MD code (including their MD-raid5).
>
> It turns out they have write-intent bitmap, which is almost the per-bg
> dirty bitmap in above spectrum.
>
> In fact, since btrfs has all the CoW and checksum for metadata (and part
> of its data), btrfs scrub can do a much better job than MD to resilver
> the range.
>
> Furthermore, we have a pretty good reserved space (1M), and has a pretty
> reasonable stripe length (1GiB).
> This means, we only need 32KiB for the bitmap for each RAID56 stripe,
> much smaller than the 1MiB we reserved.
>
> I think this can be a pretty reasonable middle ground, faster than full
> journal, while the amount to scrub should be reasonable enough to be
> done at mount time.

Furthermore, this even allows us to go something like bitmap tree, for
such write-intent bitmap.
And as long as the user is not using RAID56 for metadata (maybe even
it's OK to use RAID56 for metadata), it should be pretty safe against
most write-hole (for metadata and CoW data only though, nocow data is
still affected).

Thus I believe this can be a valid path to explore, and even have a
higher priority than full journal.

Thanks,
Qu


>
> Thanks,
> Qu
>>
>> In fact, the dirty bitmap is just a simplified version of journal (only
>> record the metadata, without data).
>> Unlike dm/dm-raid56, with btrfs scrub, we should be able to fully
>> recover the data without problem.
>>
>> Even with per-bg dirty bitmap, we still need some extra location to
>> record the bitmap. Thus it needs a on-disk format change anyway.
>>
>> Currently only sb dirty flag may be backward compatible.
>>
>> And whether we should wait for the scrub to finish before allowing use
>> to do anything into the fs is also another concern.
>>
>> Even using bitmap, we may have several GiB data needs to be scrubbed.
>> If we wait for the scrub to finish, it's the best and safest way, but
>> users won't be happy at all.
>>
>> If we go scrub resume way, it's faster but still leaves a large window
>> to allow write-hole to reduce our tolerance.
>>
>> Thanks,
>> Qu
>>>
>>> And while you're at it, add the same mechanism to all the other raid
>>> and dup modes to fix the inconsistency of NOCOW files after a crash.
>>>
>>> Regards,
>>> Lukas Straub
>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>>
>>>>>>> Paul.
>>>>>
>>>>>
>>>
>>>
>>>