Re: [PATCH RFC 00/11] btrfs: introduce write-intent bitmaps for RAID56

[Wang Yugui <wangyugui@e16-tech.com>]

Hi,

> On 2022/7/7 12:24, Wang Yugui wrote:
> > Hi,
> >
> >> [BACKGROUND]
> >> Unlike md-raid, btrfs RAID56 has nothing to sync its devices when power
> >> loss happens.
> >>
> >> For pure mirror based profiles it's fine as btrfs can utilize its csums
> >> to find the correct mirror the repair the bad ones.
> >>
> >> But for RAID56, the repair itself needs the data from other devices,
> >> thus any out-of-sync data can degrade the tolerance.
> >>
> >> Even worse, incorrect RMW can use the stale data to generate P/Q,
> >> removing the possibility of recovery the data.
> >>
> >>
> >> For md-raid, it goes with write-intent bitmap, to do faster resilver,
> >> and goes journal (partial parity log for RAID5) to ensure it can even
> >> stand a powerloss + device lose.
> >>
> >> [OBJECTIVE]
> >>
> >> This patchset will introduce a btrfs specific write-intent bitmap.
> >>
> >> The bitmap will locate at physical offset 1MiB of each device, and the
> >> content is the same between all devices.
> >>
> >> When there is a RAID56 write (currently all RAID56 write, including full
> >> stripe write), before submitting all the real bios to disks,
> >> write-intent bitmap will be updated and flushed to all writeable
> >> devices.
> >>
> >> So even if a powerloss happened, at the next mount time we know which
> >> full stripes needs to check, and can start a scrub for those involved
> >> logical bytenr ranges.
> >>
> >> [NO RECOVERY CODE YET]
> >>
> >> Unfortunately, this patchset only implements the write-intent bitmap
> >> code, the recovery part is still a place holder, as we need some scrub
> >> refactor to make it only scrub a logical bytenr range.
> >>
> >> [ADVANTAGE OF BTRFS SPECIFIC WRITE-INTENT BITMAPS]
> >>
> >> Since btrfs can utilize csum for its metadata and CoWed data, unlike
> >> dm-bitmap which can only be used for faster re-silver, we can fully
> >> rebuild the full stripe, as long as:
> >>
> >> 1) There is no missing device
> >>     For missing device case, we still need to go full journal.
> >>
> >> 2) Untouched data stays untouched
> >>     This should be mostly sane for sane hardware.
> >>
> >> And since the btrfs specific write-intent bitmaps are pretty small (4KiB
> >> in size), the overhead much lower than full journal.
> >>
> >> In the future, we may allow users to choose between just bitmaps or full
> >> journal to meet their requirement.
> >>
> >> [BITMAPS DESIGN]
> >>
> >> The bitmaps on-disk format looks like this:
> >>
> >>   [ super ][ entry 1 ][ entry 2 ] ... [entry N]
> >>   |<---------  super::size (4K) ------------->|
> >>
> >> Super block contains how many entires are in use.
> >>
> >> Each entry is 128 bits (16 bytes) in size, containing one u64 for
> >> bytenr, and u64 for one bitmap.
> >>
> >> And all utilized entries will be sorted in their bytenr order, and no
> >> bit can overlap.
> >>
> >> The blocksize is now fixed to BTRFS_STRIPE_LEN (64KiB), so each entry
> >> can contain at most 4MiB, and the whole bitmaps can contain 224 entries.
> >
> > Can we skip the write-intent bitmap log if we already log it in last N records
> > (logrotate aware) to improve the write performance?  because HDD sync
> > IOPS is very small.
> 
> I'm not aware about the logrotate idea you mentioned, mind to explain it
> more?
> 
> But the overall idea of journal/write-intent bitmaps are, always ensure
> there is something recording the full write or the write-intention
> before the real IO is submitted.
> 
> So I'm afraid such behavior can not be changed much.

The basic idea is that we recover the data by scrub, not full log,
so log *1 is same to log *2, but with less IOPS?

log *1
	write(0-64K) log
	wirte(64K-128K) log
	wirte(128K-192K) log
	wirte(192K-256K) log
log *2
	write(0-256K) log
	already write(0-256K), skip
	already write(0-256K), skip
	already write(0-256K), skip

we can search the entry we currently used, but can not search the prev
entry, because it maybe be logrotated?

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/07/07