Re: RAID56 discussion related to RST. (Was "Re: [RFC ONLY 0/8] btrfs: introduce raid-stripe-tree")

Thiago Ramon
2022-07-15T19:08:32Z

As a user of RAID6 here, let me jump in because I think this
suggestion is actually a very good compromise.

With stripes written only once, we completely eliminate any possible
write-hole, and even without any changes on the current disk layout
and allocation, there shouldn't be much wasted space (in my case, I
have a 12-disk RAID6, so each full stripe holds 640kb, and discounting
single-sector writes that should go into metadata space, any
reasonable write should fill that buffer in a few seconds).

The additional suggestion of using smaller stripe widths in case there
isn't enough data to fill a whole stripe would make it very easy to
reclaim the wasted space by rebalancing with a stripe count filter,
which can be easily automated and run very frequently.

On-disk format also wouldn't change and be fully usable by older
kernels, and it should "only" require changes on the allocator to
implement.

On Fri, Jul 15, 2022 at 2:58 PM Goffredo Baroncelli <kreijack@libero.it> wrote:
>
> On 14/07/2022 09.46, Johannes Thumshirn wrote:
> > On 14.07.22 09:32, Qu Wenruo wrote:
> >>[...]
> >
> > Again if you're doing sub-stripe size writes, you're asking stupid things and
> > then there's no reason to not give the user stupid answers.
> >
>
> Qu is right, if we consider only full stripe write the "raid hole" problem
> disappear, because if a "full stripe" is not fully written it is not
> referenced either.
>
>
> Personally I think that the ZFS variable stripe size, may be interesting
> to evaluate. Moreover, because the BTRFS disk format is quite flexible,
> we can store different BG with different number of disks. Let me to make an
> example: if we have 10 disks, we could allocate:
> 1 BG RAID1
> 1 BG RAID5, spread over 4 disks only
> 1 BG RAID5, spread over 8 disks only
> 1 BG RAID5, spread over 10 disks
>
> So if we have short writes, we could put the extents in the RAID1 BG; for longer
> writes we could use a RAID5 BG with 4 or 8 or 10 disks depending by length
> of the data.
>
> Yes this would require a sort of garbage collector to move the data to the biggest
> raid5 BG, but this would avoid (or reduce) the fragmentation which affect the
> variable stripe size.
>
> Doing so we don't need any disk format change and it would be backward compatible.
>
>
> Moreover, if we could put the smaller BG in the faster disks, we could have a
> decent tiering....
>
>
> > If a user is concerned about the write or space amplicfication of sub-stripe
> > writes on RAID56 he/she really needs to rethink the architecture.
> >
> >
> >
> > [1]
> > S. K. Mishra and P. Mohapatra,
> > "Performance study of RAID-5 disk arrays with data and parity cache,"
> > Proceedings of the 1996 ICPP Workshop on Challenges for Parallel Processing,
> > 1996, pp. 222-229 vol.1, doi: 10.1109/ICPP.1996.537164.
>
> --
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
>