Re: RAID56 discussion related to RST. (Was "Re: [RFC ONLY 0/8] btrfs: introduce raid-stripe-tree")

Goffredo Baroncelli
2022-07-19T18:58:10Z

On 18/07/2022 09.30, Johannes Thumshirn wrote:
> On 15.07.22 19:54, Goffredo Baroncelli wrote:
>> On 14/07/2022 09.46, Johannes Thumshirn wrote:
>>> On 14.07.22 09:32, Qu Wenruo wrote:
>>>> [...]
>>>
>>> Again if you're doing sub-stripe size writes, you're asking stupid things and
>>> then there's no reason to not give the user stupid answers.
>>>
>>
>> Qu is right, if we consider only full stripe write the "raid hole" problem
>> disappear, because if a "full stripe" is not fully written it is not
>> referenced either.
>
> It's not that there wil lbe a new write hole, it's just that there is sub-optimal
> space consumption until we can either re-write or garbage collect the blocks.

May be that I was not very clear. Let me to repeat: if we assume that we
can write only full stripe (padding with 0, if smaller), we don't have the
write-hole problem at all, so we can also avoid using RST.

>>
>> Personally I think that the ZFS variable stripe size, may be interesting
>
> But then we would need extra meta-data to describe the size of each stripe.

It is not needed. The stripe allocation is per extent. The layout of the stripe
depends only by the start of the extent and its length. Assuming that you
have n disks and a raid5 layout, you already know that each (n-1) data blocks,
there is a parity block in the extent.

The relation between the length of the extent and the real data stored is

extent-length = data-length + 4k*(data-length / 4k / (n-1))
extent-length += 4k if (data-length %(4k * (n-1))) > 0

extent-length = size of the extent (which contains the parity block)
data-length = the real length of consecutive data
n = number of disk

Below some examples that show better my idea:

Assuming the following logical address

Disk1 0 12k 24k
Disk2 4k 16k 28k
Disk3 8k 20k ....

first write: data size = 1 block

Disk1 D1 ...
Disk2 P1 ...
Disk3 ...

Extent = (0, 8K),

second write; data size = 3 block ( D2, D2*, D3 )

Disk1 D1 D2* P3 ...
Disk2 P1 P2 ...
Disk3 D2 D3 ...

Extent = (8k, 20K)

Write a bigger data, and shape the stripe taller:

3rd write; data size 32k: (D6, D6*, ... D9*

Disk1 D1 D2* P3 P68 P6*8* P79 P7*9* ...
Disk2 P1 P2 D6 D6* D7 D7* ...
Disk3 D2 D3 D8 D8* D9 D9* ...

Extent = (28k, 48k)

The major drawbacks are:
- you can break the extent only at stripe boundary (up to 64K * n)
- the scrub is a bit more complex, because it involves some math around
the extents start/length
- you need to have an extent that describe the stripe. I don't know if this
requirements is fulfill by metadata

The schema above, has an huge simplification if we allow BTRFS to use BG with
dedicated stripe size. Moreover this would reduce the fragmentation, even tough it
requires a GC.

>
>> to evaluate. Moreover, because the BTRFS disk format is quite flexible,
>> we can store different BG with different number of disks. Let me to make an
>> example: if we have 10 disks, we could allocate:
>> 1 BG RAID1
>> 1 BG RAID5, spread over 4 disks only
>> 1 BG RAID5, spread over 8 disks only
>> 1 BG RAID5, spread over 10 disks
>>
>> So if we have short writes, we could put the extents in the RAID1 BG; for longer
>> writes we could use a RAID5 BG with 4 or 8 or 10 disks depending by length
>> of the data.
>>
>> Yes this would require a sort of garbage collector to move the data to the biggest
>> raid5 BG, but this would avoid (or reduce) the fragmentation which affect the
>> variable stripe size.
>>
>> Doing so we don't need any disk format change and it would be backward compatible.
>>
>>
>> Moreover, if we could put the smaller BG in the faster disks, we could have a
>> decent tiering....
>>
>>
>>> If a user is concerned about the write or space amplicfication of sub-stripe
>>> writes on RAID56 he/she really needs to rethink the architecture.
>>>
>>>
>>>
>>> [1]
>>> S. K. Mishra and P. Mohapatra,
>>> "Performance study of RAID-5 disk arrays with data and parity cache,"
>>> Proceedings of the 1996 ICPP Workshop on Challenges for Parallel Processing,
>>> 1996, pp. 222-229 vol.1, doi: 10.1109/ICPP.1996.537164.
>>
>

--
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5