Re: RAID56 discussion related to RST. (Was "Re: [RFC ONLY 0/8] btrfs: introduce raid-stripe-tree")

Qu Wenruo
2022-07-13T13:47:17Z

On 2022/7/13 20:42, Johannes Thumshirn wrote:
> On 13.07.22 14:01, Qu Wenruo wrote:
>>
>>
>> On 2022/7/13 19:43, Johannes Thumshirn wrote:
>>> On 13.07.22 12:54, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/5/16 22:31, Johannes Thumshirn wrote:
>>>>> Introduce a raid-stripe-tree to record writes in a RAID environment.
>>>>>
>>>>> In essence this adds another address translation layer between the logical
>>>>> and the physical addresses in btrfs and is designed to close two gaps. The
>>>>> first is the ominous RAID-write-hole we suffer from with RAID5/6 and the
>>>>> second one is the inability of doing RAID with zoned block devices due to the
>>>>> constraints we have with REQ_OP_ZONE_APPEND writes.
>>>>
>>>> Here I want to discuss about something related to RAID56 and RST.
>>>>
>>>> One of my long existing concern is, P/Q stripes have a higher update
>>>> frequency, thus with certain transaction commit/data writeback timing,
>>>> wouldn't it cause the device storing P/Q stripes go out of space before
>>>> the data stripe devices?
>>>
>>> P/Q stripes on a dedicated drive would be RAID4, which we don't have.
>>
>> I'm just using one block group as an example.
>>
>> Sure, the next bg can definitely go somewhere else.
>>
>> But inside one bg, we are still using one zone for the bg, right?
>
> Ok maybe I'm not understanding the code in volumes.c correctly, but
> doesn't __btrfs_map_block() calculate a rotation per stripe-set?
>
> I'm looking at this code:
>
> /* Build raid_map */
> if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK && need_raid_map &&
> (need_full_stripe(op) || mirror_num > 1)) {
> u64 tmp;
> unsigned rot;
>
> /* Work out the disk rotation on this stripe-set */
> div_u64_rem(stripe_nr, num_stripes, &rot);
>
> /* Fill in the logical address of each stripe */
> tmp = stripe_nr * data_stripes;
> for (i = 0; i < data_stripes; i++)
> bioc->raid_map[(i + rot) % num_stripes] =
> em->start + (tmp + i) * map->stripe_len;
>
> bioc->raid_map[(i + rot) % map->num_stripes] = RAID5_P_STRIPE;
> if (map->type & BTRFS_BLOCK_GROUP_RAID6)
> bioc->raid_map[(i + rot + 1) % num_stripes] =
> RAID6_Q_STRIPE;
>
> sort_parity_stripes(bioc, num_stripes);
> }

That's per full-stripe. AKA, the rotation only kicks in after a full stripe.

In my example, we're inside one full stripe, no rotation, until next
full stripe.

>
>
> So then in your example we have something like this:
>
> Write of 4k D1:
>
> 0 32K 64K
> Disk 1 |D1 |
> Disk 2 | |
> Disk 3 |P1 |
>
>
> Write of 4k D2, the new parity is P2 the old P1 parity is obsolete
>
> 0 32K 64K
> Disk 1 |D1 |
> Disk 2 |P2 |
> Disk 3 |P1D2 |
>
> Write of new 4k D1 with P3
>
> 0 32K 64K
> Disk 1 |D1P3 |
> Disk 2 |P2D1 |
> Disk 3 |P1D2 |
>
> and so on.

So, not the case, at least not in the full stripe.

Thanks,
Qu