Re: RAID56 discussion related to RST. (Was "Re: [RFC ONLY 0/8] btrfs: introduce raid-stripe-tree")

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/7/13 20:42, Johannes Thumshirn wrote:
> On 13.07.22 14:01, Qu Wenruo wrote:
>>
>>
>> On 2022/7/13 19:43, Johannes Thumshirn wrote:
>>> On 13.07.22 12:54, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/5/16 22:31, Johannes Thumshirn wrote:
>>>>> Introduce a raid-stripe-tree to record writes in a RAID environment.
>>>>>
>>>>> In essence this adds another address translation layer between the logical
>>>>> and the physical addresses in btrfs and is designed to close two gaps. The
>>>>> first is the ominous RAID-write-hole we suffer from with RAID5/6 and the
>>>>> second one is the inability of doing RAID with zoned block devices due to the
>>>>> constraints we have with REQ_OP_ZONE_APPEND writes.
>>>>
>>>> Here I want to discuss about something related to RAID56 and RST.
>>>>
>>>> One of my long existing concern is, P/Q stripes have a higher update
>>>> frequency, thus with certain transaction commit/data writeback timing,
>>>> wouldn't it cause the device storing P/Q stripes go out of space before
>>>> the data stripe devices?
>>>
>>> P/Q stripes on a dedicated drive would be RAID4, which we don't have.
>>
>> I'm just using one block group as an example.
>>
>> Sure, the next bg can definitely go somewhere else.
>>
>> But inside one bg, we are still using one zone for the bg, right?
>
> Ok maybe I'm not understanding the code in volumes.c correctly, but
> doesn't __btrfs_map_block() calculate a rotation per stripe-set?
>
> I'm looking at this code:
>
> 	/* Build raid_map */
> 	if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK && need_raid_map &&
> 	    (need_full_stripe(op) || mirror_num > 1)) {
> 		u64 tmp;
> 		unsigned rot;
>
> 		/* Work out the disk rotation on this stripe-set */
> 		div_u64_rem(stripe_nr, num_stripes, &rot);
>
> 		/* Fill in the logical address of each stripe */
> 		tmp = stripe_nr * data_stripes;
> 		for (i = 0; i < data_stripes; i++)
> 			bioc->raid_map[(i + rot) % num_stripes] =
> 				em->start + (tmp + i) * map->stripe_len;
>
> 		bioc->raid_map[(i + rot) % map->num_stripes] = RAID5_P_STRIPE;
> 		if (map->type & BTRFS_BLOCK_GROUP_RAID6)
> 			bioc->raid_map[(i + rot + 1) % num_stripes] =
> 				RAID6_Q_STRIPE;
>
> 		sort_parity_stripes(bioc, num_stripes);
> 	}

That's per full-stripe. AKA, the rotation only kicks in after a full stripe.

In my example, we're inside one full stripe, no rotation, until next
full stripe.

>
>
> So then in your example we have something like this:
>
> Write of 4k D1:
>
> 	0		32K		64K
> Disk 1	|D1                             |
> Disk 2	|                               |
> Disk 3	|P1                             |
>
>
> Write of 4k D2, the new parity is P2 the old P1 parity is obsolete
>
> 	0		32K		64K
> Disk 1	|D1                             |
> Disk 2	|P2                             |
> Disk 3	|P1D2                           |
>
> Write of new 4k D1 with P3
>
> 	0		32K		64K
> Disk 1	|D1P3                           |
> Disk 2	|P2D1                           |
> Disk 3	|P1D2                           |
>
> and so on.

So, not the case, at least not in the full stripe.

Thanks,
Qu