Re: [RFC 1/1] btrfs/237: adapt the test to work with the new reclaim algorithm

[Pankaj Raghav <p.raghav@samsung.com>]

On 2022-08-22 11:40, Johannes Thumshirn wrote:
> On 19.08.22 13:53, Pankaj Raghav wrote:
>> Since 3687fcb0752a ("btrfs: zoned: make auto-reclaim less aggressive")
>> commit, reclaim algorithm has been changed to trigger auto-reclaim once
>> the fs used size is more than certain threshold. This change breaks this
>> test.
>>
>> The test has been adapted so that the new auto-reclaim algorithm can be
>> tested along with relocation.
> 
> S-o-b missing.
Oops. I hope it can be added before committing.
Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>

> 
> Thanks for doing this!
> Tested-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
>
Thanks for testing it Johannes.

I did encounter an issue when I was trying out this test in a larger zoned
device. I have explained the issue in my cover letter. I don't know if it
is an issue with QEMU. As I don't have a physical PO2 device with me, did
you see any of the issue I mentioned in the cover letter? If not, it might
be an issue with the QEMU ZNS implementation.

>> ---
>>  tests/btrfs/237 | 80 +++++++++++++++++++++++++++++++++++--------------
>>  1 file changed, 57 insertions(+), 23 deletions(-)
>>
>> diff --git a/tests/btrfs/237 b/tests/btrfs/237
>> index f96031d5..18945e78 100755
>> --- a/tests/btrfs/237
>> +++ b/tests/btrfs/237
>> @@ -54,46 +54,80 @@ if [[ "$uuid" == "" ]]; then
>>  	exit 1
>>  fi
>>  
>> +fssize=$($BTRFS_UTIL_PROG fi usage -b $SCRATCH_MNT |grep "Device size" |\
>> +	grep -Eo "[0-9]+")
>> +
>> +allocated_fssize=$($BTRFS_UTIL_PROG fi usage -b $SCRATCH_MNT |grep "Device allocated" |\
>> +	grep -Eo "[0-9]+")
>> +
>> +
>>  start_data_bg_phy=$(get_data_bg_physical)
>>  start_data_bg_phy=$((start_data_bg_phy >> 9))
>>  
>> -size=$($BLKZONE_PROG report -o $start_data_bg_phy -l 1 $SCRATCH_DEV |\
>> +zone_cap=$($BLKZONE_PROG report -o $start_data_bg_phy -l 1 $SCRATCH_DEV |\
>>  	_filter_blkzone_report |\
>>  	grep -Po "cap 0x[[:xdigit:]]+" | cut -d ' ' -f 2)
>> -size=$((size << 9))
>> +zone_cap=$((zone_cap << 9))
>>  
>> -reclaim_threshold=75
>> -echo $reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
>> -fill_percent=$((reclaim_threshold + 2))
>> -rest_percent=$((90 - fill_percent)) # make sure we're not creating a new BG
>> -fill_size=$((size * fill_percent / 100))
>> -rest=$((size * rest_percent / 100))
>> +fs_reclaim_threshold=60
>> +bg_reclaim_threshold=75
>> +echo $fs_reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
>> +echo $bg_reclaim_threshold > /sys/fs/btrfs/"$uuid"/allocation/data/bg_reclaim_threshold
>>  
>> -# step 1, fill FS over $fillsize
>> -$XFS_IO_PROG -fc "pwrite 0 $fill_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full
>> -$XFS_IO_PROG -fc "pwrite 0 $rest" $SCRATCH_MNT/$seq.test2 >> $seqres.full
>> +fs_fill_percent=$((fs_reclaim_threshold + 2))
>> +fill_size=$((fssize * fs_fill_percent / 100))
>> +
>> +# Remove the allocated size from the $fill_size
>> +fill_size=$((fill_size - allocated_fssize))
>> +
>> +bg_fill_percent=$((bg_reclaim_threshold + 2))
>> +zone_fill_size=$((zone_cap * bg_fill_percent / 100))
>> +
>> +# $fill_size might not cover the last zone block group with threshold
>> +# for reclaim. Add the remaining bytes so that it can also be reclaimed
>> +last_zone_offset=$((fill_size % zone_cap))
>> +
>> +if [ $last_zone_offset -lt $zone_fill_size ]; then
>> +	fill_size=$((fill_size + zone_fill_size - last_zone_offset))
>> +fi
>> +
>> +# This small file will be used to verify the relocation
>> +relocate_file_size=$((zone_cap * 2 / 100))
>> +
>> +# step 1, fill FS over $relocated_file_size and $fill_size
>> +$XFS_IO_PROG -fc "pwrite 0 $relocate_file_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full
>>  $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
>>  
>> -zones_before=$($BLKZONE_PROG report $SCRATCH_DEV | grep -v -e em -e nw | wc -l)
>> -echo "Before reclaim: $zones_before zones open" >> $seqres.full
>>  old_data_zone=$(get_data_bg)
>>  old_data_zone=$((old_data_zone >> 9))
>>  printf "Old data zone 0x%x\n" $old_data_zone >> $seqres.full
>>  
>> -# step 2, delete the 1st $fill_size sized file to trigger reclaim
>> -rm $SCRATCH_MNT/$seq.test1
>> +$XFS_IO_PROG -fc "pwrite 0 $fill_size" $SCRATCH_MNT/$seq.test2 >> $seqres.full
>>  $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
>> -sleep 2 # 1 transaction commit for 'rm' and 1 for balance
>> +
>> +open_zones_before_reclaim=$($BLKZONE_PROG report --offset $start_data_bg_phy $SCRATCH_DEV |\
>> +	grep -v -e em -e nw | wc -l)
>> +
>> +# sanity check
>> +if [ $open_zones_before_reclaim -eq 0 ]; then
>> +	echo "Error writing to the device"
>> +fi
>> +
>> +echo "Before reclaim: $open_zones_before_reclaim zones open" >> $seqres.full
>> +
>> +# step 2, delete the $fill_size sized file to trigger reclaim
>> +rm $SCRATCH_MNT/$seq.test2
>> +$BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
>> +sleep 5 # sleep for transaction commit for 'rm' and for balance
>>  
>>  # check that we don't have more zones open than before
>> -zones_after=$($BLKZONE_PROG report $SCRATCH_DEV | grep -v -e em -e nw | wc -l)
>> -echo "After reclaim: $zones_after zones open" >> $seqres.full
>> +open_zones_after_reclaim=$($BLKZONE_PROG report --offset $start_data_bg_phy $SCRATCH_DEV |\
>> +	grep -v -e em -e nw | wc -l)
>> +echo "After reclaim: $open_zones_after_reclaim zones open" >> $seqres.full
>>  
>> -# Check that old data zone was reset
>> -old_wptr=$($BLKZONE_PROG report -o $old_data_zone -c 1 $SCRATCH_DEV |\
>> -	grep -Eo "wptr 0x[[:xdigit:]]+" | cut -d ' ' -f 2)
>> -if [ "$old_wptr" != "0x000000" ]; then
>> -	_fail "Old wptr still at $old_wptr"
>> +# Check that data was really relocated to a different zone
>> +if [ $open_zones_after_reclaim != 1 ]; then
>> +	echo "Error relocating the data"
>>  fi
>>  
>>  new_data_zone=$(get_data_bg)
>