[PATCH blktests 0/5] Support zone capacity

[PATCH blktests 0/5] Support zone capacity

[Shin'ichiro Kawasaki]

Kernel 5.9 zone descriptor interface adds the new zone capacity field defining
the range of sectors usable within a zone. This patch series adds support for
this new field. This series depends on recent changes in blkzone and fio to
support zone capacity.

The first patch modifies the helper function _get_blkzone_report() to obtain
zone capacity of the test target zones. Following three patches adjust test
cases in zbd group for zone capacity. The last patch fixes a test condition
issue found in this work.

Shin'ichiro Kawasaki (5):
  zbd/rc: Support zone capacity report by blkzone
  zbd/002: Check validity of zone capacity
  zbd/004: Check zone boundary writes using zones without zone capacity
    gap
  zbd/005: Enable zonemode=zbd when zone capacity is less than zone size
  zbd/002: Check write pointers only when zones have valid conditions

 tests/zbd/002 | 16 ++++++++++++++--
 tests/zbd/004 |  6 +++++-
 tests/zbd/005 | 11 ++++++++---
 tests/zbd/rc  | 35 ++++++++++++++++++++++++++++++-----
 4 files changed, 57 insertions(+), 11 deletions(-)

-- 
2.26.2

[PATCH blktests 1/5] zbd/rc: Support zone capacity report by blkzone

[Shin'ichiro Kawasaki]

Linux kernel 5.9 zone descriptor interface added the new zone capacity
field defining the range of sectors usable within a zone. The blkzone
tool recently supported the zone capacity in its report zone feature.
Modify the helper function _get_blkzone_report() to support the zone
capacity field.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 tests/zbd/rc | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/tests/zbd/rc b/tests/zbd/rc
index 3124693..dafd130 100644
--- a/tests/zbd/rc
+++ b/tests/zbd/rc
@@ -104,6 +104,7 @@ _put_sysfs_variable() {
 # until put function call.
 _get_blkzone_report() {
 	local target_dev=${1}
+	local cap_idx wptr_idx conds_idx type_idx
 
 	# Initialize arrays to store parsed blkzone reports.
 	# Number of reported zones is set in REPORTED_COUNT.
@@ -111,6 +112,7 @@ _get_blkzone_report() {
 	# to simplify loop operation.
 	ZONE_STARTS=()
 	ZONE_LENGTHS=()
+	ZONE_CAPS=()
 	ZONE_WPTRS=()
 	ZONE_CONDS=()
 	ZONE_TYPES=()
@@ -123,6 +125,17 @@ _get_blkzone_report() {
 		return $?
 	fi
 
+	cap_idx=3
+	wptr_idx=5
+	conds_idx=11
+	type_idx=13
+	if grep -qe "cap 0x" "${TMP_REPORT_FILE}"; then
+		cap_idx=5
+		wptr_idx=7
+		conds_idx=13
+		type_idx=15
+	fi
+
 	local _IFS=$IFS
 	local -i loop=0
 	IFS=$' ,:'
@@ -130,9 +143,10 @@ _get_blkzone_report() {
 	do
 		ZONE_STARTS+=($((_tokens[1])))
 		ZONE_LENGTHS+=($((_tokens[3])))
-		ZONE_WPTRS+=($((_tokens[5])))
-		ZONE_CONDS+=($((${_tokens[11]%\(*})))
-		ZONE_TYPES+=($((${_tokens[13]%\(*})))
+		ZONE_CAPS+=($((_tokens[cap_idx])))
+		ZONE_WPTRS+=($((_tokens[wptr_idx])))
+		ZONE_CONDS+=($((${_tokens[conds_idx]%\(*})))
+		ZONE_TYPES+=($((${_tokens[type_idx]%\(*})))
 		if [[ ${ZONE_TYPES[-1]} -eq ${ZONE_TYPE_CONVENTIONAL} ]]; then
 			(( NR_CONV_ZONES++ ))
 		fi
@@ -150,6 +164,7 @@ _get_blkzone_report() {
 	local -i max_idx=$((REPORTED_COUNT - 1))
 	ZONE_STARTS+=( $((ZONE_STARTS[max_idx] + ZONE_LENGTHS[max_idx])) )
 	ZONE_LENGTHS+=( "${ZONE_LENGTHS[max_idx]}" )
+	ZONE_CAPS+=( "${ZONE_CAPS[max_idx]}" )
 	ZONE_WPTRS+=( "${ZONE_WPTRS[max_idx]}" )
 	ZONE_CONDS+=( "${ZONE_CONDS[max_idx]}" )
 	ZONE_TYPES+=( "${ZONE_TYPES[max_idx]}" )
@@ -160,6 +175,7 @@ _get_blkzone_report() {
 _put_blkzone_report() {
 	unset ZONE_STARTS
 	unset ZONE_LENGTHS
+	unset ZONE_CAPS
 	unset ZONE_WPTRS
 	unset ZONE_CONDS
 	unset ZONE_TYPES
-- 
2.26.2

Re: [PATCH blktests 1/5] zbd/rc: Support zone capacity report by blkzone

[Johannes Thumshirn]

Looks good,
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>

Re: [PATCH blktests 1/5] zbd/rc: Support zone capacity report by blkzone

[Klaus Jensen]

On Jul 28 19:14, Shin'ichiro Kawasaki wrote:
> Linux kernel 5.9 zone descriptor interface added the new zone capacity
> field defining the range of sectors usable within a zone. The blkzone
> tool recently supported the zone capacity in its report zone feature.
> Modify the helper function _get_blkzone_report() to support the zone
> capacity field.
> 
> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> ---

LGTM.

Reviewed-by: Klaus Jensen <k.jensen@samsung.com>

[PATCH blktests 2/5] zbd/002: Check validity of zone capacity

[Shin'ichiro Kawasaki]

Linux kernel 5.9 zone descriptor interface added the new zone capacity
field defining the range of sectors usable within a zone. Add a check to
ensure that the zone capacity is smaller than or equal to the zone size.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 tests/zbd/002 | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/zbd/002 b/tests/zbd/002
index e197827..a134434 100755
--- a/tests/zbd/002
+++ b/tests/zbd/002
@@ -24,6 +24,7 @@ _check_blkzone_report() {
 	local -i cur_start=${ZONE_STARTS[0]}
 	local -i next_start=0
 	local -i len=0
+	local -i cap=0
 	local -i wptr=0
 	local -i cond=0
 	local -i zone_type=0
@@ -50,6 +51,7 @@ _check_blkzone_report() {
 
 		next_start=${ZONE_STARTS[$((idx+1))]}
 		len=${ZONE_LENGTHS[$idx]}
+		cap=${ZONE_CAPS[$idx]}
 		wptr=${ZONE_WPTRS[$idx]}
 		cond=${ZONE_CONDS[$idx]}
 		zone_type=${ZONE_TYPES[$idx]}
@@ -70,6 +72,13 @@ _check_blkzone_report() {
 			return 1
 		fi
 
+		# Check zone capacity
+		if [[ ${cap} -gt ${len} ]]; then
+			echo -n "Zone capacity is invalid at zone ${idx}. "
+			echo "capacity: ${cap}, size: ${len}"
+			return 1
+		fi
+
 		# Check write pointer
 		if [[ ${wptr} -lt 0 || ${wptr} -gt ${len} ]]; then
 			echo -n "Write pointer is invalid at zone ${idx}. "
-- 
2.26.2

Re: [PATCH blktests 2/5] zbd/002: Check validity of zone capacity

[Johannes Thumshirn]

Looks good,
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>

Re: [PATCH blktests 2/5] zbd/002: Check validity of zone capacity

[Klaus Jensen]

On Jul 28 19:14, Shin'ichiro Kawasaki wrote:
> Linux kernel 5.9 zone descriptor interface added the new zone capacity
> field defining the range of sectors usable within a zone. Add a check to
> ensure that the zone capacity is smaller than or equal to the zone size.
> 
> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

LGTM.

Reviewed-by: Klaus Jensen <k.jensen@samsung.com>

[PATCH blktests 3/5] zbd/004: Check zone boundary writes using zones without zone capacity gap

[Shin'ichiro Kawasaki]

The test case zbd/004 checks zone boundary write handling by block layer
using two contiguous sequential write required zones. This test is valid
when the first zone has same zone capacity as zone size. However, if the
zone has zone capacity smaller than zone size, the write in the zone
beyond zone capacity limit causes write error and the test fails.

To avoid the write error, find the two zones with first zone that has
zone capacity same as zone size. If such zones are not found, skip the
test case.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 tests/zbd/004 |  6 +++++-
 tests/zbd/rc  | 11 +++++++++--
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/tests/zbd/004 b/tests/zbd/004
index ac0cf50..f09ee31 100755
--- a/tests/zbd/004
+++ b/tests/zbd/004
@@ -46,7 +46,11 @@ test_device() {
 
 	# Find target sequential required zones and reset write pointers
 	_get_blkzone_report "${TEST_DEV}" || return $?
-	idx=$(_find_two_contiguous_seq_zones) || return $?
+	if ! idx=$(_find_two_contiguous_seq_zones cap_eq_len); then
+		SKIP_REASON="No contiguous sequential write required zones"
+		_put_blkzone_report
+		return
+	fi
 	_reset_zones "${TEST_DEV}" "${idx}" "2"
 
 	# Confirm the zones are initialized
diff --git a/tests/zbd/rc b/tests/zbd/rc
index dafd130..3fd2d36 100644
--- a/tests/zbd/rc
+++ b/tests/zbd/rc
@@ -253,21 +253,28 @@ _find_sequential_zone_in_middle() {
 	return 1
 }
 
-# Search zones and find two contiguous sequential required zones.
+# Search zones and find two contiguous sequential write required zones.
 # Return index of the first zone of the found two zones.
+# When the argument cap_eq_len is specified, find the two contiguous
+# sequential write required zones with first zone that has zone capacity
+# same as zone size.
 # Call _get_blkzone_report() beforehand.
 _find_two_contiguous_seq_zones() {
+	local cap_eq_len="${1}"
 	local -i type_seq=${ZONE_TYPE_SEQ_WRITE_REQUIRED}
 
 	for ((idx = NR_CONV_ZONES; idx < REPORTED_COUNT; idx++)); do
 		if [[ ${ZONE_TYPES[idx]} -eq ${type_seq} &&
 		      ${ZONE_TYPES[idx+1]} -eq ${type_seq} ]]; then
+			if [[ -n ${cap_eq_len} ]] &&
+				   ((ZONE_CAPS[idx] != ZONE_LENGTHS[idx])); then
+				continue
+			fi
 			echo "${idx}"
 			return 0
 		fi
 	done
 
-	echo "Contiguous sequential write required zones not found"
 	return 1
 }
 
-- 
2.26.2

Re: [PATCH blktests 3/5] zbd/004: Check zone boundary writes using zones without zone capacity gap

[Johannes Thumshirn]

Looks good,
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>

Re: [PATCH blktests 3/5] zbd/004: Check zone boundary writes using zones without zone capacity gap

[Klaus Jensen]

On Jul 28 19:14, Shin'ichiro Kawasaki wrote:
> The test case zbd/004 checks zone boundary write handling by block layer
> using two contiguous sequential write required zones. This test is valid
> when the first zone has same zone capacity as zone size. However, if the
> zone has zone capacity smaller than zone size, the write in the zone
> beyond zone capacity limit causes write error and the test fails.
> 
> To avoid the write error, find the two zones with first zone that has
> zone capacity same as zone size. If such zones are not found, skip the
> test case.
> 
> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> ---

LGTM.

Reviewed-by: Klaus Jensen <k.jensen@samsung.com>

[PATCH blktests 4/5] zbd/005: Enable zonemode=zbd when zone capacity is less than zone size

[Shin'ichiro Kawasaki]

The test case zbd/005 runs fio to issue sequential write requests with
high queue depth. This workload does not require zonemode=zbd for zones
with zone capacity same as zone length. However, when the zone has
smaller zone capacity than zone size, it issues write beyond zone
capacity and triggers write errors.

To allow fio skipping the writes beyond zone capacity, specify the option
zonemode=zbd to fio when the test target zone has zone capacity smaller
than zone size.

Also remove unused sysfs access in the test case.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 tests/zbd/005 | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/tests/zbd/005 b/tests/zbd/005
index 65546a6..1e8962c 100755
--- a/tests/zbd/005
+++ b/tests/zbd/005
@@ -28,15 +28,21 @@ cleanup_fallback_device() {
 test_device() {
 	local -i zone_idx
 	local -i offset
+	local zbdmode=""
 
 	echo "Running ${TEST_NAME}"
 
-	_get_sysfs_variable "${TEST_DEV}" || return $?
 	_get_blkzone_report "${TEST_DEV}" || return $?
 
 	zone_idx=$(_find_first_sequential_zone) || return $?
 	offset=$((ZONE_STARTS[zone_idx] * 512))
 
+	# If the test target zone has smaller zone capacity than zone size,
+	# enable zonemode=zbd to have fio handle the zone capacity limit.
+	if ((ZONE_CAPS[zone_idx] != ZONE_LENGTHS[zone_idx])); then
+		zbdmode="--zonemode=zbd"
+	fi
+
 	blkzone reset -o "${ZONE_STARTS[zone_idx]}" "${TEST_DEV}"
 
 	_test_dev_queue_set scheduler deadline
@@ -45,10 +51,9 @@ test_device() {
 	FIO_PERF_FIELDS=("write io" "write iops")
 	_fio_perf --filename="${TEST_DEV}" --name zbdwo --rw=write --direct=1 \
 		  --ioengine=libaio --iodepth=128 --bs=256k \
-		  --offset="${offset}"
+		  --offset="${offset}" ${zbdmode}
 
 	_put_blkzone_report
-	_put_sysfs_variable
 
 	echo "Test complete"
 }
-- 
2.26.2

Re: [PATCH blktests 4/5] zbd/005: Enable zonemode=zbd when zone capacity is less than zone size

[Johannes Thumshirn]

Looks good,
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>

Re: [PATCH blktests 4/5] zbd/005: Enable zonemode=zbd when zone capacity is less than zone size

[Klaus Jensen]

On Jul 28 19:14, Shin'ichiro Kawasaki wrote:
> The test case zbd/005 runs fio to issue sequential write requests with
> high queue depth. This workload does not require zonemode=zbd for zones
> with zone capacity same as zone length. However, when the zone has
> smaller zone capacity than zone size, it issues write beyond zone
> capacity and triggers write errors.
> 
> To allow fio skipping the writes beyond zone capacity, specify the option
> zonemode=zbd to fio when the test target zone has zone capacity smaller
> than zone size.
> 
> Also remove unused sysfs access in the test case.
> 
> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> ---

LGTM.

Reviewed-by: Klaus Jensen <k.jensen@samsung.com>

[PATCH blktests 5/5] zbd/002: Check write pointers only when zones have valid conditions

[Shin'ichiro Kawasaki]

Per ZBC, ZAC and ZNS specifications, when zones have condition "read
only", "full" or "offline", the zones may not have valid write pointers.
In such a case, do not check validity of write pointers.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 tests/zbd/002 | 7 +++++--
 tests/zbd/rc  | 2 ++
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/tests/zbd/002 b/tests/zbd/002
index a134434..39c2ad5 100755
--- a/tests/zbd/002
+++ b/tests/zbd/002
@@ -80,9 +80,12 @@ _check_blkzone_report() {
 		fi
 
 		# Check write pointer
-		if [[ ${wptr} -lt 0 || ${wptr} -gt ${len} ]]; then
+		if ((cond != ZONE_COND_READ_ONLY &&
+		     cond != ZONE_COND_FULL &&
+		     cond != ZONE_COND_OFFLINE &&
+		     (wptr < 0 || wptr > len) )); then
 			echo -n "Write pointer is invalid at zone ${idx}. "
-			echo "wp:${wptr}"
+			echo "wp:${wptr}, cond:${cond}"
 			return 1
 		fi
 
diff --git a/tests/zbd/rc b/tests/zbd/rc
index 3fd2d36..1237363 100644
--- a/tests/zbd/rc
+++ b/tests/zbd/rc
@@ -41,7 +41,9 @@ export ZONE_TYPE_SEQ_WRITE_PREFERRED=3
 export ZONE_COND_EMPTY=1
 export ZONE_COND_IMPLICIT_OPEN=2
 export ZONE_COND_CLOSED=4
+export ZONE_COND_READ_ONLY=13
 export ZONE_COND_FULL=14
+export ZONE_COND_OFFLINE=15
 
 export ZONE_TYPE_ARRAY=(
 	[1]="CONVENTIONAL"
-- 
2.26.2

Re: [PATCH blktests 5/5] zbd/002: Check write pointers only when zones have valid conditions

[Johannes Thumshirn]

Looks good,
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>

Re: [PATCH blktests 5/5] zbd/002: Check write pointers only when zones have valid conditions

[Klaus Jensen]

On Jul 28 19:14, Shin'ichiro Kawasaki wrote:
> Per ZBC, ZAC and ZNS specifications, when zones have condition "read
> only", "full" or "offline", the zones may not have valid write pointers.
> In such a case, do not check validity of write pointers.
> 
> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> ---

LGTM.

Reviewed-by: Klaus Jensen <k.jensen@samsung.com>

Re: [PATCH blktests 0/5] Support zone capacity

[Omar Sandoval]

On Tue, Jul 28, 2020 at 07:14:47PM +0900, Shin'ichiro Kawasaki wrote:
> Kernel 5.9 zone descriptor interface adds the new zone capacity field defining
> the range of sectors usable within a zone. This patch series adds support for
> this new field. This series depends on recent changes in blkzone and fio to
> support zone capacity.
> 
> The first patch modifies the helper function _get_blkzone_report() to obtain
> zone capacity of the test target zones. Following three patches adjust test
> cases in zbd group for zone capacity. The last patch fixes a test condition
> issue found in this work.
> 
> Shin'ichiro Kawasaki (5):
>   zbd/rc: Support zone capacity report by blkzone
>   zbd/002: Check validity of zone capacity
>   zbd/004: Check zone boundary writes using zones without zone capacity
>     gap
>   zbd/005: Enable zonemode=zbd when zone capacity is less than zone size
>   zbd/002: Check write pointers only when zones have valid conditions
> 
>  tests/zbd/002 | 16 ++++++++++++++--
>  tests/zbd/004 |  6 +++++-
>  tests/zbd/005 | 11 ++++++++---
>  tests/zbd/rc  | 35 ++++++++++++++++++++++++++++++-----
>  4 files changed, 57 insertions(+), 11 deletions(-)

Thanks, applied!