[PATCH v5 0/9] fstests: add tests for btrfs' raid-stripe-tree feature

[PATCH v5 0/9] fstests: add tests for btrfs' raid-stripe-tree feature

[Johannes Thumshirn]

Add tests for btrfs' raid-stripe-tree feature. All of these test work by
writing a specific pattern to a newly created filesystem and afterwards
using `btrfs inspect-internal -t raid-stripe $SCRATCH_DEV_POOL` to verify
the placement and the layout of the metadata.

The md5sum of each file will be compared as well after a re-mount of the
filesystem.

---
Changes in v5:
- add _require_btrfs_free_space_tree helper and use in tests
- Link to v4: https://lore.kernel.org/r/20231206-btrfs-raid-v4-0-578284dd3a70@wdc.com

Changes in v4:
- add _require_btrfs_no_compress to all tests
- add _require_btrfs_no_nodatacow helper and add to btrfs/308
- add _require_btrfs_feature "free_space_tree" to all tests
- Link to v3: https://lore.kernel.org/r/20231205-btrfs-raid-v3-0-0e857a5439a2@wdc.com

Changes in v3:
- added 'raid-stripe-tree' to mkfs options, as only zoned raid gets it
  automatically
- Rename test cases as btrfs/302 and btrfs/303 already exist upstream
- Link to v2: https://lore.kernel.org/r/20231205-btrfs-raid-v2-0-25f80eea345b@wdc.com

Changes in v2:
- Re-ordered series so the newly introduced group is added before the
  tests
- Changes Filipe requested to the tests.
- Link to v1: https://lore.kernel.org/r/20231204-btrfs-raid-v1-0-b254eb1bcff8@wdc.com

---
Johannes Thumshirn (9):
      fstests: doc: add new raid-stripe-tree group
      common: add filter for btrfs raid-stripe dump
      common: add _require_btrfs_no_nodatacow helper
      common: add _require_btrfs_free_space_tree
      btrfs: add fstest for stripe-tree metadata with 4k write
      btrfs: add fstest for 8k write spanning two stripes on raid-stripe-tree
      btrfs: add fstest for writing to a file at an offset with RST
      btrfs: add fstests to write 128k to a RST filesystem
      btrfs: add fstest for overwriting a file partially with RST

 common/btrfs        |  17 +++++++++
 common/filter.btrfs |  14 +++++++
 doc/group-names.txt |   1 +
 tests/btrfs/304     |  56 +++++++++++++++++++++++++++
 tests/btrfs/304.out |  58 ++++++++++++++++++++++++++++
 tests/btrfs/305     |  61 ++++++++++++++++++++++++++++++
 tests/btrfs/305.out |  82 ++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/306     |  59 +++++++++++++++++++++++++++++
 tests/btrfs/306.out |  75 +++++++++++++++++++++++++++++++++++++
 tests/btrfs/307     |  56 +++++++++++++++++++++++++++
 tests/btrfs/307.out |  65 ++++++++++++++++++++++++++++++++
 tests/btrfs/308     |  60 +++++++++++++++++++++++++++++
 tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 13 files changed, 710 insertions(+)
---
base-commit: baca8a2b5cb6e798ce3a07e79a081031370c6cb8
change-id: 20231204-btrfs-raid-75975797f97d

Best regards,
-- 
Johannes Thumshirn <johannes.thumshirn@wdc.com>

[PATCH v5 1/9] fstests: doc: add new raid-stripe-tree group

[Johannes Thumshirn]

Add a new test group for testing the raid-stripe-tree feature of btrfs
with fstests.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 doc/group-names.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/group-names.txt b/doc/group-names.txt
index c3dcca375537..9c1624868518 100644
--- a/doc/group-names.txt
+++ b/doc/group-names.txt
@@ -94,6 +94,7 @@ punch			fallocate FALLOC_FL_PUNCH_HOLE
 qgroup			btrfs qgroup feature
 quota			filesystem usage quotas
 raid			btrfs RAID
+raid-stripe-tree	btrfs raid-stripe-tree feature
 read_repair		btrfs error correction on read failure
 realtime		XFS realtime volumes
 recoveryloop		crash recovery loops

-- 
2.43.0

[PATCH v5 2/9] common: add filter for btrfs raid-stripe dump

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 common/filter.btrfs | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/common/filter.btrfs b/common/filter.btrfs
index 02c6b92dfa94..2003ba7b7015 100644
--- a/common/filter.btrfs
+++ b/common/filter.btrfs
@@ -125,5 +125,19 @@ _filter_btrfs_cloner_error()
 	sed -e "s/\(clone failed:\) Operation not supported/\1 Invalid argument/g"
 }
 
+# filter output of "btrfs inspect-internal dump-tree -t raid-stripe"
+_filter_stripe_tree()
+{
+	sed -E -e "s/leaf [0-9]+ items [0-9]+ free space [0-9]+ generation [0-9]+ owner RAID_STRIPE_TREE/leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE/" \
+		-e "s/leaf [0-9]+ flags 0x1\(WRITTEN\) backref revision 1/leaf XXXXXXXXX flags 0x1\(WRITTEN\) backref revision 1/" \
+		-e "s/checksum stored [0-9a-f]+/checksum stored <CHECKSUM>/"  \
+		-e "s/checksum calced [0-9a-f]+/checksum calced <CHECKSUM>/"  \
+		-e "s/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<UUID>/" \
+		-e "s/item ([0-9]+) key \([0-9]+ RAID_STRIPE ([0-9]+)\) itemoff [0-9]+ itemsize ([0-9]+)/item \1 key \(XXXXXX RAID_STRIPE \2\) itemoff XXXXX itemsize \3/" \
+		-e "s/stripe ([0-9]+) devid ([0-9]+) physical [0-9]+/stripe \1 devid \2 physical XXXXXXXXX/" \
+		-e "s/total bytes [0-9]+/total bytes XXXXXXXX/" \
+		-e "s/bytes used [0-9]+/bytes used XXXXXX/"
+}
+
 # make sure this script returns success
 /bin/true

-- 
2.43.0

[PATCH v5 3/9] common: add _require_btrfs_no_nodatacow helper

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 common/btrfs | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/common/btrfs b/common/btrfs
index e3ccb4fa07d4..bf69bcee158b 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -120,6 +120,13 @@ _require_btrfs_no_compress()
 	fi
 }
 
+_require_btrfs_no_nodatacow()
+{
+	if _normalize_mount_options "$MOUNT_OPTIONS" | grep -q "nodatacow"; then
+		_notrun "This test requires no nodatacow enabled"
+	fi
+}
+
 _require_btrfs_no_block_group_tree()
 {
 	_scratch_mkfs > /dev/null 2>&1

-- 
2.43.0

[PATCH v5 4/9] common: add _require_btrfs_free_space_tree

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 common/btrfs | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/common/btrfs b/common/btrfs
index bf69bcee158b..95e46726a801 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -137,6 +137,16 @@ _require_btrfs_no_block_group_tree()
 	fi
 }
 
+_require_btrfs_free_space_tree()
+{
+	_scratch_mkfs > /dev/null 2>&1
+	if ! $BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \
+		grep -q "FREE_SPACE_TREE"
+	then
+		_notrun "This test requires a free-space-tree"
+	fi
+}
+
 _check_btrfs_filesystem()
 {
 	device=$1

-- 
2.43.0

[PATCH v5 5/9] btrfs: add fstest for stripe-tree metadata with 4k write

[Johannes Thumshirn]

Test a simple 4k write on all RAID profiles currently supported with the
raid-stripe-tree.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 tests/btrfs/304     | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/304.out | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)

diff --git a/tests/btrfs/304 b/tests/btrfs/304
new file mode 100755
index 000000000000..7774a1bd74d0
--- /dev/null
+++ b/tests/btrfs/304
@@ -0,0 +1,56 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Western Digital Cooperation.  All Rights Reserved.
+#
+# FS QA Test 304
+#
+# Test on-disk layout of RAID Stripe Tree Metadata writing 4k to a new file on
+# a pristine file system.
+#
+. ./common/preamble
+_begin_fstest auto quick raid remount volume raid-stripe-tree
+
+. ./common/filter
+. ./common/filter.btrfs
+
+_supported_fs btrfs
+_require_btrfs_command inspect-internal dump-tree
+_require_btrfs_mkfs_feature "raid-stripe-tree"
+_require_scratch_dev_pool 4
+_require_btrfs_fs_feature "raid_stripe_tree"
+_require_btrfs_fs_feature "free_space_tree"
+_require_btrfs_free_space_tree
+_require_btrfs_no_compress
+
+test_4k_write()
+{
+	local profile=$1
+	local ndevs=$2
+
+	_scratch_dev_pool_get $ndevs
+
+	echo "==== Testing $profile ===="
+	_scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree
+	_scratch_mount
+
+	$XFS_IO_PROG -fc "pwrite 0 4k" "$SCRATCH_MNT/foo" | _filter_xfs_io
+
+	_scratch_cycle_mount
+	md5sum "$SCRATCH_MNT/foo" | _filter_scratch
+
+	_scratch_unmount
+
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\
+		_filter_btrfs_version |  _filter_stripe_tree
+
+	_scratch_dev_pool_put
+}
+
+echo "= Test basic 4k write ="
+test_4k_write raid0 2
+test_4k_write raid1 2
+test_4k_write raid10 4
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/304.out b/tests/btrfs/304.out
new file mode 100644
index 000000000000..48036efbf0cf
--- /dev/null
+++ b/tests/btrfs/304.out
@@ -0,0 +1,58 @@
+QA output created by 304
+= Test basic 4k write =
+==== Testing raid0 ====
+wrote 4096/4096 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+5fed275e7617a806f94c173746a2a723  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid1 ====
+wrote 4096/4096 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+5fed275e7617a806f94c173746a2a723  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid10 ====
+wrote 4096/4096 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+5fed275e7617a806f94c173746a2a723  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>

-- 
2.43.0

[PATCH v5 6/9] btrfs: add fstest for 8k write spanning two stripes on raid-stripe-tree

[Johannes Thumshirn]

Add a test-case writing 8k to a raid-stripe-tree formatted filesystem with
one stripe pre-filled to 60k so the 8k are split into a 4k write finishing
stripe 1 and a 4k write starting the next stripe.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 tests/btrfs/305     | 61 +++++++++++++++++++++++++++++++++++++++
 tests/btrfs/305.out | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 143 insertions(+)

diff --git a/tests/btrfs/305 b/tests/btrfs/305
new file mode 100755
index 000000000000..df8a12ab9a6a
--- /dev/null
+++ b/tests/btrfs/305
@@ -0,0 +1,61 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Western Digital Cooperation.  All Rights Reserved.
+#
+# FS QA Test 305
+#
+# Test on-disk layout of RAID Stripe Tree Metadata by writing 8k to a new file
+# with a filesystem prepropulated, so that 4k of the write are written to the
+# 1st stripe and 4k start a new stripe.
+#
+. ./common/preamble
+_begin_fstest auto quick raid remount volume raid-stripe-tree
+
+. ./common/filter
+. ./common/filter.btrfs
+
+_supported_fs btrfs
+_require_btrfs_command inspect-internal dump-tree
+_require_btrfs_mkfs_feature "raid-stripe-tree"
+_require_scratch_dev_pool 4
+_require_btrfs_fs_feature "raid_stripe_tree"
+_require_btrfs_fs_feature "free_space_tree"
+_require_btrfs_free_space_tree
+_require_btrfs_no_compress
+
+test_8k_new_stripe()
+{
+	local profile=$1
+	local ndevs=$2
+
+	_scratch_dev_pool_get $ndevs
+
+	echo "==== Testing $profile ===="
+	_scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree
+	_scratch_mount
+
+	# Fill the first stripe up to 64k - 4k
+	$XFS_IO_PROG -fc "pwrite 0 60k" -c fsync "$SCRATCH_MNT/bar" | _filter_xfs_io
+
+	# The actual 8k write
+	$XFS_IO_PROG -fc "pwrite 0 8k" "$SCRATCH_MNT/foo" | _filter_xfs_io
+
+	_scratch_cycle_mount
+	md5sum "$SCRATCH_MNT/foo" | _filter_scratch
+
+	_scratch_unmount
+
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\
+		_filter_btrfs_version | _filter_stripe_tree
+
+	_scratch_dev_pool_put
+}
+
+echo "= Test 8k write to a new file so that 4k start a new stripe ="
+test_8k_new_stripe raid0 2
+test_8k_new_stripe raid1 2
+test_8k_new_stripe raid10 4
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/305.out b/tests/btrfs/305.out
new file mode 100644
index 000000000000..7460501ef40b
--- /dev/null
+++ b/tests/btrfs/305.out
@@ -0,0 +1,82 @@
+QA output created by 305
+= Test 8k write to a new file so that 4k start a new stripe =
+==== Testing raid0 ====
+wrote 61440/61440 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+9d3940adb41dd525e008a847e01b15f4  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 61440) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+	item 2 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid1 ====
+wrote 61440/61440 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+9d3940adb41dd525e008a847e01b15f4  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 61440) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 8192) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid10 ====
+wrote 61440/61440 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+9d3940adb41dd525e008a847e01b15f4  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 61440) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 2 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 3 physical XXXXXXXXX
+			stripe 1 devid 4 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>

-- 
2.43.0

[PATCH v5 7/9] btrfs: add fstest for writing to a file at an offset with RST

[Johannes Thumshirn]

Add a fstest writing 4k at offset 64k to a file with one RAID tripe
already pre-filled for a raid-stripe-tree formatted file system.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 tests/btrfs/306     | 59 +++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/306.out | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 134 insertions(+)

diff --git a/tests/btrfs/306 b/tests/btrfs/306
new file mode 100755
index 000000000000..72c066aa6940
--- /dev/null
+++ b/tests/btrfs/306
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Western Digital Cooperation.  All Rights Reserved.
+#
+# FS QA Test 306
+#
+# Test on-disk layout of RAID Stripe Tree Metadata by writing 4k to an emppty
+# file at offset 64k with one stripe pre-filled on an otherwise pristine
+# filesystem.
+#
+. ./common/preamble
+_begin_fstest auto quick raid remount volume raid-stripe-tree
+
+. ./common/filter
+. ./common/filter.btrfs
+
+_supported_fs btrfs
+_require_btrfs_command inspect-internal dump-tree
+_require_btrfs_mkfs_feature "raid-stripe-tree"
+_require_scratch_dev_pool 4
+_require_btrfs_fs_feature "raid_stripe_tree"
+_require_btrfs_fs_feature "free_space_tree"
+_require_btrfs_free_space_tree
+_require_btrfs_no_compress
+
+test_4k_write_64koff()
+{
+	local profile=$1
+	local ndevs=$2
+
+	_scratch_dev_pool_get $ndevs
+
+	echo "==== Testing $profile ===="
+	_scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree
+	_scratch_mount
+
+	# precondition one stripe
+	$XFS_IO_PROG -fc "pwrite 0 64k" "$SCRATCH_MNT/bar" | _filter_xfs_io
+
+	$XFS_IO_PROG -fc "pwrite 64k 4k" "$SCRATCH_MNT/foo" | _filter_xfs_io
+
+	_scratch_cycle_mount
+	md5sum "$SCRATCH_MNT/foo" | _filter_scratch
+
+	_scratch_unmount
+
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\
+		_filter_btrfs_version | _filter_stripe_tree
+
+	_scratch_dev_pool_put
+}
+echo "= Test 4k write to an empty file at offset 64k with one stripe prefilled ="
+test_4k_write_64koff raid0 2
+test_4k_write_64koff raid1 2
+test_4k_write_64koff raid10 4
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/306.out b/tests/btrfs/306.out
new file mode 100644
index 000000000000..8031cddb79f7
--- /dev/null
+++ b/tests/btrfs/306.out
@@ -0,0 +1,75 @@
+QA output created by 306
+= Test 4k write to an empty file at offset 64k with one stripe prefilled =
+==== Testing raid0 ====
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 4096/4096 bytes at offset 65536
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+381b0e7d72cb4f75286fe2b445e8d92a  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid1 ====
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 4096/4096 bytes at offset 65536
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+381b0e7d72cb4f75286fe2b445e8d92a  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid10 ====
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 4096/4096 bytes at offset 65536
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+381b0e7d72cb4f75286fe2b445e8d92a  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 4096) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 3 physical XXXXXXXXX
+			stripe 1 devid 4 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>

-- 
2.43.0

[PATCH v5 8/9] btrfs: add fstests to write 128k to a RST filesystem

[Johannes Thumshirn]

Add a test writing 128k to a file on an empty filesystem formatted with a
raid-stripe-tree.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 tests/btrfs/307     | 56 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/307.out | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 121 insertions(+)

diff --git a/tests/btrfs/307 b/tests/btrfs/307
new file mode 100755
index 000000000000..9509019b50cc
--- /dev/null
+++ b/tests/btrfs/307
@@ -0,0 +1,56 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Western Digital Cooperation.  All Rights Reserved.
+#
+# FS QA Test 307
+#
+# Test on-disk layout of RAID Stripe Tree Metadata by writing 128k to a new
+# file on a pristine filesystem
+#
+. ./common/preamble
+_begin_fstest auto quick raid remount volume raid-stripe-tree
+
+. ./common/filter
+. ./common/filter.btrfs
+
+_supported_fs btrfs
+_require_btrfs_command inspect-internal dump-tree
+_require_btrfs_mkfs_feature "raid-stripe-tree"
+_require_scratch_dev_pool 4
+_require_btrfs_fs_feature "raid_stripe_tree"
+_require_btrfs_fs_feature "free_space_tree"
+_require_btrfs_free_space_tree
+_require_btrfs_no_compress
+
+test_128k_write()
+{
+	local profile=$1
+	local ndevs=$2
+
+	_scratch_dev_pool_get $ndevs
+
+	echo "==== Testing $profile ===="
+	_scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree
+	_scratch_mount
+
+	$XFS_IO_PROG -fc "pwrite 0 128k" "$SCRATCH_MNT/foo" | _filter_xfs_io
+
+	_scratch_cycle_mount
+	md5sum "$SCRATCH_MNT/foo" | _filter_scratch
+
+	_scratch_unmount
+
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\
+		_filter_btrfs_version | _filter_stripe_tree
+
+	_scratch_dev_pool_put
+}
+
+echo "= Test 128k write to empty file  ="
+test_128k_write raid0 2
+test_128k_write raid1 2
+test_128k_write raid10 4
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/307.out b/tests/btrfs/307.out
new file mode 100644
index 000000000000..b39f3162bff6
--- /dev/null
+++ b/tests/btrfs/307.out
@@ -0,0 +1,65 @@
+QA output created by 307
+= Test 128k write to empty file  =
+==== Testing raid0 ====
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+d48858312a922db7eb86377f638dbc9f  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid1 ====
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+d48858312a922db7eb86377f638dbc9f  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 131072) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid10 ====
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+d48858312a922db7eb86377f638dbc9f  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 3 physical XXXXXXXXX
+			stripe 1 devid 4 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>

-- 
2.43.0

[PATCH v5 9/9] btrfs: add fstest for overwriting a file partially with RST

[Johannes Thumshirn]

Add a test writing 128k to an empty file with one stripe already
pre-filled on-disk. Then overwrite a portion of the file in the middle.

Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
 tests/btrfs/308     |  60 +++++++++++++++++++++++++++++
 tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 166 insertions(+)

diff --git a/tests/btrfs/308 b/tests/btrfs/308
new file mode 100755
index 000000000000..eb63acf6a9f2
--- /dev/null
+++ b/tests/btrfs/308
@@ -0,0 +1,60 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Western Digital Cooperation.  All Rights Reserved.
+#
+# FS QA Test 308
+#
+# Test on-disk layout of RAID Stripe Tree Metadata by writing 128k to an empty
+# file on a filesystem that has one stripe already pre-filled. Afterwards
+# overwrite a portion of the file.
+#
+. ./common/preamble
+_begin_fstest auto quick raid remount volume raid-stripe-tree
+
+. ./common/filter
+. ./common/filter.btrfs
+
+_supported_fs btrfs
+_require_btrfs_command inspect-internal dump-tree
+_require_btrfs_mkfs_feature "raid-stripe-tree"
+_require_scratch_dev_pool 4
+_require_btrfs_fs_feature "raid_stripe_tree"
+_require_btrfs_fs_feature "free_space_tree"
+_require_btrfs_free_space_tree
+_require_btrfs_no_compress
+_require_btrfs_no_nodatacow
+
+test_128k_write_overwrite()
+{
+	local profile=$1
+	local ndevs=$2
+
+	_scratch_dev_pool_get $ndevs
+
+	echo "==== Testing $profile ===="
+	_scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree
+	_scratch_mount
+
+	$XFS_IO_PROG -fc "pwrite -W 0 32k" "$SCRATCH_MNT/bar" | _filter_xfs_io
+	$XFS_IO_PROG -fc "pwrite -W 0 128k" "$SCRATCH_MNT/foo" | _filter_xfs_io
+	$XFS_IO_PROG -fc "pwrite -W 64k 8k" "$SCRATCH_MNT/foo" | _filter_xfs_io
+
+	_scratch_cycle_mount
+	md5sum "$SCRATCH_MNT/foo" | _filter_scratch
+
+	_scratch_unmount
+
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\
+		_filter_btrfs_version | _filter_stripe_tree
+
+	_scratch_dev_pool_put
+}
+
+echo "= Test 128k write to empty file with 1st stripe partially prefilled then overwrite ="
+test_128k_write_overwrite raid0 2
+test_128k_write_overwrite raid1 2
+test_128k_write_overwrite raid10 4
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/308.out b/tests/btrfs/308.out
new file mode 100644
index 000000000000..c93a3a43b248
--- /dev/null
+++ b/tests/btrfs/308.out
@@ -0,0 +1,106 @@
+QA output created by 308
+= Test 128k write to empty file with 1st stripe partially prefilled then overwrite =
+==== Testing raid0 ====
+wrote 32768/32768 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 65536
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+d48858312a922db7eb86377f638dbc9f  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 32768) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 32768) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+	item 2 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 2 physical XXXXXXXXX
+	item 3 key (XXXXXX RAID_STRIPE 32768) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+	item 4 key (XXXXXX RAID_STRIPE 8192) itemoff XXXXX itemsize 24
+			encoding: RAID0
+			stripe 0 devid 1 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid1 ====
+wrote 32768/32768 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 65536
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+d48858312a922db7eb86377f638dbc9f  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 32768) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 131072) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 2 key (XXXXXX RAID_STRIPE 8192) itemoff XXXXX itemsize 40
+			encoding: RAID1
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>
+==== Testing raid10 ====
+wrote 32768/32768 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 65536
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+d48858312a922db7eb86377f638dbc9f  SCRATCH_MNT/foo
+
+raid stripe tree key (RAID_STRIPE_TREE ROOT_ITEM 0) 
+leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE
+leaf XXXXXXXXX flags 0x1(WRITTEN) backref revision 1
+checksum stored <CHECKSUM>
+checksum calced <CHECKSUM>
+fs uuid <UUID>
+chunk uuid <UUID>
+	item 0 key (XXXXXX RAID_STRIPE 32768) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 1 key (XXXXXX RAID_STRIPE 32768) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 2 key (XXXXXX RAID_STRIPE 65536) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 3 physical XXXXXXXXX
+			stripe 1 devid 4 physical XXXXXXXXX
+	item 3 key (XXXXXX RAID_STRIPE 32768) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+	item 4 key (XXXXXX RAID_STRIPE 8192) itemoff XXXXX itemsize 40
+			encoding: RAID10
+			stripe 0 devid 1 physical XXXXXXXXX
+			stripe 1 devid 2 physical XXXXXXXXX
+total bytes XXXXXXXX
+bytes used XXXXXX
+uuid <UUID>

-- 
2.43.0

Re: [PATCH v5 0/9] fstests: add tests for btrfs' raid-stripe-tree feature

[Filipe Manana]

On Thu, Dec 7, 2023 at 9:03 AM Johannes Thumshirn
<johannes.thumshirn@wdc.com> wrote:
>
> Add tests for btrfs' raid-stripe-tree feature. All of these test work by
> writing a specific pattern to a newly created filesystem and afterwards
> using `btrfs inspect-internal -t raid-stripe $SCRATCH_DEV_POOL` to verify
> the placement and the layout of the metadata.
>
> The md5sum of each file will be compared as well after a re-mount of the
> filesystem.
>
> ---
> Changes in v5:
> - add _require_btrfs_free_space_tree helper and use in tests
> - Link to v4: https://lore.kernel.org/r/20231206-btrfs-raid-v4-0-578284dd3a70@wdc.com
>
> Changes in v4:
> - add _require_btrfs_no_compress to all tests
> - add _require_btrfs_no_nodatacow helper and add to btrfs/308
> - add _require_btrfs_feature "free_space_tree" to all tests
> - Link to v3: https://lore.kernel.org/r/20231205-btrfs-raid-v3-0-0e857a5439a2@wdc.com
>
> Changes in v3:
> - added 'raid-stripe-tree' to mkfs options, as only zoned raid gets it
>   automatically
> - Rename test cases as btrfs/302 and btrfs/303 already exist upstream
> - Link to v2: https://lore.kernel.org/r/20231205-btrfs-raid-v2-0-25f80eea345b@wdc.com
>
> Changes in v2:
> - Re-ordered series so the newly introduced group is added before the
>   tests
> - Changes Filipe requested to the tests.
> - Link to v1: https://lore.kernel.org/r/20231204-btrfs-raid-v1-0-b254eb1bcff8@wdc.com
>
> ---
> Johannes Thumshirn (9):
>       fstests: doc: add new raid-stripe-tree group
>       common: add filter for btrfs raid-stripe dump
>       common: add _require_btrfs_no_nodatacow helper
>       common: add _require_btrfs_free_space_tree
>       btrfs: add fstest for stripe-tree metadata with 4k write
>       btrfs: add fstest for 8k write spanning two stripes on raid-stripe-tree
>       btrfs: add fstest for writing to a file at an offset with RST
>       btrfs: add fstests to write 128k to a RST filesystem
>       btrfs: add fstest for overwriting a file partially with RST
>
>  common/btrfs        |  17 +++++++++
>  common/filter.btrfs |  14 +++++++
>  doc/group-names.txt |   1 +
>  tests/btrfs/304     |  56 +++++++++++++++++++++++++++
>  tests/btrfs/304.out |  58 ++++++++++++++++++++++++++++
>  tests/btrfs/305     |  61 ++++++++++++++++++++++++++++++
>  tests/btrfs/305.out |  82 ++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/306     |  59 +++++++++++++++++++++++++++++
>  tests/btrfs/306.out |  75 +++++++++++++++++++++++++++++++++++++
>  tests/btrfs/307     |  56 +++++++++++++++++++++++++++
>  tests/btrfs/307.out |  65 ++++++++++++++++++++++++++++++++
>  tests/btrfs/308     |  60 +++++++++++++++++++++++++++++
>  tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  13 files changed, 710 insertions(+)
> ---
> base-commit: baca8a2b5cb6e798ce3a07e79a081031370c6cb8

Btw this base commit does not exist in the official fstests repo.
That commit is from the staging branch at https://github.com/kdave/xfstests

A "git am" will fail because the official fstests repo doesn't have
_require_btrfs_no_block_group_tree() at common/btrfs,
so it needs to be manually adjusted when applying the 3rd patch.

I tried the tests and they look good, so:

Reviewed-by: Filipe Manana <fdmanana@suse.com>

One question I missed before. Test 304 for example does a 4K write and
expects in the golden output to get a 4K raid stripe item.
What happens on a machine with 64K page size? There the default sector
size is 64K, will the write result in a 64K raid stripe item or will
it be 4K? In the former case, it will make the test fail.

Thanks.


> change-id: 20231204-btrfs-raid-75975797f97d
>
> Best regards,
> --
> Johannes Thumshirn <johannes.thumshirn@wdc.com>
>
>

Re: [PATCH v5 0/9] fstests: add tests for btrfs' raid-stripe-tree feature

[Anand Jain]

On 12/7/23 17:41, Filipe Manana wrote:
> On Thu, Dec 7, 2023 at 9:03 AM Johannes Thumshirn
> <johannes.thumshirn@wdc.com> wrote:
>>
>> Add tests for btrfs' raid-stripe-tree feature. All of these test work by
>> writing a specific pattern to a newly created filesystem and afterwards
>> using `btrfs inspect-internal -t raid-stripe $SCRATCH_DEV_POOL` to verify
>> the placement and the layout of the metadata.
>>
>> The md5sum of each file will be compared as well after a re-mount of the
>> filesystem.
>>
>> ---
>> Changes in v5:
>> - add _require_btrfs_free_space_tree helper and use in tests
>> - Link to v4: https://lore.kernel.org/r/20231206-btrfs-raid-v4-0-578284dd3a70@wdc.com
>>
>> Changes in v4:
>> - add _require_btrfs_no_compress to all tests
>> - add _require_btrfs_no_nodatacow helper and add to btrfs/308
>> - add _require_btrfs_feature "free_space_tree" to all tests
>> - Link to v3: https://lore.kernel.org/r/20231205-btrfs-raid-v3-0-0e857a5439a2@wdc.com
>>
>> Changes in v3:
>> - added 'raid-stripe-tree' to mkfs options, as only zoned raid gets it
>>    automatically
>> - Rename test cases as btrfs/302 and btrfs/303 already exist upstream
>> - Link to v2: https://lore.kernel.org/r/20231205-btrfs-raid-v2-0-25f80eea345b@wdc.com
>>
>> Changes in v2:
>> - Re-ordered series so the newly introduced group is added before the
>>    tests
>> - Changes Filipe requested to the tests.
>> - Link to v1: https://lore.kernel.org/r/20231204-btrfs-raid-v1-0-b254eb1bcff8@wdc.com
>>
>> ---
>> Johannes Thumshirn (9):
>>        fstests: doc: add new raid-stripe-tree group
>>        common: add filter for btrfs raid-stripe dump
>>        common: add _require_btrfs_no_nodatacow helper
>>        common: add _require_btrfs_free_space_tree
>>        btrfs: add fstest for stripe-tree metadata with 4k write
>>        btrfs: add fstest for 8k write spanning two stripes on raid-stripe-tree
>>        btrfs: add fstest for writing to a file at an offset with RST
>>        btrfs: add fstests to write 128k to a RST filesystem
>>        btrfs: add fstest for overwriting a file partially with RST
>>
>>   common/btrfs        |  17 +++++++++
>>   common/filter.btrfs |  14 +++++++
>>   doc/group-names.txt |   1 +
>>   tests/btrfs/304     |  56 +++++++++++++++++++++++++++
>>   tests/btrfs/304.out |  58 ++++++++++++++++++++++++++++
>>   tests/btrfs/305     |  61 ++++++++++++++++++++++++++++++
>>   tests/btrfs/305.out |  82 ++++++++++++++++++++++++++++++++++++++++
>>   tests/btrfs/306     |  59 +++++++++++++++++++++++++++++
>>   tests/btrfs/306.out |  75 +++++++++++++++++++++++++++++++++++++
>>   tests/btrfs/307     |  56 +++++++++++++++++++++++++++
>>   tests/btrfs/307.out |  65 ++++++++++++++++++++++++++++++++
>>   tests/btrfs/308     |  60 +++++++++++++++++++++++++++++
>>   tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   13 files changed, 710 insertions(+)
>> ---
>> base-commit: baca8a2b5cb6e798ce3a07e79a081031370c6cb8
> 
> Btw this base commit does not exist in the official fstests repo.
> That commit is from the staging branch at https://github.com/kdave/xfstests
> 
> A "git am" will fail because the official fstests repo doesn't have
> _require_btrfs_no_block_group_tree() at common/btrfs,
> so it needs to be manually adjusted when applying the 3rd patch.
> 
> I tried the tests and they look good, so:
> 
> Reviewed-by: Filipe Manana <fdmanana@suse.com>
> 
> One question I missed before. Test 304 for example does a 4K write and
> expects in the golden output to get a 4K raid stripe item.
> What happens on a machine with 64K page size? There the default sector
> size is 64K, will the write result in a 64K raid stripe item or will
> it be 4K? In the former case, it will make the test fail.
> 

Testing on a 64K pagesize. Will run it. Apologies for intermittent 
responses; OOO until December 21.

Thanks Anand


> Thanks.
> 
> 
>> change-id: 20231204-btrfs-raid-75975797f97d
>>
>> Best regards,
>> --
>> Johannes Thumshirn <johannes.thumshirn@wdc.com>
>>
>>

Re: [PATCH v5 0/9] fstests: add tests for btrfs' raid-stripe-tree feature

[Johannes Thumshirn]

On 08.12.23 02:19, Anand Jain wrote:
> 
> 
> On 12/7/23 17:41, Filipe Manana wrote:
>> On Thu, Dec 7, 2023 at 9:03 AM Johannes Thumshirn
>> <johannes.thumshirn@wdc.com> wrote:
>>>
>>> Add tests for btrfs' raid-stripe-tree feature. All of these test work by
>>> writing a specific pattern to a newly created filesystem and afterwards
>>> using `btrfs inspect-internal -t raid-stripe $SCRATCH_DEV_POOL` to verify
>>> the placement and the layout of the metadata.
>>>
>>> The md5sum of each file will be compared as well after a re-mount of the
>>> filesystem.
>>>
>>> ---
>>> Changes in v5:
>>> - add _require_btrfs_free_space_tree helper and use in tests
>>> - Link to v4: https://lore.kernel.org/r/20231206-btrfs-raid-v4-0-578284dd3a70@wdc.com
>>>
>>> Changes in v4:
>>> - add _require_btrfs_no_compress to all tests
>>> - add _require_btrfs_no_nodatacow helper and add to btrfs/308
>>> - add _require_btrfs_feature "free_space_tree" to all tests
>>> - Link to v3: https://lore.kernel.org/r/20231205-btrfs-raid-v3-0-0e857a5439a2@wdc.com
>>>
>>> Changes in v3:
>>> - added 'raid-stripe-tree' to mkfs options, as only zoned raid gets it
>>>     automatically
>>> - Rename test cases as btrfs/302 and btrfs/303 already exist upstream
>>> - Link to v2: https://lore.kernel.org/r/20231205-btrfs-raid-v2-0-25f80eea345b@wdc.com
>>>
>>> Changes in v2:
>>> - Re-ordered series so the newly introduced group is added before the
>>>     tests
>>> - Changes Filipe requested to the tests.
>>> - Link to v1: https://lore.kernel.org/r/20231204-btrfs-raid-v1-0-b254eb1bcff8@wdc.com
>>>
>>> ---
>>> Johannes Thumshirn (9):
>>>         fstests: doc: add new raid-stripe-tree group
>>>         common: add filter for btrfs raid-stripe dump
>>>         common: add _require_btrfs_no_nodatacow helper
>>>         common: add _require_btrfs_free_space_tree
>>>         btrfs: add fstest for stripe-tree metadata with 4k write
>>>         btrfs: add fstest for 8k write spanning two stripes on raid-stripe-tree
>>>         btrfs: add fstest for writing to a file at an offset with RST
>>>         btrfs: add fstests to write 128k to a RST filesystem
>>>         btrfs: add fstest for overwriting a file partially with RST
>>>
>>>    common/btrfs        |  17 +++++++++
>>>    common/filter.btrfs |  14 +++++++
>>>    doc/group-names.txt |   1 +
>>>    tests/btrfs/304     |  56 +++++++++++++++++++++++++++
>>>    tests/btrfs/304.out |  58 ++++++++++++++++++++++++++++
>>>    tests/btrfs/305     |  61 ++++++++++++++++++++++++++++++
>>>    tests/btrfs/305.out |  82 ++++++++++++++++++++++++++++++++++++++++
>>>    tests/btrfs/306     |  59 +++++++++++++++++++++++++++++
>>>    tests/btrfs/306.out |  75 +++++++++++++++++++++++++++++++++++++
>>>    tests/btrfs/307     |  56 +++++++++++++++++++++++++++
>>>    tests/btrfs/307.out |  65 ++++++++++++++++++++++++++++++++
>>>    tests/btrfs/308     |  60 +++++++++++++++++++++++++++++
>>>    tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>    13 files changed, 710 insertions(+)
>>> ---
>>> base-commit: baca8a2b5cb6e798ce3a07e79a081031370c6cb8
>>
>> Btw this base commit does not exist in the official fstests repo.
>> That commit is from the staging branch at https://github.com/kdave/xfstests
>>
>> A "git am" will fail because the official fstests repo doesn't have
>> _require_btrfs_no_block_group_tree() at common/btrfs,
>> so it needs to be manually adjusted when applying the 3rd patch.
>>
>> I tried the tests and they look good, so:
>>
>> Reviewed-by: Filipe Manana <fdmanana@suse.com>
>>
>> One question I missed before. Test 304 for example does a 4K write and
>> expects in the golden output to get a 4K raid stripe item.
>> What happens on a machine with 64K page size? There the default sector
>> size is 64K, will the write result in a 64K raid stripe item or will
>> it be 4K? In the former case, it will make the test fail.
>>
> 
> Testing on a 64K pagesize. Will run it. Apologies for intermittent
> responses; OOO until December 21.

Thanks Anand!

I don't have a 64k page size system to test, but I _think_ Filipe is 
right, that will fail. I think we should skip these tests on non 4k sectors.

Re: [PATCH v5 0/9] fstests: add tests for btrfs' raid-stripe-tree feature

[Neal Gompa]

On Fri, Dec 8, 2023 at 4:19 AM Johannes Thumshirn
<Johannes.Thumshirn@wdc.com> wrote:
>
> On 08.12.23 02:19, Anand Jain wrote:
> >
> >
> > On 12/7/23 17:41, Filipe Manana wrote:
> >> On Thu, Dec 7, 2023 at 9:03 AM Johannes Thumshirn
> >> <johannes.thumshirn@wdc.com> wrote:
> >>>
> >>> Add tests for btrfs' raid-stripe-tree feature. All of these test work by
> >>> writing a specific pattern to a newly created filesystem and afterwards
> >>> using `btrfs inspect-internal -t raid-stripe $SCRATCH_DEV_POOL` to verify
> >>> the placement and the layout of the metadata.
> >>>
> >>> The md5sum of each file will be compared as well after a re-mount of the
> >>> filesystem.
> >>>
> >>> ---
> >>> Changes in v5:
> >>> - add _require_btrfs_free_space_tree helper and use in tests
> >>> - Link to v4: https://lore.kernel.org/r/20231206-btrfs-raid-v4-0-578284dd3a70@wdc.com
> >>>
> >>> Changes in v4:
> >>> - add _require_btrfs_no_compress to all tests
> >>> - add _require_btrfs_no_nodatacow helper and add to btrfs/308
> >>> - add _require_btrfs_feature "free_space_tree" to all tests
> >>> - Link to v3: https://lore.kernel.org/r/20231205-btrfs-raid-v3-0-0e857a5439a2@wdc.com
> >>>
> >>> Changes in v3:
> >>> - added 'raid-stripe-tree' to mkfs options, as only zoned raid gets it
> >>>     automatically
> >>> - Rename test cases as btrfs/302 and btrfs/303 already exist upstream
> >>> - Link to v2: https://lore.kernel.org/r/20231205-btrfs-raid-v2-0-25f80eea345b@wdc.com
> >>>
> >>> Changes in v2:
> >>> - Re-ordered series so the newly introduced group is added before the
> >>>     tests
> >>> - Changes Filipe requested to the tests.
> >>> - Link to v1: https://lore.kernel.org/r/20231204-btrfs-raid-v1-0-b254eb1bcff8@wdc.com
> >>>
> >>> ---
> >>> Johannes Thumshirn (9):
> >>>         fstests: doc: add new raid-stripe-tree group
> >>>         common: add filter for btrfs raid-stripe dump
> >>>         common: add _require_btrfs_no_nodatacow helper
> >>>         common: add _require_btrfs_free_space_tree
> >>>         btrfs: add fstest for stripe-tree metadata with 4k write
> >>>         btrfs: add fstest for 8k write spanning two stripes on raid-stripe-tree
> >>>         btrfs: add fstest for writing to a file at an offset with RST
> >>>         btrfs: add fstests to write 128k to a RST filesystem
> >>>         btrfs: add fstest for overwriting a file partially with RST
> >>>
> >>>    common/btrfs        |  17 +++++++++
> >>>    common/filter.btrfs |  14 +++++++
> >>>    doc/group-names.txt |   1 +
> >>>    tests/btrfs/304     |  56 +++++++++++++++++++++++++++
> >>>    tests/btrfs/304.out |  58 ++++++++++++++++++++++++++++
> >>>    tests/btrfs/305     |  61 ++++++++++++++++++++++++++++++
> >>>    tests/btrfs/305.out |  82 ++++++++++++++++++++++++++++++++++++++++
> >>>    tests/btrfs/306     |  59 +++++++++++++++++++++++++++++
> >>>    tests/btrfs/306.out |  75 +++++++++++++++++++++++++++++++++++++
> >>>    tests/btrfs/307     |  56 +++++++++++++++++++++++++++
> >>>    tests/btrfs/307.out |  65 ++++++++++++++++++++++++++++++++
> >>>    tests/btrfs/308     |  60 +++++++++++++++++++++++++++++
> >>>    tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>    13 files changed, 710 insertions(+)
> >>> ---
> >>> base-commit: baca8a2b5cb6e798ce3a07e79a081031370c6cb8
> >>
> >> Btw this base commit does not exist in the official fstests repo.
> >> That commit is from the staging branch at https://github.com/kdave/xfstests
> >>
> >> A "git am" will fail because the official fstests repo doesn't have
> >> _require_btrfs_no_block_group_tree() at common/btrfs,
> >> so it needs to be manually adjusted when applying the 3rd patch.
> >>
> >> I tried the tests and they look good, so:
> >>
> >> Reviewed-by: Filipe Manana <fdmanana@suse.com>
> >>
> >> One question I missed before. Test 304 for example does a 4K write and
> >> expects in the golden output to get a 4K raid stripe item.
> >> What happens on a machine with 64K page size? There the default sector
> >> size is 64K, will the write result in a 64K raid stripe item or will
> >> it be 4K? In the former case, it will make the test fail.
> >>
> >
> > Testing on a 64K pagesize. Will run it. Apologies for intermittent
> > responses; OOO until December 21.
>
> Thanks Anand!
>
> I don't have a 64k page size system to test, but I _think_ Filipe is
> right, that will fail. I think we should skip these tests on non 4k sectors.
>

Once we land the patch to default to 4k sector size[1] regardless of
page size, this should all work across all architectures, no?

[1]: https://lore.kernel.org/linux-btrfs/20231116160235.2708131-2-neal@gompa.dev/



--
真実はいつも一つ！/ Always, there's only one truth!

Re: [PATCH v5 0/9] fstests: add tests for btrfs' raid-stripe-tree feature

[Johannes Thumshirn]

On 09.12.23 20:16, Neal Gompa wrote:
> 
> Once we land the patch to default to 4k sector size[1] regardless of
> page size, this should all work across all architectures, no?
> 
> [1]: https://lore.kernel.org/linux-btrfs/20231116160235.2708131-2-neal@gompa.dev/
> 

Per default, yes. But we can still set 64k sector size manually.