* [PATCH 1/3] common/btrfs: add helper to get the bytenr for a file extent item
2023-05-09 11:52 [PATCH 0/3] btrfs: add test case for logical-resolve fdmanana
@ 2023-05-09 11:52 ` fdmanana
2023-05-10 5:50 ` Anand Jain
2023-05-09 11:52 ` [PATCH 2/3] btrfs: add a test case for the logical to ino ioctl fdmanana
2023-05-09 11:52 ` [PATCH 3/3] groups: add logical_resolve group for btrfs fdmanana
2 siblings, 1 reply; 7+ messages in thread
From: fdmanana @ 2023-05-09 11:52 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana
From: Filipe Manana <fdmanana@suse.com>
In upcoming changes there will be the need to find out the logical disk
address (bytenr) that a particular file extent item points to. This is
already implemented as local functions in the test btrfs/299, which is
a bit limited but works fine for that test. Some important or subtle
details why it works for this test:
1) It dumps all trees of the filesystem;
2) It relies on fsync'ing a file and then finding the desired file
extent item in the log tree from the dump;
3) There's a single subvolume, so it always finds the correct file extent
item. In case there were multiple subvolumes, it could pick the wrong
file extent item in case we have inodes with the same number on
multiple subvolumes (inode numbers are unique only within a subvolume,
they are not unique across an entire filesystem).
So add a helper to get the bytenr associated to a file extent item to
common/btrfs and use it at btrfs/299 and the upcoming changes.
This helper will dump only the tree of the default subvolume, will sync
the filesystem to commit any open transaction and works only in case the
filesystem is using the scratch device. This is explicitly documented.
Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
common/btrfs | 23 +++++++++++++++++++++++
tests/btrfs/299 | 18 +-----------------
2 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/common/btrfs b/common/btrfs
index 344509ce..42777df2 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -624,3 +624,26 @@ _require_btrfs_send_v2()
[ $(cat /sys/fs/btrfs/features/send_stream_version) -gt 1 ] || \
_notrun "kernel does not support send stream v2"
}
+
+# Get the bytenr associated to a file extent item at a given file offset.
+#
+# NOTE: At the moment this only works if the file is on a filesystem on top of
+# the scratch device and the file is in the default subvolume (tree id 5).
+_btrfs_get_file_extent_item_bytenr()
+{
+ local file="$1"
+ local offset="$2"
+ local ino=$(stat -c "%i" "$file")
+ local file_extent_key="($ino EXTENT_DATA $offset)"
+
+ _require_btrfs_command inspect-internal dump-tree
+
+ # The tree dump command below works on committed roots, by reading from
+ # a device directly, so we have to sync the filesystem to commit any
+ # open transaction.
+ $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
+
+ $BTRFS_UTIL_PROG inspect-internal dump-tree -t 5 $SCRATCH_DEV | \
+ grep -A4 "$file_extent_key" | grep "disk byte" | \
+ $AWK_PROG '{ print $5 }'
+}
diff --git a/tests/btrfs/299 b/tests/btrfs/299
index 8ed23ac5..2ac05957 100755
--- a/tests/btrfs/299
+++ b/tests/btrfs/299
@@ -22,25 +22,10 @@ _begin_fstest auto quick preallocrw
_supported_fs btrfs
_require_scratch
_require_xfs_io_command "falloc" "-k"
-_require_btrfs_command inspect-internal dump-tree
_require_btrfs_command inspect-internal logical-resolve
_fixed_by_kernel_commit 560840afc3e6 \
"btrfs: fix resolving backrefs for inline extent followed by prealloc"
-dump_tree() {
- $BTRFS_UTIL_PROG inspect-internal dump-tree $SCRATCH_DEV
-}
-
-get_extent_data() {
- local ino=$1
- dump_tree $SCRATCH_DEV | grep -A4 "($ino EXTENT_DATA "
-}
-
-get_prealloc_offset() {
- local ino=$1
- get_extent_data $ino | grep "disk byte" | $AWK_PROG '{print $5}'
-}
-
# This test needs to create conditions s.t. the special inode's inline extent
# is the first item in a leaf. Therefore, fix a leaf size and add
# items that are otherwise not necessary to reproduce the inline-prealloc
@@ -81,8 +66,7 @@ done
# grab the prealloc offset from dump tree while it's still the only
# extent data item for the inode
-ino=$(stat -c '%i' $f.evil)
-logical=$(get_prealloc_offset $ino)
+logical=$(_btrfs_get_file_extent_item_bytenr $f.evil 0)
# do the "small write; fsync; small write" pattern which reproduces the desired
# item pattern of an inline extent followed by a preallocated extent. The 23
--
2.35.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 2/3] btrfs: add a test case for the logical to ino ioctl
2023-05-09 11:52 [PATCH 0/3] btrfs: add test case for logical-resolve fdmanana
2023-05-09 11:52 ` [PATCH 1/3] common/btrfs: add helper to get the bytenr for a file extent item fdmanana
@ 2023-05-09 11:52 ` fdmanana
2023-05-10 5:49 ` Anand Jain
2023-05-09 11:52 ` [PATCH 3/3] groups: add logical_resolve group for btrfs fdmanana
2 siblings, 1 reply; 7+ messages in thread
From: fdmanana @ 2023-05-09 11:52 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana
From: Filipe Manana <fdmanana@suse.com>
Add a test case to exercise the logical to ino ioctl, including the v2
version (which adds the ignore offset option). This is motivated by the
fact that we have no test cases giving full coverage to that ioctl, only
two test cases partially exercise it (btrfs/004 and btrfs/299) and
absolutely no coverage for the v2 ioctl. This resulted in a recent
regression fixed by the patch mentioned in the _fixed_by_kernel_commit
tag of the introduced test case.
Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
common/btrfs | 29 +++++++++
tests/btrfs/287 | 154 ++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/287.out | 95 +++++++++++++++++++++++++++
3 files changed, 278 insertions(+)
create mode 100755 tests/btrfs/287
create mode 100644 tests/btrfs/287.out
diff --git a/common/btrfs b/common/btrfs
index 42777df2..bd4dc31f 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -647,3 +647,32 @@ _btrfs_get_file_extent_item_bytenr()
grep -A4 "$file_extent_key" | grep "disk byte" | \
$AWK_PROG '{ print $5 }'
}
+
+# Check that btrfs-progs has support for the logical-resolve command, with the
+# -o option, and that the kernel supports the logical to ino ioctl v2 (which
+# adds the ignore offset parameter).
+_require_btrfs_scratch_logical_resolve_v2()
+{
+ local bytenr
+
+ # First check if we have support for calling the v2 logical resolve
+ # ioctl in btrfs-progs. Check if the -o options exists, which makes
+ # btrfs-progs call v2 of the ioctl (because the flag
+ # BTRFS_LOGICAL_INO_ARGS_IGNORE_OFFSET is only supported in v2).
+ _require_btrfs_command inspect-internal logical-resolve -o
+ _require_scratch
+
+ _scratch_mkfs > /dev/null || \
+ _fail "_require_btrfs_scratch_logical_resolve_v2: mkfs failed"
+ _scratch_mount
+
+ $XFS_IO_PROG -f -c "pwrite -q 0 128K" "$SCRATCH_MNT/file1"
+ bytenr=$(_btrfs_get_file_extent_item_bytenr "$SCRATCH_MNT/file1" 0)
+ $BTRFS_UTIL_PROG inspect-internal logical-resolve -o $bytenr \
+ $SCRATCH_MNT > /dev/null
+ if [ $? -ne 0 ]; then
+ _scratch_unmount
+ _notrun "Logical resolve ioctl v2 not supported in the kernel"
+ fi
+ _scratch_unmount
+}
diff --git a/tests/btrfs/287 b/tests/btrfs/287
new file mode 100755
index 00000000..a7e29e2b
--- /dev/null
+++ b/tests/btrfs/287
@@ -0,0 +1,154 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 287
+#
+# Test btrfs' logical to inode ioctls (v1 and v2).
+#
+. ./common/preamble
+_begin_fstest auto quick snapshot clone punch
+
+. ./common/filter
+. ./common/reflink
+
+_supported_fs btrfs
+_require_btrfs_scratch_logical_resolve_v2
+_require_scratch_reflink
+_require_xfs_io_command "fpunch"
+
+# This is a test case to test the logical to ino ioctl in general but it also
+# serves as a regression a test for an issue fixed by the following commit.
+_fixed_by_kernel_commit XXXXXXXXXXXX \
+ "btrfs: fix backref walking not returning all inode refs"
+
+query_logical_ino()
+{
+ $BTRFS_UTIL_PROG inspect-internal logical-resolve -P $* $SCRATCH_MNT
+}
+
+_scratch_mkfs >> $seqres.full || _fail "mkfs failed"
+_scratch_mount
+
+# Create a file with two extents:
+#
+# 1) One 4M extent covering the file range [0, 4M)
+# 2) Another 4M extent covering the file range [4M, 8M)
+$XFS_IO_PROG -f -c "pwrite -S 0xab -b 4M 0 4M" \
+ -c "fsync" \
+ -c "pwrite -S 0xcd -b 4M 4M 8M" \
+ -c "fsync" $SCRATCH_MNT/foo | _filter_xfs_io
+
+echo "resolve first extent:"
+first_extent_bytenr=$(_btrfs_get_file_extent_item_bytenr "$SCRATCH_MNT/foo" 0)
+query_logical_ino $first_extent_bytenr
+
+echo "resolve second extent:"
+sz_4m=$((4 * 1024 * 1024))
+second_extent_bytenr=$(_btrfs_get_file_extent_item_bytenr "$SCRATCH_MNT/foo" $sz_4m)
+query_logical_ino $second_extent_bytenr
+
+# Now clone both extents twice to the end of the file.
+sz_8m=$((8 * 1024 * 1024))
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $sz_8m $sz_8m" $SCRATCH_MNT/foo \
+ | _filter_xfs_io
+sz_16m=$((16 * 1024 * 1024))
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $sz_16m $sz_8m" $SCRATCH_MNT/foo \
+ | _filter_xfs_io
+
+# Now lets resolve the extents again. They should now be listed 3 times each, at
+# the right file offsets.
+echo "resolve first extent:"
+query_logical_ino $first_extent_bytenr
+
+echo "resolve second extent:"
+query_logical_ino $second_extent_bytenr
+
+# Now lets punch a 2M hole at file offset 0. This changes the first file extent
+# item to point to the first extent with an offset of 2M and a length of 2M, so
+# doing a logical resolve with the bytenr of the first extent should not return
+# file offset 0.
+$XFS_IO_PROG -c "fpunch 0 2M" $SCRATCH_MNT/foo
+echo "resolve first extent after punching hole at file range [0, 2M):"
+query_logical_ino $first_extent_bytenr
+
+# Doing a logical resolve call with the BTRFS_LOGICAL_INO_ARGS_IGNORE_OFFSET
+# flag (passing -o to logical-resolve command) should ignore file extent offsets
+# and return file offsets for all file extent items that point to any section of
+# the extent (3 of them, file offsets 2M, 8M and 16M).
+echo "resolve first extent with ignore offset option:"
+query_logical_ino -o $first_extent_bytenr
+
+# Now query for file extent items containing the first extent at offset +1M.
+# Should only return the file offsets 9M and 17M.
+bytenr=$(( $first_extent_bytenr + 1024 * 1024))
+echo "resolve first extent +1M offset:"
+query_logical_ino $bytenr
+
+# Now do the same query again but with the ignore offset ioctl argument. This
+# should returns 3 results, for file offsets 2M, 8M and 16M.
+echo "resolve first extent +1M offset with ignore offset option:"
+query_logical_ino -o $bytenr
+
+# Now query for file extent items containing the first extent at offset +3M.
+# Should return the file offsets 3M and 11M and 19M.
+bytenr=$(( $first_extent_bytenr + 3 * 1024 * 1024))
+echo "resolve first extent +3M offset:"
+query_logical_ino $bytenr
+
+# Now do the same query again but with the ignore offset ioctl argument. This
+# should returns 3 results, for file offsets 2M, 8M and 16M.
+echo "resolve first extent +3M offset with ignore offset option:"
+query_logical_ino -o $bytenr
+
+# Now create two snapshots and then do some queries.
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/snap1 \
+ | _filter_scratch
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/snap2 \
+ | _filter_scratch
+
+# Query for the first extent (at offset 0). Should give two entries for each
+# root - default subvolume and the 2 snapshots, for file offsets 8M and 16M.
+echo "resolve first extent:"
+query_logical_ino $first_extent_bytenr
+
+# Query for the first extent (at offset 0) with the ignore offset option.
+# Should give 3 entries for each root - default subvolume and the 2 snapshots,
+# for file offsets 2M, 8M and 16M.
+echo "resolve first extent with ignore offset option:"
+query_logical_ino -o $first_extent_bytenr
+
+# Now lets punch a 1M hole at file offset 4M. This changes the second file
+# extent item to point to the second extent with an offset of 1M and a length
+# of 3M, so doing a logical resolve with the bytenr of the second extent should
+# not return file offset 4M for root 5 (default subvolume), bit it should return
+# file offset 4M for the files in the snapshots. For all the roots, it should
+# return file offsets 12M and 20M.
+$XFS_IO_PROG -c "fpunch 4M 1M" $SCRATCH_MNT/foo
+echo "resolve second extent after punching hole at file range [4M, 5M):"
+query_logical_ino $second_extent_bytenr
+
+# Repeat the query but with the ignore offset option. We should get 3 entries
+# for each root. For the snapshot roots, we should get entries for file offsets
+# 4M, 12M and 20M, while for the default subvolume (root 5) we should get for
+# file offsets 5M, 12M and 20M.
+echo "resolve second extent with ignore offset option:"
+query_logical_ino -o $second_extent_bytenr
+
+# Now delete the first snapshot and repeat the last 2 queries.
+$BTRFS_UTIL_PROG subvolume delete -C $SCRATCH_MNT/snap1 | _filter_scratch
+
+# Query the second extent with an offset of 0, should return file offsets 12M
+# and 20M for the default subvolume (root 5) and file offsets 4M, 12M and 20M
+# for the second snapshot root.
+echo "resolve second extent:"
+query_logical_ino $second_extent_bytenr
+
+# Query the second extent with the ignore offset option, should return file
+# offsets 5M, 12M and 20M for the default subvolume (root 5) and file offsets
+# 4M, 12M and 20M for the second snapshot root.
+echo "resolve second extent with ignore offset option:"
+query_logical_ino -o $second_extent_bytenr
+
+status=0
+exit
diff --git a/tests/btrfs/287.out b/tests/btrfs/287.out
new file mode 100644
index 00000000..683f9875
--- /dev/null
+++ b/tests/btrfs/287.out
@@ -0,0 +1,95 @@
+QA output created by 287
+wrote 4194304/4194304 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8388608/8388608 bytes at offset 4194304
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+resolve first extent:
+inode 257 offset 0 root 5
+resolve second extent:
+inode 257 offset 4194304 root 5
+linked 8388608/8388608 bytes at offset 8388608
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+linked 8388608/8388608 bytes at offset 16777216
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+resolve first extent:
+inode 257 offset 16777216 root 5
+inode 257 offset 8388608 root 5
+inode 257 offset 0 root 5
+resolve second extent:
+inode 257 offset 20971520 root 5
+inode 257 offset 12582912 root 5
+inode 257 offset 4194304 root 5
+resolve first extent after punching hole at file range [0, 2M):
+inode 257 offset 16777216 root 5
+inode 257 offset 8388608 root 5
+resolve first extent with ignore offset option:
+inode 257 offset 16777216 root 5
+inode 257 offset 8388608 root 5
+inode 257 offset 2097152 root 5
+resolve first extent +1M offset:
+inode 257 offset 17825792 root 5
+inode 257 offset 9437184 root 5
+resolve first extent +1M offset with ignore offset option:
+inode 257 offset 16777216 root 5
+inode 257 offset 8388608 root 5
+inode 257 offset 2097152 root 5
+resolve first extent +3M offset:
+inode 257 offset 19922944 root 5
+inode 257 offset 11534336 root 5
+inode 257 offset 3145728 root 5
+resolve first extent +3M offset with ignore offset option:
+inode 257 offset 16777216 root 5
+inode 257 offset 8388608 root 5
+inode 257 offset 2097152 root 5
+Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/snap1'
+Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/snap2'
+resolve first extent:
+inode 257 offset 16777216 root 257
+inode 257 offset 8388608 root 257
+inode 257 offset 16777216 root 256
+inode 257 offset 8388608 root 256
+inode 257 offset 16777216 root 5
+inode 257 offset 8388608 root 5
+resolve first extent with ignore offset option:
+inode 257 offset 16777216 root 257
+inode 257 offset 8388608 root 257
+inode 257 offset 2097152 root 257
+inode 257 offset 16777216 root 256
+inode 257 offset 8388608 root 256
+inode 257 offset 2097152 root 256
+inode 257 offset 16777216 root 5
+inode 257 offset 8388608 root 5
+inode 257 offset 2097152 root 5
+resolve second extent after punching hole at file range [4M, 5M):
+inode 257 offset 20971520 root 257
+inode 257 offset 12582912 root 257
+inode 257 offset 4194304 root 257
+inode 257 offset 20971520 root 256
+inode 257 offset 12582912 root 256
+inode 257 offset 4194304 root 256
+inode 257 offset 20971520 root 5
+inode 257 offset 12582912 root 5
+resolve second extent with ignore offset option:
+inode 257 offset 20971520 root 257
+inode 257 offset 12582912 root 257
+inode 257 offset 4194304 root 257
+inode 257 offset 20971520 root 256
+inode 257 offset 12582912 root 256
+inode 257 offset 4194304 root 256
+inode 257 offset 20971520 root 5
+inode 257 offset 12582912 root 5
+inode 257 offset 5242880 root 5
+Delete subvolume (commit): 'SCRATCH_MNT/snap1'
+resolve second extent:
+inode 257 offset 20971520 root 257
+inode 257 offset 12582912 root 257
+inode 257 offset 4194304 root 257
+inode 257 offset 20971520 root 5
+inode 257 offset 12582912 root 5
+resolve second extent with ignore offset option:
+inode 257 offset 20971520 root 257
+inode 257 offset 12582912 root 257
+inode 257 offset 4194304 root 257
+inode 257 offset 20971520 root 5
+inode 257 offset 12582912 root 5
+inode 257 offset 5242880 root 5
--
2.35.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 3/3] groups: add logical_resolve group for btrfs
2023-05-09 11:52 [PATCH 0/3] btrfs: add test case for logical-resolve fdmanana
2023-05-09 11:52 ` [PATCH 1/3] common/btrfs: add helper to get the bytenr for a file extent item fdmanana
2023-05-09 11:52 ` [PATCH 2/3] btrfs: add a test case for the logical to ino ioctl fdmanana
@ 2023-05-09 11:52 ` fdmanana
2023-05-10 4:51 ` Anand Jain
2 siblings, 1 reply; 7+ messages in thread
From: fdmanana @ 2023-05-09 11:52 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana
From: Filipe Manana <fdmanana@suse.com>
Add a 'logical_resolve' group to identify tests that use the btrfs
logical-resolve command, which exercises btrfs' logical to ino ioctl.
Signed-off-by: Filipe Manana <fdmanana@suse.com>
---
doc/group-names.txt | 1 +
tests/btrfs/004 | 2 +-
tests/btrfs/287 | 2 +-
tests/btrfs/299 | 2 +-
4 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/group-names.txt b/doc/group-names.txt
index 5fd8fe67..1c35a394 100644
--- a/doc/group-names.txt
+++ b/doc/group-names.txt
@@ -70,6 +70,7 @@ label filesystem labelling
limit resource limits
locks file locking
log metadata logging
+logical_resolve btrfs logical to ino ioctl (logical-resolve command)
logprint xfs_logprint functional tests
long_rw long-soak read write IO path exercisers
metacopy overlayfs metadata-only copy-up
diff --git a/tests/btrfs/004 b/tests/btrfs/004
index aa37c45d..ea40dbf6 100755
--- a/tests/btrfs/004
+++ b/tests/btrfs/004
@@ -10,7 +10,7 @@
# We check to end up back at the original file with the correct offset.
#
. ./common/preamble
-_begin_fstest auto rw metadata fiemap
+_begin_fstest auto rw metadata fiemap logical_resolve
noise_pid=0
diff --git a/tests/btrfs/287 b/tests/btrfs/287
index a7e29e2b..a0b71c7e 100755
--- a/tests/btrfs/287
+++ b/tests/btrfs/287
@@ -7,7 +7,7 @@
# Test btrfs' logical to inode ioctls (v1 and v2).
#
. ./common/preamble
-_begin_fstest auto quick snapshot clone punch
+_begin_fstest auto quick snapshot clone punch logical_resolve
. ./common/filter
. ./common/reflink
diff --git a/tests/btrfs/299 b/tests/btrfs/299
index 2ac05957..c4b1c7c5 100755
--- a/tests/btrfs/299
+++ b/tests/btrfs/299
@@ -15,7 +15,7 @@
# resolution is still successful.
#
. ./common/preamble
-_begin_fstest auto quick preallocrw
+_begin_fstest auto quick preallocrw logical_resolve
# real QA test starts here
# Modify as appropriate.
--
2.35.1
^ permalink raw reply related [flat|nested] 7+ messages in thread