[PATCH 0/2] btrfs-progs: tune: fix the leftover csum change item and add a test case for it

[PATCH 0/2] btrfs-progs: tune: fix the leftover csum change item and add a test case for it

[Qu Wenruo]

David's cyclic csum change tests exposed a bug even with the latest csum
change code, that I'm a total idiot and forgot to delete the csum change
item.

This results a bug that if we have run multiple csum changes to the same
target csum type (e.g. CRC32C->SHA256->CRC32C->SHA256), the second
conversion to the same type would fail due to the left over csum change items.

This series would do the fix and add a test case to cover this bug.

Qu Wenruo (2):
  btrfs-progs: tune: delete the csum change item after converting the fs
  btrfs-progs: tests/misc: add a test case for csum change

 .../058-btrfstune-csum-change/test.sh         | 26 ++++++++++++
 tune/change-csum.c                            | 42 +++++++++++++++++--
 2 files changed, 64 insertions(+), 4 deletions(-)
 create mode 100755 tests/misc-tests/058-btrfstune-csum-change/test.sh

-- 
2.40.1

[PATCH 1/2] btrfs-progs: tune: delete the csum change item after converting the fs

[Qu Wenruo]

[BUG]
Doing the following csum change in a row, it would fail:

 # mkfs.btrfs -f --csum crc32c $dev
 # btrfstune --csum sha256 $dev
 # btrfstune --csum crc32c $dev
 # btrfstune --csum sha256 $dev
 WARNING: Experimental build with unstable or unfinished features
 WARNING: Switching checksums is experimental, do not use for valuable data!

 Proceed to switch checksums
 ERROR: failed to insert csum change item: File exists
 ERROR: failed to generate new data csums: File exists
 WARNING: reserved space leaked, flag=0x4 bytes_reserved=16384
 extent buffer leak: start 30572544 len 16384
 extent buffer leak: start 30441472 len 16384
 WARNING: dirty eb leak (aborted trans): start 30441472 len 16384

[CAUSE]
During every csum change operation, btrfstune would insert an temporaray
csum change item into root tree.

But unfortunately after the conversion btrfstune doesn't properly delete
the csum change item, result the following items in the root tree:

	item 10 key (CSUM_CHANGE TEMPORARY_ITEM 0) itemoff 13423 itemsize 0
		temporary item objectid CSUM_CHANGE offset 0
		target csum type crc32c (0)
	item 11 key (CSUM_CHANGE TEMPORARY_ITEM 2) itemoff 13423 itemsize 0
		temporary item objectid CSUM_CHANGE offset 2
		target csum type sha256 (2)

Thus at the last conversion try to go back to SHA256, we failed to
insert the same item, and caused the above error.

[FIX]
After finishing the metadata csum conversion, do a proper removal of the
csum item.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tune/change-csum.c | 42 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 38 insertions(+), 4 deletions(-)

diff --git a/tune/change-csum.c b/tune/change-csum.c
index c8809300a143..b5efc3a8807f 100644
--- a/tune/change-csum.c
+++ b/tune/change-csum.c
@@ -583,10 +583,13 @@ out:
 	btrfs_release_path(&path);
 
 	/*
-	 * Finish the change by clearing the csum change flag and update the superblock
-	 * csum type.
+	 * Finish the change by clearing the csum change flag, update the superblock
+	 * csum type, and delete the csum change item in the fs with new csum type.
 	 */
 	if (ret == 0) {
+		struct btrfs_root *tree_root = fs_info->tree_root;
+		struct btrfs_trans_handle *trans;
+
 		u64 super_flags = btrfs_super_flags(fs_info->super_copy);
 
 		btrfs_set_super_csum_type(fs_info->super_copy, new_csum_type);
@@ -596,11 +599,42 @@ out:
 
 		fs_info->csum_type = new_csum_type;
 		fs_info->csum_size = btrfs_csum_type_size(new_csum_type);
+		fs_info->skip_csum_check = 0;
 
-		ret = write_all_supers(fs_info);
+		trans = btrfs_start_transaction(tree_root, 1);
+		if (IS_ERR(trans)) {
+			ret = PTR_ERR(trans);
+			errno = -ret;
+			error("failed to start new transaction with new csum type: %m");
+			return ret;
+		}
+		key.objectid = BTRFS_CSUM_CHANGE_OBJECTID;
+		key.type = BTRFS_TEMPORARY_ITEM_KEY;
+		key.offset = new_csum_type;
+
+		ret = btrfs_search_slot(trans, tree_root, &key, &path, -1, 1);
+		if (ret > 0)
+			ret = -ENOENT;
 		if (ret < 0) {
 			errno = -ret;
-			error("failed to write super blocks: %m");
+			error("failed to locate the csum change item: %m");
+			btrfs_release_path(&path);
+			btrfs_abort_transaction(trans, ret);
+			return ret;
+		}
+		ret = btrfs_del_item(trans, tree_root, &path);
+		if (ret < 0) {
+			errno = -ret;
+			error("failed to delete the csum change item: %m");
+			btrfs_release_path(&path);
+			btrfs_abort_transaction(trans, ret);
+			return ret;
+		}
+		btrfs_release_path(&path);
+		ret = btrfs_commit_transaction(trans, tree_root);
+		if (ret < 0) {
+			errno = -ret;
+			error("failed to finalize the csum change: %m");
 		}
 	}
 	return ret;
-- 
2.40.1

[PATCH 2/2] btrfs-progs: tests/misc: add a test case for csum change

[Qu Wenruo]

The test case would:
- Populate the fs
- Run csum change for all the target csum type
- Run above csum change again

This should more or less cover the common paths, and detect any
conflicting csum change item.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 .../058-btrfstune-csum-change/test.sh         | 26 +++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100755 tests/misc-tests/058-btrfstune-csum-change/test.sh

diff --git a/tests/misc-tests/058-btrfstune-csum-change/test.sh b/tests/misc-tests/058-btrfstune-csum-change/test.sh
new file mode 100755
index 000000000000..925e71b86285
--- /dev/null
+++ b/tests/misc-tests/058-btrfstune-csum-change/test.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+# Test btrfstune --csum option
+
+source "$TEST_TOP/common" || exit
+source "$TEST_TOP/common.convert" || exit
+
+check_prereq mkfs.btrfs
+check_prereq btrfstune
+check_prereq btrfs
+
+setup_root_helper
+prepare_test_dev
+
+run_check_mkfs_test_dev --csum crc32c
+run_check_mount_test_dev
+populate_fs
+run_check_umount_test_dev
+
+for hash in "blake2" "crc32c" "xxhash" "sha256"; do
+	run_check $SUDO_HELPER "$TOP/btrfstune" --csum $hash "$TEST_DEV"
+	run_check $SUDO_HELPER "$TOP/btrfs" check --check-data-csum "$TEST_DEV"
+done
+for hash in "blake2" "crc32c" "xxhash" "sha256"; do
+	run_check $SUDO_HELPER "$TOP/btrfstune" --csum $hash "$TEST_DEV"
+	run_check $SUDO_HELPER "$TOP/btrfs" check --check-data-csum "$TEST_DEV"
+done
-- 
2.40.1

Re: [PATCH 0/2] btrfs-progs: tune: fix the leftover csum change item and add a test case for it

[David Sterba]

On Tue, May 23, 2023 at 08:37:11AM +0800, Qu Wenruo wrote:
> David's cyclic csum change tests exposed a bug even with the latest csum
> change code, that I'm a total idiot and forgot to delete the csum change
> item.
> 
> This results a bug that if we have run multiple csum changes to the same
> target csum type (e.g. CRC32C->SHA256->CRC32C->SHA256), the second
> conversion to the same type would fail due to the left over csum change items.
> 
> This series would do the fix and add a test case to cover this bug.
> 
> Qu Wenruo (2):
>   btrfs-progs: tune: delete the csum change item after converting the fs
>   btrfs-progs: tests/misc: add a test case for csum change

Added to devel thanks, no more errors reported.

Re: [PATCH 0/2] btrfs-progs: tune: fix the leftover csum change item and add a test case for it

[Qu Wenruo]

On 2023/5/23 18:46, David Sterba wrote:
> On Tue, May 23, 2023 at 08:37:11AM +0800, Qu Wenruo wrote:
>> David's cyclic csum change tests exposed a bug even with the latest csum
>> change code, that I'm a total idiot and forgot to delete the csum change
>> item.
>>
>> This results a bug that if we have run multiple csum changes to the same
>> target csum type (e.g. CRC32C->SHA256->CRC32C->SHA256), the second
>> conversion to the same type would fail due to the left over csum change items.
>>
>> This series would do the fix and add a test case to cover this bug.
>>
>> Qu Wenruo (2):
>>    btrfs-progs: tune: delete the csum change item after converting the fs
>>    btrfs-progs: tests/misc: add a test case for csum change
>
> Added to devel thanks, no more errors reported.

Finally I can work on the resume part (which is not much compared to the
main workflow).

Thanks,
Qu