[PATCH 0/2] btrfs: add a delayed ref self test

[PATCH 0/2] btrfs: add a delayed ref self test

[Josef Bacik]

Hello,

I made a silly mistake when refactoring the delayed ref code to make it easier
to understand, and that resulted in a whole lot of not fun trying to find what
went wrong with boxes started falling over when we were deploying 6.11.  This
style of bug is easy to catch with basic unit testing, so add a variety of unit
tests for delayed refs to make sure I don't break things again.  One patch moves
an important helper and exports it so that we can do the testing, the other
patch is a giant patch of all the tests with a few changes that are necessary to
make everything work.  I validated these work properly and catch a variety of
bugs that I hand introduced to the delayed ref code.  Thanks,

Josef Bacik (2):
  btrfs: move select_delayed_ref and export it
  btrfs: add delayed ref self tests

 fs/btrfs/Makefile                   |    2 +-
 fs/btrfs/delayed-ref.c              |   41 +-
 fs/btrfs/delayed-ref.h              |    2 +
 fs/btrfs/extent-tree.c              |   26 +-
 fs/btrfs/tests/btrfs-tests.c        |   18 +
 fs/btrfs/tests/btrfs-tests.h        |    6 +
 fs/btrfs/tests/delayed-refs-tests.c | 1012 +++++++++++++++++++++++++++
 7 files changed, 1078 insertions(+), 29 deletions(-)
 create mode 100644 fs/btrfs/tests/delayed-refs-tests.c

-- 
2.43.0

[PATCH 1/2] btrfs: move select_delayed_ref and export it

[Josef Bacik]

This helper is how we select the delayed ref to run once we've selected
the delayed ref head.  I need this exported to add a unit test for
delayed refs, and it's more natural home is in delayed-ref.c.  Rename it
to btrfs_select_delayed_ref and move it into delayed-ref.c.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 fs/btrfs/delayed-ref.c | 27 +++++++++++++++++++++++++++
 fs/btrfs/delayed-ref.h |  2 ++
 fs/btrfs/extent-tree.c | 26 +-------------------------
 3 files changed, 30 insertions(+), 25 deletions(-)

diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
index 0d878dbbabba..63e0a7f660da 100644
--- a/fs/btrfs/delayed-ref.c
+++ b/fs/btrfs/delayed-ref.c
@@ -555,6 +555,33 @@ void btrfs_delete_ref_head(const struct btrfs_fs_info *fs_info,
 		delayed_refs->num_heads_ready--;
 }
 
+struct btrfs_delayed_ref_node *btrfs_select_delayed_ref(
+		struct btrfs_delayed_ref_head *head)
+{
+	struct btrfs_delayed_ref_node *ref;
+
+	lockdep_assert_held(&head->mutex);
+	lockdep_assert_held(&head->lock);
+
+	if (RB_EMPTY_ROOT(&head->ref_tree.rb_root))
+		return NULL;
+
+	/*
+	 * Select a delayed ref of type BTRFS_ADD_DELAYED_REF first.
+	 * This is to prevent a ref count from going down to zero, which deletes
+	 * the extent item from the extent tree, when there still are references
+	 * to add, which would fail because they would not find the extent item.
+	 */
+	if (!list_empty(&head->ref_add_list))
+		return list_first_entry(&head->ref_add_list,
+				struct btrfs_delayed_ref_node, add_list);
+
+	ref = rb_entry(rb_first_cached(&head->ref_tree),
+		       struct btrfs_delayed_ref_node, ref_node);
+	ASSERT(list_empty(&ref->add_list));
+	return ref;
+}
+
 /*
  * Helper to insert the ref_node to the tail or merge with tail.
  *
diff --git a/fs/btrfs/delayed-ref.h b/fs/btrfs/delayed-ref.h
index 611fb3388f82..ca9111a5779d 100644
--- a/fs/btrfs/delayed-ref.h
+++ b/fs/btrfs/delayed-ref.h
@@ -402,6 +402,8 @@ struct btrfs_delayed_ref_head *btrfs_select_ref_head(
 		struct btrfs_delayed_ref_root *delayed_refs);
 void btrfs_unselect_ref_head(struct btrfs_delayed_ref_root *delayed_refs,
 			     struct btrfs_delayed_ref_head *head);
+struct btrfs_delayed_ref_node *btrfs_select_delayed_ref(
+		struct btrfs_delayed_ref_head *head);
 
 int btrfs_check_delayed_seq(struct btrfs_fs_info *fs_info, u64 seq);
 
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 43a771f7bd7a..f765c74c14d9 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -1803,30 +1803,6 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
-static inline struct btrfs_delayed_ref_node *
-select_delayed_ref(struct btrfs_delayed_ref_head *head)
-{
-	struct btrfs_delayed_ref_node *ref;
-
-	if (RB_EMPTY_ROOT(&head->ref_tree.rb_root))
-		return NULL;
-
-	/*
-	 * Select a delayed ref of type BTRFS_ADD_DELAYED_REF first.
-	 * This is to prevent a ref count from going down to zero, which deletes
-	 * the extent item from the extent tree, when there still are references
-	 * to add, which would fail because they would not find the extent item.
-	 */
-	if (!list_empty(&head->ref_add_list))
-		return list_first_entry(&head->ref_add_list,
-				struct btrfs_delayed_ref_node, add_list);
-
-	ref = rb_entry(rb_first_cached(&head->ref_tree),
-		       struct btrfs_delayed_ref_node, ref_node);
-	ASSERT(list_empty(&ref->add_list));
-	return ref;
-}
-
 static struct btrfs_delayed_extent_op *cleanup_extent_op(
 				struct btrfs_delayed_ref_head *head)
 {
@@ -1959,7 +1935,7 @@ static int btrfs_run_delayed_refs_for_head(struct btrfs_trans_handle *trans,
 	lockdep_assert_held(&locked_ref->mutex);
 	lockdep_assert_held(&locked_ref->lock);
 
-	while ((ref = select_delayed_ref(locked_ref))) {
+	while ((ref = btrfs_select_delayed_ref(locked_ref))) {
 		if (ref->seq &&
 		    btrfs_check_delayed_seq(fs_info, ref->seq)) {
 			spin_unlock(&locked_ref->lock);
-- 
2.43.0

[PATCH 2/2] btrfs: add delayed ref self tests

[Josef Bacik]

The recent fix for a stupid mistake I made uncovered the fact that we
don't have adequate testing in the delayed refs code, as it took a
pretty extensive and long running stress test to uncover something that
a unit test would have uncovered right away.

Fix this by adding a delayed refs self test suite.  This will validate
that the btrfs_ref transformation does the correct thing, that we do the
correct thing when merging delayed refs, and that we get the delayed
refs in the order that we expect.  These are all crucial to how the
delayed refs operate.

I introduced various bugs (including the original bug) into the delayed
refs code to validate that these tests caught all of the shenanigans
that I could think of.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 fs/btrfs/Makefile                   |    2 +-
 fs/btrfs/delayed-ref.c              |   14 +-
 fs/btrfs/tests/btrfs-tests.c        |   18 +
 fs/btrfs/tests/btrfs-tests.h        |    6 +
 fs/btrfs/tests/delayed-refs-tests.c | 1012 +++++++++++++++++++++++++++
 5 files changed, 1048 insertions(+), 4 deletions(-)
 create mode 100644 fs/btrfs/tests/delayed-refs-tests.c

diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 3cfc440c636c..2d5f0482678b 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -44,4 +44,4 @@ btrfs-$(CONFIG_BTRFS_FS_RUN_SANITY_TESTS) += tests/free-space-tests.o \
 	tests/extent-buffer-tests.o tests/btrfs-tests.o \
 	tests/extent-io-tests.o tests/inode-tests.o tests/qgroup-tests.o \
 	tests/free-space-tree-tests.o tests/extent-map-tests.o \
-	tests/raid-stripe-tree-tests.o
+	tests/raid-stripe-tree-tests.o tests/delayed-refs-tests.o
diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
index 63e0a7f660da..3ec0468d1d94 100644
--- a/fs/btrfs/delayed-ref.c
+++ b/fs/btrfs/delayed-ref.c
@@ -93,6 +93,9 @@ void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans)
 	u64 num_bytes;
 	u64 reserved_bytes;
 
+	if (btrfs_is_testing(fs_info))
+		return;
+
 	num_bytes = btrfs_calc_delayed_ref_bytes(fs_info, trans->delayed_ref_updates);
 	num_bytes += btrfs_calc_delayed_ref_csum_bytes(fs_info,
 						       trans->delayed_ref_csum_deletions);
@@ -1261,6 +1264,7 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
 {
 	struct btrfs_delayed_ref_root *delayed_refs = &trans->delayed_refs;
 	struct btrfs_fs_info *fs_info = trans->fs_info;
+	bool testing = btrfs_is_testing(fs_info);
 
 	spin_lock(&delayed_refs->lock);
 	while (true) {
@@ -1290,7 +1294,7 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
 		spin_unlock(&delayed_refs->lock);
 		mutex_unlock(&head->mutex);
 
-		if (pin_bytes) {
+		if (!testing && pin_bytes) {
 			struct btrfs_block_group *bg;
 
 			bg = btrfs_lookup_block_group(fs_info, head->bytenr);
@@ -1322,12 +1326,16 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
 			btrfs_error_unpin_extent_range(fs_info, head->bytenr,
 				head->bytenr + head->num_bytes - 1);
 		}
-		btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs, head);
+		if (!testing)
+			btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs,
+							  head);
 		btrfs_put_delayed_ref_head(head);
 		cond_resched();
 		spin_lock(&delayed_refs->lock);
 	}
-	btrfs_qgroup_destroy_extent_records(trans);
+
+	if (!testing)
+		btrfs_qgroup_destroy_extent_records(trans);
 
 	spin_unlock(&delayed_refs->lock);
 }
diff --git a/fs/btrfs/tests/btrfs-tests.c b/fs/btrfs/tests/btrfs-tests.c
index e607b5d52fb1..a8549c5685ad 100644
--- a/fs/btrfs/tests/btrfs-tests.c
+++ b/fs/btrfs/tests/btrfs-tests.c
@@ -142,6 +142,11 @@ struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize)
 	fs_info->nodesize = nodesize;
 	fs_info->sectorsize = sectorsize;
 	fs_info->sectorsize_bits = ilog2(sectorsize);
+
+	/* CRC32C csum size. */
+	fs_info->csum_size = 4;
+	fs_info->csums_per_leaf = BTRFS_MAX_ITEM_SIZE(fs_info) /
+		fs_info->csum_size;
 	set_bit(BTRFS_FS_STATE_DUMMY_FS_INFO, &fs_info->fs_state);
 
 	test_mnt->mnt_sb->s_fs_info = fs_info;
@@ -247,6 +252,16 @@ void btrfs_free_dummy_block_group(struct btrfs_block_group *cache)
 	kfree(cache);
 }
 
+void btrfs_init_dummy_transaction(struct btrfs_transaction *trans,
+				  struct btrfs_fs_info *fs_info)
+{
+	memset(trans, 0, sizeof(*trans));
+	trans->fs_info = fs_info;
+	xa_init(&trans->delayed_refs.head_refs);
+	xa_init(&trans->delayed_refs.dirty_extents);
+	spin_lock_init(&trans->delayed_refs.lock);
+}
+
 void btrfs_init_dummy_trans(struct btrfs_trans_handle *trans,
 			    struct btrfs_fs_info *fs_info)
 {
@@ -295,6 +310,9 @@ int btrfs_run_sanity_tests(void)
 			ret = btrfs_test_raid_stripe_tree(sectorsize, nodesize);
 			if (ret)
 				goto out;
+			ret = btrfs_test_delayed_refs(sectorsize, nodesize);
+			if (ret)
+				goto out;
 		}
 	}
 	ret = btrfs_test_extent_map();
diff --git a/fs/btrfs/tests/btrfs-tests.h b/fs/btrfs/tests/btrfs-tests.h
index b524ecf2f452..713f1b26222b 100644
--- a/fs/btrfs/tests/btrfs-tests.h
+++ b/fs/btrfs/tests/btrfs-tests.h
@@ -6,6 +6,8 @@
 #ifndef BTRFS_TESTS_H
 #define BTRFS_TESTS_H
 
+#include <linux/types.h>
+
 #ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
 int btrfs_run_sanity_tests(void);
 
@@ -31,6 +33,7 @@ extern const char *test_error[];
 
 struct btrfs_root;
 struct btrfs_trans_handle;
+struct btrfs_transaction;
 
 int btrfs_test_extent_buffer_operations(u32 sectorsize, u32 nodesize);
 int btrfs_test_free_space_cache(u32 sectorsize, u32 nodesize);
@@ -40,6 +43,7 @@ int btrfs_test_qgroups(u32 sectorsize, u32 nodesize);
 int btrfs_test_free_space_tree(u32 sectorsize, u32 nodesize);
 int btrfs_test_raid_stripe_tree(u32 sectorsize, u32 nodesize);
 int btrfs_test_extent_map(void);
+int btrfs_test_delayed_refs(u32 sectorsize, u32 nodesize);
 struct inode *btrfs_new_test_inode(void);
 struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize);
 void btrfs_free_dummy_fs_info(struct btrfs_fs_info *fs_info);
@@ -49,6 +53,8 @@ btrfs_alloc_dummy_block_group(struct btrfs_fs_info *fs_info, unsigned long lengt
 void btrfs_free_dummy_block_group(struct btrfs_block_group *cache);
 void btrfs_init_dummy_trans(struct btrfs_trans_handle *trans,
 			    struct btrfs_fs_info *fs_info);
+void btrfs_init_dummy_transaction(struct btrfs_transaction *trans,
+				  struct btrfs_fs_info *fs_info);
 struct btrfs_device *btrfs_alloc_dummy_device(struct btrfs_fs_info *fs_info);
 #else
 static inline int btrfs_run_sanity_tests(void)
diff --git a/fs/btrfs/tests/delayed-refs-tests.c b/fs/btrfs/tests/delayed-refs-tests.c
new file mode 100644
index 000000000000..9b469791c20a
--- /dev/null
+++ b/fs/btrfs/tests/delayed-refs-tests.c
@@ -0,0 +1,1012 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/sizes.h>
+#include "btrfs-tests.h"
+#include "../transaction.h"
+#include "../delayed-ref.h"
+#include "../extent-tree.h"
+
+#define FAKE_ROOT_OBJECTID 256
+#define FAKE_BYTENR 0
+#define FAKE_LEVEL 1
+#define FAKE_INO 256
+#define FAKE_FILE_OFFSET 0
+#define FAKE_PARENT SZ_1M
+
+struct ref_head_check {
+	u64 bytenr;
+	u64 num_bytes;
+	int ref_mod;
+	int total_ref_mod;
+	int must_insert;
+};
+
+struct ref_node_check {
+	u64 bytenr;
+	u64 num_bytes;
+	int ref_mod;
+	enum btrfs_delayed_ref_action action;
+	u8 type;
+	u64 parent;
+	u64 root;
+	u64 owner;
+	u64 offset;
+};
+
+static enum btrfs_ref_type ref_type_from_disk_ref_type(u8 type)
+{
+	if ((type == BTRFS_TREE_BLOCK_REF_KEY) ||
+	    (type == BTRFS_SHARED_BLOCK_REF_KEY))
+		return BTRFS_REF_METADATA;
+	return BTRFS_REF_DATA;
+}
+
+static void delete_delayed_ref_head(struct btrfs_trans_handle *trans,
+				    struct btrfs_delayed_ref_head *head)
+{
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct btrfs_delayed_ref_root *delayed_refs =
+		&trans->transaction->delayed_refs;
+
+	spin_lock(&delayed_refs->lock);
+	spin_lock(&head->lock);
+	btrfs_delete_ref_head(fs_info, delayed_refs, head);
+	spin_unlock(&head->lock);
+	spin_unlock(&delayed_refs->lock);
+
+	btrfs_delayed_ref_unlock(head);
+	btrfs_put_delayed_ref_head(head);
+}
+
+static void delete_delayed_ref_node(struct btrfs_delayed_ref_head *head,
+				    struct btrfs_delayed_ref_node *node)
+{
+	rb_erase_cached(&node->ref_node, &head->ref_tree);
+	RB_CLEAR_NODE(&node->ref_node);
+	if (!list_empty(&node->add_list))
+		list_del_init(&node->add_list);
+	btrfs_put_delayed_ref(node);
+}
+
+static int validate_ref_head(struct btrfs_delayed_ref_head *head,
+			     struct ref_head_check *check)
+{
+	if (head->bytenr != check->bytenr) {
+		test_err("invalid bytenr have: %llu want: %llu", head->bytenr,
+			 check->bytenr);
+		return -EINVAL;
+	}
+
+	if (head->num_bytes != check->num_bytes) {
+		test_err("invalid num_bytes have: %llu want: %llu",
+			 head->num_bytes, check->num_bytes);
+		return -EINVAL;
+	}
+
+	if (head->ref_mod != check->ref_mod) {
+		test_err("invalid ref_mod have: %d want: %d", head->ref_mod,
+			 check->ref_mod);
+		return -EINVAL;
+	}
+
+	if (head->total_ref_mod != check->total_ref_mod) {
+		test_err("invalid total_ref_mod have: %d want: %d",
+			 head->total_ref_mod, check->total_ref_mod);
+		return -EINVAL;
+	}
+
+	if (head->must_insert_reserved != check->must_insert) {
+		test_err("invalid must_insert have: %d want: %d",
+			 head->must_insert_reserved, check->must_insert);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int validate_ref_node(struct btrfs_delayed_ref_node *node,
+			     struct ref_node_check *check)
+{
+	if (node->bytenr != check->bytenr) {
+		test_err("invalid bytenr have: %llu want: %llu", node->bytenr,
+			 check->bytenr);
+		return -EINVAL;
+	}
+
+	if (node->num_bytes != check->num_bytes) {
+		test_err("invalid num_bytes have: %llu want: %llu",
+			 node->num_bytes, check->num_bytes);
+		return -EINVAL;
+	}
+
+	if (node->ref_mod != check->ref_mod) {
+		test_err("invalid ref_mod have: %d want: %d", node->ref_mod,
+			 check->ref_mod);
+		return -EINVAL;
+	}
+
+	if (node->action != check->action) {
+		test_err("invalid action have: %d want: %d", node->action,
+			 check->action);
+		return -EINVAL;
+	}
+
+	if (node->parent != check->parent) {
+		test_err("invalid parent have: %llu want: %llu", node->parent,
+			 check->parent);
+		return -EINVAL;
+	}
+
+	if (node->ref_root != check->root) {
+		test_err("invalid root have: %llu want: %llu", node->ref_root,
+			 check->root);
+		return -EINVAL;
+	}
+
+	if (node->type != check->type) {
+		test_err("invalid type have: %d want: %d", node->type,
+			 check->type);
+		return -EINVAL;
+	}
+
+	if (btrfs_delayed_ref_owner(node) != check->owner) {
+		test_err("invalid owner have: %llu want: %llu",
+			 btrfs_delayed_ref_owner(node), check->owner);
+		return -EINVAL;
+	}
+
+	if (btrfs_delayed_ref_offset(node) != check->offset) {
+		test_err("invalid offset have: %llu want: %llu",
+			 btrfs_delayed_ref_offset(node), check->offset);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int simple_test(struct btrfs_trans_handle *trans,
+		       struct ref_head_check *head_check,
+		       struct ref_node_check *node_check)
+{
+	struct btrfs_delayed_ref_root *delayed_refs =
+		&trans->transaction->delayed_refs;
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct btrfs_delayed_ref_head *head;
+	struct btrfs_delayed_ref_node *node;
+	struct btrfs_ref ref = {
+		.type = ref_type_from_disk_ref_type(node_check->type),
+		.action = node_check->action,
+		.parent = node_check->parent,
+		.ref_root = node_check->root,
+		.bytenr = node_check->bytenr,
+		.num_bytes = fs_info->nodesize,
+	};
+	int ret;
+
+	if (ref.type == BTRFS_REF_METADATA)
+		btrfs_init_tree_ref(&ref, node_check->owner, node_check->root,
+				    false);
+	else
+		btrfs_init_data_ref(&ref, node_check->owner, node_check->offset,
+				    node_check->root, true);
+
+	if (ref.type == BTRFS_REF_METADATA)
+		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	else
+		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		return ret;
+	}
+
+	head = btrfs_select_ref_head(fs_info, delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		return -EINVAL;
+	}
+
+	ret = -EINVAL;
+	if (validate_ref_head(head, head_check))
+		goto out;
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	if (validate_ref_node(node, node_check))
+		goto out;
+	ret = 0;
+out:
+	btrfs_unselect_ref_head(delayed_refs, head);
+	btrfs_destroy_delayed_refs(trans->transaction);
+	return ret;
+}
+
+/*
+ * These are simple tests, make sure that our btrfs_ref's get turned into the
+ * appropriate btrfs_delayed_ref_node based on their settings and action.
+ */
+static int simple_tests(struct btrfs_trans_handle *trans)
+{
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct ref_head_check head_check = {
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+		.ref_mod = 1,
+		.total_ref_mod = 1,
+	};
+	struct ref_node_check node_check = {
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+		.ref_mod = 1,
+		.action = BTRFS_ADD_DELAYED_REF,
+		.type = BTRFS_TREE_BLOCK_REF_KEY,
+		.parent = 0,
+		.root = FAKE_ROOT_OBJECTID,
+		.owner = FAKE_LEVEL,
+		.offset = 0,
+	};
+
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single add tree block failed");
+		return -EINVAL;
+	}
+
+	node_check.type = BTRFS_EXTENT_DATA_REF_KEY;
+	node_check.owner = FAKE_INO;
+	node_check.offset = FAKE_FILE_OFFSET;
+
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single add extent data failed");
+		return -EINVAL;
+	}
+
+	node_check.parent = FAKE_PARENT;
+	node_check.type = BTRFS_SHARED_BLOCK_REF_KEY;
+	node_check.owner = FAKE_LEVEL;
+	node_check.offset = 0;
+
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single add shared block failed");
+		return -EINVAL;
+	}
+
+	node_check.type = BTRFS_SHARED_DATA_REF_KEY;
+	node_check.owner = FAKE_INO;
+	node_check.offset = FAKE_FILE_OFFSET;
+
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single add shared data failed");
+		return -EINVAL;
+	}
+
+	head_check.ref_mod = -1;
+	head_check.total_ref_mod = -1;
+	node_check.action = BTRFS_DROP_DELAYED_REF;
+	node_check.type = BTRFS_TREE_BLOCK_REF_KEY;
+	node_check.owner = FAKE_LEVEL;
+	node_check.offset = 0;
+	node_check.parent = 0;
+
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single drop tree block failed");
+		return -EINVAL;
+	}
+
+	node_check.type = BTRFS_EXTENT_DATA_REF_KEY;
+	node_check.owner = FAKE_INO;
+	node_check.offset = FAKE_FILE_OFFSET;
+
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single drop extent data failed");
+		return -EINVAL;
+	}
+
+	node_check.parent = FAKE_PARENT;
+	node_check.type = BTRFS_SHARED_BLOCK_REF_KEY;
+	node_check.owner = FAKE_LEVEL;
+	node_check.offset = 0;
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single drop shared block failed");
+		return -EINVAL;
+	}
+
+	node_check.type = BTRFS_SHARED_DATA_REF_KEY;
+	node_check.owner = FAKE_INO;
+	node_check.offset = FAKE_FILE_OFFSET;
+	if (simple_test(trans, &head_check, &node_check)) {
+		test_err("single drop shared data failed");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/*
+ * Merge tests, validate that we do delayed ref merging properly, the ref counts
+ * all end up properly, and delayed refs are deleted once they're no longer
+ * needed.
+ */
+static int merge_tests(struct btrfs_trans_handle *trans,
+		       enum btrfs_ref_type type)
+{
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct btrfs_delayed_ref_head *head = NULL;
+	struct btrfs_delayed_ref_node *node;
+	struct btrfs_ref ref = {
+		.type = type,
+		.action = BTRFS_ADD_DELAYED_REF,
+		.parent = 0,
+		.ref_root = FAKE_ROOT_OBJECTID,
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+	};
+	struct ref_head_check head_check = {
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+		.ref_mod = 0,
+		.total_ref_mod = 0,
+	};
+	struct ref_node_check node_check = {
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+		.ref_mod = 2,
+		.action = BTRFS_ADD_DELAYED_REF,
+		.parent = 0,
+		.root = FAKE_ROOT_OBJECTID,
+	};
+	int ret;
+
+	/*
+	 * First add a ref and then drop it, make sure we get a head ref with a
+	 * 0 total ref mod and no nodes.
+	 */
+	if (type == BTRFS_REF_METADATA) {
+		node_check.type = BTRFS_TREE_BLOCK_REF_KEY;
+		node_check.owner = FAKE_LEVEL;
+		btrfs_init_tree_ref(&ref, FAKE_LEVEL, FAKE_ROOT_OBJECTID, false);
+	} else {
+		node_check.type = BTRFS_EXTENT_DATA_REF_KEY;
+		node_check.owner = FAKE_INO;
+		node_check.offset = FAKE_FILE_OFFSET;
+		btrfs_init_data_ref(&ref, FAKE_INO, FAKE_FILE_OFFSET,
+				    FAKE_ROOT_OBJECTID, true);
+	}
+
+	if (type == BTRFS_REF_METADATA)
+		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	else
+		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		return ret;
+	}
+
+	ref.action = BTRFS_DROP_DELAYED_REF;
+	if (type == BTRFS_REF_METADATA)
+		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	else
+		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		head = NULL;
+		goto out;
+	}
+
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("single add and drop failed");
+		goto out;
+	}
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (node) {
+		test_err("found node when none should exist");
+		goto out;
+	}
+
+	delete_delayed_ref_head(trans, head);
+	head = NULL;
+
+	/*
+	 * Add a ref, then add another ref, make sure we get a head ref with a
+	 * 2 total ref mod and 1 node.
+	 */
+	ref.action = BTRFS_ADD_DELAYED_REF;
+	if (type == BTRFS_REF_METADATA)
+		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	else
+		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	if (type == BTRFS_REF_METADATA)
+		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	else
+		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		head = NULL;
+		goto out;
+	}
+
+	head_check.ref_mod = 2;
+	head_check.total_ref_mod = 2;
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("double add failed");
+		goto out;
+	}
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+
+	delete_delayed_ref_node(head, node);
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (node) {
+		test_err("found node when none should exist");
+		goto out;
+	}
+	delete_delayed_ref_head(trans, head);
+	head = NULL;
+
+	/* Add two drop refs, make sure they are merged properly. */
+	ref.action = BTRFS_DROP_DELAYED_REF;
+	if (type == BTRFS_REF_METADATA)
+		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	else
+		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	if (type == BTRFS_REF_METADATA)
+		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	else
+		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		goto out;
+	}
+
+	head_check.ref_mod = -2;
+	head_check.total_ref_mod = -2;
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("double drop failed");
+		goto out;
+	}
+
+	node_check.action = BTRFS_DROP_DELAYED_REF;
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+
+	delete_delayed_ref_node(head, node);
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (node) {
+		test_err("found node when none should exist");
+		goto out;
+	}
+	delete_delayed_ref_head(trans, head);
+	head = NULL;
+
+	/* Add multiple refs, then drop until we go negative again. */
+	ref.action = BTRFS_ADD_DELAYED_REF;
+	for (int i = 0; i < 10; i++) {
+		if (type == BTRFS_REF_METADATA)
+			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+		else
+			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+		if (ret) {
+			test_err("failed ref action %d", ret);
+			goto out;
+		}
+	}
+
+	ref.action = BTRFS_DROP_DELAYED_REF;
+	for (int i = 0; i < 12; i++) {
+		if (type == BTRFS_REF_METADATA)
+			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+		else
+			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+		if (ret) {
+			test_err("failed ref action %d", ret);
+			goto out;
+		}
+	}
+
+	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		head = NULL;
+		ret = -EINVAL;
+		goto out;
+	}
+
+	head_check.ref_mod = -2;
+	head_check.total_ref_mod = -2;
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("double drop failed");
+		goto out;
+	}
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+
+	delete_delayed_ref_node(head, node);
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (node) {
+		test_err("found node when none should exist");
+		goto out;
+	}
+
+	delete_delayed_ref_head(trans, head);
+	head = NULL;
+
+	/* Drop multiple refs, then add until we go positive again. */
+	ref.action = BTRFS_DROP_DELAYED_REF;
+	for (int i = 0; i < 10; i++) {
+		if (type == BTRFS_REF_METADATA)
+			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+		else
+			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+		if (ret) {
+			test_err("failed ref action %d", ret);
+			goto out;
+		}
+	}
+
+	ref.action = BTRFS_ADD_DELAYED_REF;
+	for (int i = 0; i < 12; i++) {
+		if (type == BTRFS_REF_METADATA)
+			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+		else
+			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+		if (ret) {
+			test_err("failed ref action %d", ret);
+			goto out;
+		}
+	}
+
+	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		ret = -EINVAL;
+		head = NULL;
+		goto out;
+	}
+
+	head_check.ref_mod = 2;
+	head_check.total_ref_mod = 2;
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("add and drop to positive failed");
+		goto out;
+	}
+
+	node_check.action = BTRFS_ADD_DELAYED_REF;
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+
+	delete_delayed_ref_node(head, node);
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (node) {
+		test_err("found node when none should exist");
+		goto out;
+	}
+	delete_delayed_ref_head(trans, head);
+	head = NULL;
+
+	/*
+	 * Add a bunch of refs with different roots and parents, then drop them
+	 * all, make sure everything is properly merged.
+	 */
+	ref.action = BTRFS_ADD_DELAYED_REF;
+	for (int i = 0; i < 50; i++) {
+		if (!(i % 2)) {
+			ref.parent = 0;
+			ref.ref_root = FAKE_ROOT_OBJECTID + i;
+		} else {
+			ref.parent = FAKE_PARENT + (i * fs_info->nodesize);
+		}
+		if (type == BTRFS_REF_METADATA)
+			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+		else
+			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+		if (ret) {
+			test_err("failed ref action %d", ret);
+			goto out;
+		}
+	}
+
+	ref.action = BTRFS_DROP_DELAYED_REF;
+	for (int i = 0; i < 50; i++) {
+		if (!(i % 2)) {
+			ref.parent = 0;
+			ref.ref_root = FAKE_ROOT_OBJECTID + i;
+		} else {
+			ref.parent = FAKE_PARENT + (i * fs_info->nodesize);
+		}
+		if (type == BTRFS_REF_METADATA)
+			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+		else
+			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
+		if (ret) {
+			test_err("failed ref action %d", ret);
+			goto out;
+		}
+	}
+
+	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		head = NULL;
+		ret = -EINVAL;
+		goto out;
+	}
+
+	head_check.ref_mod = 0;
+	head_check.total_ref_mod = 0;
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("add and drop multiple failed");
+		goto out;
+	}
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (node) {
+		test_err("found node when none should exist");
+		goto out;
+	}
+	ret = 0;
+out:
+	if (head)
+		btrfs_unselect_ref_head(&trans->transaction->delayed_refs, head);
+	btrfs_destroy_delayed_refs(trans->transaction);
+	return ret;
+}
+
+/*
+ * Basic test to validate we always get the add operations first followed by any
+ * delete operations.
+ */
+static int select_delayed_refs_test(struct btrfs_trans_handle *trans)
+{
+	struct btrfs_delayed_ref_root *delayed_refs =
+		&trans->transaction->delayed_refs;
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct btrfs_delayed_ref_head *head = NULL;
+	struct btrfs_delayed_ref_node *node;
+	struct btrfs_ref ref = {
+		.type = BTRFS_REF_METADATA,
+		.action = BTRFS_DROP_DELAYED_REF,
+		.parent = 0,
+		.ref_root = FAKE_ROOT_OBJECTID,
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+	};
+	struct ref_head_check head_check = {
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+		.ref_mod = 0,
+		.total_ref_mod = 0,
+	};
+	struct ref_node_check node_check = {
+		.bytenr = FAKE_BYTENR,
+		.num_bytes = fs_info->nodesize,
+		.ref_mod = 1,
+		.action = BTRFS_ADD_DELAYED_REF,
+		.type = BTRFS_TREE_BLOCK_REF_KEY,
+		.parent = 0,
+		.owner = FAKE_LEVEL,
+		.offset = 0,
+	};
+	int ret;
+
+	/* Add the drop first. */
+	btrfs_init_tree_ref(&ref, FAKE_LEVEL, FAKE_ROOT_OBJECTID, false);
+	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		return ret;
+	}
+
+	/*
+	 * Now add the add, and make it a different root so it's logically later
+	 * in the rb tree.
+	 */
+	ref.action = BTRFS_ADD_DELAYED_REF;
+	ref.ref_root = FAKE_ROOT_OBJECTID + 1;
+	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	head = btrfs_select_ref_head(fs_info, delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		ret = -EINVAL;
+		head = NULL;
+		goto out;
+	}
+
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("head check failed");
+		goto out;
+	}
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	node_check.root = FAKE_ROOT_OBJECTID + 1;
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+	delete_delayed_ref_node(head, node);
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	node_check.action = BTRFS_DROP_DELAYED_REF;
+	node_check.root = FAKE_ROOT_OBJECTID;
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+	delete_delayed_ref_node(head, node);
+	delete_delayed_ref_head(trans, head);
+	head = NULL;
+
+	/*
+	 * Now we're going to do the same thing, but we're going to have an add
+	 * that gets deleted because of a merge, and make sure we still have
+	 * another add in place.
+	 */
+	ref.action = BTRFS_DROP_DELAYED_REF;
+	ref.ref_root = FAKE_ROOT_OBJECTID;
+	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	ref.action = BTRFS_ADD_DELAYED_REF;
+	ref.ref_root = FAKE_ROOT_OBJECTID + 1;
+	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	ref.action = BTRFS_DROP_DELAYED_REF;
+	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	ref.action = BTRFS_ADD_DELAYED_REF;
+	ref.ref_root = FAKE_ROOT_OBJECTID + 2;
+	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
+	if (ret) {
+		test_err("failed ref action %d", ret);
+		goto out;
+	}
+
+	head = btrfs_select_ref_head(fs_info, delayed_refs);
+	if (IS_ERR_OR_NULL(head)) {
+		if (IS_ERR(head))
+			test_err("failed to select delayed ref head: %ld",
+				 PTR_ERR(head));
+		else
+			test_err("failed to find delayed ref head");
+		ret = -EINVAL;
+		head = NULL;
+		goto out;
+	}
+
+	ret = -EINVAL;
+	if (validate_ref_head(head, &head_check)) {
+		test_err("head check failed");
+		goto out;
+	}
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	node_check.action = BTRFS_ADD_DELAYED_REF;
+	node_check.root = FAKE_ROOT_OBJECTID + 2;
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+	delete_delayed_ref_node(head, node);
+
+	spin_lock(&head->lock);
+	node = btrfs_select_delayed_ref(head);
+	spin_unlock(&head->lock);
+	if (!node) {
+		test_err("failed to select delayed ref");
+		goto out;
+	}
+
+	node_check.action = BTRFS_DROP_DELAYED_REF;
+	node_check.root = FAKE_ROOT_OBJECTID;
+	if (validate_ref_node(node, &node_check)) {
+		test_err("node check failed");
+		goto out;
+	}
+	delete_delayed_ref_node(head, node);
+	ret = 0;
+out:
+	if (head)
+		btrfs_unselect_ref_head(delayed_refs, head);
+	btrfs_destroy_delayed_refs(trans->transaction);
+	return ret;
+}
+
+int btrfs_test_delayed_refs(u32 sectorsize, u32 nodesize)
+{
+	struct btrfs_transaction transaction;
+	struct btrfs_trans_handle trans;
+	struct btrfs_fs_info *fs_info;
+	int ret;
+
+	test_msg("running delayed refs tests");
+
+	fs_info = btrfs_alloc_dummy_fs_info(nodesize, sectorsize);
+	if (!fs_info) {
+		test_std_err(TEST_ALLOC_FS_INFO);
+		return -ENOMEM;
+	}
+	btrfs_init_dummy_trans(&trans, fs_info);
+	btrfs_init_dummy_transaction(&transaction, fs_info);
+	trans.transaction = &transaction;
+
+	ret = simple_tests(&trans);
+	if (!ret) {
+		test_msg("running delayed refs merg tests on metadata refs");
+		ret = merge_tests(&trans, BTRFS_REF_METADATA);
+	}
+
+	if (!ret) {
+		test_msg("running delayed refs merg tests on data refs");
+		ret = merge_tests(&trans, BTRFS_REF_DATA);
+	}
+
+	if (!ret)
+		ret = select_delayed_refs_test(&trans);
+	btrfs_free_dummy_fs_info(fs_info);
+	return ret;
+}
-- 
2.43.0

Re: [PATCH 2/2] btrfs: add delayed ref self tests

[Boris Burkov]

On Thu, Nov 14, 2024 at 02:57:49PM -0500, Josef Bacik wrote:
> The recent fix for a stupid mistake I made uncovered the fact that we
> don't have adequate testing in the delayed refs code, as it took a
> pretty extensive and long running stress test to uncover something that
> a unit test would have uncovered right away.
> 
> Fix this by adding a delayed refs self test suite.  This will validate
> that the btrfs_ref transformation does the correct thing, that we do the
> correct thing when merging delayed refs, and that we get the delayed
> refs in the order that we expect.  These are all crucial to how the
> delayed refs operate.
> 
> I introduced various bugs (including the original bug) into the delayed
> refs code to validate that these tests caught all of the shenanigans
> that I could think of.

These tests look reasonably exhaustive for us handling expected
situations in the expected way.

I think it's also a good practice to try to test various error paths
that are part of the intended behavior of the function.

Is that something we could feasibly do at this point? (i.e.,
would it generally BUG_ON and do we have the vocabulary to assert the
expected failures?)

OTOH, I don't see any too interesting (not enomem) failures in add/select
ref anyway, so that might apply more to testing an aspect of delayed
refs that actually cares about the extent tree, for example.

> 
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> ---
>  fs/btrfs/Makefile                   |    2 +-
>  fs/btrfs/delayed-ref.c              |   14 +-
>  fs/btrfs/tests/btrfs-tests.c        |   18 +
>  fs/btrfs/tests/btrfs-tests.h        |    6 +
>  fs/btrfs/tests/delayed-refs-tests.c | 1012 +++++++++++++++++++++++++++
>  5 files changed, 1048 insertions(+), 4 deletions(-)
>  create mode 100644 fs/btrfs/tests/delayed-refs-tests.c
> 
> diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
> index 3cfc440c636c..2d5f0482678b 100644
> --- a/fs/btrfs/Makefile
> +++ b/fs/btrfs/Makefile
> @@ -44,4 +44,4 @@ btrfs-$(CONFIG_BTRFS_FS_RUN_SANITY_TESTS) += tests/free-space-tests.o \
>  	tests/extent-buffer-tests.o tests/btrfs-tests.o \
>  	tests/extent-io-tests.o tests/inode-tests.o tests/qgroup-tests.o \
>  	tests/free-space-tree-tests.o tests/extent-map-tests.o \
> -	tests/raid-stripe-tree-tests.o
> +	tests/raid-stripe-tree-tests.o tests/delayed-refs-tests.o
> diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
> index 63e0a7f660da..3ec0468d1d94 100644
> --- a/fs/btrfs/delayed-ref.c
> +++ b/fs/btrfs/delayed-ref.c
> @@ -93,6 +93,9 @@ void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans)
>  	u64 num_bytes;
>  	u64 reserved_bytes;
>  
> +	if (btrfs_is_testing(fs_info))
> +		return;
> +

Not new as of this patch or anything, but this is really gross. Is this
really better than different modes and blocking off/replacing stuff with
macros, for example? I guess that doesn't work with running the tests
while loading the module.

Optimally, we'd have to refactor stuff to have a unit-testable core, but
that is a much bigger lift that also leads to a bunch of bugs while we
do it.

>  	num_bytes = btrfs_calc_delayed_ref_bytes(fs_info, trans->delayed_ref_updates);
>  	num_bytes += btrfs_calc_delayed_ref_csum_bytes(fs_info,
>  						       trans->delayed_ref_csum_deletions);
> @@ -1261,6 +1264,7 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
>  {
>  	struct btrfs_delayed_ref_root *delayed_refs = &trans->delayed_refs;
>  	struct btrfs_fs_info *fs_info = trans->fs_info;
> +	bool testing = btrfs_is_testing(fs_info);
>  
>  	spin_lock(&delayed_refs->lock);
>  	while (true) {
> @@ -1290,7 +1294,7 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
>  		spin_unlock(&delayed_refs->lock);
>  		mutex_unlock(&head->mutex);
>  
> -		if (pin_bytes) {
> +		if (!testing && pin_bytes) {
>  			struct btrfs_block_group *bg;
>  
>  			bg = btrfs_lookup_block_group(fs_info, head->bytenr);
> @@ -1322,12 +1326,16 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
>  			btrfs_error_unpin_extent_range(fs_info, head->bytenr,
>  				head->bytenr + head->num_bytes - 1);
>  		}
> -		btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs, head);
> +		if (!testing)
> +			btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs,
> +							  head);
>  		btrfs_put_delayed_ref_head(head);
>  		cond_resched();
>  		spin_lock(&delayed_refs->lock);
>  	}
> -	btrfs_qgroup_destroy_extent_records(trans);
> +
> +	if (!testing)

What is the principle that decides whether something is appropriate for
testing mode?

> +		btrfs_qgroup_destroy_extent_records(trans);
>  
>  	spin_unlock(&delayed_refs->lock);
>  }
> diff --git a/fs/btrfs/tests/btrfs-tests.c b/fs/btrfs/tests/btrfs-tests.c
> index e607b5d52fb1..a8549c5685ad 100644
> --- a/fs/btrfs/tests/btrfs-tests.c
> +++ b/fs/btrfs/tests/btrfs-tests.c
> @@ -142,6 +142,11 @@ struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize)
>  	fs_info->nodesize = nodesize;
>  	fs_info->sectorsize = sectorsize;
>  	fs_info->sectorsize_bits = ilog2(sectorsize);
> +
> +	/* CRC32C csum size. */
> +	fs_info->csum_size = 4;
> +	fs_info->csums_per_leaf = BTRFS_MAX_ITEM_SIZE(fs_info) /
> +		fs_info->csum_size;
>  	set_bit(BTRFS_FS_STATE_DUMMY_FS_INFO, &fs_info->fs_state);
>  
>  	test_mnt->mnt_sb->s_fs_info = fs_info;
> @@ -247,6 +252,16 @@ void btrfs_free_dummy_block_group(struct btrfs_block_group *cache)
>  	kfree(cache);
>  }
>  
> +void btrfs_init_dummy_transaction(struct btrfs_transaction *trans,
> +				  struct btrfs_fs_info *fs_info)
> +{
> +	memset(trans, 0, sizeof(*trans));
> +	trans->fs_info = fs_info;
> +	xa_init(&trans->delayed_refs.head_refs);
> +	xa_init(&trans->delayed_refs.dirty_extents);
> +	spin_lock_init(&trans->delayed_refs.lock);
> +}
> +
>  void btrfs_init_dummy_trans(struct btrfs_trans_handle *trans,
>  			    struct btrfs_fs_info *fs_info)
>  {
> @@ -295,6 +310,9 @@ int btrfs_run_sanity_tests(void)
>  			ret = btrfs_test_raid_stripe_tree(sectorsize, nodesize);
>  			if (ret)
>  				goto out;
> +			ret = btrfs_test_delayed_refs(sectorsize, nodesize);
> +			if (ret)
> +				goto out;
>  		}
>  	}
>  	ret = btrfs_test_extent_map();
> diff --git a/fs/btrfs/tests/btrfs-tests.h b/fs/btrfs/tests/btrfs-tests.h
> index b524ecf2f452..713f1b26222b 100644
> --- a/fs/btrfs/tests/btrfs-tests.h
> +++ b/fs/btrfs/tests/btrfs-tests.h
> @@ -6,6 +6,8 @@
>  #ifndef BTRFS_TESTS_H
>  #define BTRFS_TESTS_H
>  
> +#include <linux/types.h>
> +
>  #ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
>  int btrfs_run_sanity_tests(void);
>  
> @@ -31,6 +33,7 @@ extern const char *test_error[];
>  
>  struct btrfs_root; >  struct btrfs_trans_handle;
> +struct btrfs_transaction;
>  
>  int btrfs_test_extent_buffer_operations(u32 sectorsize, u32 nodesize);
>  int btrfs_test_free_space_cache(u32 sectorsize, u32 nodesize);
> @@ -40,6 +43,7 @@ int btrfs_test_qgroups(u32 sectorsize, u32 nodesize);
>  int btrfs_test_free_space_tree(u32 sectorsize, u32 nodesize);
>  int btrfs_test_raid_stripe_tree(u32 sectorsize, u32 nodesize);
>  int btrfs_test_extent_map(void);
> +int btrfs_test_delayed_refs(u32 sectorsize, u32 nodesize);
>  struct inode *btrfs_new_test_inode(void);
>  struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize);
>  void btrfs_free_dummy_fs_info(struct btrfs_fs_info *fs_info);
> @@ -49,6 +53,8 @@ btrfs_alloc_dummy_block_group(struct btrfs_fs_info *fs_info, unsigned long lengt
>  void btrfs_free_dummy_block_group(struct btrfs_block_group *cache);
>  void btrfs_init_dummy_trans(struct btrfs_trans_handle *trans,
>  			    struct btrfs_fs_info *fs_info);
> +void btrfs_init_dummy_transaction(struct btrfs_transaction *trans,
> +				  struct btrfs_fs_info *fs_info);
>  struct btrfs_device *btrfs_alloc_dummy_device(struct btrfs_fs_info *fs_info);
>  #else
>  static inline int btrfs_run_sanity_tests(void)
> diff --git a/fs/btrfs/tests/delayed-refs-tests.c b/fs/btrfs/tests/delayed-refs-tests.c
> new file mode 100644
> index 000000000000..9b469791c20a
> --- /dev/null
> +++ b/fs/btrfs/tests/delayed-refs-tests.c
> @@ -0,0 +1,1012 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <linux/sizes.h>
> +#include "btrfs-tests.h"
> +#include "../transaction.h"
> +#include "../delayed-ref.h"
> +#include "../extent-tree.h"
> +
> +#define FAKE_ROOT_OBJECTID 256
> +#define FAKE_BYTENR 0
> +#define FAKE_LEVEL 1
> +#define FAKE_INO 256
> +#define FAKE_FILE_OFFSET 0
> +#define FAKE_PARENT SZ_1M
> +
> +struct ref_head_check {
> +	u64 bytenr;
> +	u64 num_bytes;
> +	int ref_mod;
> +	int total_ref_mod;
> +	int must_insert;
> +};
> +
> +struct ref_node_check {
> +	u64 bytenr;
> +	u64 num_bytes;
> +	int ref_mod;
> +	enum btrfs_delayed_ref_action action;
> +	u8 type;
> +	u64 parent;
> +	u64 root;
> +	u64 owner;
> +	u64 offset;
> +};
> +
> +static enum btrfs_ref_type ref_type_from_disk_ref_type(u8 type)
> +{
> +	if ((type == BTRFS_TREE_BLOCK_REF_KEY) ||
> +	    (type == BTRFS_SHARED_BLOCK_REF_KEY))
> +		return BTRFS_REF_METADATA;
> +	return BTRFS_REF_DATA;
> +}
> +
> +static void delete_delayed_ref_head(struct btrfs_trans_handle *trans,
> +				    struct btrfs_delayed_ref_head *head)
> +{
> +	struct btrfs_fs_info *fs_info = trans->fs_info;
> +	struct btrfs_delayed_ref_root *delayed_refs =
> +		&trans->transaction->delayed_refs;
> +
> +	spin_lock(&delayed_refs->lock);
> +	spin_lock(&head->lock);
> +	btrfs_delete_ref_head(fs_info, delayed_refs, head);
> +	spin_unlock(&head->lock);
> +	spin_unlock(&delayed_refs->lock);
> +
> +	btrfs_delayed_ref_unlock(head);
> +	btrfs_put_delayed_ref_head(head);
> +}
> +
> +static void delete_delayed_ref_node(struct btrfs_delayed_ref_head *head,
> +				    struct btrfs_delayed_ref_node *node)
> +{
> +	rb_erase_cached(&node->ref_node, &head->ref_tree);
> +	RB_CLEAR_NODE(&node->ref_node);
> +	if (!list_empty(&node->add_list))
> +		list_del_init(&node->add_list);
> +	btrfs_put_delayed_ref(node);
> +}
> +
> +static int validate_ref_head(struct btrfs_delayed_ref_head *head,
> +			     struct ref_head_check *check)
> +{
> +	if (head->bytenr != check->bytenr) {
> +		test_err("invalid bytenr have: %llu want: %llu", head->bytenr,
> +			 check->bytenr);
> +		return -EINVAL;
> +	}
> +
> +	if (head->num_bytes != check->num_bytes) {
> +		test_err("invalid num_bytes have: %llu want: %llu",
> +			 head->num_bytes, check->num_bytes);
> +		return -EINVAL;
> +	}
> +
> +	if (head->ref_mod != check->ref_mod) {
> +		test_err("invalid ref_mod have: %d want: %d", head->ref_mod,
> +			 check->ref_mod);
> +		return -EINVAL;
> +	}
> +
> +	if (head->total_ref_mod != check->total_ref_mod) {
> +		test_err("invalid total_ref_mod have: %d want: %d",
> +			 head->total_ref_mod, check->total_ref_mod);
> +		return -EINVAL;
> +	}
> +
> +	if (head->must_insert_reserved != check->must_insert) {
> +		test_err("invalid must_insert have: %d want: %d",
> +			 head->must_insert_reserved, check->must_insert);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int validate_ref_node(struct btrfs_delayed_ref_node *node,
> +			     struct ref_node_check *check)
> +{
> +	if (node->bytenr != check->bytenr) {
> +		test_err("invalid bytenr have: %llu want: %llu", node->bytenr,
> +			 check->bytenr);
> +		return -EINVAL;
> +	}
> +
> +	if (node->num_bytes != check->num_bytes) {
> +		test_err("invalid num_bytes have: %llu want: %llu",
> +			 node->num_bytes, check->num_bytes);
> +		return -EINVAL;
> +	}
> +
> +	if (node->ref_mod != check->ref_mod) {
> +		test_err("invalid ref_mod have: %d want: %d", node->ref_mod,
> +			 check->ref_mod);
> +		return -EINVAL;
> +	}
> +
> +	if (node->action != check->action) {
> +		test_err("invalid action have: %d want: %d", node->action,
> +			 check->action);
> +		return -EINVAL;
> +	}
> +
> +	if (node->parent != check->parent) {
> +		test_err("invalid parent have: %llu want: %llu", node->parent,
> +			 check->parent);
> +		return -EINVAL;
> +	}
> +
> +	if (node->ref_root != check->root) {
> +		test_err("invalid root have: %llu want: %llu", node->ref_root,
> +			 check->root);
> +		return -EINVAL;
> +	}
> +
> +	if (node->type != check->type) {
> +		test_err("invalid type have: %d want: %d", node->type,
> +			 check->type);
> +		return -EINVAL;
> +	}
> +
> +	if (btrfs_delayed_ref_owner(node) != check->owner) {
> +		test_err("invalid owner have: %llu want: %llu",
> +			 btrfs_delayed_ref_owner(node), check->owner);
> +		return -EINVAL;
> +	}
> +
> +	if (btrfs_delayed_ref_offset(node) != check->offset) {
> +		test_err("invalid offset have: %llu want: %llu",
> +			 btrfs_delayed_ref_offset(node), check->offset);
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +static int simple_test(struct btrfs_trans_handle *trans,
> +		       struct ref_head_check *head_check,
> +		       struct ref_node_check *node_check)
> +{
> +	struct btrfs_delayed_ref_root *delayed_refs =
> +		&trans->transaction->delayed_refs;
> +	struct btrfs_fs_info *fs_info = trans->fs_info;
> +	struct btrfs_delayed_ref_head *head;
> +	struct btrfs_delayed_ref_node *node;
> +	struct btrfs_ref ref = {
> +		.type = ref_type_from_disk_ref_type(node_check->type),
> +		.action = node_check->action,
> +		.parent = node_check->parent,
> +		.ref_root = node_check->root,
> +		.bytenr = node_check->bytenr,
> +		.num_bytes = fs_info->nodesize,
> +	};
> +	int ret;
> +
> +	if (ref.type == BTRFS_REF_METADATA)
> +		btrfs_init_tree_ref(&ref, node_check->owner, node_check->root,
> +				    false);
> +	else
> +		btrfs_init_data_ref(&ref, node_check->owner, node_check->offset,
> +				    node_check->root, true);
> +
> +	if (ref.type == BTRFS_REF_METADATA)
> +		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	else
> +		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		return ret;
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		return -EINVAL;
> +	}
> +
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, head_check))
> +		goto out;
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	if (validate_ref_node(node, node_check))
> +		goto out;
> +	ret = 0;
> +out:
> +	btrfs_unselect_ref_head(delayed_refs, head);
> +	btrfs_destroy_delayed_refs(trans->transaction);
> +	return ret;
> +}
> +
> +/*
> + * These are simple tests, make sure that our btrfs_ref's get turned into the
> + * appropriate btrfs_delayed_ref_node based on their settings and action.
> + */
> +static int simple_tests(struct btrfs_trans_handle *trans)
> +{
> +	struct btrfs_fs_info *fs_info = trans->fs_info;
> +	struct ref_head_check head_check = {
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +		.ref_mod = 1,
> +		.total_ref_mod = 1,
> +	};
> +	struct ref_node_check node_check = {
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +		.ref_mod = 1,
> +		.action = BTRFS_ADD_DELAYED_REF,
> +		.type = BTRFS_TREE_BLOCK_REF_KEY,
> +		.parent = 0,
> +		.root = FAKE_ROOT_OBJECTID,
> +		.owner = FAKE_LEVEL,
> +		.offset = 0,
> +	};
> +
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single add tree block failed");
> +		return -EINVAL;
> +	}
> +
> +	node_check.type = BTRFS_EXTENT_DATA_REF_KEY;
> +	node_check.owner = FAKE_INO;
> +	node_check.offset = FAKE_FILE_OFFSET;
> +
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single add extent data failed");
> +		return -EINVAL;
> +	}
> +
> +	node_check.parent = FAKE_PARENT;
> +	node_check.type = BTRFS_SHARED_BLOCK_REF_KEY;
> +	node_check.owner = FAKE_LEVEL;
> +	node_check.offset = 0;
> +
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single add shared block failed");
> +		return -EINVAL;
> +	}
> +
> +	node_check.type = BTRFS_SHARED_DATA_REF_KEY;
> +	node_check.owner = FAKE_INO;
> +	node_check.offset = FAKE_FILE_OFFSET;
> +
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single add shared data failed");
> +		return -EINVAL;
> +	}
> +
> +	head_check.ref_mod = -1;
> +	head_check.total_ref_mod = -1;
> +	node_check.action = BTRFS_DROP_DELAYED_REF;
> +	node_check.type = BTRFS_TREE_BLOCK_REF_KEY;
> +	node_check.owner = FAKE_LEVEL;
> +	node_check.offset = 0;
> +	node_check.parent = 0;
> +
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single drop tree block failed");
> +		return -EINVAL;
> +	}
> +
> +	node_check.type = BTRFS_EXTENT_DATA_REF_KEY;
> +	node_check.owner = FAKE_INO;
> +	node_check.offset = FAKE_FILE_OFFSET;
> +
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single drop extent data failed");
> +		return -EINVAL;
> +	}
> +
> +	node_check.parent = FAKE_PARENT;
> +	node_check.type = BTRFS_SHARED_BLOCK_REF_KEY;
> +	node_check.owner = FAKE_LEVEL;
> +	node_check.offset = 0;
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single drop shared block failed");
> +		return -EINVAL;
> +	}
> +
> +	node_check.type = BTRFS_SHARED_DATA_REF_KEY;
> +	node_check.owner = FAKE_INO;
> +	node_check.offset = FAKE_FILE_OFFSET;
> +	if (simple_test(trans, &head_check, &node_check)) {
> +		test_err("single drop shared data failed");
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * Merge tests, validate that we do delayed ref merging properly, the ref counts
> + * all end up properly, and delayed refs are deleted once they're no longer
> + * needed.
> + */
> +static int merge_tests(struct btrfs_trans_handle *trans,
> +		       enum btrfs_ref_type type)
> +{
> +	struct btrfs_fs_info *fs_info = trans->fs_info;
> +	struct btrfs_delayed_ref_head *head = NULL;
> +	struct btrfs_delayed_ref_node *node;
> +	struct btrfs_ref ref = {
> +		.type = type,
> +		.action = BTRFS_ADD_DELAYED_REF,
> +		.parent = 0,
> +		.ref_root = FAKE_ROOT_OBJECTID,
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +	};
> +	struct ref_head_check head_check = {
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +		.ref_mod = 0,
> +		.total_ref_mod = 0,
> +	};
> +	struct ref_node_check node_check = {
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +		.ref_mod = 2,
> +		.action = BTRFS_ADD_DELAYED_REF,
> +		.parent = 0,
> +		.root = FAKE_ROOT_OBJECTID,
> +	};
> +	int ret;
> +
> +	/*
> +	 * First add a ref and then drop it, make sure we get a head ref with a
> +	 * 0 total ref mod and no nodes.
> +	 */
> +	if (type == BTRFS_REF_METADATA) {
> +		node_check.type = BTRFS_TREE_BLOCK_REF_KEY;
> +		node_check.owner = FAKE_LEVEL;
> +		btrfs_init_tree_ref(&ref, FAKE_LEVEL, FAKE_ROOT_OBJECTID, false);
> +	} else {
> +		node_check.type = BTRFS_EXTENT_DATA_REF_KEY;
> +		node_check.owner = FAKE_INO;
> +		node_check.offset = FAKE_FILE_OFFSET;
> +		btrfs_init_data_ref(&ref, FAKE_INO, FAKE_FILE_OFFSET,
> +				    FAKE_ROOT_OBJECTID, true);
> +	}
> +
> +	if (type == BTRFS_REF_METADATA)
> +		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	else
> +		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		return ret;
> +	}
> +
> +	ref.action = BTRFS_DROP_DELAYED_REF;
> +	if (type == BTRFS_REF_METADATA)
> +		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	else
> +		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		head = NULL;
> +		goto out;
> +	}
> +
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("single add and drop failed");
> +		goto out;
> +	}
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (node) {
> +		test_err("found node when none should exist");
> +		goto out;
> +	}
> +
> +	delete_delayed_ref_head(trans, head);
> +	head = NULL;
> +
> +	/*
> +	 * Add a ref, then add another ref, make sure we get a head ref with a
> +	 * 2 total ref mod and 1 node.
> +	 */
> +	ref.action = BTRFS_ADD_DELAYED_REF;
> +	if (type == BTRFS_REF_METADATA)
> +		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	else
> +		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	if (type == BTRFS_REF_METADATA)
> +		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	else
> +		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		head = NULL;
> +		goto out;
> +	}
> +
> +	head_check.ref_mod = 2;
> +	head_check.total_ref_mod = 2;
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("double add failed");
> +		goto out;
> +	}
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +
> +	delete_delayed_ref_node(head, node);
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (node) {
> +		test_err("found node when none should exist");
> +		goto out;
> +	}
> +	delete_delayed_ref_head(trans, head);
> +	head = NULL;
> +
> +	/* Add two drop refs, make sure they are merged properly. */
> +	ref.action = BTRFS_DROP_DELAYED_REF;
> +	if (type == BTRFS_REF_METADATA)
> +		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	else
> +		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	if (type == BTRFS_REF_METADATA)
> +		ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	else
> +		ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		goto out;
> +	}
> +
> +	head_check.ref_mod = -2;
> +	head_check.total_ref_mod = -2;
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("double drop failed");
> +		goto out;
> +	}
> +
> +	node_check.action = BTRFS_DROP_DELAYED_REF;
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +
> +	delete_delayed_ref_node(head, node);
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (node) {
> +		test_err("found node when none should exist");
> +		goto out;
> +	}
> +	delete_delayed_ref_head(trans, head);
> +	head = NULL;
> +
> +	/* Add multiple refs, then drop until we go negative again. */
> +	ref.action = BTRFS_ADD_DELAYED_REF;
> +	for (int i = 0; i < 10; i++) {
> +		if (type == BTRFS_REF_METADATA)
> +			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +		else
> +			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +		if (ret) {
> +			test_err("failed ref action %d", ret);
> +			goto out;
> +		}
> +	}
> +
> +	ref.action = BTRFS_DROP_DELAYED_REF;
> +	for (int i = 0; i < 12; i++) {
> +		if (type == BTRFS_REF_METADATA)
> +			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +		else
> +			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +		if (ret) {
> +			test_err("failed ref action %d", ret);
> +			goto out;
> +		}
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		head = NULL;
> +		ret = -EINVAL;
> +		goto out;
> +	}
> +
> +	head_check.ref_mod = -2;
> +	head_check.total_ref_mod = -2;
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("double drop failed");
> +		goto out;
> +	}
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +
> +	delete_delayed_ref_node(head, node);
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (node) {
> +		test_err("found node when none should exist");
> +		goto out;
> +	}
> +
> +	delete_delayed_ref_head(trans, head);
> +	head = NULL;
> +
> +	/* Drop multiple refs, then add until we go positive again. */
> +	ref.action = BTRFS_DROP_DELAYED_REF;
> +	for (int i = 0; i < 10; i++) {
> +		if (type == BTRFS_REF_METADATA)
> +			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +		else
> +			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +		if (ret) {
> +			test_err("failed ref action %d", ret);
> +			goto out;
> +		}
> +	}
> +
> +	ref.action = BTRFS_ADD_DELAYED_REF;
> +	for (int i = 0; i < 12; i++) {
> +		if (type == BTRFS_REF_METADATA)
> +			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +		else
> +			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +		if (ret) {
> +			test_err("failed ref action %d", ret);
> +			goto out;
> +		}
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		ret = -EINVAL;
> +		head = NULL;
> +		goto out;
> +	}
> +
> +	head_check.ref_mod = 2;
> +	head_check.total_ref_mod = 2;
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("add and drop to positive failed");
> +		goto out;
> +	}
> +
> +	node_check.action = BTRFS_ADD_DELAYED_REF;
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +
> +	delete_delayed_ref_node(head, node);
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (node) {
> +		test_err("found node when none should exist");
> +		goto out;
> +	}
> +	delete_delayed_ref_head(trans, head);
> +	head = NULL;
> +
> +	/*
> +	 * Add a bunch of refs with different roots and parents, then drop them
> +	 * all, make sure everything is properly merged.
> +	 */
> +	ref.action = BTRFS_ADD_DELAYED_REF;
> +	for (int i = 0; i < 50; i++) {
> +		if (!(i % 2)) {
> +			ref.parent = 0;
> +			ref.ref_root = FAKE_ROOT_OBJECTID + i;
> +		} else {
> +			ref.parent = FAKE_PARENT + (i * fs_info->nodesize);
> +		}
> +		if (type == BTRFS_REF_METADATA)
> +			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +		else
> +			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +		if (ret) {
> +			test_err("failed ref action %d", ret);
> +			goto out;
> +		}
> +	}
> +
> +	ref.action = BTRFS_DROP_DELAYED_REF;
> +	for (int i = 0; i < 50; i++) {
> +		if (!(i % 2)) {
> +			ref.parent = 0;
> +			ref.ref_root = FAKE_ROOT_OBJECTID + i;
> +		} else {
> +			ref.parent = FAKE_PARENT + (i * fs_info->nodesize);
> +		}
> +		if (type == BTRFS_REF_METADATA)
> +			ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +		else
> +			ret = btrfs_add_delayed_data_ref(trans, &ref, 0);
> +		if (ret) {
> +			test_err("failed ref action %d", ret);
> +			goto out;
> +		}
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, &trans->transaction->delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		head = NULL;
> +		ret = -EINVAL;
> +		goto out;
> +	}
> +
> +	head_check.ref_mod = 0;
> +	head_check.total_ref_mod = 0;
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("add and drop multiple failed");
> +		goto out;
> +	}
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (node) {
> +		test_err("found node when none should exist");
> +		goto out;
> +	}
> +	ret = 0;
> +out:
> +	if (head)
> +		btrfs_unselect_ref_head(&trans->transaction->delayed_refs, head);
> +	btrfs_destroy_delayed_refs(trans->transaction);
> +	return ret;
> +}
> +
> +/*
> + * Basic test to validate we always get the add operations first followed by any
> + * delete operations.
> + */
> +static int select_delayed_refs_test(struct btrfs_trans_handle *trans)
> +{
> +	struct btrfs_delayed_ref_root *delayed_refs =
> +		&trans->transaction->delayed_refs;
> +	struct btrfs_fs_info *fs_info = trans->fs_info;
> +	struct btrfs_delayed_ref_head *head = NULL;
> +	struct btrfs_delayed_ref_node *node;
> +	struct btrfs_ref ref = {
> +		.type = BTRFS_REF_METADATA,
> +		.action = BTRFS_DROP_DELAYED_REF,
> +		.parent = 0,
> +		.ref_root = FAKE_ROOT_OBJECTID,
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +	};
> +	struct ref_head_check head_check = {
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +		.ref_mod = 0,
> +		.total_ref_mod = 0,
> +	};
> +	struct ref_node_check node_check = {
> +		.bytenr = FAKE_BYTENR,
> +		.num_bytes = fs_info->nodesize,
> +		.ref_mod = 1,
> +		.action = BTRFS_ADD_DELAYED_REF,
> +		.type = BTRFS_TREE_BLOCK_REF_KEY,
> +		.parent = 0,
> +		.owner = FAKE_LEVEL,
> +		.offset = 0,
> +	};
> +	int ret;
> +
> +	/* Add the drop first. */
> +	btrfs_init_tree_ref(&ref, FAKE_LEVEL, FAKE_ROOT_OBJECTID, false);
> +	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		return ret;
> +	}
> +
> +	/*
> +	 * Now add the add, and make it a different root so it's logically later
> +	 * in the rb tree.
> +	 */
> +	ref.action = BTRFS_ADD_DELAYED_REF;
> +	ref.ref_root = FAKE_ROOT_OBJECTID + 1;
> +	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		ret = -EINVAL;
> +		head = NULL;
> +		goto out;
> +	}
> +
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("head check failed");
> +		goto out;
> +	}
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	node_check.root = FAKE_ROOT_OBJECTID + 1;
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +	delete_delayed_ref_node(head, node);
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	node_check.action = BTRFS_DROP_DELAYED_REF;
> +	node_check.root = FAKE_ROOT_OBJECTID;
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +	delete_delayed_ref_node(head, node);
> +	delete_delayed_ref_head(trans, head);
> +	head = NULL;
> +
> +	/*
> +	 * Now we're going to do the same thing, but we're going to have an add
> +	 * that gets deleted because of a merge, and make sure we still have
> +	 * another add in place.
> +	 */
> +	ref.action = BTRFS_DROP_DELAYED_REF;
> +	ref.ref_root = FAKE_ROOT_OBJECTID;
> +	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	ref.action = BTRFS_ADD_DELAYED_REF;
> +	ref.ref_root = FAKE_ROOT_OBJECTID + 1;
> +	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	ref.action = BTRFS_DROP_DELAYED_REF;
> +	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	ref.action = BTRFS_ADD_DELAYED_REF;
> +	ref.ref_root = FAKE_ROOT_OBJECTID + 2;
> +	ret = btrfs_add_delayed_tree_ref(trans, &ref, NULL);
> +	if (ret) {
> +		test_err("failed ref action %d", ret);
> +		goto out;
> +	}
> +
> +	head = btrfs_select_ref_head(fs_info, delayed_refs);
> +	if (IS_ERR_OR_NULL(head)) {
> +		if (IS_ERR(head))
> +			test_err("failed to select delayed ref head: %ld",
> +				 PTR_ERR(head));
> +		else
> +			test_err("failed to find delayed ref head");
> +		ret = -EINVAL;
> +		head = NULL;
> +		goto out;
> +	}
> +
> +	ret = -EINVAL;
> +	if (validate_ref_head(head, &head_check)) {
> +		test_err("head check failed");
> +		goto out;
> +	}
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	node_check.action = BTRFS_ADD_DELAYED_REF;
> +	node_check.root = FAKE_ROOT_OBJECTID + 2;
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +	delete_delayed_ref_node(head, node);
> +
> +	spin_lock(&head->lock);
> +	node = btrfs_select_delayed_ref(head);
> +	spin_unlock(&head->lock);
> +	if (!node) {
> +		test_err("failed to select delayed ref");
> +		goto out;
> +	}
> +
> +	node_check.action = BTRFS_DROP_DELAYED_REF;
> +	node_check.root = FAKE_ROOT_OBJECTID;
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +	delete_delayed_ref_node(head, node);
> +	ret = 0;
> +out:
> +	if (head)
> +		btrfs_unselect_ref_head(delayed_refs, head);
> +	btrfs_destroy_delayed_refs(trans->transaction);
> +	return ret;
> +}
> +
> +int btrfs_test_delayed_refs(u32 sectorsize, u32 nodesize)
> +{
> +	struct btrfs_transaction transaction;
> +	struct btrfs_trans_handle trans;
> +	struct btrfs_fs_info *fs_info;
> +	int ret;
> +
> +	test_msg("running delayed refs tests");
> +
> +	fs_info = btrfs_alloc_dummy_fs_info(nodesize, sectorsize);
> +	if (!fs_info) {
> +		test_std_err(TEST_ALLOC_FS_INFO);
> +		return -ENOMEM;
> +	}
> +	btrfs_init_dummy_trans(&trans, fs_info);
> +	btrfs_init_dummy_transaction(&transaction, fs_info);
> +	trans.transaction = &transaction;
> +
> +	ret = simple_tests(&trans);
> +	if (!ret) {
> +		test_msg("running delayed refs merg tests on metadata refs");
> +		ret = merge_tests(&trans, BTRFS_REF_METADATA);
> +	}
> +
> +	if (!ret) {
> +		test_msg("running delayed refs merg tests on data refs");
> +		ret = merge_tests(&trans, BTRFS_REF_DATA);
> +	}
> +
> +	if (!ret)
> +		ret = select_delayed_refs_test(&trans);
> +	btrfs_free_dummy_fs_info(fs_info);
> +	return ret;
> +}
> -- 
> 2.43.0
> 

Re: [PATCH 2/2] btrfs: add delayed ref self tests

[Josef Bacik]

On Thu, Nov 14, 2024 at 02:23:53PM -0800, Boris Burkov wrote:
> On Thu, Nov 14, 2024 at 02:57:49PM -0500, Josef Bacik wrote:
> > The recent fix for a stupid mistake I made uncovered the fact that we
> > don't have adequate testing in the delayed refs code, as it took a
> > pretty extensive and long running stress test to uncover something that
> > a unit test would have uncovered right away.
> > 
> > Fix this by adding a delayed refs self test suite.  This will validate
> > that the btrfs_ref transformation does the correct thing, that we do the
> > correct thing when merging delayed refs, and that we get the delayed
> > refs in the order that we expect.  These are all crucial to how the
> > delayed refs operate.
> > 
> > I introduced various bugs (including the original bug) into the delayed
> > refs code to validate that these tests caught all of the shenanigans
> > that I could think of.
> 
> These tests look reasonably exhaustive for us handling expected
> situations in the expected way.
> 
> I think it's also a good practice to try to test various error paths
> that are part of the intended behavior of the function.
> 
> Is that something we could feasibly do at this point? (i.e.,
> would it generally BUG_ON and do we have the vocabulary to assert the
> expected failures?)
> 
> OTOH, I don't see any too interesting (not enomem) failures in add/select
> ref anyway, so that might apply more to testing an aspect of delayed
> refs that actually cares about the extent tree, for example.

Funnily enough I did this at first to make sure it caught the wrong ref type,
and forgot we have ASSERT()'s to validate that.  So we could do this, but we'd
have to go through and remove all the ASSERT()'s we put into place to keep us
from doing obviously wrong things.  ENOMEM and other related errors that are
valid is good things to test for, but unfortunately outside our abilities with
the self test.  Maybe kunit test has a way to do this, so perhaps that's
something to put on the list to do in the future instead of doing our own hand
rolled unit testing.

> 
> > 
> > Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> > ---
> >  fs/btrfs/Makefile                   |    2 +-
> >  fs/btrfs/delayed-ref.c              |   14 +-
> >  fs/btrfs/tests/btrfs-tests.c        |   18 +
> >  fs/btrfs/tests/btrfs-tests.h        |    6 +
> >  fs/btrfs/tests/delayed-refs-tests.c | 1012 +++++++++++++++++++++++++++
> >  5 files changed, 1048 insertions(+), 4 deletions(-)
> >  create mode 100644 fs/btrfs/tests/delayed-refs-tests.c
> > 
> > diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
> > index 3cfc440c636c..2d5f0482678b 100644
> > --- a/fs/btrfs/Makefile
> > +++ b/fs/btrfs/Makefile
> > @@ -44,4 +44,4 @@ btrfs-$(CONFIG_BTRFS_FS_RUN_SANITY_TESTS) += tests/free-space-tests.o \
> >  	tests/extent-buffer-tests.o tests/btrfs-tests.o \
> >  	tests/extent-io-tests.o tests/inode-tests.o tests/qgroup-tests.o \
> >  	tests/free-space-tree-tests.o tests/extent-map-tests.o \
> > -	tests/raid-stripe-tree-tests.o
> > +	tests/raid-stripe-tree-tests.o tests/delayed-refs-tests.o
> > diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
> > index 63e0a7f660da..3ec0468d1d94 100644
> > --- a/fs/btrfs/delayed-ref.c
> > +++ b/fs/btrfs/delayed-ref.c
> > @@ -93,6 +93,9 @@ void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans)
> >  	u64 num_bytes;
> >  	u64 reserved_bytes;
> >  
> > +	if (btrfs_is_testing(fs_info))
> > +		return;
> > +
> 
> Not new as of this patch or anything, but this is really gross. Is this
> really better than different modes and blocking off/replacing stuff with
> macros, for example? I guess that doesn't work with running the tests
> while loading the module.
> 
> Optimally, we'd have to refactor stuff to have a unit-testable core, but
> that is a much bigger lift that also leads to a bunch of bugs while we
> do it.

There's two things you'll see here, there's this style of "block this off for
testing", and then there's what I did in btrfs_alloc_dummy_fs_info() where I
stub out the fs_info->csum_size and ->csums_per_leaf.  Those tripped up some of
the random helpers, and I could have easily just slapped a "if
(btrfs_is_testing(fs_info)) return;" in there, but it's way more straightforward
to just stub out what we need for the test.

> 
> >  	num_bytes = btrfs_calc_delayed_ref_bytes(fs_info, trans->delayed_ref_updates);
> >  	num_bytes += btrfs_calc_delayed_ref_csum_bytes(fs_info,
> >  						       trans->delayed_ref_csum_deletions);
> > @@ -1261,6 +1264,7 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
> >  {
> >  	struct btrfs_delayed_ref_root *delayed_refs = &trans->delayed_refs;
> >  	struct btrfs_fs_info *fs_info = trans->fs_info;
> > +	bool testing = btrfs_is_testing(fs_info);
> >  
> >  	spin_lock(&delayed_refs->lock);
> >  	while (true) {
> > @@ -1290,7 +1294,7 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
> >  		spin_unlock(&delayed_refs->lock);
> >  		mutex_unlock(&head->mutex);
> >  
> > -		if (pin_bytes) {
> > +		if (!testing && pin_bytes) {
> >  			struct btrfs_block_group *bg;
> >  
> >  			bg = btrfs_lookup_block_group(fs_info, head->bytenr);
> > @@ -1322,12 +1326,16 @@ void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans)
> >  			btrfs_error_unpin_extent_range(fs_info, head->bytenr,
> >  				head->bytenr + head->num_bytes - 1);
> >  		}
> > -		btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs, head);
> > +		if (!testing)
> > +			btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs,
> > +							  head);
> >  		btrfs_put_delayed_ref_head(head);
> >  		cond_resched();
> >  		spin_lock(&delayed_refs->lock);
> >  	}
> > -	btrfs_qgroup_destroy_extent_records(trans);
> > +
> > +	if (!testing)
> 
> What is the principle that decides whether something is appropriate for
> testing mode?

The principle is "do I care about this right now for the test I'm writing" ;).

The reality is that our codebase isn't well suited for unit testing.  I added
the self tests over a decade ago, we didn't have kunit or any of the
infrastructure/experience to do proper unit testing.  We could prioritize
removing the btrfs_is_testing() stuff, we know where it's all used, and we could
rework the code to be more testable now that we know where the sore spots are.
Unfortunately right now it's more a balance between getting things tested and
making it as painless as possible.  Thanks,

Josef

Re: [PATCH 2/2] btrfs: add delayed ref self tests

[David Sterba]

On Thu, Nov 14, 2024 at 02:57:49PM -0500, Josef Bacik wrote:
> +	node_check.root = FAKE_ROOT_OBJECTID;
> +	if (validate_ref_node(node, &node_check)) {
> +		test_err("node check failed");
> +		goto out;
> +	}
> +	delete_delayed_ref_node(head, node);
> +	ret = 0;
> +out:
> +	if (head)
> +		btrfs_unselect_ref_head(delayed_refs, head);
> +	btrfs_destroy_delayed_refs(trans->transaction);
> +	return ret;
> +}
> +
> +int btrfs_test_delayed_refs(u32 sectorsize, u32 nodesize)
> +{
> +	struct btrfs_transaction transaction;
> +	struct btrfs_trans_handle trans;

Build complains

  CC [M]  fs/btrfs/tests/delayed-refs-tests.o
fs/btrfs/tests/delayed-refs-tests.c: In function ‘btrfs_test_delayed_refs’:
fs/btrfs/tests/delayed-refs-tests.c:1012:1: warning: the frame size of 2056 bytes is larger than 2048 bytes [-Wframe-larger-than=]

Please change that to kmalloc so we don't get warning reports on configs that
bloat data structures. On release config the sizes are like 480 + 168, which is ok.

Re: [PATCH 2/2] btrfs: add delayed ref self tests

[David Sterba]

On Fri, Nov 15, 2024 at 07:33:05PM +0100, David Sterba wrote:
> On Thu, Nov 14, 2024 at 02:57:49PM -0500, Josef Bacik wrote:
> > +	node_check.root = FAKE_ROOT_OBJECTID;
> > +	if (validate_ref_node(node, &node_check)) {
> > +		test_err("node check failed");
> > +		goto out;
> > +	}
> > +	delete_delayed_ref_node(head, node);
> > +	ret = 0;
> > +out:
> > +	if (head)
> > +		btrfs_unselect_ref_head(delayed_refs, head);
> > +	btrfs_destroy_delayed_refs(trans->transaction);
> > +	return ret;
> > +}
> > +
> > +int btrfs_test_delayed_refs(u32 sectorsize, u32 nodesize)
> > +{
> > +	struct btrfs_transaction transaction;
> > +	struct btrfs_trans_handle trans;
> 
> Build complains
> 
>   CC [M]  fs/btrfs/tests/delayed-refs-tests.o
> fs/btrfs/tests/delayed-refs-tests.c: In function ‘btrfs_test_delayed_refs’:
> fs/btrfs/tests/delayed-refs-tests.c:1012:1: warning: the frame size of 2056 bytes is larger than 2048 bytes [-Wframe-larger-than=]
> 
> Please change that to kmalloc so we don't get warning reports on configs that
> bloat data structures. On release config the sizes are like 480 + 168, which is ok.

Fixed in for-next.

Re: [PATCH 2/2] btrfs: add delayed ref self tests

[Josef Bacik]

On Fri, Dec 06, 2024 at 08:51:00PM +0100, David Sterba wrote:
> On Fri, Nov 15, 2024 at 07:33:05PM +0100, David Sterba wrote:
> > On Thu, Nov 14, 2024 at 02:57:49PM -0500, Josef Bacik wrote:
> > > +	node_check.root = FAKE_ROOT_OBJECTID;
> > > +	if (validate_ref_node(node, &node_check)) {
> > > +		test_err("node check failed");
> > > +		goto out;
> > > +	}
> > > +	delete_delayed_ref_node(head, node);
> > > +	ret = 0;
> > > +out:
> > > +	if (head)
> > > +		btrfs_unselect_ref_head(delayed_refs, head);
> > > +	btrfs_destroy_delayed_refs(trans->transaction);
> > > +	return ret;
> > > +}
> > > +
> > > +int btrfs_test_delayed_refs(u32 sectorsize, u32 nodesize)
> > > +{
> > > +	struct btrfs_transaction transaction;
> > > +	struct btrfs_trans_handle trans;
> > 
> > Build complains
> > 
> >   CC [M]  fs/btrfs/tests/delayed-refs-tests.o
> > fs/btrfs/tests/delayed-refs-tests.c: In function ‘btrfs_test_delayed_refs’:
> > fs/btrfs/tests/delayed-refs-tests.c:1012:1: warning: the frame size of 2056 bytes is larger than 2048 bytes [-Wframe-larger-than=]
> > 
> > Please change that to kmalloc so we don't get warning reports on configs that
> > bloat data structures. On release config the sizes are like 480 + 168, which is ok.
> 
> Fixed in for-next.

Aaaand you're cleaning up all my messes, thanks for this,

Josef

Re: [PATCH 2/2] btrfs: add delayed ref self tests

[David Sterba]

On Thu, Nov 14, 2024 at 02:57:49PM -0500, Josef Bacik wrote:
> +++ b/fs/btrfs/tests/btrfs-tests.c
> @@ -142,6 +142,11 @@ struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize)
>  	fs_info->nodesize = nodesize;
>  	fs_info->sectorsize = sectorsize;
>  	fs_info->sectorsize_bits = ilog2(sectorsize);
> +
> +	/* CRC32C csum size. */
> +	fs_info->csum_size = 4;

While this is correct it would be cleaner to do btrfs_csum_type_size(BTRFS_CSUM_TYPE_CRC32)

Re: [PATCH 0/2] btrfs: add a delayed ref self test

[Boris Burkov]

On Thu, Nov 14, 2024 at 02:57:47PM -0500, Josef Bacik wrote:
> Hello,
> 
> I made a silly mistake when refactoring the delayed ref code to make it easier
> to understand, and that resulted in a whole lot of not fun trying to find what
> went wrong with boxes started falling over when we were deploying 6.11.  This
> style of bug is easy to catch with basic unit testing, so add a variety of unit
> tests for delayed refs to make sure I don't break things again.  One patch moves
> an important helper and exports it so that we can do the testing, the other
> patch is a giant patch of all the tests with a few changes that are necessary to
> make everything work.  I validated these work properly and catch a variety of
> bugs that I hand introduced to the delayed ref code.  Thanks,

These are great, thanks.

I brought up some high level but non-blocking thoughts/questions
on the test patch, but please feel free to add

Reviewed-by: Boris Burkov <boris@bur.io>

> 
> Josef Bacik (2):
>   btrfs: move select_delayed_ref and export it
>   btrfs: add delayed ref self tests
> 
>  fs/btrfs/Makefile                   |    2 +-
>  fs/btrfs/delayed-ref.c              |   41 +-
>  fs/btrfs/delayed-ref.h              |    2 +
>  fs/btrfs/extent-tree.c              |   26 +-
>  fs/btrfs/tests/btrfs-tests.c        |   18 +
>  fs/btrfs/tests/btrfs-tests.h        |    6 +
>  fs/btrfs/tests/delayed-refs-tests.c | 1012 +++++++++++++++++++++++++++
>  7 files changed, 1078 insertions(+), 29 deletions(-)
>  create mode 100644 fs/btrfs/tests/delayed-refs-tests.c
> 
> -- 
> 2.43.0
>