[PATCH v2 00/13] btrfs-progs: extent tree v2 global root support prep work

[PATCH v2 00/13] btrfs-progs: extent tree v2 global root support prep work

[Josef Bacik]

v1->v2:
- Added a "btrfs-progs: check: fix set_extent_dirty range" which I discovered
  when adding the next part of the changes.
- Dropped all of the global root stuff, pushing it into the extent tree v2
  specific changes.
- Reworked the super block root helper to take a level as suggested by Qu.

--- Original email ---

Hello,

This is a series of patches to do all the prep work needed to support extent
tree v2.  These patches are independent of the actual extent tree v2 support,
some of them are fixes, some of them are purely to pave the way for the global
root support.  These patches are mostly around stopping direct access of
->extent_root/->csum_root/->free_space_root, putting these roots into a rb_tree,
and changing the code to look up the roots in the rb_tree instead of accessing
them directly.  There are a variety of fixes to help make this easier, mostly
removing access to these roots that are strictly necessary.  Thanks,

Josef

Josef Bacik (13):
  btrfs-progs: check: fix set_extent_dirty range
  btrfs-progs: simplify btrfs_make_block_group
  btrfs-progs: check: don't walk down non fs-trees for qgroup check
  btrfs-progs: filesystem-show: close ctree once we're done
  btrfs-progs: add a helper for setting up a root node
  btrfs-progs: btrfs-shared: stop passing root to csum related functions
  btrfs-progs: check: stop passing csum root around
  btrfs-progs: stop accessing ->csum_root directly
  btrfs-progs: image: keep track of seen blocks when walking trees
  btrfs-progs: common: move btrfs_fix_block_accounting to repair.c
  btrfs-progs: check: abstract out the used marking helpers
  btrfs-progs: check: move btrfs_mark_used_tree_blocks to common
  btrfs-progs: mark reloc roots as used

 btrfs-corrupt-block.c       |   2 +-
 check/main.c                |  37 ++++----
 check/mode-common.c         | 124 ++-----------------------
 check/qgroup-verify.c       |   4 +
 cmds/filesystem.c           |  19 ++--
 cmds/rescue-chunk-recover.c |  17 ++--
 common/repair.c             | 176 ++++++++++++++++++++++++++++++++++++
 common/repair.h             |   3 +
 convert/main.c              |   2 +-
 image/main.c                |  12 +++
 kernel-shared/ctree.h       |   8 +-
 kernel-shared/disk-io.c     | 101 +++++++++++++--------
 kernel-shared/disk-io.h     |   2 +-
 kernel-shared/extent-tree.c | 117 ++++--------------------
 kernel-shared/file-item.c   |   6 +-
 mkfs/main.c                 |   3 +-
 mkfs/rootdir.c              |   2 +-
 17 files changed, 326 insertions(+), 309 deletions(-)

-- 
2.26.3

[PATCH v2 01/13] btrfs-progs: check: fix set_extent_dirty range

[Josef Bacik]

I screwed up a fix where we're setting the bytenr range as dirty when
marking all tree blocks used, I was looking at btrfs_pin_extent and put
->nodesize for end instead of the actual end, which is bytenr +
->nodesize - 1.  Fix this up so it's correct.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 check/mode-common.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/check/mode-common.c b/check/mode-common.c
index 0059672c..02807101 100644
--- a/check/mode-common.c
+++ b/check/mode-common.c
@@ -664,7 +664,10 @@ static int traverse_tree_blocks(struct extent_buffer *eb, int tree_root, int pin
 			if (ret)
 				return ret;
 		} else {
+			u64 end;
+
 			bytenr = btrfs_node_blockptr(eb, i);
+			end = bytenr + gfs_info->nodesize - 1;
 
 			/* If we aren't the tree root don't read the block */
 			if (level == 1 && !tree_root) {
@@ -672,8 +675,7 @@ static int traverse_tree_blocks(struct extent_buffer *eb, int tree_root, int pin
 					btrfs_pin_extent(gfs_info, bytenr,
 							 gfs_info->nodesize);
 				else
-					set_extent_dirty(tree, bytenr,
-							 gfs_info->nodesize);
+					set_extent_dirty(tree, bytenr, end);
 				continue;
 			}
 
-- 
2.26.3

[PATCH v2 02/13] btrfs-progs: simplify btrfs_make_block_group

[Josef Bacik]

This is doing the same work as insert_block_group_item, rework it to
call the helper instead.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 kernel-shared/extent-tree.c | 43 ++++++++++++++-----------------------
 1 file changed, 16 insertions(+), 27 deletions(-)

diff --git a/kernel-shared/extent-tree.c b/kernel-shared/extent-tree.c
index 8e0614e0..a918e5aa 100644
--- a/kernel-shared/extent-tree.c
+++ b/kernel-shared/extent-tree.c
@@ -2791,33 +2791,6 @@ btrfs_add_block_group(struct btrfs_fs_info *fs_info, u64 bytes_used, u64 type,
 	return cache;
 }
 
-int btrfs_make_block_group(struct btrfs_trans_handle *trans,
-			   struct btrfs_fs_info *fs_info, u64 bytes_used,
-			   u64 type, u64 chunk_offset, u64 size)
-{
-	int ret;
-	struct btrfs_root *extent_root = fs_info->extent_root;
-	struct btrfs_block_group *cache;
-	struct btrfs_block_group_item bgi;
-	struct btrfs_key key;
-
-	cache = btrfs_add_block_group(fs_info, bytes_used, type, chunk_offset,
-				      size);
-	btrfs_set_stack_block_group_used(&bgi, cache->used);
-	btrfs_set_stack_block_group_flags(&bgi, cache->flags);
-	btrfs_set_stack_block_group_chunk_objectid(&bgi,
-			BTRFS_FIRST_CHUNK_TREE_OBJECTID);
-	key.objectid = cache->start;
-	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
-	key.offset = cache->length;
-	ret = btrfs_insert_item(trans, extent_root, &key, &bgi, sizeof(bgi));
-	BUG_ON(ret);
-
-	add_block_group_free_space(trans, cache);
-
-	return 0;
-}
-
 static int insert_block_group_item(struct btrfs_trans_handle *trans,
 				   struct btrfs_block_group *block_group)
 {
@@ -2838,6 +2811,22 @@ static int insert_block_group_item(struct btrfs_trans_handle *trans,
 	return btrfs_insert_item(trans, root, &key, &bgi, sizeof(bgi));
 }
 
+int btrfs_make_block_group(struct btrfs_trans_handle *trans,
+			   struct btrfs_fs_info *fs_info, u64 bytes_used,
+			   u64 type, u64 chunk_offset, u64 size)
+{
+	struct btrfs_block_group *cache;
+	int ret;
+
+	cache = btrfs_add_block_group(fs_info, bytes_used, type, chunk_offset,
+				      size);
+	ret = insert_block_group_item(trans, cache);
+	if (ret)
+		return ret;
+	add_block_group_free_space(trans, cache);
+	return 0;
+}
+
 /*
  * This is for converter use only.
  *
-- 
2.26.3

[PATCH v2 03/13] btrfs-progs: check: don't walk down non fs-trees for qgroup check

[Josef Bacik]

We will lookup implied refs for every root id for every block that we
find when verifying qgroups, but we don't need to worry about non
fstrees, so skip them here.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 check/qgroup-verify.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/check/qgroup-verify.c b/check/qgroup-verify.c
index 14645c85..a514fee1 100644
--- a/check/qgroup-verify.c
+++ b/check/qgroup-verify.c
@@ -765,6 +765,10 @@ static int add_refs_for_implied(struct btrfs_fs_info *info, u64 bytenr,
 	struct btrfs_root *root;
 	struct btrfs_key key;
 
+	/* If this is a global tree skip it. */
+	if (!is_fstree(root_id))
+		return 0;
+
 	/* Tree reloc tree doesn't contribute qgroup, skip it */
 	if (root_id == BTRFS_TREE_RELOC_OBJECTID)
 		return 0;
-- 
2.26.3

[PATCH v2 04/13] btrfs-progs: filesystem-show: close ctree once we're done

[Josef Bacik]

Running with ASAN we won't pass the self tests because we leak the whole
fs_info with btrfs filesystem show.  Fix this by making sure we close
out the fs_info and clean up all of the memory and such.

Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 cmds/filesystem.c | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/cmds/filesystem.c b/cmds/filesystem.c
index 6a9e46d2..624d0288 100644
--- a/cmds/filesystem.c
+++ b/cmds/filesystem.c
@@ -655,6 +655,7 @@ static int cmd_filesystem_show(const struct cmd_struct *cmd,
 {
 	LIST_HEAD(all_uuids);
 	struct btrfs_fs_devices *fs_devices;
+	struct btrfs_root *root = NULL;
 	char *search = NULL;
 	int ret;
 	/* default, search both kernel and udev */
@@ -753,12 +754,8 @@ static int cmd_filesystem_show(const struct cmd_struct *cmd,
 
 devs_only:
 	if (type == BTRFS_ARG_REG) {
-		/*
-		 * Given input (search) is regular file.
-		 * We don't close the fs_info because it will free the device,
-		 * this is not a long-running process so it's fine
-		 */
-		if (open_ctree(search, btrfs_sb_offset(0), 0))
+		root = open_ctree(search, btrfs_sb_offset(0), 0);
+		if (root)
 			ret = 0;
 		else
 			ret = 1;
@@ -768,7 +765,7 @@ devs_only:
 
 	if (ret) {
 		error("blkid device scan returned %d", ret);
-		return 1;
+		goto out;
 	}
 
 	/*
@@ -779,13 +776,13 @@ devs_only:
 	ret = search_umounted_fs_uuids(&all_uuids, search, &found);
 	if (ret < 0) {
 		error("searching target device returned error %d", ret);
-		return 1;
+		goto out;
 	}
 
 	ret = map_seed_devices(&all_uuids);
 	if (ret) {
 		error("mapping seed devices returned error %d", ret);
-		return 1;
+		goto out;
 	}
 
 	list_for_each_entry(fs_devices, &all_uuids, list)
@@ -801,8 +798,10 @@ devs_only:
 		free_fs_devices(fs_devices);
 	}
 out:
+	if (root)
+		close_ctree(root);
 	free_seen_fsid(seen_fsid_hash);
-	return ret;
+	return !!ret;
 }
 static DEFINE_SIMPLE_COMMAND(filesystem_show, "show");
 
-- 
2.26.3

[PATCH v2 05/13] btrfs-progs: add a helper for setting up a root node

[Josef Bacik]

We use this pattern in a few places, and will use it more with different
roots in the future.  Extract out this helper to read the root nodes.

There is a behavior change here in that we're now checking the root
levels, whereas before we were not.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 kernel-shared/disk-io.c | 73 +++++++++++++++++++++++++----------------
 1 file changed, 45 insertions(+), 28 deletions(-)

diff --git a/kernel-shared/disk-io.c b/kernel-shared/disk-io.c
index 3b3c4523..7a797801 100644
--- a/kernel-shared/disk-io.c
+++ b/kernel-shared/disk-io.c
@@ -578,12 +578,31 @@ void btrfs_setup_root(struct btrfs_root *root, struct btrfs_fs_info *fs_info,
 	root->root_key.objectid = objectid;
 }
 
+static int read_root_node(struct btrfs_fs_info *fs_info,
+			  struct btrfs_root *root, u64 bytenr, u64 gen,
+			  int level)
+{
+	root->node = read_tree_block(fs_info, bytenr, gen);
+	if (!extent_buffer_uptodate(root->node))
+		goto err;
+	if (btrfs_header_level(root->node) != level) {
+		error("root [%llu %llu] level %d does not match %d\n",
+		      root->root_key.objectid, root->root_key.offset,
+		      btrfs_header_level(root->node), level);
+		goto err;
+	}
+	return 0;
+err:
+	free_extent_buffer(root->node);
+	root->node = NULL;
+	return -EIO;
+}
+
 static int find_and_setup_root(struct btrfs_root *tree_root,
 			       struct btrfs_fs_info *fs_info,
 			       u64 objectid, struct btrfs_root *root)
 {
 	int ret;
-	u64 generation;
 
 	btrfs_setup_root(root, fs_info, objectid);
 	ret = btrfs_find_last_root(tree_root, objectid,
@@ -591,13 +610,10 @@ static int find_and_setup_root(struct btrfs_root *tree_root,
 	if (ret)
 		return ret;
 
-	generation = btrfs_root_generation(&root->root_item);
-	root->node = read_tree_block(fs_info,
-			btrfs_root_bytenr(&root->root_item), generation);
-	if (!extent_buffer_uptodate(root->node))
-		return -EIO;
-
-	return 0;
+	return read_root_node(fs_info, root,
+			      btrfs_root_bytenr(&root->root_item),
+			      btrfs_root_generation(&root->root_item),
+			      btrfs_root_level(&root->root_item));
 }
 
 static int find_and_setup_log_root(struct btrfs_root *tree_root,
@@ -606,6 +622,7 @@ static int find_and_setup_log_root(struct btrfs_root *tree_root,
 {
 	u64 blocknr = btrfs_super_log_root(disk_super);
 	struct btrfs_root *log_root = malloc(sizeof(struct btrfs_root));
+	int ret;
 
 	if (!log_root)
 		return -ENOMEM;
@@ -615,20 +632,16 @@ static int find_and_setup_log_root(struct btrfs_root *tree_root,
 		return 0;
 	}
 
-	btrfs_setup_root(log_root, fs_info,
-			 BTRFS_TREE_LOG_OBJECTID);
-
-	log_root->node = read_tree_block(fs_info, blocknr,
-				     btrfs_super_generation(disk_super) + 1);
-
-	fs_info->log_root_tree = log_root;
-
-	if (!extent_buffer_uptodate(log_root->node)) {
-		free_extent_buffer(log_root->node);
+	btrfs_setup_root(log_root, fs_info, BTRFS_TREE_LOG_OBJECTID);
+	ret = read_root_node(fs_info, log_root, blocknr,
+			     btrfs_super_generation(disk_super) + 1,
+			     btrfs_super_log_root_level(disk_super));
+	if (ret) {
 		free(log_root);
 		fs_info->log_root_tree = NULL;
-		return -EIO;
+		return ret;
 	}
+	fs_info->log_root_tree = log_root;
 
 	return 0;
 }
@@ -704,9 +717,10 @@ out:
 		return ERR_PTR(ret);
 	}
 	generation = btrfs_root_generation(&root->root_item);
-	root->node = read_tree_block(fs_info,
-			btrfs_root_bytenr(&root->root_item), generation);
-	if (!extent_buffer_uptodate(root->node)) {
+	ret = read_root_node(fs_info, root,
+			     btrfs_root_bytenr(&root->root_item), generation,
+			     btrfs_root_level(&root->root_item));
+	if (ret) {
 		free(root);
 		return ERR_PTR(-EIO);
 	}
@@ -950,11 +964,13 @@ int btrfs_setup_all_roots(struct btrfs_fs_info *fs_info, u64 root_tree_bytenr,
 	struct btrfs_root *root;
 	struct btrfs_key key;
 	u64 generation;
+	int level;
 	int ret;
 
 	root = fs_info->tree_root;
 	btrfs_setup_root(root, fs_info, BTRFS_ROOT_TREE_OBJECTID);
 	generation = btrfs_super_generation(sb);
+	level = btrfs_super_root_level(sb);
 
 	if (!root_tree_bytenr && !(flags & OPEN_CTREE_BACKUP_ROOT)) {
 		root_tree_bytenr = btrfs_super_root(sb);
@@ -968,10 +984,12 @@ int btrfs_setup_all_roots(struct btrfs_fs_info *fs_info, u64 root_tree_bytenr,
 		backup = fs_info->super_copy->super_roots + index;
 		root_tree_bytenr = btrfs_backup_tree_root(backup);
 		generation = btrfs_backup_tree_root_gen(backup);
+		level = btrfs_backup_tree_root_level(backup);
 	}
 
-	root->node = read_tree_block(fs_info, root_tree_bytenr, generation);
-	if (!extent_buffer_uptodate(root->node)) {
+	ret = read_root_node(fs_info, root, root_tree_bytenr, generation,
+			     level);
+	if (ret) {
 		fprintf(stderr, "Couldn't read tree root\n");
 		return -EIO;
 	}
@@ -1179,10 +1197,9 @@ int btrfs_setup_chunk_tree_and_device_map(struct btrfs_fs_info *fs_info,
 	else
 		generation = 0;
 
-	fs_info->chunk_root->node = read_tree_block(fs_info,
-						    chunk_root_bytenr,
-						    generation);
-	if (!extent_buffer_uptodate(fs_info->chunk_root->node)) {
+	ret = read_root_node(fs_info, fs_info->chunk_root, chunk_root_bytenr,
+			     generation, btrfs_super_chunk_root_level(sb));
+	if (ret) {
 		if (fs_info->ignore_chunk_tree_error) {
 			warning("cannot read chunk root, continue anyway");
 			fs_info->chunk_root = NULL;
-- 
2.26.3

[PATCH v2 06/13] btrfs-progs: btrfs-shared: stop passing root to csum related functions

[Josef Bacik]

We are going to need to start looking up the csum root based on the
bytenr with extent tree v2.  To that end stop passing the root to the
csum related functions so that can be done in the helper functions
themselves.

There's an unrelated deletion of a function prototype that no longer
exists, if desired I can break that out from this patch.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 check/main.c              | 4 ++--
 convert/main.c            | 1 -
 kernel-shared/ctree.h     | 6 +-----
 kernel-shared/file-item.c | 4 ++--
 mkfs/rootdir.c            | 2 +-
 5 files changed, 6 insertions(+), 11 deletions(-)

diff --git a/check/main.c b/check/main.c
index 235a9bab..08810c5f 100644
--- a/check/main.c
+++ b/check/main.c
@@ -9484,8 +9484,8 @@ static int populate_csum(struct btrfs_trans_handle *trans,
 				       &sectorsize, 0);
 		if (ret)
 			break;
-		ret = btrfs_csum_file_block(trans, csum_root, start + len,
-					    start + offset, buf, sectorsize);
+		ret = btrfs_csum_file_block(trans, start + len, start + offset,
+					    buf, sectorsize);
 		if (ret)
 			break;
 		offset += sectorsize;
diff --git a/convert/main.c b/convert/main.c
index 223eebad..7f33d4e1 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -187,7 +187,6 @@ static int csum_disk_extent(struct btrfs_trans_handle *trans,
 		if (ret)
 			break;
 		ret = btrfs_csum_file_block(trans,
-					    root->fs_info->csum_root,
 					    disk_bytenr + num_bytes,
 					    disk_bytenr + offset,
 					    buffer, blocksize);
diff --git a/kernel-shared/ctree.h b/kernel-shared/ctree.h
index 3c0743cc..73904255 100644
--- a/kernel-shared/ctree.h
+++ b/kernel-shared/ctree.h
@@ -2847,12 +2847,8 @@ int btrfs_insert_file_extent(struct btrfs_trans_handle *trans,
 int btrfs_insert_inline_extent(struct btrfs_trans_handle *trans,
 				struct btrfs_root *root, u64 objectid,
 				u64 offset, const char *buffer, size_t size);
-int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
-			  struct btrfs_root *root, u64 alloc_end,
+int btrfs_csum_file_block(struct btrfs_trans_handle *trans, u64 alloc_end,
 			  u64 bytenr, char *data, size_t len);
-int btrfs_csum_truncate(struct btrfs_trans_handle *trans,
-			struct btrfs_root *root, struct btrfs_path *path,
-			u64 isize);
 
 /* uuid-tree.c, interface for mounted mounted filesystem */
 int btrfs_lookup_uuid_subvol_item(int fd, const u8 *uuid, u64 *subvol_id);
diff --git a/kernel-shared/file-item.c b/kernel-shared/file-item.c
index c910e27e..5bf6aab4 100644
--- a/kernel-shared/file-item.c
+++ b/kernel-shared/file-item.c
@@ -183,9 +183,9 @@ fail:
 }
 
 int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
-			  struct btrfs_root *root, u64 alloc_end,
-			  u64 bytenr, char *data, size_t len)
+			  u64 alloc_end, u64 bytenr, char *data, size_t len)
 {
+	struct btrfs_root *root = trans->fs_info->csum_root;
 	int ret = 0;
 	struct btrfs_key file_key;
 	struct btrfs_key found_key;
diff --git a/mkfs/rootdir.c b/mkfs/rootdir.c
index 16ff257a..92be32ea 100644
--- a/mkfs/rootdir.c
+++ b/mkfs/rootdir.c
@@ -403,7 +403,7 @@ again:
 		 * we're doing the csum before we record the extent, but
 		 * that's ok
 		 */
-		ret = btrfs_csum_file_block(trans, root->fs_info->csum_root,
+		ret = btrfs_csum_file_block(trans,
 				first_block + bytes_read + sectorsize,
 				first_block + bytes_read,
 				eb->data, sectorsize);
-- 
2.26.3

Re: [PATCH v2 06/13] btrfs-progs: btrfs-shared: stop passing root to csum related functions

[David Sterba]

On Wed, Nov 10, 2021 at 03:07:57PM -0500, Josef Bacik wrote:
> We are going to need to start looking up the csum root based on the
> bytenr with extent tree v2.  To that end stop passing the root to the
> csum related functions so that can be done in the helper functions
> themselves.
> 
> There's an unrelated deletion of a function prototype that no longer
> exists, if desired I can break that out from this patch.

I'd care namely about things that could break bisectability or are
completely unrelated.

> --- a/kernel-shared/ctree.h
> +++ b/kernel-shared/ctree.h
> @@ -2847,12 +2847,8 @@ int btrfs_insert_file_extent(struct btrfs_trans_handle *trans,
>  int btrfs_insert_inline_extent(struct btrfs_trans_handle *trans,
>  				struct btrfs_root *root, u64 objectid,
>  				u64 offset, const char *buffer, size_t size);
> -int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
> -			  struct btrfs_root *root, u64 alloc_end,
> +int btrfs_csum_file_block(struct btrfs_trans_handle *trans, u64 alloc_end,
>  			  u64 bytenr, char *data, size_t len);
> -int btrfs_csum_truncate(struct btrfs_trans_handle *trans,
> -			struct btrfs_root *root, struct btrfs_path *path,
> -			u64 isize);

Here it's touching the code near and is trivial, so mentioning it in the
changelog is fine.

[PATCH v2 07/13] btrfs-progs: check: stop passing csum root around

[Josef Bacik]

We pass the csum root from way high in the call chain in check down to
where we actually need it.  However we can just get it from the fs_info
in these places, so clean up the functions to skip passing around the
csum root needlessly.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 check/main.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)

diff --git a/check/main.c b/check/main.c
index 08810c5f..22306cf4 100644
--- a/check/main.c
+++ b/check/main.c
@@ -9494,9 +9494,9 @@ static int populate_csum(struct btrfs_trans_handle *trans,
 }
 
 static int fill_csum_tree_from_one_fs_root(struct btrfs_trans_handle *trans,
-				      struct btrfs_root *csum_root,
-				      struct btrfs_root *cur_root)
+					   struct btrfs_root *cur_root)
 {
+	struct btrfs_root *csum_root = gfs_info->csum_root;
 	struct btrfs_path path;
 	struct btrfs_key key;
 	struct extent_buffer *node;
@@ -9557,8 +9557,7 @@ out:
 	return ret;
 }
 
-static int fill_csum_tree_from_fs(struct btrfs_trans_handle *trans,
-				  struct btrfs_root *csum_root)
+static int fill_csum_tree_from_fs(struct btrfs_trans_handle *trans)
 {
 	struct btrfs_path path;
 	struct btrfs_root *tree_root = gfs_info->tree_root;
@@ -9598,8 +9597,7 @@ static int fill_csum_tree_from_fs(struct btrfs_trans_handle *trans,
 				key.objectid);
 			goto out;
 		}
-		ret = fill_csum_tree_from_one_fs_root(trans, csum_root,
-				cur_root);
+		ret = fill_csum_tree_from_one_fs_root(trans, cur_root);
 		if (ret < 0)
 			goto out;
 next:
@@ -9617,10 +9615,10 @@ out:
 	return ret;
 }
 
-static int fill_csum_tree_from_extent(struct btrfs_trans_handle *trans,
-				      struct btrfs_root *csum_root)
+static int fill_csum_tree_from_extent(struct btrfs_trans_handle *trans)
 {
 	struct btrfs_root *extent_root = gfs_info->extent_root;
+	struct btrfs_root *csum_root = gfs_info->csum_root;
 	struct btrfs_path path;
 	struct btrfs_extent_item *ei;
 	struct extent_buffer *leaf;
@@ -9690,13 +9688,12 @@ static int fill_csum_tree_from_extent(struct btrfs_trans_handle *trans,
  * will use fs/subvol trees to init the csum tree.
  */
 static int fill_csum_tree(struct btrfs_trans_handle *trans,
-			  struct btrfs_root *csum_root,
 			  int search_fs_tree)
 {
 	if (search_fs_tree)
-		return fill_csum_tree_from_fs(trans, csum_root);
+		return fill_csum_tree_from_fs(trans);
 	else
-		return fill_csum_tree_from_extent(trans, csum_root);
+		return fill_csum_tree_from_extent(trans);
 }
 
 static void free_roots_info_cache(void)
@@ -10700,8 +10697,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 				goto close_out;
 			}
 
-			ret = fill_csum_tree(trans, gfs_info->csum_root,
-					     init_extent_tree);
+			ret = fill_csum_tree(trans, init_extent_tree);
 			err |= !!ret;
 			if (ret) {
 				error("checksum tree refilling failed: %d", ret);
-- 
2.26.3

[PATCH v2 08/13] btrfs-progs: stop accessing ->csum_root directly

[Josef Bacik]

With extent tree v2 we will have per-block group checksums, so add a
helper to access the csum root and rename the fs_info csum_root to
_csum_root to catch all the places that are accessing it directly.
Convert everybody to use the helper except for internal things.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 btrfs-corrupt-block.c       |  2 +-
 check/main.c                | 15 ++++++++++-----
 check/mode-common.c         |  6 +++---
 cmds/rescue-chunk-recover.c | 17 ++++++++---------
 kernel-shared/ctree.h       |  2 +-
 kernel-shared/disk-io.c     | 28 +++++++++++++++++-----------
 kernel-shared/disk-io.h     |  2 +-
 kernel-shared/file-item.c   |  4 ++--
 mkfs/main.c                 |  3 ++-
 9 files changed, 45 insertions(+), 34 deletions(-)

diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
index c1624ee1..d7904b6f 100644
--- a/btrfs-corrupt-block.c
+++ b/btrfs-corrupt-block.c
@@ -1031,7 +1031,7 @@ static int delete_csum(struct btrfs_root *root, u64 bytenr, u64 bytes)
 	struct btrfs_trans_handle *trans;
 	int ret;
 
-	root = root->fs_info->csum_root;
+	root = btrfs_csum_root(root->fs_info, bytenr);
 	trans = btrfs_start_transaction(root, 1);
 	if (IS_ERR(trans)) {
 		fprintf(stderr, "Couldn't start transaction %ld\n",
diff --git a/check/main.c b/check/main.c
index 22306cf4..632dfba0 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5988,7 +5988,7 @@ static int check_csums(struct btrfs_root *root)
 	unsigned long leaf_offset;
 	bool verify_csum = !!check_data_csum;
 
-	root = gfs_info->csum_root;
+	root = btrfs_csum_root(gfs_info, 0);
 	if (!extent_buffer_uptodate(root->node)) {
 		fprintf(stderr, "No valid csum tree found\n");
 		return -ENOENT;
@@ -9496,7 +9496,7 @@ static int populate_csum(struct btrfs_trans_handle *trans,
 static int fill_csum_tree_from_one_fs_root(struct btrfs_trans_handle *trans,
 					   struct btrfs_root *cur_root)
 {
-	struct btrfs_root *csum_root = gfs_info->csum_root;
+	struct btrfs_root *csum_root;
 	struct btrfs_path path;
 	struct btrfs_key key;
 	struct extent_buffer *node;
@@ -9532,6 +9532,7 @@ static int fill_csum_tree_from_one_fs_root(struct btrfs_trans_handle *trans,
 		start = btrfs_file_extent_disk_bytenr(node, fi);
 		len = btrfs_file_extent_disk_num_bytes(node, fi);
 
+		csum_root = btrfs_csum_root(gfs_info, start);
 		ret = populate_csum(trans, csum_root, buf, start, len);
 		if (ret == -EEXIST)
 			ret = 0;
@@ -9618,7 +9619,7 @@ out:
 static int fill_csum_tree_from_extent(struct btrfs_trans_handle *trans)
 {
 	struct btrfs_root *extent_root = gfs_info->extent_root;
-	struct btrfs_root *csum_root = gfs_info->csum_root;
+	struct btrfs_root *csum_root;
 	struct btrfs_path path;
 	struct btrfs_extent_item *ei;
 	struct extent_buffer *leaf;
@@ -9668,6 +9669,7 @@ static int fill_csum_tree_from_extent(struct btrfs_trans_handle *trans)
 			continue;
 		}
 
+		csum_root = btrfs_csum_root(gfs_info, key.objectid);
 		ret = populate_csum(trans, csum_root, buf, key.objectid,
 				    key.offset);
 		if (ret)
@@ -10688,7 +10690,8 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 
 		if (init_csum_tree) {
 			printf("Reinitialize checksum tree\n");
-			ret = btrfs_fsck_reinit_root(trans, gfs_info->csum_root);
+			ret = btrfs_fsck_reinit_root(trans,
+						btrfs_csum_root(gfs_info, 0));
 			if (ret) {
 				error("checksum tree initialization failed: %d",
 						ret);
@@ -10719,7 +10722,9 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 		err |= !!ret;
 		goto close_out;
 	}
-	if (!extent_buffer_uptodate(gfs_info->csum_root->node)) {
+
+	root = btrfs_csum_root(gfs_info, 0);
+	if (!extent_buffer_uptodate(root->node)) {
 		error("critical: csum_root, unable to check the filesystem");
 		ret = -EIO;
 		err |= !!ret;
diff --git a/check/mode-common.c b/check/mode-common.c
index 02807101..47b141a7 100644
--- a/check/mode-common.c
+++ b/check/mode-common.c
@@ -287,6 +287,7 @@ out:
  */
 int count_csum_range(u64 start, u64 len, u64 *found)
 {
+	struct btrfs_root *csum_root = btrfs_csum_root(gfs_info, start);
 	struct btrfs_key key;
 	struct btrfs_path path;
 	struct extent_buffer *leaf;
@@ -302,8 +303,7 @@ int count_csum_range(u64 start, u64 len, u64 *found)
 	key.offset = start;
 	key.type = BTRFS_EXTENT_CSUM_KEY;
 
-	ret = btrfs_search_slot(NULL, gfs_info->csum_root,
-				&key, &path, 0, 0);
+	ret = btrfs_search_slot(NULL, csum_root, &key, &path, 0, 0);
 	if (ret < 0)
 		goto out;
 	if (ret > 0 && path.slots[0] > 0) {
@@ -317,7 +317,7 @@ int count_csum_range(u64 start, u64 len, u64 *found)
 	while (len > 0) {
 		leaf = path.nodes[0];
 		if (path.slots[0] >= btrfs_header_nritems(leaf)) {
-			ret = btrfs_next_leaf(gfs_info->csum_root, &path);
+			ret = btrfs_next_leaf(csum_root, &path);
 			if (ret > 0)
 				break;
 			else if (ret < 0)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 03c7f48f..115d91d7 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1801,7 +1801,7 @@ static int btrfs_rebuild_chunk_stripes(struct recover_control *rc,
 	return ret;
 }
 
-static int next_csum(struct btrfs_root *root,
+static int next_csum(struct btrfs_root *csum_root,
 		     struct extent_buffer **leaf,
 		     struct btrfs_path *path,
 		     int *slot,
@@ -1811,10 +1811,9 @@ static int next_csum(struct btrfs_root *root,
 		     struct btrfs_key *key)
 {
 	int ret = 0;
-	struct btrfs_root *csum_root = root->fs_info->csum_root;
 	struct btrfs_csum_item *csum_item;
-	u32 blocksize = root->fs_info->sectorsize;
-	u16 csum_size = btrfs_super_csum_size(root->fs_info->super_copy);
+	u32 blocksize = csum_root->fs_info->sectorsize;
+	u16 csum_size = btrfs_super_csum_size(csum_root->fs_info->super_copy);
 	int csums_in_item = btrfs_item_size_nr(*leaf, *slot) / csum_size;
 
 	if (*csum_offset >= csums_in_item) {
@@ -1994,7 +1993,6 @@ static int rebuild_raid_data_chunk_stripes(struct recover_control *rc,
 	LIST_HEAD(unordered);
 	LIST_HEAD(candidates);
 
-	csum_root = root->fs_info->csum_root;
 	btrfs_init_path(&path);
 	list_splice_init(&chunk->dextents, &candidates);
 again:
@@ -2005,6 +2003,7 @@ again:
 	key.type = BTRFS_EXTENT_CSUM_KEY;
 	key.offset = start;
 
+	csum_root = btrfs_csum_root(root->fs_info, start);
 	ret = btrfs_search_slot(NULL, csum_root, &key, &path, 0, 0);
 	if (ret < 0) {
 		fprintf(stderr, "Search csum failed(%d)\n", ret);
@@ -2022,8 +2021,8 @@ again:
 			} else if (ret > 0) {
 				slot = btrfs_header_nritems(leaf) - 1;
 				btrfs_item_key_to_cpu(leaf, &key, slot);
-				if (item_end_offset(root, &key, leaf, slot)
-								> start) {
+				if (item_end_offset(csum_root, &key,
+						    leaf, slot) > start) {
 					csum_offset = start - key.offset;
 					csum_offset /= blocksize;
 					goto next_csum;
@@ -2061,8 +2060,8 @@ again:
 			goto out;
 	}
 next_csum:
-	ret = next_csum(root, &leaf, &path, &slot, &csum_offset, &tree_csum,
-			end, &key);
+	ret = next_csum(csum_root, &leaf, &path, &slot, &csum_offset,
+			&tree_csum, end, &key);
 	if (ret < 0) {
 		fprintf(stderr, "Fetch csum failed\n");
 		goto fail_out;
diff --git a/kernel-shared/ctree.h b/kernel-shared/ctree.h
index 73904255..d2d1a006 100644
--- a/kernel-shared/ctree.h
+++ b/kernel-shared/ctree.h
@@ -1161,7 +1161,7 @@ struct btrfs_fs_info {
 	struct btrfs_root *tree_root;
 	struct btrfs_root *chunk_root;
 	struct btrfs_root *dev_root;
-	struct btrfs_root *csum_root;
+	struct btrfs_root *_csum_root;
 	struct btrfs_root *quota_root;
 	struct btrfs_root *free_space_root;
 	struct btrfs_root *uuid_root;
diff --git a/kernel-shared/disk-io.c b/kernel-shared/disk-io.c
index 7a797801..f157442d 100644
--- a/kernel-shared/disk-io.c
+++ b/kernel-shared/disk-io.c
@@ -755,6 +755,12 @@ int btrfs_fs_roots_compare_roots(struct rb_node *node1, struct rb_node *node2)
 	return btrfs_fs_roots_compare_objectids(node1, (void *)&root->objectid);
 }
 
+struct btrfs_root *btrfs_csum_root(struct btrfs_fs_info *fs_info,
+				   u64 bytenr)
+{
+	return fs_info->_csum_root;
+}
+
 struct btrfs_root *btrfs_read_fs_root(struct btrfs_fs_info *fs_info,
 				      struct btrfs_key *location)
 {
@@ -772,7 +778,7 @@ struct btrfs_root *btrfs_read_fs_root(struct btrfs_fs_info *fs_info,
 	if (location->objectid == BTRFS_DEV_TREE_OBJECTID)
 		return fs_info->dev_root;
 	if (location->objectid == BTRFS_CSUM_TREE_OBJECTID)
-		return fs_info->csum_root;
+		return btrfs_csum_root(fs_info, location->offset);
 	if (location->objectid == BTRFS_UUID_TREE_OBJECTID)
 		return fs_info->uuid_root ? fs_info->uuid_root : ERR_PTR(-ENOENT);
 	if (location->objectid == BTRFS_QUOTA_TREE_OBJECTID)
@@ -808,7 +814,7 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info)
 	free(fs_info->extent_root);
 	free(fs_info->chunk_root);
 	free(fs_info->dev_root);
-	free(fs_info->csum_root);
+	free(fs_info->_csum_root);
 	free(fs_info->free_space_root);
 	free(fs_info->uuid_root);
 	free(fs_info->super_copy);
@@ -828,7 +834,7 @@ struct btrfs_fs_info *btrfs_new_fs_info(int writable, u64 sb_bytenr)
 	fs_info->extent_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->chunk_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->dev_root = calloc(1, sizeof(struct btrfs_root));
-	fs_info->csum_root = calloc(1, sizeof(struct btrfs_root));
+	fs_info->_csum_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->quota_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->free_space_root = calloc(1, sizeof(struct btrfs_root));
 	fs_info->uuid_root = calloc(1, sizeof(struct btrfs_root));
@@ -836,7 +842,7 @@ struct btrfs_fs_info *btrfs_new_fs_info(int writable, u64 sb_bytenr)
 
 	if (!fs_info->tree_root || !fs_info->extent_root ||
 	    !fs_info->chunk_root || !fs_info->dev_root ||
-	    !fs_info->csum_root || !fs_info->quota_root ||
+	    !fs_info->_csum_root || !fs_info->quota_root ||
 	    !fs_info->free_space_root || !fs_info->uuid_root ||
 	    !fs_info->super_copy)
 		goto free_all;
@@ -1008,11 +1014,11 @@ int btrfs_setup_all_roots(struct btrfs_fs_info *fs_info, u64 root_tree_bytenr,
 	}
 	fs_info->dev_root->track_dirty = 1;
 
-	ret = setup_root_or_create_block(fs_info, flags, fs_info->csum_root,
+	ret = setup_root_or_create_block(fs_info, flags, fs_info->_csum_root,
 					 BTRFS_CSUM_TREE_OBJECTID, "csum");
 	if (ret)
 		return ret;
-	fs_info->csum_root->track_dirty = 1;
+	fs_info->_csum_root->track_dirty = 1;
 
 	ret = find_and_setup_root(root, fs_info, BTRFS_UUID_TREE_OBJECTID,
 				  fs_info->uuid_root);
@@ -1087,8 +1093,8 @@ void btrfs_release_all_roots(struct btrfs_fs_info *fs_info)
 		free_extent_buffer(fs_info->free_space_root->node);
 	if (fs_info->quota_root)
 		free_extent_buffer(fs_info->quota_root->node);
-	if (fs_info->csum_root)
-		free_extent_buffer(fs_info->csum_root->node);
+	if (fs_info->_csum_root)
+		free_extent_buffer(fs_info->_csum_root->node);
 	if (fs_info->dev_root)
 		free_extent_buffer(fs_info->dev_root->node);
 	if (fs_info->extent_root)
@@ -1869,11 +1875,11 @@ static void backup_super_roots(struct btrfs_fs_info *info)
 	btrfs_set_backup_dev_root_level(root_backup,
 				       btrfs_header_level(info->dev_root->node));
 
-	btrfs_set_backup_csum_root(root_backup, info->csum_root->node->start);
+	btrfs_set_backup_csum_root(root_backup, info->_csum_root->node->start);
 	btrfs_set_backup_csum_root_gen(root_backup,
-			       btrfs_header_generation(info->csum_root->node));
+			       btrfs_header_generation(info->_csum_root->node));
 	btrfs_set_backup_csum_root_level(root_backup,
-			       btrfs_header_level(info->csum_root->node));
+			       btrfs_header_level(info->_csum_root->node));
 
 	btrfs_set_backup_total_bytes(root_backup,
 			     btrfs_super_total_bytes(info->super_copy));
diff --git a/kernel-shared/disk-io.h b/kernel-shared/disk-io.h
index e113d842..38abbca5 100644
--- a/kernel-shared/disk-io.h
+++ b/kernel-shared/disk-io.h
@@ -217,5 +217,5 @@ int btrfs_fs_roots_compare_roots(struct rb_node *node1, struct rb_node *node2);
 struct btrfs_root *btrfs_create_tree(struct btrfs_trans_handle *trans,
 				     struct btrfs_fs_info *fs_info,
 				     u64 objectid);
-
+struct btrfs_root *btrfs_csum_root(struct btrfs_fs_info *fs_info, u64 bytenr);
 #endif
diff --git a/kernel-shared/file-item.c b/kernel-shared/file-item.c
index 5bf6aab4..12932e1d 100644
--- a/kernel-shared/file-item.c
+++ b/kernel-shared/file-item.c
@@ -185,7 +185,7 @@ fail:
 int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
 			  u64 alloc_end, u64 bytenr, char *data, size_t len)
 {
-	struct btrfs_root *root = trans->fs_info->csum_root;
+	struct btrfs_root *root = btrfs_csum_root(trans->fs_info, bytenr);
 	int ret = 0;
 	struct btrfs_key file_key;
 	struct btrfs_key found_key;
@@ -401,7 +401,7 @@ int btrfs_del_csums(struct btrfs_trans_handle *trans, u64 bytenr, u64 len)
 	int ret;
 	u16 csum_size = btrfs_super_csum_size(trans->fs_info->super_copy);
 	int blocksize = trans->fs_info->sectorsize;
-	struct btrfs_root *csum_root = trans->fs_info->csum_root;
+	struct btrfs_root *csum_root = btrfs_csum_root(trans->fs_info, bytenr);
 
 
 	path = btrfs_alloc_path();
diff --git a/mkfs/main.c b/mkfs/main.c
index ce86a0db..c6cc51e9 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -268,6 +268,7 @@ static int recow_roots(struct btrfs_trans_handle *trans,
 		       struct btrfs_root *root)
 {
 	struct btrfs_fs_info *info = root->fs_info;
+	struct btrfs_root *csum_root = btrfs_csum_root(info, 0);
 	int ret;
 
 	ret = __recow_root(trans, info->fs_root);
@@ -285,7 +286,7 @@ static int recow_roots(struct btrfs_trans_handle *trans,
 	ret = __recow_root(trans, info->dev_root);
 	if (ret)
 		return ret;
-	ret = __recow_root(trans, info->csum_root);
+	ret = __recow_root(trans, csum_root);
 	if (ret)
 		return ret;
 	if (info->free_space_root) {
-- 
2.26.3

[PATCH v2 09/13] btrfs-progs: image: keep track of seen blocks when walking trees

[Josef Bacik]

Extent tree v2 no longer tracks all allocated blocks on the file system,
so we'll have to default to walking trees to generate metadata images.
There's an annoying drawback with walking trees with btrfs-image where
we'll happily copy multiple blocks over and over again if there are
snapshots.  Fix this by keeping track of blocks we've seen and simply
skipping blocks that we've already queued up for copying.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 image/main.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/image/main.c b/image/main.c
index dbce17e7..57e0cb6c 100644
--- a/image/main.c
+++ b/image/main.c
@@ -93,6 +93,8 @@ struct metadump_struct {
 	pthread_cond_t cond;
 	struct rb_root name_tree;
 
+	struct extent_io_tree seen;
+
 	struct list_head list;
 	struct list_head ordered;
 	size_t num_items;
@@ -461,6 +463,7 @@ static void metadump_destroy(struct metadump_struct *md, int num_threads)
 		free(name->sub);
 		free(name);
 	}
+	extent_io_tree_cleanup(&md->seen);
 }
 
 static int metadump_init(struct metadump_struct *md, struct btrfs_root *root,
@@ -476,6 +479,7 @@ static int metadump_init(struct metadump_struct *md, struct btrfs_root *root,
 	memset(md, 0, sizeof(*md));
 	INIT_LIST_HEAD(&md->list);
 	INIT_LIST_HEAD(&md->ordered);
+	extent_io_tree_init(&md->seen);
 	md->root = root;
 	md->out = out;
 	md->pending_start = (u64)-1;
@@ -771,6 +775,14 @@ static int copy_tree_blocks(struct btrfs_root *root, struct extent_buffer *eb,
 	int i = 0;
 	int ret;
 
+	bytenr = btrfs_header_bytenr(eb);
+	if (test_range_bit(&metadump->seen, bytenr,
+			   bytenr + fs_info->nodesize - 1, EXTENT_DIRTY, 1))
+		return 0;
+
+	set_extent_dirty(&metadump->seen, bytenr,
+			 bytenr + fs_info->nodesize - 1);
+
 	ret = add_extent(btrfs_header_bytenr(eb), fs_info->nodesize,
 			 metadump, 0);
 	if (ret) {
-- 
2.26.3

[PATCH v2 10/13] btrfs-progs: common: move btrfs_fix_block_accounting to repair.c

[Josef Bacik]

We have this helper sitting in extent-tree.c, but it's a repair
function.  I'm going to need to make changes to this for extent-tree-v2
and would rather this live outside of the code we need to share with the
kernel.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 common/repair.c             | 75 +++++++++++++++++++++++++++++++++++++
 common/repair.h             |  1 +
 convert/main.c              |  1 +
 kernel-shared/extent-tree.c | 74 ------------------------------------
 4 files changed, 77 insertions(+), 74 deletions(-)

diff --git a/common/repair.c b/common/repair.c
index 6eed7cec..a5ba43e2 100644
--- a/common/repair.c
+++ b/common/repair.c
@@ -17,6 +17,7 @@
  */
 
 #include "kernel-shared/ctree.h"
+#include "kernel-shared/transaction.h"
 #include "common/extent-cache.h"
 #include "common/utils.h"
 #include "common/repair.h"
@@ -50,3 +51,77 @@ int btrfs_add_corrupt_extent_record(struct btrfs_fs_info *info,
 	return ret;
 }
 
+/*
+ * Fixup block accounting. The initial block accounting created by
+ * make_block_groups isn't accuracy in this case.
+ */
+int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
+{
+	int ret = 0;
+	int slot;
+	u64 start = 0;
+	u64 bytes_used = 0;
+	struct btrfs_path path;
+	struct btrfs_key key;
+	struct extent_buffer *leaf;
+	struct btrfs_block_group *cache;
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct btrfs_root *root = fs_info->extent_root;
+
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
+	while(1) {
+		cache = btrfs_lookup_first_block_group(fs_info, start);
+		if (!cache)
+			break;
+
+		start = cache->start + cache->length;
+		cache->used = 0;
+		cache->space_info->bytes_used = 0;
+		if (list_empty(&cache->dirty_list))
+			list_add_tail(&cache->dirty_list, &trans->dirty_bgs);
+	}
+
+	btrfs_init_path(&path);
+	key.offset = 0;
+	key.objectid = 0;
+	key.type = BTRFS_EXTENT_ITEM_KEY;
+	ret = btrfs_search_slot(trans, root->fs_info->extent_root,
+				&key, &path, 0, 0);
+	if (ret < 0)
+		return ret;
+	while(1) {
+		leaf = path.nodes[0];
+		slot = path.slots[0];
+		if (slot >= btrfs_header_nritems(leaf)) {
+			ret = btrfs_next_leaf(root, &path);
+			if (ret < 0)
+				return ret;
+			if (ret > 0)
+				break;
+			leaf = path.nodes[0];
+			slot = path.slots[0];
+		}
+		btrfs_item_key_to_cpu(leaf, &key, slot);
+		if (key.type == BTRFS_EXTENT_ITEM_KEY) {
+			bytes_used += key.offset;
+			ret = btrfs_update_block_group(trans,
+				  key.objectid, key.offset, 1, 0);
+			BUG_ON(ret);
+		} else if (key.type == BTRFS_METADATA_ITEM_KEY) {
+			bytes_used += fs_info->nodesize;
+			ret = btrfs_update_block_group(trans,
+				  key.objectid, fs_info->nodesize, 1, 0);
+			if (ret)
+				goto out;
+		}
+		path.slots[0]++;
+	}
+	btrfs_set_super_bytes_used(root->fs_info->super_copy, bytes_used);
+	ret = 0;
+out:
+	btrfs_release_path(&path);
+	return ret;
+}
diff --git a/common/repair.h b/common/repair.h
index d1794610..4e1fa3e7 100644
--- a/common/repair.h
+++ b/common/repair.h
@@ -32,5 +32,6 @@ struct btrfs_corrupt_block {
 int btrfs_add_corrupt_extent_record(struct btrfs_fs_info *info,
 				    struct btrfs_key *first_key,
 				    u64 start, u64 len, int level);
+int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans);
 
 #endif
diff --git a/convert/main.c b/convert/main.c
index 7f33d4e1..d480f166 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -109,6 +109,7 @@
 #include "common/device-scan.h"
 #include "common/box.h"
 #include "common/open-utils.h"
+#include "common/repair.h"
 
 extern const struct btrfs_convert_operations ext2_convert_ops;
 extern const struct btrfs_convert_operations reiserfs_convert_ops;
diff --git a/kernel-shared/extent-tree.c b/kernel-shared/extent-tree.c
index a918e5aa..402904d1 100644
--- a/kernel-shared/extent-tree.c
+++ b/kernel-shared/extent-tree.c
@@ -3247,80 +3247,6 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
-/*
- * Fixup block accounting. The initial block accounting created by
- * make_block_groups isn't accuracy in this case.
- */
-int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
-{
-	int ret = 0;
-	int slot;
-	u64 start = 0;
-	u64 bytes_used = 0;
-	struct btrfs_path path;
-	struct btrfs_key key;
-	struct extent_buffer *leaf;
-	struct btrfs_block_group *cache;
-	struct btrfs_fs_info *fs_info = trans->fs_info;
-	struct btrfs_root *root = fs_info->extent_root;
-
-	ret = btrfs_run_delayed_refs(trans, -1);
-	if (ret)
-		return ret;
-
-	while(1) {
-		cache = btrfs_lookup_first_block_group(fs_info, start);
-		if (!cache)
-			break;
-
-		start = cache->start + cache->length;
-		cache->used = 0;
-		cache->space_info->bytes_used = 0;
-		if (list_empty(&cache->dirty_list))
-			list_add_tail(&cache->dirty_list, &trans->dirty_bgs);
-	}
-
-	btrfs_init_path(&path);
-	key.offset = 0;
-	key.objectid = 0;
-	key.type = BTRFS_EXTENT_ITEM_KEY;
-	ret = btrfs_search_slot(trans, root->fs_info->extent_root,
-				&key, &path, 0, 0);
-	if (ret < 0)
-		return ret;
-	while(1) {
-		leaf = path.nodes[0];
-		slot = path.slots[0];
-		if (slot >= btrfs_header_nritems(leaf)) {
-			ret = btrfs_next_leaf(root, &path);
-			if (ret < 0)
-				return ret;
-			if (ret > 0)
-				break;
-			leaf = path.nodes[0];
-			slot = path.slots[0];
-		}
-		btrfs_item_key_to_cpu(leaf, &key, slot);
-		if (key.type == BTRFS_EXTENT_ITEM_KEY) {
-			bytes_used += key.offset;
-			ret = btrfs_update_block_group(trans,
-				  key.objectid, key.offset, 1, 0);
-			BUG_ON(ret);
-		} else if (key.type == BTRFS_METADATA_ITEM_KEY) {
-			bytes_used += fs_info->nodesize;
-			ret = btrfs_update_block_group(trans,
-				  key.objectid, fs_info->nodesize, 1, 0);
-			if (ret)
-				goto out;
-		}
-		path.slots[0]++;
-	}
-	btrfs_set_super_bytes_used(root->fs_info->super_copy, bytes_used);
-	ret = 0;
-out:
-	btrfs_release_path(&path);
-	return ret;
-}
 
 static void __get_extent_size(struct btrfs_root *root, struct btrfs_path *path,
 			      u64 *start, u64 *len)
-- 
2.26.3

[PATCH v2 11/13] btrfs-progs: check: abstract out the used marking helpers

[Josef Bacik]

We will walk all referenced tree blocks during check in order to avoid
writing over any referenced blocks during fsck.  However in the future
we're going to need to do this for things like fixing block group
accounting with extent tree v2.  This is because extent tree v2 will not
refer to all of the allocated blocks in the extent tree.  Refactor the
code some to allow us to send down an arbitrary extent_io_tree so we can
use this helper for any case where we need to figure out where all the
used space is on an extent tree v2 file system.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 check/mode-common.c | 54 ++++++++++++++++++---------------------------
 1 file changed, 22 insertions(+), 32 deletions(-)

diff --git a/check/mode-common.c b/check/mode-common.c
index 47b141a7..5dc41e8d 100644
--- a/check/mode-common.c
+++ b/check/mode-common.c
@@ -599,23 +599,21 @@ void reset_cached_block_groups()
 	}
 }
 
-static int traverse_tree_blocks(struct extent_buffer *eb, int tree_root, int pin)
+static int traverse_tree_blocks(struct extent_io_tree *tree,
+				struct extent_buffer *eb, int tree_root)
 {
+	struct btrfs_fs_info *fs_info = eb->fs_info;
 	struct extent_buffer *tmp;
 	struct btrfs_root_item *ri;
 	struct btrfs_key key;
-	struct extent_io_tree *tree;
 	u64 bytenr;
 	int level = btrfs_header_level(eb);
 	int nritems;
 	int ret;
 	int i;
 	u64 end = eb->start + eb->len;
+	bool pin = tree == &fs_info->pinned_extents;
 
-	if (pin)
-		tree = &gfs_info->pinned_extents;
-	else
-		tree = gfs_info->excluded_extents;
 	/*
 	 * If we have pinned/excluded this block before, don't do it again.
 	 * This can not only avoid forever loop with broken filesystem
@@ -625,7 +623,7 @@ static int traverse_tree_blocks(struct extent_buffer *eb, int tree_root, int pin
 		return 0;
 
 	if (pin)
-		btrfs_pin_extent(gfs_info, eb->start, eb->len);
+		btrfs_pin_extent(fs_info, eb->start, eb->len);
 	else
 		set_extent_dirty(tree, eb->start, end - 1);
 
@@ -654,12 +652,12 @@ static int traverse_tree_blocks(struct extent_buffer *eb, int tree_root, int pin
 			 * in, but for now this doesn't actually use the root so
 			 * just pass in extent_root.
 			 */
-			tmp = read_tree_block(gfs_info, bytenr, 0);
+			tmp = read_tree_block(fs_info, bytenr, 0);
 			if (!extent_buffer_uptodate(tmp)) {
 				fprintf(stderr, "Error reading root block\n");
 				return -EIO;
 			}
-			ret = traverse_tree_blocks(tmp, 0, pin);
+			ret = traverse_tree_blocks(tree, tmp, 0);
 			free_extent_buffer(tmp);
 			if (ret)
 				return ret;
@@ -667,24 +665,24 @@ static int traverse_tree_blocks(struct extent_buffer *eb, int tree_root, int pin
 			u64 end;
 
 			bytenr = btrfs_node_blockptr(eb, i);
-			end = bytenr + gfs_info->nodesize - 1;
+			end = bytenr + fs_info->nodesize - 1;
 
 			/* If we aren't the tree root don't read the block */
 			if (level == 1 && !tree_root) {
 				if (pin)
-					btrfs_pin_extent(gfs_info, bytenr,
-							 gfs_info->nodesize);
+					btrfs_pin_extent(fs_info, bytenr,
+							 fs_info->nodesize);
 				else
 					set_extent_dirty(tree, bytenr, end);
 				continue;
 			}
 
-			tmp = read_tree_block(gfs_info, bytenr, 0);
+			tmp = read_tree_block(fs_info, bytenr, 0);
 			if (!extent_buffer_uptodate(tmp)) {
 				fprintf(stderr, "Error reading tree block\n");
 				return -EIO;
 			}
-			ret = traverse_tree_blocks(tmp, tree_root, pin);
+			ret = traverse_tree_blocks(tree, tmp, tree_root);
 			free_extent_buffer(tmp);
 			if (ret)
 				return ret;
@@ -694,30 +692,25 @@ static int traverse_tree_blocks(struct extent_buffer *eb, int tree_root, int pin
 	return 0;
 }
 
-static int pin_down_tree_blocks(struct extent_buffer *eb, int tree_root)
-{
-	return traverse_tree_blocks(eb, tree_root, 1);
-}
-
-int pin_metadata_blocks(void)
+int btrfs_mark_used_tree_blocks(struct btrfs_fs_info *fs_info,
+				struct extent_io_tree *tree)
 {
 	int ret;
 
-	ret = pin_down_tree_blocks(gfs_info->chunk_root->node, 0);
-	if (ret)
-		return ret;
-
-	return pin_down_tree_blocks(gfs_info->tree_root->node, 1);
+	ret = traverse_tree_blocks(tree, fs_info->chunk_root->node, 0);
+	if (!ret)
+		ret = traverse_tree_blocks(tree, fs_info->tree_root->node, 1);
+	return ret;
 }
 
-static int exclude_tree_blocks(struct extent_buffer *eb, int tree_root)
+int pin_metadata_blocks(void)
 {
-	return traverse_tree_blocks(eb, tree_root, 0);
+	return btrfs_mark_used_tree_blocks(gfs_info,
+					   &gfs_info->pinned_extents);
 }
 
 int exclude_metadata_blocks(void)
 {
-	int ret;
 	struct extent_io_tree *excluded_extents;
 
 	excluded_extents = malloc(sizeof(*excluded_extents));
@@ -726,10 +719,7 @@ int exclude_metadata_blocks(void)
 	extent_io_tree_init(excluded_extents);
 	gfs_info->excluded_extents = excluded_extents;
 
-	ret = exclude_tree_blocks(gfs_info->chunk_root->node, 0);
-	if (ret)
-		return ret;
-	return exclude_tree_blocks(gfs_info->tree_root->node, 1);
+	return btrfs_mark_used_tree_blocks(gfs_info, excluded_extents);
 }
 
 void cleanup_excluded_extents(void)
-- 
2.26.3

[PATCH v2 12/13] btrfs-progs: check: move btrfs_mark_used_tree_blocks to common

[Josef Bacik]

This is going to be used for the extent tree v2 stuff more commonly, so
move it out so that it is accessible from everywhere that we need it.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 check/mode-common.c | 104 -------------------------------------------
 common/repair.c     | 105 ++++++++++++++++++++++++++++++++++++++++++++
 common/repair.h     |   2 +
 3 files changed, 107 insertions(+), 104 deletions(-)

diff --git a/check/mode-common.c b/check/mode-common.c
index 5dc41e8d..56377840 100644
--- a/check/mode-common.c
+++ b/check/mode-common.c
@@ -599,110 +599,6 @@ void reset_cached_block_groups()
 	}
 }
 
-static int traverse_tree_blocks(struct extent_io_tree *tree,
-				struct extent_buffer *eb, int tree_root)
-{
-	struct btrfs_fs_info *fs_info = eb->fs_info;
-	struct extent_buffer *tmp;
-	struct btrfs_root_item *ri;
-	struct btrfs_key key;
-	u64 bytenr;
-	int level = btrfs_header_level(eb);
-	int nritems;
-	int ret;
-	int i;
-	u64 end = eb->start + eb->len;
-	bool pin = tree == &fs_info->pinned_extents;
-
-	/*
-	 * If we have pinned/excluded this block before, don't do it again.
-	 * This can not only avoid forever loop with broken filesystem
-	 * but also give us some speedups.
-	 */
-	if (test_range_bit(tree, eb->start, end - 1, EXTENT_DIRTY, 0))
-		return 0;
-
-	if (pin)
-		btrfs_pin_extent(fs_info, eb->start, eb->len);
-	else
-		set_extent_dirty(tree, eb->start, end - 1);
-
-	nritems = btrfs_header_nritems(eb);
-	for (i = 0; i < nritems; i++) {
-		if (level == 0) {
-			bool is_extent_root;
-			btrfs_item_key_to_cpu(eb, &key, i);
-			if (key.type != BTRFS_ROOT_ITEM_KEY)
-				continue;
-			/* Skip the extent root and reloc roots */
-			if (key.objectid == BTRFS_TREE_RELOC_OBJECTID ||
-			    key.objectid == BTRFS_DATA_RELOC_TREE_OBJECTID)
-				continue;
-			is_extent_root =
-				key.objectid == BTRFS_EXTENT_TREE_OBJECTID;
-			/* If pin, skip the extent root */
-			if (pin && is_extent_root)
-				continue;
-			ri = btrfs_item_ptr(eb, i, struct btrfs_root_item);
-			bytenr = btrfs_disk_root_bytenr(eb, ri);
-
-			/*
-			 * If at any point we start needing the real root we
-			 * will have to build a stump root for the root we are
-			 * in, but for now this doesn't actually use the root so
-			 * just pass in extent_root.
-			 */
-			tmp = read_tree_block(fs_info, bytenr, 0);
-			if (!extent_buffer_uptodate(tmp)) {
-				fprintf(stderr, "Error reading root block\n");
-				return -EIO;
-			}
-			ret = traverse_tree_blocks(tree, tmp, 0);
-			free_extent_buffer(tmp);
-			if (ret)
-				return ret;
-		} else {
-			u64 end;
-
-			bytenr = btrfs_node_blockptr(eb, i);
-			end = bytenr + fs_info->nodesize - 1;
-
-			/* If we aren't the tree root don't read the block */
-			if (level == 1 && !tree_root) {
-				if (pin)
-					btrfs_pin_extent(fs_info, bytenr,
-							 fs_info->nodesize);
-				else
-					set_extent_dirty(tree, bytenr, end);
-				continue;
-			}
-
-			tmp = read_tree_block(fs_info, bytenr, 0);
-			if (!extent_buffer_uptodate(tmp)) {
-				fprintf(stderr, "Error reading tree block\n");
-				return -EIO;
-			}
-			ret = traverse_tree_blocks(tree, tmp, tree_root);
-			free_extent_buffer(tmp);
-			if (ret)
-				return ret;
-		}
-	}
-
-	return 0;
-}
-
-int btrfs_mark_used_tree_blocks(struct btrfs_fs_info *fs_info,
-				struct extent_io_tree *tree)
-{
-	int ret;
-
-	ret = traverse_tree_blocks(tree, fs_info->chunk_root->node, 0);
-	if (!ret)
-		ret = traverse_tree_blocks(tree, fs_info->tree_root->node, 1);
-	return ret;
-}
-
 int pin_metadata_blocks(void)
 {
 	return btrfs_mark_used_tree_blocks(gfs_info,
diff --git a/common/repair.c b/common/repair.c
index a5ba43e2..e30fd8e8 100644
--- a/common/repair.c
+++ b/common/repair.c
@@ -18,6 +18,7 @@
 
 #include "kernel-shared/ctree.h"
 #include "kernel-shared/transaction.h"
+#include "kernel-shared/disk-io.h"
 #include "common/extent-cache.h"
 #include "common/utils.h"
 #include "common/repair.h"
@@ -51,6 +52,110 @@ int btrfs_add_corrupt_extent_record(struct btrfs_fs_info *info,
 	return ret;
 }
 
+static int traverse_tree_blocks(struct extent_io_tree *tree,
+				struct extent_buffer *eb, int tree_root)
+{
+	struct btrfs_fs_info *fs_info = eb->fs_info;
+	struct extent_buffer *tmp;
+	struct btrfs_root_item *ri;
+	struct btrfs_key key;
+	u64 bytenr;
+	int level = btrfs_header_level(eb);
+	int nritems;
+	int ret;
+	int i;
+	u64 end = eb->start + eb->len;
+	bool pin = tree == &fs_info->pinned_extents;
+
+	/*
+	 * If we have pinned/excluded this block before, don't do it again.
+	 * This can not only avoid forever loop with broken filesystem
+	 * but also give us some speedups.
+	 */
+	if (test_range_bit(tree, eb->start, end - 1, EXTENT_DIRTY, 0))
+		return 0;
+
+	if (pin)
+		btrfs_pin_extent(fs_info, eb->start, eb->len);
+	else
+		set_extent_dirty(tree, eb->start, end - 1);
+
+	nritems = btrfs_header_nritems(eb);
+	for (i = 0; i < nritems; i++) {
+		if (level == 0) {
+			bool is_extent_root;
+			btrfs_item_key_to_cpu(eb, &key, i);
+			if (key.type != BTRFS_ROOT_ITEM_KEY)
+				continue;
+			/* Skip the extent root and reloc roots */
+			if (key.objectid == BTRFS_TREE_RELOC_OBJECTID ||
+			    key.objectid == BTRFS_DATA_RELOC_TREE_OBJECTID)
+				continue;
+			is_extent_root =
+				key.objectid == BTRFS_EXTENT_TREE_OBJECTID;
+			/* If pin, skip the extent root */
+			if (pin && is_extent_root)
+				continue;
+			ri = btrfs_item_ptr(eb, i, struct btrfs_root_item);
+			bytenr = btrfs_disk_root_bytenr(eb, ri);
+
+			/*
+			 * If at any point we start needing the real root we
+			 * will have to build a stump root for the root we are
+			 * in, but for now this doesn't actually use the root so
+			 * just pass in extent_root.
+			 */
+			tmp = read_tree_block(fs_info, bytenr, 0);
+			if (!extent_buffer_uptodate(tmp)) {
+				fprintf(stderr, "Error reading root block\n");
+				return -EIO;
+			}
+			ret = traverse_tree_blocks(tree, tmp, 0);
+			free_extent_buffer(tmp);
+			if (ret)
+				return ret;
+		} else {
+			u64 end;
+
+			bytenr = btrfs_node_blockptr(eb, i);
+			end = bytenr + fs_info->nodesize - 1;
+
+			/* If we aren't the tree root don't read the block */
+			if (level == 1 && !tree_root) {
+				if (pin)
+					btrfs_pin_extent(fs_info, bytenr,
+							 fs_info->nodesize);
+				else
+					set_extent_dirty(tree, bytenr, end);
+				continue;
+			}
+
+			tmp = read_tree_block(fs_info, bytenr, 0);
+			if (!extent_buffer_uptodate(tmp)) {
+				fprintf(stderr, "Error reading tree block\n");
+				return -EIO;
+			}
+			ret = traverse_tree_blocks(tree, tmp, tree_root);
+			free_extent_buffer(tmp);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
+int btrfs_mark_used_tree_blocks(struct btrfs_fs_info *fs_info,
+				struct extent_io_tree *tree)
+{
+	int ret;
+
+	ret = traverse_tree_blocks(tree, fs_info->chunk_root->node, 0);
+	if (!ret)
+		ret = traverse_tree_blocks(tree, fs_info->tree_root->node, 1);
+	return ret;
+}
+
 /*
  * Fixup block accounting. The initial block accounting created by
  * make_block_groups isn't accuracy in this case.
diff --git a/common/repair.h b/common/repair.h
index 4e1fa3e7..793ebcd2 100644
--- a/common/repair.h
+++ b/common/repair.h
@@ -33,5 +33,7 @@ int btrfs_add_corrupt_extent_record(struct btrfs_fs_info *info,
 				    struct btrfs_key *first_key,
 				    u64 start, u64 len, int level);
 int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans);
+int btrfs_mark_used_tree_blocks(struct btrfs_fs_info *fs_info,
+				struct extent_io_tree *tree);
 
 #endif
-- 
2.26.3

[PATCH v2 13/13] btrfs-progs: mark reloc roots as used

[Josef Bacik]

btrfs_mark_used_tree_blocks skips the reloc roots for some reason, which
causes problems because these blocks are in use, and we use this helper
to determine if the block accounting is correct with extent tree v2.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 common/repair.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/common/repair.c b/common/repair.c
index e30fd8e8..176665bb 100644
--- a/common/repair.c
+++ b/common/repair.c
@@ -87,10 +87,6 @@ static int traverse_tree_blocks(struct extent_io_tree *tree,
 			btrfs_item_key_to_cpu(eb, &key, i);
 			if (key.type != BTRFS_ROOT_ITEM_KEY)
 				continue;
-			/* Skip the extent root and reloc roots */
-			if (key.objectid == BTRFS_TREE_RELOC_OBJECTID ||
-			    key.objectid == BTRFS_DATA_RELOC_TREE_OBJECTID)
-				continue;
 			is_extent_root =
 				key.objectid == BTRFS_EXTENT_TREE_OBJECTID;
 			/* If pin, skip the extent root */
-- 
2.26.3

Re: [PATCH v2 00/13] btrfs-progs: extent tree v2 global root support prep work

[Josef Bacik]

On Wed, Nov 10, 2021 at 03:07:51PM -0500, Josef Bacik wrote:
> v1->v2:
> - Added a "btrfs-progs: check: fix set_extent_dirty range" which I discovered
>   when adding the next part of the changes.
> - Dropped all of the global root stuff, pushing it into the extent tree v2
>   specific changes.
> - Reworked the super block root helper to take a level as suggested by Qu.
> 

Sorry I'm an idiot, this should be v3, not v2, I forgot I had refreshed things
and sent out another v2 already.  Thanks,

Josef