[PATCH 0/9] Remaining part of mkfs --rootdir rework

[PATCH 0/9] Remaining part of mkfs --rootdir rework

[Qu Wenruo]

Can be fetched from my github repo:
https://github.com/adam900710/btrfs-progs/tree/mkfs_rootdir_rework

Based on the following commit head of David's devel branch:
------
 commit af322ba5aa1dd0b2a3422e1c4acd8082948efa7b (david/devel)
 Author: Su Yue <suy.fnst@cn.fujitsu.com>
 Date:   Tue Nov 28 17:14:48 2017 +0800

    btrfs-progs: fi defrag: clean up duplicate code if find errors
    
    In function cmd_filesystem_defrag(), lines of code for error handling
    are duplicate and hard to expand in further.
    
    Create a jump label for errors.
    
    Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
    Signed-off-by: David Sterba <dsterba@suse.com>
------

Patch 1 is split from original patch.
Patch 2~5 are the patches rebased. Only minor conflicts.

Patch 6~7 are regression fix and its test case. Which prohibits
mkfs.btrfs --rootdir from creating new file.

Patch 8~9 are fix and test case for incorrect shrink behavior, which
will shrink the fs even --shrink is not specified.

Qu Wenruo (9):
  btrfs-progs: mkfs: Cleanup temporary chunks before filling rootdir
  btrfs-progs: mkfs: Don't use custom chunk allocator for rootdir
  btrfs-progs: mkfs/rootdir: Use over-reserve method to make size
    estimate easier
  btrfs-progs: mkfs/rootdir: Shrink fs for rootdir option
  btrfs-progs: mkfs: Separate shrink from rootdir
  btrfs-progs: mkfs: Fix regression preventing --rootdir to create file
  btrfs-progs: tests/mkfs: Introduce test case to check if mkfs rootdir
    can create new file
  btrfs-progs: mkfs: Use the whole file or block device to mkfs for
    rootdir
  btrfs-progs: tests/mkfs: Introduce test case to verify if mkfs.btrfs
    rootdir shrink behaves correctly

 Documentation/mkfs.btrfs.asciidoc                |  11 +
 mkfs/main.c                                      | 231 ++++++++++-----------
 mkfs/rootdir.c                                   | 249 +++++++++++++++++++----
 mkfs/rootdir.h                                   |   7 +-
 tests/mkfs-tests/011-rootdir-create-file/test.sh |  14 ++
 tests/mkfs-tests/012-rootdir-no-shrink/test.sh   |  38 ++++
 utils.c                                          |  17 +-
 utils.h                                          |   2 +
 8 files changed, 402 insertions(+), 167 deletions(-)
 create mode 100755 tests/mkfs-tests/011-rootdir-create-file/test.sh
 create mode 100755 tests/mkfs-tests/012-rootdir-no-shrink/test.sh

-- 
2.15.0

[PATCH 1/9] btrfs-progs: mkfs: Cleanup temporary chunks before filling rootdir

[Qu Wenruo]

Cleanup those temporary chunks should be as soon as possible, and it
should be especially before doing large tree operations, like filling
rootdir.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/mkfs/main.c b/mkfs/main.c
index 2da38cb9490e..a81718d1517d 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -1248,6 +1248,13 @@ raid_groups:
 		goto out;
 	}
 
+	ret = cleanup_temp_chunks(fs_info, &allocation, data_profile,
+				  metadata_profile, metadata_profile);
+	if (ret < 0) {
+		error("failed to cleanup temporary chunks: %d", ret);
+		goto out;
+	}
+
 	if (source_dir_set) {
 		trans = btrfs_start_transaction(root, 1);
 		BUG_ON(IS_ERR(trans));
@@ -1311,12 +1318,6 @@ raid_groups:
 			}
 		}
 	}
-	ret = cleanup_temp_chunks(fs_info, &allocation, data_profile,
-				  metadata_profile, metadata_profile);
-	if (ret < 0) {
-		error("failed to cleanup temporary chunks: %d", ret);
-		goto out;
-	}
 
 	if (verbose) {
 		char features_buf[64];
-- 
2.15.0

[PATCH 2/9] btrfs-progs: mkfs: Don't use custom chunk allocator for rootdir

[Qu Wenruo]

Remove these custom chunk allocator for mkfs.
Use generic btrfs chunk allocator instead.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c | 62 -------------------------------------------------------------
 1 file changed, 62 deletions(-)

diff --git a/mkfs/main.c b/mkfs/main.c
index a81718d1517d..716395c4b6b4 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -413,53 +413,6 @@ static char *parse_subvol_name(const char *input)
 	return strdup(input);
 }
 
-static int create_chunks(struct btrfs_trans_handle *trans,
-			 struct btrfs_root *root, u64 num_of_meta_chunks,
-			 u64 size_of_data,
-			 struct mkfs_allocation *allocation)
-{
-	struct btrfs_fs_info *fs_info = root->fs_info;
-	u64 chunk_start;
-	u64 chunk_size;
-	u64 meta_type = BTRFS_BLOCK_GROUP_METADATA;
-	u64 data_type = BTRFS_BLOCK_GROUP_DATA;
-	u64 minimum_data_chunk_size = SZ_8M;
-	u64 i;
-	int ret;
-
-	for (i = 0; i < num_of_meta_chunks; i++) {
-		ret = btrfs_alloc_chunk(trans, fs_info,
-					&chunk_start, &chunk_size, meta_type);
-		if (ret)
-			return ret;
-		ret = btrfs_make_block_group(trans, fs_info, 0,
-					     meta_type, BTRFS_FIRST_CHUNK_TREE_OBJECTID,
-					     chunk_start, chunk_size);
-		allocation->metadata += chunk_size;
-		if (ret)
-			return ret;
-		set_extent_dirty(&root->fs_info->free_space_cache,
-				 chunk_start, chunk_start + chunk_size - 1);
-	}
-
-	if (size_of_data < minimum_data_chunk_size)
-		size_of_data = minimum_data_chunk_size;
-
-	ret = btrfs_alloc_data_chunk(trans, fs_info,
-				     &chunk_start, size_of_data, data_type, 0);
-	if (ret)
-		return ret;
-	ret = btrfs_make_block_group(trans, fs_info, 0,
-				     data_type, BTRFS_FIRST_CHUNK_TREE_OBJECTID,
-				     chunk_start, size_of_data);
-	allocation->data += size_of_data;
-	if (ret)
-		return ret;
-	set_extent_dirty(&root->fs_info->free_space_cache,
-			 chunk_start, chunk_start + size_of_data - 1);
-	return ret;
-}
-
 static int zero_output_file(int out_fd, u64 size)
 {
 	int loop_num;
@@ -1256,21 +1209,6 @@ raid_groups:
 	}
 
 	if (source_dir_set) {
-		trans = btrfs_start_transaction(root, 1);
-		BUG_ON(IS_ERR(trans));
-		ret = create_chunks(trans, root,
-				    num_of_meta_chunks, size_of_data,
-				    &allocation);
-		if (ret) {
-			error("unable to create chunks: %d", ret);
-			goto out;
-		}
-		ret = btrfs_commit_transaction(trans, root);
-		if (ret) {
-			error("transaction commit failed: %d", ret);
-			goto out;
-		}
-
 		if (subvol_name_set) {
 			u64 dirid, objectid;
 			struct btrfs_root *file_root;
-- 
2.15.0

[PATCH 3/9] btrfs-progs: mkfs/rootdir: Use over-reserve method to make size estimate easier

[Qu Wenruo]

Use an easier method to calculate the estimate device for mkfs.btrfs
--rootdir.

The new method will over-estimate, but should ensure we won't encounter
ENOSPC.

It relies on the following data to estimate:
1) number of inodes
   for metadata chunk size
2) rounded up data size of each regular inode
   for data chunk size.

Total meta chunk size = round_up(nr_inode * (PATH_MAX * 3 + sectorsize),
min_chunk_size) * profile_multiplier

PATH_MAX is the maximum size possible for INODE_REF/DIR_INDEX/DIR_ITEM.
Sectorsize is the maximum size possible for inline extent.
min_chunk_size is 8M for SINGLE, and 32M for DUP, get from
btrfs_alloc_chunk().
profile_multiplier is 1 for Single, 2 for DUP.

Total data chunk size is much easier.
Total data chunk size = round_up(total_data_usage, min_chunk_size) *
profile_multiplier

Total_data_usage is the sum of *rounded up* size of each regular inode
use.
min_chunk_size is 8M for SINGLE, 64M for DUP, get from
btrfS_alloc_chunk().
Same profile_multiplier for meta.

This over-estimate calculate is, of course, over-estimate.
But since we will later shrink the fs to its real usage, it doesn't
matter much now.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c    | 109 ++++++++++++++++++++++++++--------------------------
 mkfs/rootdir.c | 119 +++++++++++++++++++++++++++++++++++++++------------------
 mkfs/rootdir.h |   5 +--
 3 files changed, 139 insertions(+), 94 deletions(-)

diff --git a/mkfs/main.c b/mkfs/main.c
index 716395c4b6b4..eb49182ebe81 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -747,8 +747,6 @@ int main(int argc, char **argv)
 	int subvol_name_set = 0;
 	char *source_dir = NULL;
 	int source_dir_set = 0;
-	u64 num_of_meta_chunks = 0;
-	u64 size_of_data = 0;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
 	int dev_cnt = 0;
@@ -977,6 +975,34 @@ int main(int argc, char **argv)
 
 	min_dev_size = btrfs_min_dev_size(nodesize, mixed, metadata_profile,
 					  data_profile);
+	/*
+	 * Enlarge the destination file or create new one, using the
+	 * size calculated from source dir.
+	 *
+	 * This must be done before minimal device size check.
+	 */
+	if (source_dir_set) {
+		fd = open(file, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR | S_IRGRP |
+			  S_IWGRP | S_IROTH);
+		if (fd < 0) {
+			error("unable to open %s: %s", file, strerror(errno));
+			goto error;
+		}
+
+		source_dir_size = btrfs_mkfs_size_dir(source_dir, sectorsize,
+				min_dev_size, metadata_profile, data_profile);
+		if (block_count < source_dir_size)
+			block_count = source_dir_size;
+		ret = zero_output_file(fd, block_count);
+		if (ret) {
+			error("unable to zero the output file");
+			close(fd);
+			goto error;
+		}
+		/* our "device" is the new image file */
+		dev_block_count = block_count;
+		close(fd);
+	}
 	/* Check device/block_count after the nodesize is determined */
 	if (block_count && block_count < min_dev_size) {
 		error("size %llu is too small to make a usable filesystem",
@@ -1010,51 +1036,28 @@ int main(int argc, char **argv)
 
 	dev_cnt--;
 
-	if (!source_dir_set) {
-		/*
-		 * open without O_EXCL so that the problem should not
-		 * occur by the following processing.
-		 * (btrfs_register_one_device() fails if O_EXCL is on)
-		 */
-		fd = open(file, O_RDWR);
-		if (fd < 0) {
-			error("unable to open %s: %s", file, strerror(errno));
-			goto error;
-		}
-		ret = btrfs_prepare_device(fd, file, &dev_block_count,
-				block_count,
-				(zero_end ? PREP_DEVICE_ZERO_END : 0) |
-				(discard ? PREP_DEVICE_DISCARD : 0) |
-				(verbose ? PREP_DEVICE_VERBOSE : 0));
-		if (ret) {
-			goto error;
-		}
-		if (block_count && block_count > dev_block_count) {
-			error("%s is smaller than requested size, expected %llu, found %llu",
-					file,
-					(unsigned long long)block_count,
-					(unsigned long long)dev_block_count);
-			goto error;
-		}
-	} else {
-		fd = open(file, O_CREAT | O_RDWR,
-				S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH);
-		if (fd < 0) {
-			error("unable to open %s: %s", file, strerror(errno));
-			goto error;
-		}
-
-		source_dir_size = btrfs_mkfs_size_dir(source_dir, sectorsize,
-					&num_of_meta_chunks, &size_of_data);
-		if(block_count < source_dir_size)
-			block_count = source_dir_size;
-		ret = zero_output_file(fd, block_count);
-		if (ret) {
-			error("unable to zero the output file");
-			goto error;
-		}
-		/* our "device" is the new image file */
-		dev_block_count = block_count;
+	/*
+	 * open without O_EXCL so that the problem should not
+	 * occur by the following processing.
+	 * (btrfs_register_one_device() fails if O_EXCL is on)
+	 */
+	fd = open(file, O_RDWR);
+	if (fd < 0) {
+		error("unable to open %s: %s", file, strerror(errno));
+		goto error;
+	}
+	ret = btrfs_prepare_device(fd, file, &dev_block_count,
+			block_count,
+			(zero_end ? PREP_DEVICE_ZERO_END : 0) |
+			(discard ? PREP_DEVICE_DISCARD : 0) |
+			(verbose ? PREP_DEVICE_VERBOSE : 0));
+	if (ret)
+		goto error;
+	if (block_count && block_count > dev_block_count) {
+		error("%s is smaller than requested size, expected %llu, found %llu",
+		      file, (unsigned long long)block_count,
+		      (unsigned long long)dev_block_count);
+		goto error;
 	}
 
 	/* To create the first block group and chunk 0 in make_btrfs */
@@ -1180,13 +1183,11 @@ int main(int argc, char **argv)
 	}
 
 raid_groups:
-	if (!source_dir_set) {
-		ret = create_raid_groups(trans, root, data_profile,
-				 metadata_profile, mixed, &allocation);
-		if (ret) {
-			error("unable to create raid groups: %d", ret);
-			goto out;
-		}
+	ret = create_raid_groups(trans, root, data_profile,
+			 metadata_profile, mixed, &allocation);
+	if (ret) {
+		error("unable to create raid groups: %d", ret);
+		goto out;
 	}
 
 	ret = create_tree(trans, root, BTRFS_DATA_RELOC_TREE_OBJECTID);
diff --git a/mkfs/rootdir.c b/mkfs/rootdir.c
index 0eb3f8f26139..f8cff44d06eb 100644
--- a/mkfs/rootdir.c
+++ b/mkfs/rootdir.c
@@ -33,19 +33,29 @@
 #include "transaction.h"
 #include "utils.h"
 #include "mkfs/rootdir.h"
+#include "mkfs/common.h"
 #include "send-utils.h"
 
-/*
- * This ignores symlinks with unreadable targets and subdirs that can't
- * be read.  It's a best-effort to give a rough estimate of the size of
- * a subdir.  It doesn't guarantee that prepopulating btrfs from this
- * tree won't still run out of space.
- */
-static u64 global_total_size;
-static u64 fs_block_size;
+static u32 fs_block_size;
 
 static u64 index_cnt = 2;
 
+/*
+ * Size estimate will be done using the following data:
+ * 1) Number of inodes
+ *    Since we will later shrink the fs, over-estimate is completely fine here
+ *    as long as our estimate ensure we can populate the image without ENOSPC.
+ *    So we only records how many inodes there is, and use the maximum space
+ *    usage for every inode.
+ *
+ * 2) Data space each (regular) inode uses
+ *    To estimate data chunk size.
+ *    Don't care if it can fit as inline extent.
+ *    Always round them up to sectorsize.
+ */
+static u64 ftw_meta_nr_inode;
+static u64 ftw_data_size;
+
 static int add_directory_items(struct btrfs_trans_handle *trans,
 			       struct btrfs_root *root, u64 objectid,
 			       ino_t parent_inum, const char *name,
@@ -685,53 +695,88 @@ out:
 static int ftw_add_entry_size(const char *fpath, const struct stat *st,
 			      int type)
 {
-	if (type == FTW_F || type == FTW_D)
-		global_total_size += round_up(st->st_size, fs_block_size);
+	/*
+	 * Failed to read dir, mostly due to EPERM.
+	 * Abort ASAP, so we don't need to populate the fs
+	 */
+	if (type == FTW_DNR || type == FTW_NS)
+		return -EPERM;
+
+	if (S_ISREG(st->st_mode))
+		ftw_data_size += round_up(st->st_size, fs_block_size);
+	ftw_meta_nr_inode++;
 
 	return 0;
 }
 
-u64 btrfs_mkfs_size_dir(const char *dir_name, u64 sectorsize,
-			u64 *num_of_meta_chunks_ret, u64 *size_of_data_ret)
+u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
+			u64 meta_profile, u64 data_profile)
 {
-	u64 dir_size = 0;
 	u64 total_size = 0;
 	int ret;
-	u64 default_chunk_size = SZ_8M;
-	u64 allocated_meta_size = SZ_8M;
-	u64 allocated_total_size = 20 * SZ_1M;	/* 20MB */
-	u64 num_of_meta_chunks = 0;
-	u64 num_of_data_chunks = 0;
-	u64 num_of_allocated_meta_chunks =
-			allocated_meta_size / default_chunk_size;
-
-	global_total_size = 0;
+
+	u64 meta_size = 0;	/* Based on @ftw_meta_nr_inode */
+	u64 meta_chunk_size = 0;/* Based on @meta_size */
+	u64 data_chunk_size = 0;/* Based on @ftw_data_size */
+
+	u64 meta_threshold = SZ_8M;
+	u64 data_threshold = SZ_8M;
+
+	float data_multipler = 1;
+	float meta_multipler = 1;
+
 	fs_block_size = sectorsize;
+	ftw_data_size = 0;
+	ftw_meta_nr_inode = 0;
 	ret = ftw(dir_name, ftw_add_entry_size, 10);
-	dir_size = global_total_size;
 	if (ret < 0) {
 		error("ftw subdir walk of %s failed: %s", dir_name,
 			strerror(errno));
 		exit(1);
 	}
 
-	num_of_data_chunks = (dir_size + default_chunk_size - 1) /
-		default_chunk_size;
 
-	num_of_meta_chunks = (dir_size / 2) / default_chunk_size;
-	if (((dir_size / 2) % default_chunk_size) != 0)
-		num_of_meta_chunks++;
-	if (num_of_meta_chunks <= num_of_allocated_meta_chunks)
-		num_of_meta_chunks = 0;
-	else
-		num_of_meta_chunks -= num_of_allocated_meta_chunks;
+	/*
+	 * Maximum metadata useage for every inode, which will be PATH_MAX
+	 * for the following items:
+	 * 1) DIR_ITEM
+	 * 2) DIR_INDEX
+	 * 3) INODE_REF
+	 *
+	 * Plus possible inline extent size, which is sectorsize.
+	 *
+	 * And finally, allow metadata usage to increase with data size.
+	 * Follow the old kernel 8:1 data:meta ratio.
+	 * This is especially important for --rootdir, as the file extent size
+	 * up limit is 1M, instead of 128M in kernel.
+	 * This can bump meta usage easily.
+	 */
+	meta_size = ftw_meta_nr_inode * (PATH_MAX * 3 + sectorsize) +
+		    ftw_data_size / 8;
 
-	total_size = allocated_total_size +
-		     (num_of_data_chunks * default_chunk_size) +
-		     (num_of_meta_chunks * default_chunk_size);
+	/* Minimal chunk size from btrfs_alloc_chunk(). */
+	if (meta_profile & BTRFS_BLOCK_GROUP_DUP) {
+		meta_threshold = SZ_32M;
+		meta_multipler = 2;
+	}
+	if (data_profile & BTRFS_BLOCK_GROUP_DUP) {
+		data_threshold = SZ_64M;
+		data_multipler = 2;
+	}
 
-	*num_of_meta_chunks_ret = num_of_meta_chunks;
-	*size_of_data_ret = num_of_data_chunks * default_chunk_size;
+	/*
+	 * Only when the usage is larger than the minimal chunk size (threshold)
+	 * we need to allocate new chunk, or the initial chunk in the image is
+	 * large enough.
+	 */
+	if (meta_size > meta_threshold)
+		meta_chunk_size = (round_up(meta_size, meta_threshold) -
+				   meta_threshold) * meta_multipler;
+	if (ftw_data_size > data_threshold)
+		data_chunk_size = (round_up(ftw_data_size, data_threshold) -
+				   data_threshold) * data_multipler;
+
+	total_size = data_chunk_size + meta_chunk_size + min_dev_size;
 	return total_size;
 }
 
diff --git a/mkfs/rootdir.h b/mkfs/rootdir.h
index c4b120c5d1e4..ada50ccb6ac4 100644
--- a/mkfs/rootdir.h
+++ b/mkfs/rootdir.h
@@ -30,7 +30,6 @@ struct directory_name_entry {
 
 int btrfs_mkfs_fill_dir(const char *source_dir, struct btrfs_root *root,
 			bool verbose);
-u64 btrfs_mkfs_size_dir(const char *dir_name, u64 sectorsize,
-			u64 *num_of_meta_chunks_ret, u64 *size_of_data_ret);
-
+u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
+			u64 meta_profile, u64 data_profile);
 #endif
-- 
2.15.0

[PATCH 4/9] btrfs-progs: mkfs/rootdir: Shrink fs for rootdir option

[Qu Wenruo]

Use the new dev extent based shrink method for rootdir option.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c    |   5 +++
 mkfs/rootdir.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 mkfs/rootdir.h |   1 +
 3 files changed, 117 insertions(+)

diff --git a/mkfs/main.c b/mkfs/main.c
index eb49182ebe81..eb3e5bb1f92e 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -1256,6 +1256,11 @@ raid_groups:
 				goto out;
 			}
 		}
+		ret = btrfs_mkfs_shrink_fs(fs_info, NULL);
+		if (ret < 0) {
+			error("error while shrinking filesystem: %d", ret);
+			goto out;
+		}
 	}
 
 	if (verbose) {
diff --git a/mkfs/rootdir.c b/mkfs/rootdir.c
index f8cff44d06eb..271167fe0c02 100644
--- a/mkfs/rootdir.c
+++ b/mkfs/rootdir.c
@@ -822,3 +822,114 @@ out:
 
 	return ret;
 }
+
+/*
+ * Set device size to @new_size.
+ *
+ * Only used for --rootdir option.
+ * We will need to reset the following values:
+ * 1) dev item in chunk tree
+ * 2) super->dev_item
+ * 3) super->total_bytes
+ */
+static int set_device_size(struct btrfs_fs_info *fs_info,
+			   struct btrfs_device *device, u64 new_size)
+{
+	struct btrfs_root *chunk_root = fs_info->chunk_root;
+	struct btrfs_trans_handle *trans;
+	struct btrfs_dev_item *di;
+	struct btrfs_path path;
+	struct btrfs_key key;
+	int ret;
+
+	/*
+	 * Update in-meory device->total_bytes, so that at trans commit time,
+	 * it super->dev_item will also get updated
+	 */
+	device->total_bytes = new_size;
+	btrfs_init_path(&path);
+
+	/* Update device item in chunk tree */
+	trans = btrfs_start_transaction(chunk_root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("failed to start transaction: %d (%s)", ret,
+			strerror(-ret));
+		return ret;
+	}
+	key.objectid = BTRFS_DEV_ITEMS_OBJECTID;
+	key.type = BTRFS_DEV_ITEM_KEY;
+	key.offset = device->devid;
+
+	ret = btrfs_search_slot(trans, chunk_root, &key, &path, 0, 1);
+	if (ret < 0)
+		goto err;
+	if (ret > 0)
+		ret = -ENOENT;
+	di = btrfs_item_ptr(path.nodes[0], path.slots[0],
+			    struct btrfs_dev_item);
+	btrfs_set_device_total_bytes(path.nodes[0], di, new_size);
+	btrfs_mark_buffer_dirty(path.nodes[0]);
+
+	/*
+	 * Update super->total_bytes, since it's only used for --rootdir,
+	 * there is only one device, just use the @new_size.
+	 */
+	btrfs_set_super_total_bytes(fs_info->super_copy, new_size);
+
+	/*
+	 * Commit transaction to reflect the updated super->total_bytes and
+	 * super->dev_item
+	 */
+	ret = btrfs_commit_transaction(trans, chunk_root);
+	if (ret < 0)
+		error("failed to commit current transaction: %d (%s)",
+			ret, strerror(-ret));
+	btrfs_release_path(&path);
+	return ret;
+
+err:
+	btrfs_release_path(&path);
+	/*
+	 * Commit trans here won't cause problem since the fs still has
+	 * bad magic, and something wrong already happened, we don't
+	 * care the return value anyway.
+	 */
+	btrfs_commit_transaction(trans, chunk_root);
+	return ret;
+}
+
+int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret)
+{
+	u64 new_size;
+	struct btrfs_device *device;
+	struct list_head *cur;
+	int nr_devs = 0;
+	int ret;
+
+	list_for_each(cur, &fs_info->fs_devices->devices)
+		nr_devs++;
+
+	if (nr_devs > 1) {
+		error("cannot shrink fs with more than 1 device");
+		return -ENOTTY;
+	}
+
+	ret = get_device_extent_end(fs_info, 1, &new_size);
+	if (ret < 0) {
+		error("failed to get minimal device size: %d (%s)",
+			ret, strerror(-ret));
+		return ret;
+	}
+
+	BUG_ON(!IS_ALIGNED(new_size, fs_info->sectorsize));
+
+	device = list_entry(fs_info->fs_devices->devices.next,
+			   struct btrfs_device, dev_list);
+	ret = set_device_size(fs_info, device, new_size);
+	if (ret < 0)
+		return ret;
+	if (new_size_ret)
+		*new_size_ret = new_size;
+	return ret;
+}
diff --git a/mkfs/rootdir.h b/mkfs/rootdir.h
index ada50ccb6ac4..91bebc507f02 100644
--- a/mkfs/rootdir.h
+++ b/mkfs/rootdir.h
@@ -32,4 +32,5 @@ int btrfs_mkfs_fill_dir(const char *source_dir, struct btrfs_root *root,
 			bool verbose);
 u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
 			u64 meta_profile, u64 data_profile);
+int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret);
 #endif
-- 
2.15.0

[PATCH 5/9] btrfs-progs: mkfs: Separate shrink from rootdir

[Qu Wenruo]

Make --shrink a separate option for --rootdir, and make it default to
off.
So this will cause less confusion.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 Documentation/mkfs.btrfs.asciidoc | 11 +++++++++++
 mkfs/main.c                       | 26 ++++++++++++++++++++------
 mkfs/rootdir.c                    | 21 ++++++++++++++++++++-
 mkfs/rootdir.h                    |  3 ++-
 4 files changed, 53 insertions(+), 8 deletions(-)

diff --git a/Documentation/mkfs.btrfs.asciidoc b/Documentation/mkfs.btrfs.asciidoc
index 62f5a5bebaee..ded798216a5b 100644
--- a/Documentation/mkfs.btrfs.asciidoc
+++ b/Documentation/mkfs.btrfs.asciidoc
@@ -106,6 +106,17 @@ Please see the mount option 'discard' for that in `btrfs`(5).
 *-r|--rootdir <rootdir>*::
 Populate the toplevel subvolume with files from 'rootdir'.  This does not
 require root permissions and does not mount the filesystem.
++
+NOTE: This option may enlarge the image or file to ensure it's large enough to
+contain the files from 'rootdir'.
+
+*--shrink*:
+Shrink the filesystem to its minimal size, only works with *-r|--rootdir*
+option.
++
+NOTE: If the destination is regular file, this option will also reduce the
+file size. Or it will only reduce the filesystem available space.
+Extra space will not be usable unless resized using 'btrfs filesystem resize'.
 
 *-O|--features <feature1>[,<feature2>...]*::
 A list of filesystem features turned on at mkfs time. Not all features are
diff --git a/mkfs/main.c b/mkfs/main.c
index eb3e5bb1f92e..64cf03289188 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -746,9 +746,11 @@ int main(int argc, char **argv)
 	char *subvol_name = NULL;
 	int subvol_name_set = 0;
 	char *source_dir = NULL;
-	int source_dir_set = 0;
+	bool source_dir_set = false;
+	bool shrink_rootdir = false;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
+	u64 shrink_size;
 	int dev_cnt = 0;
 	int saved_optind;
 	char fs_uuid[BTRFS_UUID_UNPARSED_SIZE] = { 0 };
@@ -758,6 +760,7 @@ int main(int argc, char **argv)
 
 	while(1) {
 		int c;
+		enum { GETOPT_VAL_SHRINK = 257 };
 		static const struct option long_options[] = {
 			{ "alloc-start", required_argument, NULL, 'A'},
 			{ "byte-count", required_argument, NULL, 'b' },
@@ -776,6 +779,7 @@ int main(int argc, char **argv)
 			{ "features", required_argument, NULL, 'O' },
 			{ "uuid", required_argument, NULL, 'U' },
 			{ "quiet", 0, NULL, 'q' },
+			{ "shrink", no_argument, NULL, GETOPT_VAL_SHRINK },
 			{ "help", no_argument, NULL, GETOPT_VAL_HELP },
 			{ NULL, 0, NULL, 0}
 		};
@@ -847,7 +851,7 @@ int main(int argc, char **argv)
 				break;
 			case 'r':
 				source_dir = optarg;
-				source_dir_set = 1;
+				source_dir_set = true;
 				break;
 			case 'U':
 				strncpy(fs_uuid, optarg,
@@ -859,6 +863,9 @@ int main(int argc, char **argv)
 			case 'q':
 				verbose = 0;
 				break;
+			case GETOPT_VAL_SHRINK:
+				shrink_rootdir = true;
+				break;
 			case GETOPT_VAL_HELP:
 			default:
 				print_usage(c != GETOPT_VAL_HELP);
@@ -886,6 +893,10 @@ int main(int argc, char **argv)
 		error("the option -r is limited to a single device");
 		goto error;
 	}
+	if (shrink_rootdir && !source_dir_set) {
+		error("the option --shrink can only be paired with -r");
+		goto error;
+	}
 
 	if (*fs_uuid) {
 		uuid_t dummy_uuid;
@@ -1256,10 +1267,13 @@ raid_groups:
 				goto out;
 			}
 		}
-		ret = btrfs_mkfs_shrink_fs(fs_info, NULL);
-		if (ret < 0) {
-			error("error while shrinking filesystem: %d", ret);
-			goto out;
+		if (shrink_rootdir) {
+			ret = btrfs_mkfs_shrink_fs(fs_info, &shrink_size,
+						   shrink_rootdir);
+			if (ret < 0) {
+				error("error while shrinking filesystem: %d", ret);
+				goto out;
+			}
 		}
 	}
 
diff --git a/mkfs/rootdir.c b/mkfs/rootdir.c
index 271167fe0c02..b91900834316 100644
--- a/mkfs/rootdir.c
+++ b/mkfs/rootdir.c
@@ -899,11 +899,13 @@ err:
 	return ret;
 }
 
-int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret)
+int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret,
+			 bool shrink_file_size)
 {
 	u64 new_size;
 	struct btrfs_device *device;
 	struct list_head *cur;
+	struct stat64 file_stat;
 	int nr_devs = 0;
 	int ret;
 
@@ -931,5 +933,22 @@ int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret)
 		return ret;
 	if (new_size_ret)
 		*new_size_ret = new_size;
+
+	if (shrink_file_size) {
+		ret = fstat64(device->fd, &file_stat);
+		if (ret < 0) {
+			error("failed to stat devid %llu: %s", device->devid,
+				strerror(errno));
+			return ret;
+		}
+		if (!S_ISREG(file_stat.st_mode))
+			return ret;
+		ret = ftruncate64(device->fd, new_size);
+		if (ret < 0) {
+			error("failed to truncate device file of devid %llu: %s",
+				device->devid, strerror(errno));
+			return ret;
+		}
+	}
 	return ret;
 }
diff --git a/mkfs/rootdir.h b/mkfs/rootdir.h
index 91bebc507f02..7ab53c292c9d 100644
--- a/mkfs/rootdir.h
+++ b/mkfs/rootdir.h
@@ -32,5 +32,6 @@ int btrfs_mkfs_fill_dir(const char *source_dir, struct btrfs_root *root,
 			bool verbose);
 u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
 			u64 meta_profile, u64 data_profile);
-int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret);
+int btrfs_mkfs_shrink_fs(struct btrfs_fs_info *fs_info, u64 *new_size_ret,
+			 bool shrink_file_size);
 #endif
-- 
2.15.0

[PATCH 6/9] btrfs-progs: mkfs: Fix regression preventing --rootdir to create file

[Qu Wenruo]

Commit 460e93f25754 ("btrfs-progs: mkfs: check the status of file at mkfs")
will try to check the file state before creating fs on it.

The check is mostly fine for normal mkfs case, while for --rootdir
option, it's allowed to create new file if destination file doesn't
exist.

Fix it by allowing non-existing file if --rootdir is specified.

Fixes: 460e93f25754 ("btrfs-progs: mkfs: check the status of file at mkfs")
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c |  6 ++++--
 utils.c     | 15 +++++++++++++++
 utils.h     |  1 +
 3 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/mkfs/main.c b/mkfs/main.c
index 64cf03289188..a34b73bfb845 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -733,7 +733,7 @@ int main(int argc, char **argv)
 	u32 stripesize = 4096;
 	int zero_end = 1;
 	int fd = -1;
-	int ret;
+	int ret = 0;
 	int close_ret;
 	int i;
 	int mixed = 0;
@@ -913,7 +913,9 @@ int main(int argc, char **argv)
 
 	while (dev_cnt-- > 0) {
 		file = argv[optind++];
-		if (is_block_device(file) == 1)
+		if (source_dir_set && is_path_exist(file) == 0)
+			ret = 0;
+		else if (is_block_device(file) == 1)
 			ret = test_dev_for_mkfs(file, force_overwrite);
 		else
 			ret = test_status_for_mkfs(file, force_overwrite);
diff --git a/utils.c b/utils.c
index 524f463d3140..22c137514592 100644
--- a/utils.c
+++ b/utils.c
@@ -456,6 +456,21 @@ static int is_reg_file(const char *path)
 	return S_ISREG(statbuf.st_mode);
 }
 
+int is_path_exist(const char *path)
+{
+	struct stat statbuf;
+	int ret;
+
+	ret = stat(path, &statbuf);
+	if (ret < 0) {
+		if (errno == ENOENT)
+			return 0;
+		else
+			return -errno;
+	}
+	return 1;
+}
+
 /*
  * This function checks if the given input parameter is
  * an uuid or a path
diff --git a/utils.h b/utils.h
index a82d46f6d7cc..860c25d3cdba 100644
--- a/utils.h
+++ b/utils.h
@@ -122,6 +122,7 @@ int set_label(const char *btrfs_dev, const char *label);
 char *__strncpy_null(char *dest, const char *src, size_t n);
 int is_block_device(const char *file);
 int is_mount_point(const char *file);
+int is_path_exist(const char *file);
 int check_arg_type(const char *input);
 int open_path_or_dev_mnt(const char *path, DIR **dirstream, int verbose);
 int btrfs_open(const char *path, DIR **dirstream, int verbose, int dir_only);
-- 
2.15.0

[PATCH 7/9] btrfs-progs: tests/mkfs: Introduce test case to check if mkfs rootdir can create new file

[Qu Wenruo]

To test regression 460e93f25754 ("btrfs-progs: mkfs: check the status of file at
mkfs").

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/mkfs-tests/011-rootdir-create-file/test.sh | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100755 tests/mkfs-tests/011-rootdir-create-file/test.sh

diff --git a/tests/mkfs-tests/011-rootdir-create-file/test.sh b/tests/mkfs-tests/011-rootdir-create-file/test.sh
new file mode 100755
index 000000000000..c6f3d74da885
--- /dev/null
+++ b/tests/mkfs-tests/011-rootdir-create-file/test.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+# Regression test for mkfs.btrfs --rootdir on non-exist file
+# Designed behavior is, it should create a new file if destination doesn't exist
+# Regression 460e93f25754 ("btrfs-progs: mkfs: check the status of file at mkfs")
+
+
+source "$TOP/tests/common"
+
+check_prereq mkfs.btrfs
+
+tmp=$(mktemp -d --tmpdir btrfs-progs-mkfs.rootdirXXXXXXX)
+run_check "$TOP/mkfs.btrfs" -f -r "$TOP/Documentation/" $tmp/new_file
+
+rm -rf -- "$tmp"
-- 
2.15.0

[PATCH 8/9] btrfs-progs: mkfs: Use the whole file or block device to mkfs for rootdir

[Qu Wenruo]

For --rootdir, even for large existing file or block device, it will
always shrink the result filesystem.

The problem is, mkfs.btrfs will try to calculate the dir size, and use
it as @block_count to mkfs, which makes the result filesystem shrinked.

Fix by trying to get the original block device or file size as
@block_count, so mkfs.btrfs can use the full file/block device for
--rootdir option.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c | 22 ++++++++++++++++++++--
 utils.c     |  2 +-
 utils.h     |  1 +
 3 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/mkfs/main.c b/mkfs/main.c
index a34b73bfb845..eeeba9292856 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -995,13 +995,31 @@ int main(int argc, char **argv)
 	 * This must be done before minimal device size check.
 	 */
 	if (source_dir_set) {
-		fd = open(file, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR | S_IRGRP |
-			  S_IWGRP | S_IROTH);
+		int oflags = O_RDWR;
+		struct stat statbuf;
+
+		if (is_path_exist(file) == 0)
+			oflags |= O_CREAT;
+
+		fd = open(file, oflags , S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP |
+					 S_IROTH);
 		if (fd < 0) {
 			error("unable to open %s: %s", file, strerror(errno));
 			goto error;
 		}
+		ret = fstat(fd, &statbuf);
+		if (ret < 0) {
+			error("unable to stat %s: %s", file, strerror(errno));
+			ret = -errno;
+			goto error;
+		}
 
+		/*
+		 * Block_count not specified, use file/dev size first.
+		 * Or we will always use source_dir_size calculated for mkfs.
+		 */
+		if (!block_count)
+			block_count = btrfs_device_size(fd, &statbuf);
 		source_dir_size = btrfs_mkfs_size_dir(source_dir, sectorsize,
 				min_dev_size, metadata_profile, data_profile);
 		if (block_count < source_dir_size)
diff --git a/utils.c b/utils.c
index 22c137514592..80071e23fe2b 100644
--- a/utils.c
+++ b/utils.c
@@ -447,7 +447,7 @@ int is_mount_point(const char *path)
 	return ret;
 }
 
-static int is_reg_file(const char *path)
+int is_reg_file(const char *path)
 {
 	struct stat statbuf;
 
diff --git a/utils.h b/utils.h
index 860c25d3cdba..bd7484fd3620 100644
--- a/utils.h
+++ b/utils.h
@@ -123,6 +123,7 @@ char *__strncpy_null(char *dest, const char *src, size_t n);
 int is_block_device(const char *file);
 int is_mount_point(const char *file);
 int is_path_exist(const char *file);
+int is_reg_file(const char *path);
 int check_arg_type(const char *input);
 int open_path_or_dev_mnt(const char *path, DIR **dirstream, int verbose);
 int btrfs_open(const char *path, DIR **dirstream, int verbose, int dir_only);
-- 
2.15.0

[PATCH 9/9] btrfs-progs: tests/mkfs: Introduce test case to verify if mkfs.btrfs rootdir shrink behaves correctly

[Qu Wenruo]

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/mkfs-tests/012-rootdir-no-shrink/test.sh | 38 ++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100755 tests/mkfs-tests/012-rootdir-no-shrink/test.sh

diff --git a/tests/mkfs-tests/012-rootdir-no-shrink/test.sh b/tests/mkfs-tests/012-rootdir-no-shrink/test.sh
new file mode 100755
index 000000000000..1dfa55b376f6
--- /dev/null
+++ b/tests/mkfs-tests/012-rootdir-no-shrink/test.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+# Test if mkfs.btrfs --rootdir will skip shrinking correctly
+
+source "$TOP/tests/common"
+
+check_prereq mkfs.btrfs
+
+setup_root_helper
+
+fs_size=$((512 * 1024 * 1024))
+bs=$((1024 * 1024))
+tmp=$(mktemp -d --tmpdir btrfs-progs-mkfs.rootdirXXXXXXX)
+
+prepare_test_dev $fs_size
+
+### no shrink case ###
+
+run_check "$TOP/mkfs.btrfs" -f -r $tmp "$TEST_DEV"
+run_check_mount_test_dev
+
+# We should be able to write at least half of the fs size data since
+# the fs is not shrunk
+run_check $SUDO_HELPER dd if=/dev/zero bs=$bs count=$(($fs_size / $bs / 2)) \
+	of="$TEST_MNT/file"
+
+run_check_umount_test_dev
+
+### shrink case ###
+run_check "$TOP/mkfs.btrfs" -f -r $tmp --shrink "$TEST_DEV"
+run_check_mount_test_dev
+
+run_mustfail "mkfs.btrfs for shrink rootdir" \
+	$SUDO_HELPER dd if=/dev/zero bs=$bs count=$(($fs_size / $bs / 2)) \
+	of="$TEST_MNT/file"
+
+run_check_umount_test_dev
+
+rm -rf -- "$tmp"
-- 
2.15.0

Re: [PATCH 6/9] btrfs-progs: mkfs: Fix regression preventing --rootdir to create file

[Misono, Tomohiro]

On 2017/11/29 18:16, Qu Wenruo wrote:
> Commit 460e93f25754 ("btrfs-progs: mkfs: check the status of file at mkfs")
> will try to check the file state before creating fs on it.
> 
> The check is mostly fine for normal mkfs case, while for --rootdir
> option, it's allowed to create new file if destination file doesn't
> exist.
> 
> Fix it by allowing non-existing file if --rootdir is specified.
> 
> Fixes: 460e93f25754 ("btrfs-progs: mkfs: check the status of file at mkfs")
> Signed-off-by: Qu Wenruo <wqu@suse.com>

Sorry, I didn't notice that. Thanks.
Reviewed-by: Tomohiro Misono <misono.tomohiro@jp.fujitsu.com>

> ---
>  mkfs/main.c |  6 ++++--
>  utils.c     | 15 +++++++++++++++
>  utils.h     |  1 +
>  3 files changed, 20 insertions(+), 2 deletions(-)
> 
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 64cf03289188..a34b73bfb845 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -733,7 +733,7 @@ int main(int argc, char **argv)
>  	u32 stripesize = 4096;
>  	int zero_end = 1;
>  	int fd = -1;
> -	int ret;
> +	int ret = 0;
>  	int close_ret;
>  	int i;
>  	int mixed = 0;
> @@ -913,7 +913,9 @@ int main(int argc, char **argv)
>  
>  	while (dev_cnt-- > 0) {
>  		file = argv[optind++];
> -		if (is_block_device(file) == 1)
> +		if (source_dir_set && is_path_exist(file) == 0)
> +			ret = 0;
> +		else if (is_block_device(file) == 1)
>  			ret = test_dev_for_mkfs(file, force_overwrite);
>  		else
>  			ret = test_status_for_mkfs(file, force_overwrite);
> diff --git a/utils.c b/utils.c
> index 524f463d3140..22c137514592 100644
> --- a/utils.c
> +++ b/utils.c
> @@ -456,6 +456,21 @@ static int is_reg_file(const char *path)
>  	return S_ISREG(statbuf.st_mode);
>  }
>  
> +int is_path_exist(const char *path)
> +{
> +	struct stat statbuf;
> +	int ret;
> +
> +	ret = stat(path, &statbuf);
> +	if (ret < 0) {
> +		if (errno == ENOENT)
> +			return 0;
> +		else
> +			return -errno;
> +	}
> +	return 1;
> +}
> +
>  /*
>   * This function checks if the given input parameter is
>   * an uuid or a path
> diff --git a/utils.h b/utils.h
> index a82d46f6d7cc..860c25d3cdba 100644
> --- a/utils.h
> +++ b/utils.h
> @@ -122,6 +122,7 @@ int set_label(const char *btrfs_dev, const char *label);
>  char *__strncpy_null(char *dest, const char *src, size_t n);
>  int is_block_device(const char *file);
>  int is_mount_point(const char *file);
> +int is_path_exist(const char *file);
>  int check_arg_type(const char *input);
>  int open_path_or_dev_mnt(const char *path, DIR **dirstream, int verbose);
>  int btrfs_open(const char *path, DIR **dirstream, int verbose, int dir_only);
> 

Re: [PATCH 0/9] Remaining part of mkfs --rootdir rework

[David Sterba]

On Wed, Nov 29, 2017 at 05:15:55PM +0800, Qu Wenruo wrote:
> Qu Wenruo (9):
>   btrfs-progs: mkfs: Cleanup temporary chunks before filling rootdir
>   btrfs-progs: mkfs: Don't use custom chunk allocator for rootdir
>   btrfs-progs: mkfs/rootdir: Use over-reserve method to make size
>     estimate easier
>   btrfs-progs: mkfs/rootdir: Shrink fs for rootdir option
>   btrfs-progs: mkfs: Separate shrink from rootdir
>   btrfs-progs: mkfs: Fix regression preventing --rootdir to create file
>   btrfs-progs: tests/mkfs: Introduce test case to check if mkfs rootdir
>     can create new file
>   btrfs-progs: mkfs: Use the whole file or block device to mkfs for
>     rootdir
>   btrfs-progs: tests/mkfs: Introduce test case to verify if mkfs.btrfs
>     rootdir shrink behaves correctly

Patches reviewed and merged. The basic test coverage is there, I'm CCing
Austin: we'd appreciate if you could also test it for your usecase.
Thanks.