[RFC PATCH v2 0/6] btrfs send stream version 2

[RFC PATCH v2 0/6] btrfs send stream version 2

[Howard McLauchlan]

This is v2 of send stream version 2. The goal is to provide proper
versioning/compatibility as new features are implemented. v1 can be found here
[1].

v2 adds BTRFS_SEND_C_WRITE_COMPRESSED to patch 1 and style fixes/logic
simplifications to patches 5,6.

v2 also updates btrfs-progs to reflect BTRFS_SEND_C_WRITE_COMPRESSED and updates
the pertinent xfstests to also test the total size command.

As of 4.17-rc7, these changes pass all "send" group xfstests

Cheers,

1: https://patchwork.kernel.org/patch/10388003/

Filipe David Borba Manana (5):
  btrfs: send, bump stream version
  btrfs: send, implement total data size command to allow for progress
    estimation
  btrfs: send, use fallocate command to punch holes
  btrfs: send, use fallocate command to allocate extents
  btrfs: add send_stream_version attribute to sysfs

Howard McLauchlan (1):
  btrfs: add chattr support for send/receive

 fs/btrfs/ctree.h           |   2 +
 fs/btrfs/ioctl.c           |   2 +-
 fs/btrfs/send.c            | 495 +++++++++++++++++++++++++++++++------
 fs/btrfs/send.h            |  27 +-
 fs/btrfs/sysfs.c           |  27 ++
 include/uapi/linux/btrfs.h |  21 +-
 6 files changed, 493 insertions(+), 81 deletions(-)

-- 
2.17.0

[RFC PATCH v2 1/6] btrfs: send, bump stream version

[Howard McLauchlan]

From: Filipe David Borba Manana <fdmanana@gmail.com>

This increases the send stream version from version 1 to version 2, adding
new commands:

1) total data size - used to tell the receiver how much file data the stream
   will add or update;

2) fallocate - used to pre-allocate space for files and to punch holes in files;

3) inode set flags (chattr);

4) set inode otime;

5) sending compressed writes.

This is preparation work for subsequent changes that implement the new features.

A version 2 stream is only produced if the send ioctl caller passes in one of the
new flags (BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE | BTRFS_SEND_FLAG_STREAM_V2), meaning
old clients are unaffected.

[Howard: rebased on 4.17-rc7]
Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Omar Sandoval <osandov@fb.com>
---
 fs/btrfs/send.c            |  7 ++++++-
 fs/btrfs/send.h            | 22 +++++++++++++++++++++-
 include/uapi/linux/btrfs.h | 21 ++++++++++++++++++++-
 3 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index c0074d2d7d6d..eccd69387065 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -649,7 +649,10 @@ static int send_header(struct send_ctx *sctx)
 	struct btrfs_stream_header hdr;
 
 	strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC);
-	hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION);
+	if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2)
+		hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION_2);
+	else
+		hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION_1);
 
 	return write_buf(sctx->send_filp, &hdr, sizeof(hdr),
 					&sctx->send_off);
@@ -6535,6 +6538,8 @@ long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg)
 	INIT_LIST_HEAD(&sctx->name_cache_list);
 
 	sctx->flags = arg->flags;
+	if (sctx->flags & BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE)
+		sctx->flags |= BTRFS_SEND_FLAG_STREAM_V2;
 
 	sctx->send_filp = fget(arg->send_fd);
 	if (!sctx->send_filp) {
diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h
index ead397f7034f..152180304078 100644
--- a/fs/btrfs/send.h
+++ b/fs/btrfs/send.h
@@ -10,7 +10,8 @@
 #include "ctree.h"
 
 #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream"
-#define BTRFS_SEND_STREAM_VERSION 1
+#define BTRFS_SEND_STREAM_VERSION_1 1
+#define BTRFS_SEND_STREAM_VERSION_2 2
 
 #define BTRFS_SEND_BUF_SIZE SZ_64K
 #define BTRFS_SEND_READ_SIZE (48 * SZ_1K)
@@ -77,6 +78,16 @@ enum btrfs_send_cmd {
 
 	BTRFS_SEND_C_END,
 	BTRFS_SEND_C_UPDATE_EXTENT,
+
+	/*
+	 * The following commands were added in stream version 2.
+	 */
+	BTRFS_SEND_C_TOTAL_DATA_SIZE,
+	BTRFS_SEND_C_FALLOCATE,
+	BTRFS_SEND_C_CHATTR,
+	BTRFS_SEND_C_UTIMES2, /* Same as UTIMES, but it includes OTIME too. */
+	BTRFS_SEND_C_WRITE_COMPRESSED, /* to be implemented */
+
 	__BTRFS_SEND_C_MAX,
 };
 #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1)
@@ -115,10 +126,19 @@ enum {
 	BTRFS_SEND_A_CLONE_OFFSET,
 	BTRFS_SEND_A_CLONE_LEN,
 
+	/*
+	 * The following attributes were added in stream version 2.
+	 */
+	BTRFS_SEND_A_FALLOCATE_FLAGS,
+	BTRFS_SEND_A_CHATTR,
+
 	__BTRFS_SEND_A_MAX,
 };
 #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1)
 
+#define BTRFS_SEND_A_FALLOCATE_FLAG_KEEP_SIZE   (1 << 0)
+#define BTRFS_SEND_A_FALLOCATE_FLAG_PUNCH_HOLE  (1 << 1)
+
 #ifdef __KERNEL__
 long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg);
 #endif
diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h
index c8d99b9ca550..ed63176660d2 100644
--- a/include/uapi/linux/btrfs.h
+++ b/include/uapi/linux/btrfs.h
@@ -711,10 +711,29 @@ struct btrfs_ioctl_received_subvol_args {
  */
 #define BTRFS_SEND_FLAG_OMIT_END_CMD		0x4
 
+/*
+ * Calculate the amount (in bytes) of new file data between the send and
+ * parent snapshots, or in case of a full send, the total amount of file data
+ * we will send.
+ * This corresponds to the sum of the data lengths of each write, clone and
+ * fallocate commands that are sent through the send stream. The receiving end
+ * can use this information to compute progress.
+ *
+ * Added in send stream version 2, and implies producing a version 2 stream.
+ */
+#define BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE	0x8
+
+/*
+ * Used by a client to request a version 2 of the send stream.
+ */
+#define BTRFS_SEND_FLAG_STREAM_V2              0x10
+
 #define BTRFS_SEND_FLAG_MASK \
 	(BTRFS_SEND_FLAG_NO_FILE_DATA | \
 	 BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \
-	 BTRFS_SEND_FLAG_OMIT_END_CMD)
+	 BTRFS_SEND_FLAG_OMIT_END_CMD | \
+	 BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE | \
+	 BTRFS_SEND_FLAG_STREAM_V2)
 
 struct btrfs_ioctl_send_args {
 	__s64 send_fd;			/* in */
-- 
2.17.0

[RFC PATCH v2 2/6] btrfs: send, implement total data size command to allow for progress estimation

[Howard McLauchlan]

From: Filipe David Borba Manana <fdmanana@gmail.com>

This new send flag makes send calculate first the amount of new file data (in bytes)
the send root has relatively to the parent root, or for the case of a non-incremental
send, the total amount of file data the stream will create (including holes and prealloc
extents). In other words, it computes the sum of the lengths of all write, clone and
fallocate operations that will be sent through the send stream.

This data size value is sent in a new command, named BTRFS_SEND_C_TOTAL_DATA_SIZE, that
immediately follows a BTRFS_SEND_C_SUBVOL or BTRFS_SEND_C_SNAPSHOT command, and precedes
any command that changes a file or the filesystem hierarchy. Upon receiving a write, clone
or fallocate command, the receiving end can increment a counter by the data length of that
command and therefore report progress by comparing the counter's value with the data size
value received in the BTRFS_SEND_C_TOTAL_DATA_SIZE command.

The approach is simple, before the normal operation of send, do a scan in the file system
tree for new inodes and new/changed file extent items, just like in send's normal operation,
and keep incrementing a counter with new inodes' size and the size of file extents (and file
holes)  that are going to be written, cloned or fallocated. This is actually a simpler and
more lightweight tree scan/processing than the one we do when sending the changes, as it
doesn't process inode references nor does any lookups in the extent tree for example.

After modifying btrfs-progs to understand this new command and report progress, here's an
example (the -o flag tells btrfs send to pass the new flag to the kernel's send ioctl):

    $ btrfs send -s --stream-version 2 /mnt/sdd/snap_base | btrfs receive /mnt/sdc
    At subvol /mnt/sdd/snap_base
    At subvol snap_base
    About to receive 9212392667 bytes
    Subvolume /mnt/sdc//snap_base, 4059722426 / 9212392667 bytes received, 44.07%, 40.32MB/s

    $ btrfs send -s --stream-version 2 -p /mnt/sdd/snap_base /mnt/sdd/snap_incr | btrfs receive /mnt/sdc
    At subvol /mnt/sdd/snap_incr
    At subvol snap_incr
    About to receive 9571342213 bytes
    Subvolume /mnt/sdc//snap_incr, 6557345221 / 9571342213 bytes received, 68.51%, 51.04MB/s

[Howard: rebased on 4.17-rc7]
Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
 fs/btrfs/send.c | 189 ++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 157 insertions(+), 32 deletions(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index eccd69387065..7b184831812b 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -66,7 +66,13 @@ struct clone_root {
 #define SEND_CTX_MAX_NAME_CACHE_SIZE 128
 #define SEND_CTX_NAME_CACHE_CLEAN_SIZE (SEND_CTX_MAX_NAME_CACHE_SIZE * 2)
 
+enum btrfs_send_phase {
+	SEND_PHASE_STREAM_CHANGES,
+	SEND_PHASE_COMPUTE_DATA_SIZE,
+};
+
 struct send_ctx {
+	enum btrfs_send_phase phase;
 	struct file *send_filp;
 	loff_t send_off;
 	char *send_buf;
@@ -102,6 +108,7 @@ struct send_ctx {
 	u64 cur_inode_next_write_offset;
 
 	u64 send_progress;
+	u64 total_data_size;
 
 	struct list_head new_refs;
 	struct list_head deleted_refs;
@@ -709,6 +716,8 @@ static int send_rename(struct send_ctx *sctx,
 	struct btrfs_fs_info *fs_info = sctx->send_root->fs_info;
 	int ret;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(fs_info, "send_rename %s -> %s", from->start, to->start);
 
 	ret = begin_cmd(sctx, BTRFS_SEND_C_RENAME);
@@ -758,6 +767,8 @@ static int send_unlink(struct send_ctx *sctx, struct fs_path *path)
 	struct btrfs_fs_info *fs_info = sctx->send_root->fs_info;
 	int ret;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(fs_info, "send_unlink %s", path->start);
 
 	ret = begin_cmd(sctx, BTRFS_SEND_C_UNLINK);
@@ -780,6 +791,7 @@ static int send_rmdir(struct send_ctx *sctx, struct fs_path *path)
 {
 	struct btrfs_fs_info *fs_info = sctx->send_root->fs_info;
 	int ret;
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
 
 	btrfs_debug(fs_info, "send_rmdir %s", path->start);
 
@@ -2419,6 +2431,8 @@ static int send_truncate(struct send_ctx *sctx, u64 ino, u64 gen, u64 size)
 	int ret = 0;
 	struct fs_path *p;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(fs_info, "send_truncate %llu size=%llu", ino, size);
 
 	p = fs_path_alloc();
@@ -2449,6 +2463,8 @@ static int send_chmod(struct send_ctx *sctx, u64 ino, u64 gen, u64 mode)
 	int ret = 0;
 	struct fs_path *p;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(fs_info, "send_chmod %llu mode=%llu", ino, mode);
 
 	p = fs_path_alloc();
@@ -2479,6 +2495,8 @@ static int send_chown(struct send_ctx *sctx, u64 ino, u64 gen, u64 uid, u64 gid)
 	int ret = 0;
 	struct fs_path *p;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(fs_info, "send_chown %llu uid=%llu, gid=%llu",
 		    ino, uid, gid);
 
@@ -2516,6 +2534,8 @@ static int send_utimes(struct send_ctx *sctx, u64 ino, u64 gen)
 	struct btrfs_key key;
 	int slot;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(fs_info, "send_utimes %llu", ino);
 
 	p = fs_path_alloc();
@@ -2578,6 +2598,8 @@ static int send_create_inode(struct send_ctx *sctx, u64 ino)
 	u64 mode;
 	u64 rdev;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(fs_info, "send_create_inode %llu", ino);
 
 	p = fs_path_alloc();
@@ -2725,6 +2747,8 @@ static int send_create_inode_if_needed(struct send_ctx *sctx)
 {
 	int ret;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	if (S_ISDIR(sctx->cur_inode_mode)) {
 		ret = did_create_dir(sctx, sctx->cur_ino);
 		if (ret < 0)
@@ -2825,6 +2849,8 @@ static int orphanize_inode(struct send_ctx *sctx, u64 ino, u64 gen,
 	int ret;
 	struct fs_path *orphan;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	orphan = fs_path_alloc();
 	if (!orphan)
 		return -ENOMEM;
@@ -3200,6 +3226,8 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 	bool is_orphan;
 	int ret;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	name = fs_path_alloc();
 	from_path = fs_path_alloc();
 	if (!name || !from_path) {
@@ -3792,6 +3820,9 @@ static int process_recorded_refs(struct send_ctx *sctx, int *pending_move)
 	bool orphanized_dir = false;
 	bool orphanized_ancestor = false;
 
+	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE)
+		return 0;
+
 	btrfs_debug(fs_info, "process_recorded_refs %llu", sctx->cur_ino);
 
 	/*
@@ -4391,6 +4422,8 @@ static int process_all_refs(struct send_ctx *sctx,
 	iterate_inode_ref_t cb;
 	int pending_move = 0;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	path = alloc_path_for_send();
 	if (!path)
 		return -ENOMEM;
@@ -4708,6 +4741,8 @@ static int process_all_new_xattrs(struct send_ctx *sctx)
 	struct extent_buffer *eb;
 	int slot;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	path = alloc_path_for_send();
 	if (!path)
 		return -ENOMEM;
@@ -4849,6 +4884,8 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len)
 	struct fs_path *p;
 	ssize_t num_read = 0;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	p = fs_path_alloc();
 	if (!p)
 		return -ENOMEM;
@@ -4884,6 +4921,22 @@ static int send_write(struct send_ctx *sctx, u64 offset, u32 len)
 	return num_read;
 }
 
+static int send_total_data_size(struct send_ctx *sctx, u64 data_size)
+{
+	int ret;
+
+	ret = begin_cmd(sctx, BTRFS_SEND_C_TOTAL_DATA_SIZE);
+	if (ret < 0)
+		goto out;
+
+	TLV_PUT_U64(sctx, BTRFS_SEND_A_SIZE, data_size);
+	ret = send_cmd(sctx);
+
+tlv_put_failure:
+out:
+	return ret;
+}
+
 /*
  * Send a clone command to user space.
  */
@@ -4895,6 +4948,8 @@ static int send_clone(struct send_ctx *sctx,
 	struct fs_path *p;
 	u64 gen;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	btrfs_debug(sctx->send_root->fs_info,
 		    "send_clone offset=%llu, len=%d, clone_root=%llu, clone_inode=%llu, clone_offset=%llu",
 		    offset, len, clone_root->root->objectid, clone_root->ino,
@@ -4966,6 +5021,8 @@ static int send_update_extent(struct send_ctx *sctx,
 	int ret = 0;
 	struct fs_path *p;
 
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
 	p = fs_path_alloc();
 	if (!p)
 		return -ENOMEM;
@@ -4997,6 +5054,10 @@ static int send_hole(struct send_ctx *sctx, u64 end)
 	u64 len;
 	int ret = 0;
 
+	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE) {
+		sctx->total_data_size += end - offset;
+		return 0;
+	}
 	if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA)
 		return send_update_extent(sctx, offset, end - offset);
 
@@ -5250,6 +5311,12 @@ static int send_write_or_clone(struct send_ctx *sctx,
 		goto out;
 	}
 
+	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE) {
+		if (offset < sctx->cur_inode_size)
+			sctx->total_data_size += len;
+		goto out;
+	}
+
 	if (clone_root && IS_ALIGNED(offset + len, bs)) {
 		u64 disk_byte;
 		u64 data_offset;
@@ -5681,10 +5748,12 @@ static int process_extent(struct send_ctx *sctx,
 		}
 	}
 
-	ret = find_extent_clone(sctx, path, key->objectid, key->offset,
-			sctx->cur_inode_size, &found_clone);
-	if (ret != -ENOENT && ret < 0)
-		goto out;
+	if (sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE) {
+		ret = find_extent_clone(sctx, path, key->objectid, key->offset,
+					sctx->cur_inode_size, &found_clone);
+		if (ret != -ENOENT && ret < 0)
+			goto out;
+	}
 
 	ret = send_write_or_clone(sctx, path, key, found_clone);
 	if (ret)
@@ -5815,6 +5884,9 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 	if (!at_end && sctx->cmp_key->objectid == sctx->cur_ino)
 		goto out;
 
+	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE)
+		goto truncate_inode;
+
 	ret = get_inode_info(sctx->send_root, sctx->cur_ino, NULL, NULL,
 			&left_mode, &left_uid, &left_gid, NULL);
 	if (ret < 0)
@@ -5845,6 +5917,7 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 			need_truncate = 0;
 	}
 
+truncate_inode:
 	if (S_ISREG(sctx->cur_inode_mode)) {
 		if (need_send_hole(sctx)) {
 			if (sctx->cur_inode_last_extent == (u64)-1 ||
@@ -5887,7 +5960,8 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 	 * If other directory inodes depended on our current directory
 	 * inode's move/rename, now do their move/rename operations.
 	 */
-	if (!is_waiting_for_move(sctx, sctx->cur_ino)) {
+	if (sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE &&
+	    !is_waiting_for_move(sctx, sctx->cur_ino)) {
 		ret = apply_children_dir_moves(sctx);
 		if (ret)
 			goto out;
@@ -5972,7 +6046,8 @@ static int changed_inode(struct send_ctx *sctx,
 				sctx->left_path->nodes[0], left_ii);
 		sctx->cur_inode_rdev = btrfs_inode_rdev(
 				sctx->left_path->nodes[0], left_ii);
-		if (sctx->cur_ino != BTRFS_FIRST_FREE_OBJECTID)
+		if (sctx->cur_ino != BTRFS_FIRST_FREE_OBJECTID &&
+		    sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE)
 			ret = send_create_inode_if_needed(sctx);
 	} else if (result == BTRFS_COMPARE_TREE_DELETED) {
 		sctx->cur_inode_gen = right_gen;
@@ -5994,17 +6069,19 @@ static int changed_inode(struct send_ctx *sctx,
 			/*
 			 * First, process the inode as if it was deleted.
 			 */
-			sctx->cur_inode_gen = right_gen;
-			sctx->cur_inode_new = 0;
-			sctx->cur_inode_deleted = 1;
-			sctx->cur_inode_size = btrfs_inode_size(
+			if (sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE) {
+				sctx->cur_inode_gen = right_gen;
+				sctx->cur_inode_new = 0;
+				sctx->cur_inode_deleted = 1;
+				sctx->cur_inode_size = btrfs_inode_size(
 					sctx->right_path->nodes[0], right_ii);
-			sctx->cur_inode_mode = btrfs_inode_mode(
+				sctx->cur_inode_mode = btrfs_inode_mode(
 					sctx->right_path->nodes[0], right_ii);
-			ret = process_all_refs(sctx,
-					BTRFS_COMPARE_TREE_DELETED);
-			if (ret < 0)
-				goto out;
+				ret = process_all_refs(sctx,
+						   BTRFS_COMPARE_TREE_DELETED);
+				if (ret < 0)
+					goto out;
+			}
 
 			/*
 			 * Now process the inode as if it was new.
@@ -6018,29 +6095,38 @@ static int changed_inode(struct send_ctx *sctx,
 					sctx->left_path->nodes[0], left_ii);
 			sctx->cur_inode_rdev = btrfs_inode_rdev(
 					sctx->left_path->nodes[0], left_ii);
-			ret = send_create_inode_if_needed(sctx);
-			if (ret < 0)
-				goto out;
-
-			ret = process_all_refs(sctx, BTRFS_COMPARE_TREE_NEW);
-			if (ret < 0)
-				goto out;
+			if (sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE) {
+				ret = send_create_inode_if_needed(sctx);
+				if (ret < 0)
+					goto out;
+				ret = process_all_refs(sctx,
+						       BTRFS_COMPARE_TREE_NEW);
+				if (ret < 0)
+					goto out;
+			}
 			/*
 			 * Advance send_progress now as we did not get into
 			 * process_recorded_refs_if_needed in the new_gen case.
 			 */
 			sctx->send_progress = sctx->cur_ino + 1;
 
-			/*
-			 * Now process all extents and xattrs of the inode as if
-			 * they were all new.
-			 */
-			ret = process_all_extents(sctx);
-			if (ret < 0)
-				goto out;
-			ret = process_all_new_xattrs(sctx);
-			if (ret < 0)
-				goto out;
+			if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE) {
+				if (S_ISREG(sctx->cur_inode_mode))
+					sctx->total_data_size +=
+						sctx->cur_inode_size;
+				/* TODO: maybe account for xattrs one day too */
+			} else {
+				/*
+				 * Now process all extents and xattrs of the
+				 * inode as if they were all new.
+				 */
+				ret = process_all_extents(sctx);
+				if (ret < 0)
+					goto out;
+				ret = process_all_new_xattrs(sctx);
+				if (ret < 0)
+					goto out;
+			}
 		} else {
 			sctx->cur_inode_gen = left_gen;
 			sctx->cur_inode_new = 0;
@@ -6077,6 +6163,9 @@ static int changed_ref(struct send_ctx *sctx,
 		return -EIO;
 	}
 
+	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE)
+		return 0;
+
 	if (!sctx->cur_inode_new_gen &&
 	    sctx->cur_ino != BTRFS_FIRST_FREE_OBJECTID) {
 		if (result == BTRFS_COMPARE_TREE_NEW)
@@ -6105,6 +6194,9 @@ static int changed_xattr(struct send_ctx *sctx,
 		return -EIO;
 	}
 
+	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE)
+		return 0;
+
 	if (!sctx->cur_inode_new_gen && !sctx->cur_inode_deleted) {
 		if (result == BTRFS_COMPARE_TREE_NEW)
 			ret = process_new_xattr(sctx);
@@ -6273,6 +6365,8 @@ static int changed_cb(struct btrfs_path *left_path,
 	if (result == BTRFS_COMPARE_TREE_SAME) {
 		if (key->type == BTRFS_INODE_REF_KEY ||
 		    key->type == BTRFS_INODE_EXTREF_KEY) {
+			if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE)
+				return 0;
 			ret = compare_refs(sctx, left_path, key);
 			if (!ret)
 				return 0;
@@ -6369,6 +6463,24 @@ static int full_send_tree(struct send_ctx *sctx)
 	return ret;
 }
 
+static int compute_total_data_size(struct send_ctx *sctx)
+{
+	int ret;
+
+	sctx->total_data_size = 0;
+
+	if (sctx->parent_root) {
+		ret = btrfs_compare_trees(sctx->send_root, sctx->parent_root,
+					  changed_cb, sctx);
+		if (!ret)
+			ret = finish_inode_if_needed(sctx, 1);
+	} else {
+		ret = full_send_tree(sctx);
+	}
+
+	return ret;
+}
+
 static int send_subvol(struct send_ctx *sctx)
 {
 	int ret;
@@ -6383,6 +6495,19 @@ static int send_subvol(struct send_ctx *sctx)
 	if (ret < 0)
 		goto out;
 
+	if (sctx->flags & BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE) {
+		sctx->phase = SEND_PHASE_COMPUTE_DATA_SIZE;
+		ret = compute_total_data_size(sctx);
+		if (ret < 0)
+			goto out;
+		ret = send_total_data_size(sctx, sctx->total_data_size);
+		if (ret < 0)
+			goto out;
+		sctx->phase = SEND_PHASE_STREAM_CHANGES;
+		sctx->cur_ino = 0;
+		sctx->send_progress = 0;
+	}
+
 	if (sctx->parent_root) {
 		ret = btrfs_compare_trees(sctx->send_root, sctx->parent_root,
 				changed_cb, sctx);
-- 
2.17.0

[RFC PATCH v2 3/6] btrfs: send, use fallocate command to punch holes

[Howard McLauchlan]

From: Filipe David Borba Manana <fdmanana@gmail.com>

Instead of sending a write command with a data buffer filled with 0 value bytes,
use the fallocate command, introduced in the send stream version 2, to tell the
receiver to punch a file hole using the fallocate system call.

[Howard: rebased on 4.17-rc7]
Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
 fs/btrfs/send.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++---
 fs/btrfs/send.h |  4 ++++
 2 files changed, 55 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 7b184831812b..328c7a2857ae 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -581,6 +581,7 @@ static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len)
 		return tlv_put(sctx, attr, &__tmp, sizeof(__tmp));	\
 	}
 
+TLV_PUT_DEFINE_INT(32)
 TLV_PUT_DEFINE_INT(64)
 
 static int tlv_put_string(struct send_ctx *sctx, u16 attr,
@@ -5047,17 +5048,57 @@ static int send_update_extent(struct send_ctx *sctx,
 	return ret;
 }
 
+static int send_fallocate(struct send_ctx *sctx, u32 flags,
+			  u64 offset, u64 len)
+{
+	struct fs_path *p = NULL;
+	int ret = 0;
+
+	ASSERT(sctx->flags & BTRFS_SEND_FLAG_STREAM_V2);
+
+	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE) {
+		sctx->total_data_size += len;
+		return 0;
+	}
+
+	p = fs_path_alloc();
+	if (!p)
+		return -ENOMEM;
+	ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, p);
+	if (ret < 0)
+		goto out;
+
+	ret = begin_cmd(sctx, BTRFS_SEND_C_FALLOCATE);
+	if (ret < 0)
+		goto out;
+	TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p);
+	TLV_PUT_U32(sctx, BTRFS_SEND_A_FALLOCATE_FLAGS, flags);
+	TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset);
+	TLV_PUT_U64(sctx, BTRFS_SEND_A_SIZE, len);
+	ret = send_cmd(sctx);
+
+tlv_put_failure:
+out:
+	fs_path_free(p);
+	return ret;
+}
+
+
 static int send_hole(struct send_ctx *sctx, u64 end)
 {
 	struct fs_path *p = NULL;
 	u64 offset = sctx->cur_inode_last_extent;
-	u64 len;
+	u64 len = end - offset;
 	int ret = 0;
 
 	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE) {
-		sctx->total_data_size += end - offset;
+		sctx->total_data_size += len;
 		return 0;
 	}
+	if (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2)
+		return send_fallocate(sctx, BTRFS_SEND_PUNCH_HOLE_FALLOC_FLAGS,
+				      offset, len);
+
 	if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA)
 		return send_update_extent(sctx, offset, end - offset);
 
@@ -5304,7 +5345,8 @@ static int send_write_or_clone(struct send_ctx *sctx,
 		ret = 0;
 		goto out;
 	}
-	if (offset + len > sctx->cur_inode_size)
+	if (offset < sctx->cur_inode_size &&
+	    offset + len > sctx->cur_inode_size)
 		len = sctx->cur_inode_size - offset;
 	if (len == 0) {
 		ret = 0;
@@ -5325,6 +5367,12 @@ static int send_write_or_clone(struct send_ctx *sctx,
 		data_offset = btrfs_file_extent_offset(path->nodes[0], ei);
 		ret = clone_range(sctx, clone_root, disk_byte, data_offset,
 				  offset, len);
+	} else if (btrfs_file_extent_disk_bytenr(path->nodes[0], ei) == 0 &&
+		 type != BTRFS_FILE_EXTENT_INLINE &&
+		 (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) &&
+		 offset < sctx->cur_inode_size) {
+		ret = send_fallocate(sctx, BTRFS_SEND_PUNCH_HOLE_FALLOC_FLAGS,
+				     offset, len);
 	} else {
 		ret = send_extent_data(sctx, offset, len);
 	}
diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h
index 152180304078..b6e9281f171a 100644
--- a/fs/btrfs/send.h
+++ b/fs/btrfs/send.h
@@ -139,6 +139,10 @@ enum {
 #define BTRFS_SEND_A_FALLOCATE_FLAG_KEEP_SIZE   (1 << 0)
 #define BTRFS_SEND_A_FALLOCATE_FLAG_PUNCH_HOLE  (1 << 1)
 
+#define BTRFS_SEND_PUNCH_HOLE_FALLOC_FLAGS        \
+	(BTRFS_SEND_A_FALLOCATE_FLAG_KEEP_SIZE |  \
+	 BTRFS_SEND_A_FALLOCATE_FLAG_PUNCH_HOLE)
+
 #ifdef __KERNEL__
 long btrfs_ioctl_send(struct file *mnt_file, struct btrfs_ioctl_send_args *arg);
 #endif
-- 
2.17.0

[RFC PATCH v2 4/6] btrfs: send, use fallocate command to allocate extents

[Howard McLauchlan]

From: Filipe David Borba Manana <fdmanana@gmail.com>

The send stream version 2 adds the fallocate command, which can be used to
allocate extents for a file or punch holes in a file. Previously we were
ignoring file prealloc extents or treating them as extents filled with 0
bytes and sending a regular write command to the stream.

After this change, together with my previous change titled:

    "Btrfs: send, use fallocate command to punch holes"

an incremental send preserves the hole and data structure of files, which can
be seen via calls to lseek with the whence parameter set to SEEK_DATA or SEEK_HOLE,
as the example below shows:

    mkfs.btrfs -f /dev/sdc
    mount /dev/sdc /mnt
    xfs_io -f -c "pwrite -S 0x01 -b 300000 0 300000" /mnt/foo
    btrfs subvolume snapshot -r /mnt /mnt/mysnap1

    xfs_io -c "fpunch 100000 50000" /mnt/foo
    xfs_io -c "falloc 100000 50000" /mnt/foo
    xfs_io -c "pwrite -S 0xff -b 1000 120000 1000" /mnt/foo
    xfs_io -c "fpunch 250000 20000" /mnt/foo

    # prealloc extents that start beyond the inode's size
    xfs_io -c "falloc -k 300000 1000000" /mnt/foo
    xfs_io -c "falloc -k 9000000 2000000" /mnt/foo

    btrfs subvolume snapshot -r /mnt /mnt/mysnap2

    btrfs send /mnt/mysnap1 -f /tmp/1.snap
    btrfs send -p /mnt/mysnap1 /mnt/mysnap2 -f /tmp/2.snap

    mkfs.btrfs -f /dev/sdd
    mount /dev/sdd /mnt2
    btrfs receive /mnt2 -f /tmp/1.snap
    btrfs receive /mnt2 -f /tmp/2.snap

Before this change the hole/data structure differed between both filesystems:

    $ xfs_io -r -c 'seek -r -a 0' /mnt/mysnap2/foo
    Whence  Result
    DATA    0
    HOLE    102400
    DATA    118784
    HOLE    122880
    DATA    147456
    HOLE    253952
    DATA    266240
    HOLE    300000

    $ xfs_io -r -c 'seek -r -a 0' /mnt2/mysnap2/foo
    Whence  Result
    DATA    0
    HOLE    300000

After this change the second filesystem (/dev/sdd) ends up with the same hole/data
structure as the first filesystem.

Also, after this change, prealloc extents that lie beyond the inode's size (were
allocated with fallocate + keep size flag) are also replicated by an incremental
send. For the above test, it can be observed via fiemap (or btrfs-debug-tree):

    $ xfs_io -r -c 'fiemap -l' /mnt2/mysnap2/foo
	0: [0..191]: 25096..25287 192 blocks
	1: [192..199]: 24672..24679 8 blocks
	2: [200..231]: 24584..24615 32 blocks
	3: [232..239]: 24680..24687 8 blocks
	4: [240..287]: 24616..24663 48 blocks
	5: [288..295]: 24688..24695 8 blocks
	6: [296..487]: 25392..25583 192 blocks
	7: [488..495]: 24696..24703 8 blocks
	8: [496..519]: hole 24 blocks
	9: [520..527]: 24704..24711 8 blocks
	10: [528..583]: 25624..25679 56 blocks
	11: [584..591]: 24712..24719 8 blocks
	12: [592..2543]: 26192..28143 1952 blocks
	13: [2544..17575]: hole 15032 blocks
	14: [17576..21487]: 28144..32055 3912 blocks

The proposed xfstest can be found at:

    xfstests: btrfs, test send's ability to punch holes and prealloc extents

This test verifies that send-stream version 2 does space pre-allocation
and hole punching.

[Howard: rebased on 4.17-rc7]
Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
 fs/btrfs/send.c | 70 ++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 52 insertions(+), 18 deletions(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 328c7a2857ae..84dacb20d832 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -98,9 +98,10 @@ struct send_ctx {
 	 */
 	u64 cur_ino;
 	u64 cur_inode_gen;
-	int cur_inode_new;
-	int cur_inode_new_gen;
-	int cur_inode_deleted;
+	u8 cur_inode_new:1;
+	u8 cur_inode_new_gen:1;
+	u8 cur_inode_skip_truncate:1;
+	u8 cur_inode_deleted:1;
 	u64 cur_inode_size;
 	u64 cur_inode_mode;
 	u64 cur_inode_rdev;
@@ -5313,6 +5314,19 @@ static int clone_range(struct send_ctx *sctx,
 	return ret;
 }
 
+static int truncate_before_falloc(struct send_ctx *sctx)
+{
+	int ret = 0;
+
+	if (!sctx->cur_inode_skip_truncate) {
+		ret = send_truncate(sctx, sctx->cur_ino,
+				    sctx->cur_inode_gen,
+				    sctx->cur_inode_size);
+		sctx->cur_inode_skip_truncate = 1;
+	}
+	return ret;
+}
+
 static int send_write_or_clone(struct send_ctx *sctx,
 			       struct btrfs_path *path,
 			       struct btrfs_key *key,
@@ -5354,8 +5368,7 @@ static int send_write_or_clone(struct send_ctx *sctx,
 	}
 
 	if (sctx->phase == SEND_PHASE_COMPUTE_DATA_SIZE) {
-		if (offset < sctx->cur_inode_size)
-			sctx->total_data_size += len;
+		sctx->total_data_size += len;
 		goto out;
 	}
 
@@ -5373,6 +5386,21 @@ static int send_write_or_clone(struct send_ctx *sctx,
 		 offset < sctx->cur_inode_size) {
 		ret = send_fallocate(sctx, BTRFS_SEND_PUNCH_HOLE_FALLOC_FLAGS,
 				     offset, len);
+	} else if (type == BTRFS_FILE_EXTENT_PREALLOC &&
+		   (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2)) {
+		u32 flags = 0;
+		if (offset < sctx->cur_inode_size) {
+			ret = send_fallocate(sctx,
+					     BTRFS_SEND_PUNCH_HOLE_FALLOC_FLAGS,
+					     offset, len);
+		} else {
+			flags |= BTRFS_SEND_A_FALLOCATE_FLAG_KEEP_SIZE;
+			ret = truncate_before_falloc(sctx);
+		}
+		if (ret)
+			goto out;
+		ret = send_fallocate(sctx, flags, offset, len);
+
 	} else {
 		ret = send_extent_data(sctx, offset, len);
 	}
@@ -5775,19 +5803,24 @@ static int process_extent(struct send_ctx *sctx,
 		ei = btrfs_item_ptr(path->nodes[0], path->slots[0],
 				    struct btrfs_file_extent_item);
 		type = btrfs_file_extent_type(path->nodes[0], ei);
-		if (type == BTRFS_FILE_EXTENT_PREALLOC ||
-		    type == BTRFS_FILE_EXTENT_REG) {
-			/*
-			 * The send spec does not have a prealloc command yet,
-			 * so just leave a hole for prealloc'ed extents until
-			 * we have enough commands queued up to justify rev'ing
-			 * the send spec.
-			 */
-			if (type == BTRFS_FILE_EXTENT_PREALLOC) {
-				ret = 0;
-				goto out;
+		if (type == BTRFS_FILE_EXTENT_PREALLOC &&
+		    (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2)) {
+			u64 len;
+			u32 flags = 0;
+
+			len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
+			if (key->offset >= sctx->cur_inode_size) {
+				flags |= BTRFS_SEND_A_FALLOCATE_FLAG_KEEP_SIZE;
+				ret = truncate_before_falloc(sctx);
+				if (ret)
+					goto out;
 			}
-
+			ret = send_fallocate(sctx, flags, key->offset, len);
+			goto out;
+		} else if (type == BTRFS_FILE_EXTENT_PREALLOC) {
+			ret = 0;
+			goto out;
+		} else if (type == BTRFS_FILE_EXTENT_REG) {
 			/* Have a hole, just skip it. */
 			if (btrfs_file_extent_disk_bytenr(path->nodes[0], ei) == 0) {
 				ret = 0;
@@ -5982,7 +6015,7 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 					goto out;
 			}
 		}
-		if (need_truncate) {
+		if (need_truncate && !sctx->cur_inode_skip_truncate) {
 			ret = send_truncate(sctx, sctx->cur_ino,
 					    sctx->cur_inode_gen,
 					    sctx->cur_inode_size);
@@ -6044,6 +6077,7 @@ static int changed_inode(struct send_ctx *sctx,
 	sctx->cur_inode_new_gen = 0;
 	sctx->cur_inode_last_extent = (u64)-1;
 	sctx->cur_inode_next_write_offset = 0;
+	sctx->cur_inode_skip_truncate = 0;
 
 	/*
 	 * Set send_progress to current inode. This will tell all get_cur_xxx
-- 
2.17.0

[RFC PATCH v2 5/6] btrfs: add send_stream_version attribute to sysfs

[Howard McLauchlan]

From: Filipe David Borba Manana <fdmanana@gmail.com>

So that applications can find out what's the highest send stream
version supported/implemented by the running kernel:

    $ cat /sys/fs/btrfs/send/stream_version
    2

[Howard: rebased on 4.17-rc7]
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: David Sterba <dsterba@suse.cz>
---
 fs/btrfs/send.h  |  1 +
 fs/btrfs/sysfs.c | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)

diff --git a/fs/btrfs/send.h b/fs/btrfs/send.h
index b6e9281f171a..267674d3e954 100644
--- a/fs/btrfs/send.h
+++ b/fs/btrfs/send.h
@@ -12,6 +12,7 @@
 #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream"
 #define BTRFS_SEND_STREAM_VERSION_1 1
 #define BTRFS_SEND_STREAM_VERSION_2 2
+#define BTRFS_SEND_STREAM_VERSION_LATEST BTRFS_SEND_STREAM_VERSION_2
 
 #define BTRFS_SEND_BUF_SIZE SZ_64K
 #define BTRFS_SEND_READ_SIZE (48 * SZ_1K)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 4848a4318fb5..718bc927ea13 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -18,6 +18,7 @@
 #include "transaction.h"
 #include "sysfs.h"
 #include "volumes.h"
+#include "send.h"
 
 static inline struct btrfs_fs_info *to_fs_info(struct kobject *kobj);
 static inline struct btrfs_fs_devices *to_fs_devs(struct kobject *kobj);
@@ -884,6 +885,26 @@ static int btrfs_init_debugfs(void)
 	return 0;
 }
 
+static ssize_t send_stream_version_show(struct kobject *kobj,
+					struct kobj_attribute *a,
+					char *buf)
+{
+	return snprintf(buf, PAGE_SIZE, "%d\n",
+			BTRFS_SEND_STREAM_VERSION_LATEST);
+}
+
+BTRFS_ATTR(, stream_version, send_stream_version_show);
+
+static struct attribute *btrfs_send_attrs[] = {
+	BTRFS_ATTR_PTR(, stream_version),
+	NULL
+};
+
+static const struct attribute_group btrfs_send_attr_group = {
+	.name = "send",
+	.attrs = btrfs_send_attrs,
+};
+
 int __init btrfs_init_sysfs(void)
 {
 	int ret;
@@ -900,8 +921,13 @@ int __init btrfs_init_sysfs(void)
 	ret = sysfs_create_group(&btrfs_kset->kobj, &btrfs_feature_attr_group);
 	if (ret)
 		goto out2;
+	ret = sysfs_create_group(&btrfs_kset->kobj, &btrfs_send_attr_group);
+	if (ret)
+		goto out3;
 
 	return 0;
+out3:
+	sysfs_remove_group(&btrfs_kset->kobj, &btrfs_feature_attr_group);
 out2:
 	debugfs_remove_recursive(btrfs_debugfs_root_dentry);
 out1:
@@ -913,6 +939,7 @@ int __init btrfs_init_sysfs(void)
 void __cold btrfs_exit_sysfs(void)
 {
 	sysfs_remove_group(&btrfs_kset->kobj, &btrfs_feature_attr_group);
+	sysfs_remove_group(&btrfs_kset->kobj, &btrfs_send_attr_group);
 	kset_unregister(btrfs_kset);
 	debugfs_remove_recursive(btrfs_debugfs_root_dentry);
 }
-- 
2.17.0

[RFC PATCH v2 6/6] btrfs: add chattr support for send/receive

[Howard McLauchlan]

From: Howard McLauchlan <hmclauchlan@fb.com>

Presently btrfs send/receive does not propagate inode attribute flags;
all chattr operations are effectively discarded upon transmission.

This patch adds kernel support for inode attribute flags. Userspace
support can be found under the commit:

    btrfs-progs: add chattr support for send/receive

An associated xfstest can be found at:

    btrfs: add verify chattr support for send/receive test

These changes are only enabled for send stream version 2

Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
---
 fs/btrfs/ctree.h |   2 +
 fs/btrfs/ioctl.c |   2 +-
 fs/btrfs/send.c  | 183 +++++++++++++++++++++++++++++++++++++++--------
 3 files changed, 158 insertions(+), 29 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 0d422c9908b8..002fe3ad193a 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -1461,6 +1461,8 @@ struct btrfs_map_token {
 	unsigned long offset;
 };
 
+unsigned int btrfs_flags_to_ioctl(unsigned int flags);
+
 #define BTRFS_BYTES_TO_BLKS(fs_info, bytes) \
 				((bytes) >> (fs_info)->sb->s_blocksize_bits)
 
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 632e26d6f7ce..36ce1e589f9e 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -106,7 +106,7 @@ static unsigned int btrfs_mask_flags(umode_t mode, unsigned int flags)
 /*
  * Export inode flags to the format expected by the FS_IOC_GETFLAGS ioctl.
  */
-static unsigned int btrfs_flags_to_ioctl(unsigned int flags)
+unsigned int btrfs_flags_to_ioctl(unsigned int flags)
 {
 	unsigned int iflags = 0;
 
diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 84dacb20d832..a36a2983b34a 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -108,6 +108,13 @@ struct send_ctx {
 	u64 cur_inode_last_extent;
 	u64 cur_inode_next_write_offset;
 
+	/*
+	 * state for chattr purposes
+	 */
+	u64 cur_inode_flip_flags;
+	u64 cur_inode_receive_flags;
+	bool receive_flags_valid;
+
 	u64 send_progress;
 	u64 total_data_size;
 
@@ -815,7 +822,7 @@ static int send_rmdir(struct send_ctx *sctx, struct fs_path *path)
  */
 static int __get_inode_info(struct btrfs_root *root, struct btrfs_path *path,
 			  u64 ino, u64 *size, u64 *gen, u64 *mode, u64 *uid,
-			  u64 *gid, u64 *rdev)
+			  u64 *gid, u64 *rdev, u64 *flags)
 {
 	int ret;
 	struct btrfs_inode_item *ii;
@@ -845,6 +852,8 @@ static int __get_inode_info(struct btrfs_root *root, struct btrfs_path *path,
 		*gid = btrfs_inode_gid(path->nodes[0], ii);
 	if (rdev)
 		*rdev = btrfs_inode_rdev(path->nodes[0], ii);
+	if (flags)
+		*flags = btrfs_inode_flags(path->nodes[0], ii);
 
 	return ret;
 }
@@ -852,7 +861,7 @@ static int __get_inode_info(struct btrfs_root *root, struct btrfs_path *path,
 static int get_inode_info(struct btrfs_root *root,
 			  u64 ino, u64 *size, u64 *gen,
 			  u64 *mode, u64 *uid, u64 *gid,
-			  u64 *rdev)
+			  u64 *rdev, u64 *flags)
 {
 	struct btrfs_path *path;
 	int ret;
@@ -861,7 +870,7 @@ static int get_inode_info(struct btrfs_root *root,
 	if (!path)
 		return -ENOMEM;
 	ret = __get_inode_info(root, path, ino, size, gen, mode, uid, gid,
-			       rdev);
+			       rdev, flags);
 	btrfs_free_path(path);
 	return ret;
 }
@@ -1250,7 +1259,7 @@ static int __iterate_backrefs(u64 ino, u64 offset, u64 root, void *ctx_)
 	 * accept clones from these extents.
 	 */
 	ret = __get_inode_info(found->root, bctx->path, ino, &i_size, NULL, NULL,
-			       NULL, NULL, NULL);
+			       NULL, NULL, NULL, NULL);
 	btrfs_release_path(bctx->path);
 	if (ret < 0)
 		return ret;
@@ -1610,7 +1619,7 @@ static int get_cur_inode_state(struct send_ctx *sctx, u64 ino, u64 gen)
 	u64 right_gen;
 
 	ret = get_inode_info(sctx->send_root, ino, NULL, &left_gen, NULL, NULL,
-			NULL, NULL);
+			NULL, NULL, NULL);
 	if (ret < 0 && ret != -ENOENT)
 		goto out;
 	left_ret = ret;
@@ -1619,7 +1628,7 @@ static int get_cur_inode_state(struct send_ctx *sctx, u64 ino, u64 gen)
 		right_ret = -ENOENT;
 	} else {
 		ret = get_inode_info(sctx->parent_root, ino, NULL, &right_gen,
-				NULL, NULL, NULL, NULL);
+				NULL, NULL, NULL, NULL, NULL);
 		if (ret < 0 && ret != -ENOENT)
 			goto out;
 		right_ret = ret;
@@ -1788,7 +1797,7 @@ static int get_first_ref(struct btrfs_root *root, u64 ino,
 
 	if (dir_gen) {
 		ret = get_inode_info(root, parent_dir, NULL, dir_gen, NULL,
-				     NULL, NULL, NULL);
+				     NULL, NULL, NULL, NULL);
 		if (ret < 0)
 			goto out;
 	}
@@ -1861,7 +1870,7 @@ static int will_overwrite_ref(struct send_ctx *sctx, u64 dir, u64 dir_gen,
 	 */
 	if (sctx->parent_root && dir != BTRFS_FIRST_FREE_OBJECTID) {
 		ret = get_inode_info(sctx->parent_root, dir, NULL, &gen, NULL,
-				     NULL, NULL, NULL);
+				     NULL, NULL, NULL, NULL);
 		if (ret < 0 && ret != -ENOENT)
 			goto out;
 		if (ret) {
@@ -1889,7 +1898,7 @@ static int will_overwrite_ref(struct send_ctx *sctx, u64 dir, u64 dir_gen,
 	if (other_inode > sctx->send_progress ||
 	    is_waiting_for_move(sctx, other_inode)) {
 		ret = get_inode_info(sctx->parent_root, other_inode, NULL,
-				who_gen, who_mode, NULL, NULL, NULL);
+				who_gen, who_mode, NULL, NULL, NULL, NULL);
 		if (ret < 0)
 			goto out;
 
@@ -1929,7 +1938,7 @@ static int did_overwrite_ref(struct send_ctx *sctx,
 
 	if (dir != BTRFS_FIRST_FREE_OBJECTID) {
 		ret = get_inode_info(sctx->send_root, dir, NULL, &gen, NULL,
-				     NULL, NULL, NULL);
+				     NULL, NULL, NULL, NULL);
 		if (ret < 0 && ret != -ENOENT)
 			goto out;
 		if (ret) {
@@ -1952,7 +1961,7 @@ static int did_overwrite_ref(struct send_ctx *sctx,
 	}
 
 	ret = get_inode_info(sctx->send_root, ow_inode, NULL, &gen, NULL, NULL,
-			NULL, NULL);
+			NULL, NULL, NULL);
 	if (ret < 0)
 		goto out;
 
@@ -2525,6 +2534,41 @@ static int send_chown(struct send_ctx *sctx, u64 ino, u64 gen, u64 uid, u64 gid)
 	return ret;
 }
 
+static int send_chattr(struct send_ctx *sctx, u64 ino, u64 gen, u64 flags)
+{
+	struct btrfs_fs_info *fs_info = sctx->send_root->fs_info;
+	int ret = 0;
+	int __flags;
+	struct fs_path *p;
+
+	ASSERT(sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE);
+
+	__flags = btrfs_flags_to_ioctl(flags);
+
+	btrfs_debug(fs_info, "send_chattr %llu flags=%llu", ino, flags);
+
+	p = fs_path_alloc();
+	if (!p)
+		return -ENOMEM;
+
+	ret = begin_cmd(sctx, BTRFS_SEND_C_CHATTR);
+	if (ret < 0)
+		goto out;
+
+	ret = get_cur_path(sctx, ino, gen, p);
+	if (ret < 0)
+		goto out;
+	TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p);
+	TLV_PUT_U64(sctx, BTRFS_SEND_A_CHATTR, __flags);
+
+	ret = send_cmd(sctx);
+
+tlv_put_failure:
+out:
+	fs_path_free(p);
+	return ret;
+}
+
 static int send_utimes(struct send_ctx *sctx, u64 ino, u64 gen)
 {
 	struct btrfs_fs_info *fs_info = sctx->send_root->fs_info;
@@ -2610,7 +2654,7 @@ static int send_create_inode(struct send_ctx *sctx, u64 ino)
 
 	if (ino != sctx->cur_ino) {
 		ret = get_inode_info(sctx->send_root, ino, NULL, &gen, &mode,
-				     NULL, NULL, &rdev);
+				     NULL, NULL, &rdev, NULL);
 		if (ret < 0)
 			goto out;
 	} else {
@@ -3332,7 +3376,7 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 		 * The parent inode might have been deleted in the send snapshot
 		 */
 		ret = get_inode_info(sctx->send_root, cur->dir, NULL,
-				     NULL, NULL, NULL, NULL, NULL);
+				     NULL, NULL, NULL, NULL, NULL, NULL);
 		if (ret == -ENOENT) {
 			ret = 0;
 			continue;
@@ -3502,11 +3546,11 @@ static int wait_for_dest_dir_move(struct send_ctx *sctx,
 	}
 
 	ret = get_inode_info(sctx->parent_root, di_key.objectid, NULL,
-			     &left_gen, NULL, NULL, NULL, NULL);
+			     &left_gen, NULL, NULL, NULL, NULL, NULL);
 	if (ret < 0)
 		goto out;
 	ret = get_inode_info(sctx->send_root, di_key.objectid, NULL,
-			     &right_gen, NULL, NULL, NULL, NULL);
+			     &right_gen, NULL, NULL, NULL, NULL, NULL);
 	if (ret < 0) {
 		if (ret == -ENOENT)
 			ret = 0;
@@ -3650,7 +3694,7 @@ static int is_ancestor(struct btrfs_root *root,
 			}
 
 			ret = get_inode_info(root, parent, NULL, &parent_gen,
-					     NULL, NULL, NULL, NULL);
+					     NULL, NULL, NULL, NULL, NULL);
 			if (ret < 0)
 				goto out;
 			ret = check_ino_in_path(root, ino1, ino1_gen,
@@ -3740,7 +3784,7 @@ static int wait_for_parent_move(struct send_ctx *sctx,
 
 			ret = get_inode_info(sctx->parent_root, ino, NULL,
 					     &parent_ino_gen, NULL, NULL, NULL,
-					     NULL);
+					     NULL, NULL);
 			if (ret < 0)
 				goto out;
 			if (ino_gen == parent_ino_gen) {
@@ -4220,7 +4264,7 @@ static int record_ref(struct btrfs_root *root, u64 dir, struct fs_path *name,
 		return -ENOMEM;
 
 	ret = get_inode_info(root, dir, NULL, &gen, NULL, NULL,
-			NULL, NULL);
+			NULL, NULL, NULL);
 	if (ret < 0)
 		goto out;
 
@@ -4308,7 +4352,7 @@ static int __find_iref(int num, u64 dir, int index,
 		 * else matches.
 		 */
 		ret = get_inode_info(ctx->root, dir, NULL, &dir_gen, NULL,
-				     NULL, NULL, NULL);
+				     NULL, NULL, NULL, NULL);
 		if (ret)
 			return ret;
 		if (dir_gen != ctx->dir_gen)
@@ -4352,7 +4396,7 @@ static int __record_changed_new_ref(int num, u64 dir, int index,
 	struct send_ctx *sctx = ctx;
 
 	ret = get_inode_info(sctx->send_root, dir, NULL, &dir_gen, NULL,
-			     NULL, NULL, NULL);
+			     NULL, NULL, NULL, NULL);
 	if (ret)
 		return ret;
 
@@ -4375,7 +4419,7 @@ static int __record_changed_deleted_ref(int num, u64 dir, int index,
 	struct send_ctx *sctx = ctx;
 
 	ret = get_inode_info(sctx->parent_root, dir, NULL, &dir_gen, NULL,
-			     NULL, NULL, NULL);
+			     NULL, NULL, NULL, NULL);
 	if (ret)
 		return ret;
 
@@ -4975,7 +5019,7 @@ static int send_clone(struct send_ctx *sctx,
 
 	if (clone_root->root == sctx->send_root) {
 		ret = get_inode_info(sctx->send_root, clone_root->ino, NULL,
-				&gen, NULL, NULL, NULL, NULL);
+				&gen, NULL, NULL, NULL, NULL, NULL);
 		if (ret < 0)
 			goto out;
 		ret = get_cur_path(sctx, clone_root->ino, gen, p);
@@ -5934,9 +5978,11 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 	u64 right_mode;
 	u64 right_uid;
 	u64 right_gid;
+	u64 left_flags;
 	int need_chmod = 0;
 	int need_chown = 0;
 	int need_truncate = 1;
+	int need_chattr = 0;
 	int pending_move = 0;
 	int refs_processed = 0;
 
@@ -5944,7 +5990,6 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 					      &refs_processed);
 	if (ret < 0)
 		goto out;
-
 	/*
 	 * We have processed the refs and thus need to advance send_progress.
 	 * Now, calls to get_cur_xxx will take the updated refs of the current
@@ -5962,6 +6007,65 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 
 	if (sctx->cur_ino == 0 || sctx->cur_inode_deleted)
 		goto out;
+
+	/*
+	 * If possible, we want to know what flags are set for this inode on the
+	 * receiving end.
+	 */
+	if (sctx->parent_root && !sctx->receive_flags_valid &&
+	    (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2)) {
+		ret = get_inode_info(sctx->parent_root, sctx->cur_ino, NULL,
+				     NULL, NULL, NULL, NULL, NULL,
+				     &sctx->cur_inode_receive_flags);
+		if (ret < 0)
+			goto out;
+		sctx->receive_flags_valid = true;
+	}
+
+	/*
+	 * The change is going to modify data and the inode already exists
+	 * !at_end prevents unnecessary chattr.
+	 */
+	if (!at_end && sctx->parent_root && !sctx->cur_inode_new &&
+	    (sctx->cmp_key->type == BTRFS_EXTENT_DATA_KEY ||
+	    sctx->cmp_key->type == BTRFS_XATTR_ITEM_KEY) &&
+	    (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) &&
+	    sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE) {
+
+		ret = get_inode_info(sctx->send_root, sctx->cur_ino, NULL, NULL,
+				NULL, NULL, NULL, NULL, &left_flags);
+		if (ret < 0)
+			goto out;
+		/*
+		 * We check against the receive flags first; then check against
+		 * the left flags to see if we can save a chattr later on
+		 */
+		if (sctx->cur_inode_receive_flags & BTRFS_INODE_IMMUTABLE) {
+			sctx->cur_inode_flip_flags |= (left_flags &
+						       BTRFS_INODE_IMMUTABLE);
+			left_flags &= ~BTRFS_INODE_IMMUTABLE;
+			need_chattr = 1;
+		}
+		if (sctx->cur_inode_receive_flags & BTRFS_INODE_APPEND) {
+			sctx->cur_inode_flip_flags |= (left_flags &
+						       BTRFS_INODE_APPEND);
+			left_flags &= ~BTRFS_INODE_APPEND;
+			need_chattr = 1;
+		}
+		if (need_chattr) {
+			need_chattr = 0;
+			ret = send_chattr(sctx, sctx->cur_ino,
+					  sctx->cur_inode_gen, left_flags);
+			if (ret < 0)
+				goto out;
+			/*
+			 * left_flags is now an accurate rep of what the
+			 * receiving inode's flags are
+			 */
+			sctx->cur_inode_receive_flags = left_flags;
+		}
+	}
+
 	if (!at_end && sctx->cmp_key->objectid == sctx->cur_ino)
 		goto out;
 
@@ -5969,7 +6073,7 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 		goto truncate_inode;
 
 	ret = get_inode_info(sctx->send_root, sctx->cur_ino, NULL, NULL,
-			&left_mode, &left_uid, &left_gid, NULL);
+			&left_mode, &left_uid, &left_gid, NULL, &left_flags);
 	if (ret < 0)
 		goto out;
 
@@ -5984,7 +6088,7 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 
 		ret = get_inode_info(sctx->parent_root, sctx->cur_ino,
 				&old_size, NULL, &right_mode, &right_uid,
-				&right_gid, NULL);
+				&right_gid, NULL, NULL);
 		if (ret < 0)
 			goto out;
 
@@ -6015,7 +6119,8 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 					goto out;
 			}
 		}
-		if (need_truncate && !sctx->cur_inode_skip_truncate) {
+		if (sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE &&
+		    need_truncate && !sctx->cur_inode_skip_truncate) {
 			ret = send_truncate(sctx, sctx->cur_ino,
 					    sctx->cur_inode_gen,
 					    sctx->cur_inode_size);
@@ -6059,6 +6164,28 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end)
 			goto out;
 	}
 
+	/*
+	 * At this point, if we toggled stuff earlier, untoggle it
+	 * force a chattr and fix the flags
+	 */
+	if (sctx->cur_inode_flip_flags)
+		left_flags |= sctx->cur_inode_flip_flags;
+
+	/*
+	 * We either need a chattr because this inode is new, or we need to make
+	 * a change due to a discrepancy between left_flags and receive_flags
+	 */
+	if ((need_chattr || (sctx->cur_inode_receive_flags != left_flags)) &&
+	    (sctx->flags & BTRFS_SEND_FLAG_STREAM_V2) &&
+	    sctx->phase != SEND_PHASE_COMPUTE_DATA_SIZE) {
+		ret = send_chattr(sctx, sctx->cur_ino, sctx->cur_inode_gen,
+				  left_flags);
+		if (ret < 0)
+			goto out;
+	}
+	sctx->cur_inode_flip_flags = 0;
+	sctx->cur_inode_receive_flags = 0;
+	sctx->receive_flags_valid = false;
 out:
 	return ret;
 }
@@ -6379,12 +6506,12 @@ static int dir_changed(struct send_ctx *sctx, u64 dir)
 	int ret;
 
 	ret = get_inode_info(sctx->send_root, dir, NULL, &new_gen, NULL, NULL,
-			     NULL, NULL);
+			     NULL, NULL, NULL);
 	if (ret)
 		return ret;
 
 	ret = get_inode_info(sctx->parent_root, dir, NULL, &orig_gen, NULL,
-			     NULL, NULL, NULL);
+			     NULL, NULL, NULL, NULL);
 	if (ret)
 		return ret;
 
-- 
2.17.0

[PATCH v2 1/6] Btrfs-progs: send, bump stream version

[Howard McLauchlan]

From: Filipe Manana <fdmanana@gmail.com>

This increases the send stream version from version 1 to version 2, adding
new commands:

1) total data size - used to tell the receiver how much file data the stream
   will add or update;

2) fallocate - used to pre-allocate space for files and to punch holes in files;

3) inode set flags (chattr);

4) set inode otime;

5) sending compressed writes

This is preparation work for subsequent changes that implement the new features.

This doesn't break compatibility with older kernels or clients. In order to get
a version 2 send stream, new flags must be passed to the send ioctl.

Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
 cmds-send.c   | 61 ++++++++++++++++++++++++++++++++++++---------------
 ioctl.h       | 15 +++++++++++++
 send-stream.c |  2 +-
 send.h        | 24 +++++++++++++++++++-
 4 files changed, 82 insertions(+), 20 deletions(-)

diff --git a/cmds-send.c b/cmds-send.c
index c5ecdaa1..0ec557c7 100644
--- a/cmds-send.c
+++ b/cmds-send.c
@@ -52,6 +52,7 @@
  * the 'At subvol' message.
  */
 static int g_verbose = 1;
+static int g_stream_version = BTRFS_SEND_STREAM_VERSION_1;
 
 struct btrfs_send {
 	int send_fd;
@@ -343,6 +344,8 @@ static int do_send(struct btrfs_send *send, u64 parent_root_id,
 		io_send.flags |= BTRFS_SEND_FLAG_OMIT_STREAM_HEADER;
 	if (!is_last_subvol)
 		io_send.flags |= BTRFS_SEND_FLAG_OMIT_END_CMD;
+	if (g_stream_version == BTRFS_SEND_STREAM_VERSION_2)
+		io_send.flags |= BTRFS_SEND_FLAG_STREAM_V2;
 	ret = ioctl(subvol_fd, BTRFS_IOC_SEND, &io_send);
 	if (ret < 0) {
 		ret = -errno;
@@ -513,7 +516,8 @@ int cmd_send(int argc, char **argv)
 		static const struct option long_options[] = {
 			{ "verbose", no_argument, NULL, 'v' },
 			{ "quiet", no_argument, NULL, 'q' },
-			{ "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA }
+			{ "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA },
+			{ "stream-version", 1, NULL, 'V' },
 		};
 		int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL);
 
@@ -597,6 +601,24 @@ int cmd_send(int argc, char **argv)
 			error("option -i was removed, use -c instead");
 			ret = 1;
 			goto out;
+		case 'V':
+			if (sscanf(optarg, "%d", &g_stream_version) != 1) {
+				fprintf(stderr,
+					"ERROR: invalid value for stream version: %s\n",
+					optarg);
+				ret = 1;
+				goto out;
+			}
+			if (g_stream_version <= 0 ||
+			    g_stream_version > BTRFS_SEND_STREAM_VERSION_MAX) {
+				fprintf(stderr,
+					"ERROR: unsupported stream version %d, minimum: 1, maximum: %d\n",
+					g_stream_version,
+					BTRFS_SEND_STREAM_VERSION_MAX);
+				ret = 1;
+				goto out;
+			}
+			break;
 		case GETOPT_VAL_SEND_NO_DATA:
 			send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA;
 			break;
@@ -776,7 +798,7 @@ out:
 }
 
 const char * const cmd_send_usage[] = {
-	"btrfs send [-ve] [-p <parent>] [-c <clone-src>] [-f <outfile>] <subvol> [<subvol>...]",
+	"btrfs send [-ve] [--stream-version <version>] [-p <parent>] [-c <clone-src>] [-f <outfile>] <subvol> [<subvol>...]",
 	"Send the subvolume(s) to stdout.",
 	"Sends the subvolume(s) specified by <subvol> to stdout.",
 	"<subvol> should be read-only here.",
@@ -790,21 +812,24 @@ const char * const cmd_send_usage[] = {
 	"which case 'btrfs send' will determine a suitable parent among the",
 	"clone sources itself.",
 	"\n",
-	"-e               If sending multiple subvols at once, use the new",
-	"                 format and omit the end-cmd between the subvols.",
-	"-p <parent>      Send an incremental stream from <parent> to",
-	"                 <subvol>.",
-	"-c <clone-src>   Use this snapshot as a clone source for an ",
-	"                 incremental send (multiple allowed)",
-	"-f <outfile>     Output is normally written to stdout. To write to",
-	"                 a file, use this option. An alternative would be to",
-	"                 use pipes.",
-	"--no-data        send in NO_FILE_DATA mode, Note: the output stream",
-	"                 does not contain any file data and thus cannot be used",
-	"                 to transfer changes. This mode is faster and useful to",
-	"                 show the differences in metadata.",
-	"-v|--verbose     enable verbose output to stderr, each occurrence of",
-	"                 this option increases verbosity",
-	"-q|--quiet       suppress all messages, except errors",
+	"-e                          If sending multiple subvols at once, use the new",
+	"                            format and omit the end-cmd between the subvols.",
+	"-p <parent>                 Send an incremental stream from <parent> to",
+	"                            <subvol>.",
+	"-c <clone-src>              Use this snapshot as a clone source for an ",
+	"                            incremental send (multiple allowed)",
+	"-f <outfile>                Output is normally written to stdout. To write to",
+	"                            a file, use this option. An alternative would be to",
+	"                            use pipes.",
+	"--no-data                   send in NO_FILE_DATA mode, Note: the output stream",
+	"                            does not contain any file data and thus cannot be used",
+	"                            to transfer changes. This mode is faster and useful to",
+	"                            show the differences in metadata.",
+	"-v|--verbose                enable verbose output to stderr, each occurrence of",
+	"                            this option increases verbosity",
+	"-q|--quiet       	     suppress all messages, except errors",
+	"--stream-version <version>  Ask the kernel to produce a specific send stream",
+	"                            version. More recent stream versions provide new",
+	"                            features and better performance. Default value is 1.",
 	NULL
 };
diff --git a/ioctl.h b/ioctl.h
index 709e996f..9d490aca 100644
--- a/ioctl.h
+++ b/ioctl.h
@@ -632,6 +632,21 @@ BUILD_ASSERT(sizeof(struct btrfs_ioctl_received_subvol_args_32) == 192);
 	 BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \
 	 BTRFS_SEND_FLAG_OMIT_END_CMD)
 
+/*
+ * The sum of all length fields the receiver will get in write, clone and
+ * fallocate commands.
+ * This can be used by the receiver to compute progress, at the expense of some
+ * initial metadata scan performed by the sender (kernel).
+ *
+ * Added in send stream version 2.
+ */
+#define BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE	0x8
+
+/*
+ * Used by a client to request a version 2 of the send stream.
+ */
+#define BTRFS_SEND_FLAG_STREAM_V2               0x10
+
 struct btrfs_ioctl_send_args {
 	__s64 send_fd;			/* in */
 	__u64 clone_sources_count;	/* in */
diff --git a/send-stream.c b/send-stream.c
index 78f2571a..86956d28 100644
--- a/send-stream.c
+++ b/send-stream.c
@@ -503,7 +503,7 @@ int btrfs_read_and_process_send_stream(int fd,
 	}
 
 	sctx.version = le32_to_cpu(hdr.version);
-	if (sctx.version > BTRFS_SEND_STREAM_VERSION) {
+	if (sctx.version > BTRFS_SEND_STREAM_VERSION_MAX) {
 		ret = -EINVAL;
 		error("stream version %d not supported, please use newer version",
 				sctx.version);
diff --git a/send.h b/send.h
index fe613cbb..c9752bb8 100644
--- a/send.h
+++ b/send.h
@@ -27,7 +27,10 @@ extern "C" {
 #endif
 
 #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream"
-#define BTRFS_SEND_STREAM_VERSION 1
+#define BTRFS_SEND_STREAM_VERSION_1 1
+#define BTRFS_SEND_STREAM_VERSION_2 2
+/* Max supported stream version. */
+#define BTRFS_SEND_STREAM_VERSION_MAX BTRFS_SEND_STREAM_VERSION_2
 
 #define BTRFS_SEND_BUF_SIZE SZ_64K
 #define BTRFS_SEND_READ_SIZE (1024 * 48)
@@ -94,6 +97,16 @@ enum btrfs_send_cmd {
 
 	BTRFS_SEND_C_END,
 	BTRFS_SEND_C_UPDATE_EXTENT,
+
+	/*
+	 * The following commands were added in stream version 2.
+	 */
+	BTRFS_SEND_C_TOTAL_DATA_SIZE,
+	BTRFS_SEND_C_FALLOCATE,
+	BTRFS_SEND_C_CHATTR,
+	BTRFS_SEND_C_UTIMES2, /* Same as UTIMES, but it includes OTIME too. */
+	BTRFS_SEND_C_WRITE_COMPRESSED, /* to be implemented */
+
 	__BTRFS_SEND_C_MAX,
 };
 #define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1)
@@ -132,10 +145,19 @@ enum {
 	BTRFS_SEND_A_CLONE_OFFSET,
 	BTRFS_SEND_A_CLONE_LEN,
 
+	/*
+	 * The following attributes were added in stream version 2.
+	 */
+	BTRFS_SEND_A_FALLOCATE_FLAGS, /* 32 bits */
+	BTRFS_SEND_A_CHATTR,	      /* 32 bits */
+
 	__BTRFS_SEND_A_MAX,
 };
 #define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1)
 
+#define BTRFS_SEND_A_FALLOCATE_FLAG_KEEP_SIZE   (1 << 0)
+#define BTRFS_SEND_A_FALLOCATE_FLAG_PUNCH_HOLE  (1 << 1)
+
 #ifdef __KERNEL__
 long btrfs_ioctl_send(struct file *mnt_file, void __user *arg);
 #endif
-- 
2.17.0

[PATCH v2 2/6] Btrfs-progs: send, implement total data size callback and progress report

[Howard McLauchlan]

From: Filipe Manana <fdmanana@gmail.com>

This is a followup to the kernel patch titled:

    Btrfs: send, implement total data size command to allow for progress estimation

This makes the btrfs send and receive commands aware of the new send flag,
named BTRFS_SEND_C_TOTAL_DATA_SIZE, which tells us the amount of file data
that is new between the parent and send snapshots/roots. As this command
immediately follows the commands to start a snapshot/subvolume, it can be
used to report and compute progress, by keeping a counter that is incremented
with the data length of each write, clone and fallocate command that is received
from the stream.

Example:

    $ btrfs send -s --stream-version 2 /mnt/sdd/snap_base | btrfs receive /mnt/sdc
    At subvol /mnt/sdd/snap_base
    At subvol snap_base
    About to receive 9212392667 bytes
    Subvolume /mnt/sdc//snap_base, 4059722426 / 9212392667 bytes received, 44.07%, 40.32MB/s

    $ btrfs send -s --stream-version 2 -p /mnt/sdd/snap_base /mnt/sdd/snap_incr | btrfs receive /mnt/sdc
    At subvol /mnt/sdd/snap_incr
    At subvol snap_incr
    About to receive 9571342213 bytes
    Subvolume /mnt/sdc//snap_incr, 6557345221 / 9571342213 bytes received, 68.51%, 51.04MB/s

At the moment progress is only reported by btrfs-receive, but it is possible and simple
to do it for btrfs-send too, so that we can get progress report when not piping btrfs-send
output to btrfs-receive (directly to a file).

Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
 cmds-receive.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++
 cmds-send.c    | 23 +++++++++++--
 send-stream.c  |  4 +++
 send-stream.h  |  1 +
 4 files changed, 117 insertions(+), 2 deletions(-)

diff --git a/cmds-receive.c b/cmds-receive.c
index 68123a31..d8ff5194 100644
--- a/cmds-receive.c
+++ b/cmds-receive.c
@@ -30,6 +30,7 @@
 #include <assert.h>
 #include <getopt.h>
 #include <limits.h>
+#include <time.h>
 
 #include <sys/stat.h>
 #include <sys/types.h>
@@ -79,6 +80,14 @@ struct btrfs_receive
 
 	int honor_end_cmd;
 
+	/* For the subvolume/snapshot we're currently receiving. */
+	u64 total_data_size;
+	u64 bytes_received;
+	time_t last_progress_update;
+	u64 bytes_received_last_update;
+	float progress;
+	const char *target;
+
 	/*
 	 * Buffer to store capabilities from security.capabilities xattr,
 	 * usually 20 bytes, but make same room for potentially larger
@@ -156,6 +165,16 @@ out:
 	return ret;
 }
 
+static void reset_progress(struct btrfs_receive *rctx, const char *dest)
+{
+	rctx->total_data_size = 0;
+	rctx->bytes_received = 0;
+	rctx->progress = 0.0;
+	rctx->last_progress_update = 0;
+	rctx->bytes_received_last_update = 0;
+	rctx->target = dest;
+}
+
 static int process_subvol(const char *path, const u8 *uuid, u64 ctransid,
 			  void *user)
 {
@@ -180,6 +199,7 @@ static int process_subvol(const char *path, const u8 *uuid, u64 ctransid,
 		ret = -EINVAL;
 		goto out;
 	}
+	reset_progress(rctx, "Subvolume");
 
 	if (*rctx->dest_dir_path == 0) {
 		strncpy_null(rctx->cur_subvol_path, path);
@@ -249,6 +269,7 @@ static int process_snapshot(const char *path, const u8 *uuid, u64 ctransid,
 		ret = -EINVAL;
 		goto out;
 	}
+	reset_progress(rctx, "Snapshot");
 
 	if (*rctx->dest_dir_path == 0) {
 		strncpy_null(rctx->cur_subvol_path, path);
@@ -388,6 +409,73 @@ out:
 	return ret;
 }
 
+static int process_total_data_size(u64 size, void *user)
+{
+	struct btrfs_receive *rctx = user;
+
+	rctx->total_data_size = size;
+	fprintf(stdout, "About to receive %llu bytes\n", size);
+
+	return 0;
+}
+
+static void update_progress(struct btrfs_receive *rctx, u64 bytes)
+{
+	float new_progress;
+	time_t now;
+	time_t tdiff;
+
+	if (rctx->total_data_size == 0)
+		return;
+
+	rctx->bytes_received += bytes;
+
+	now = time(NULL);
+	tdiff = now - rctx->last_progress_update;
+	if (tdiff < 1) {
+		if (rctx->bytes_received == rctx->total_data_size)
+			fprintf(stdout, "\n");
+		return;
+	}
+
+	new_progress = ((float)rctx->bytes_received / rctx->total_data_size) * 100.0;
+
+	if ((int)(new_progress * 100) > (int)(rctx->progress * 100) ||
+	    rctx->bytes_received == rctx->total_data_size) {
+		char line[5000];
+		float rate = rctx->bytes_received - rctx->bytes_received_last_update;
+		const char *rate_units;
+
+		rate /= tdiff;
+		if (rate > (1024 * 1024)) {
+			rate_units = "MB/s";
+			rate /= 1024 * 1024;
+		} else if (rate > 1024) {
+			rate_units = "KB/s";
+			rate /= 1024;
+		} else {
+			rate_units = "B/s";
+		}
+
+		snprintf(line, sizeof(line),
+			 "%s%s %s, %llu / %llu bytes received, %5.2f%%, %5.2f%s%s",
+			 (g_verbose ? "" : "\r"),
+			 rctx->target,
+			 rctx->full_subvol_path,
+			 rctx->bytes_received, rctx->total_data_size,
+			 new_progress, rate, rate_units,
+			 (g_verbose ? "\n" : ""));
+		fprintf(stdout, "%s%s", line, (g_verbose ? "" : "        "));
+		fflush(stdout);
+	}
+
+	if (rctx->bytes_received == rctx->total_data_size)
+		fprintf(stdout, "\n");
+	rctx->progress = new_progress;
+	rctx->last_progress_update = now;
+	rctx->bytes_received_last_update = rctx->bytes_received;
+}
+
 static int process_mkfile(const char *path, void *user)
 {
 	int ret;
@@ -722,6 +810,7 @@ static int process_write(const char *path, const void *data, u64 offset,
 		}
 		pos += w;
 	}
+	update_progress(rctx, len);
 
 out:
 	return ret;
@@ -827,6 +916,7 @@ static int process_clone(const char *path, u64 offset, u64 len,
 				path, strerror(-ret));
 		goto out;
 	}
+	update_progress(rctx, len);
 
 out:
 	if (si) {
@@ -1081,6 +1171,7 @@ static struct btrfs_send_ops send_ops = {
 	.chown = process_chown,
 	.utimes = process_utimes,
 	.update_extent = process_update_extent,
+	.total_data_size = process_total_data_size,
 };
 
 static int do_receive(struct btrfs_receive *rctx, const char *tomnt,
diff --git a/cmds-send.c b/cmds-send.c
index 0ec557c7..45e30f53 100644
--- a/cmds-send.c
+++ b/cmds-send.c
@@ -53,6 +53,7 @@
  */
 static int g_verbose = 1;
 static int g_stream_version = BTRFS_SEND_STREAM_VERSION_1;
+static int g_total_data_size = 0;
 
 struct btrfs_send {
 	int send_fd;
@@ -346,6 +347,8 @@ static int do_send(struct btrfs_send *send, u64 parent_root_id,
 		io_send.flags |= BTRFS_SEND_FLAG_OMIT_END_CMD;
 	if (g_stream_version == BTRFS_SEND_STREAM_VERSION_2)
 		io_send.flags |= BTRFS_SEND_FLAG_STREAM_V2;
+	if (g_total_data_size)
+		io_send.flags |= BTRFS_SEND_FLAG_CALCULATE_DATA_SIZE;
 	ret = ioctl(subvol_fd, BTRFS_IOC_SEND, &io_send);
 	if (ret < 0) {
 		ret = -errno;
@@ -519,7 +522,7 @@ int cmd_send(int argc, char **argv)
 			{ "no-data", no_argument, NULL, GETOPT_VAL_SEND_NO_DATA },
 			{ "stream-version", 1, NULL, 'V' },
 		};
-		int c = getopt_long(argc, argv, "vqec:f:i:p:", long_options, NULL);
+		int c = getopt_long(argc, argv, "vqesc:f:i:p:", long_options, NULL);
 
 		if (c < 0)
 			break;
@@ -619,6 +622,9 @@ int cmd_send(int argc, char **argv)
 				goto out;
 			}
 			break;
+		case 's':
+			g_total_data_size = 1;
+			break;
 		case GETOPT_VAL_SEND_NO_DATA:
 			send_flags |= BTRFS_SEND_FLAG_NO_FILE_DATA;
 			break;
@@ -633,6 +639,14 @@ int cmd_send(int argc, char **argv)
 	if (check_argc_min(argc - optind, 1))
 		usage(cmd_send_usage);
 
+	if (g_total_data_size &&
+	    g_stream_version < BTRFS_SEND_STREAM_VERSION_2) {
+		fprintf(stderr,
+			"ERROR: option total data size (-s) requires use of the send stream version 2 or higher\n");
+		ret = 1;
+		goto out;
+	}
+
 	if (outname[0]) {
 		int tmpfd;
 
@@ -798,7 +812,7 @@ out:
 }
 
 const char * const cmd_send_usage[] = {
-	"btrfs send [-ve] [--stream-version <version>] [-p <parent>] [-c <clone-src>] [-f <outfile>] <subvol> [<subvol>...]",
+	"btrfs send [-ves] [--stream-version <version>] [-p <parent>] [-c <clone-src>] [-f <outfile>] <subvol> [<subvol>...]",
 	"Send the subvolume(s) to stdout.",
 	"Sends the subvolume(s) specified by <subvol> to stdout.",
 	"<subvol> should be read-only here.",
@@ -831,5 +845,10 @@ const char * const cmd_send_usage[] = {
 	"--stream-version <version>  Ask the kernel to produce a specific send stream",
 	"                            version. More recent stream versions provide new",
 	"                            features and better performance. Default value is 1.",
+	"-s                          Obtain the total data size for each subvolume or ",
+	"                            snapshot to send. This demands additional processing",
+	"                            (mostly IO bound) but is useful for the receive ",
+	"                            command to report progress. This option requires use",
+	"                            of the send stream version 2 or higher.",
 	NULL
 };
diff --git a/send-stream.c b/send-stream.c
index 86956d28..d30fd5a7 100644
--- a/send-stream.c
+++ b/send-stream.c
@@ -453,6 +453,10 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx)
 		TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp);
 		ret = sctx->ops->update_extent(path, offset, tmp, sctx->user);
 		break;
+	case BTRFS_SEND_C_TOTAL_DATA_SIZE:
+		TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp);
+		ret = sctx->ops->total_data_size(tmp, sctx->user);
+		break;
 	case BTRFS_SEND_C_END:
 		ret = 1;
 		break;
diff --git a/send-stream.h b/send-stream.h
index 39901f86..5b244ab6 100644
--- a/send-stream.h
+++ b/send-stream.h
@@ -66,6 +66,7 @@ struct btrfs_send_ops {
 		      struct timespec *mt, struct timespec *ct,
 		      void *user);
 	int (*update_extent)(const char *path, u64 offset, u64 len, void *user);
+	int (*total_data_size)(u64 size, void *user);
 };
 
 int btrfs_read_and_process_send_stream(int fd,
-- 
2.17.0

[PATCH v2 3/6] Btrfs-progs: send, implement fallocate command callback

[Howard McLauchlan]

From: Filipe Manana <fdmanana@gmail.com>

The fallocate send stream command, added in stream version 2, is used to
pre-allocate space for files and punch file holes. This change implements
the callback for that new command, using the fallocate function from the
standard C library to carry out the specified action (allocate file space
or punch a file hole).

Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
 cmds-receive.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 send-stream.c  | 13 +++++++++++++
 send-stream.h  |  2 ++
 3 files changed, 59 insertions(+)

diff --git a/cmds-receive.c b/cmds-receive.c
index d8ff5194..510b6bc8 100644
--- a/cmds-receive.c
+++ b/cmds-receive.c
@@ -39,6 +39,7 @@
 #include <sys/types.h>
 #include <sys/xattr.h>
 #include <uuid/uuid.h>
+#include <linux/falloc.h>
 
 #include "ctree.h"
 #include "ioctl.h"
@@ -1149,6 +1150,48 @@ static int process_update_extent(const char *path, u64 offset, u64 len,
 	return 0;
 }
 
+static int process_fallocate(const char *path, u32 flags, u64 offset,
+			     u64 len, void *user)
+{
+	struct btrfs_receive *rctx = user;
+	int mode = 0;
+	int ret;
+	char full_path[PATH_MAX];
+
+	ret = path_cat_out(full_path, rctx->full_subvol_path, path);
+	if (ret < 0) {
+		error("fallocate: path invalid: %s", path);
+		goto out;
+	}
+
+	if (flags & BTRFS_SEND_A_FALLOCATE_FLAG_KEEP_SIZE)
+		mode |= FALLOC_FL_KEEP_SIZE;
+	if (flags & BTRFS_SEND_A_FALLOCATE_FLAG_PUNCH_HOLE)
+		mode |= FALLOC_FL_PUNCH_HOLE;
+
+	if (g_verbose >= 2)
+		fprintf(stderr,
+			"fallocate %s - flags %u, offset %llu, len %llu\n",
+			path, flags, offset, len);
+
+	ret = open_inode_for_write(rctx, full_path);
+	if (ret < 0)
+		goto out;
+
+	ret = fallocate(rctx->write_fd, mode, offset, len);
+	if (ret) {
+		ret = -errno;
+		fprintf(stderr,
+			"ERROR: fallocate against %s failed. %s\n",
+			path, strerror(-ret));
+		goto out;
+	}
+	update_progress(rctx, len);
+
+out:
+	return ret;
+}
+
 static struct btrfs_send_ops send_ops = {
 	.subvol = process_subvol,
 	.snapshot = process_snapshot,
@@ -1172,6 +1215,7 @@ static struct btrfs_send_ops send_ops = {
 	.utimes = process_utimes,
 	.update_extent = process_update_extent,
 	.total_data_size = process_total_data_size,
+	.fallocate = process_fallocate,
 };
 
 static int do_receive(struct btrfs_receive *rctx, const char *tomnt,
diff --git a/send-stream.c b/send-stream.c
index d30fd5a7..74ec37dd 100644
--- a/send-stream.c
+++ b/send-stream.c
@@ -457,6 +457,19 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx)
 		TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &tmp);
 		ret = sctx->ops->total_data_size(tmp, sctx->user);
 		break;
+	case BTRFS_SEND_C_FALLOCATE:
+		{
+			u32 flags;
+			u64 len;
+
+			TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path);
+			TLV_GET_U32(sctx, BTRFS_SEND_A_FALLOCATE_FLAGS, &flags);
+			TLV_GET_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, &offset);
+			TLV_GET_U64(sctx, BTRFS_SEND_A_SIZE, &len);
+			ret = sctx->ops->fallocate(path, flags, offset, len,
+						sctx->user);
+		}
+		break;
 	case BTRFS_SEND_C_END:
 		ret = 1;
 		break;
diff --git a/send-stream.h b/send-stream.h
index 5b244ab6..89e64043 100644
--- a/send-stream.h
+++ b/send-stream.h
@@ -67,6 +67,8 @@ struct btrfs_send_ops {
 		      void *user);
 	int (*update_extent)(const char *path, u64 offset, u64 len, void *user);
 	int (*total_data_size)(u64 size, void *user);
+	int (*fallocate)(const char *path, u32 flags, u64 offset,
+			 u64 len, void *user);
 };
 
 int btrfs_read_and_process_send_stream(int fd,
-- 
2.17.0

[PATCH v2 4/6] Btrfs-progs: add write and clone commands debug info to receive

[Howard McLauchlan]

From: Filipe Manana <fdmanana@gmail.com>

When specifying -vv print information about received write and clone commands too,
as we do this for other commands already and it's very useful for debugging and
troubleshooting.

Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
 cmds-receive.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/cmds-receive.c b/cmds-receive.c
index 510b6bc8..20e593f7 100644
--- a/cmds-receive.c
+++ b/cmds-receive.c
@@ -790,6 +790,10 @@ static int process_write(const char *path, const void *data, u64 offset,
 	u64 pos = 0;
 	int w;
 
+	if (g_verbose >= 2)
+		fprintf(stderr, "write %s, offset %llu, len %llu\n",
+			path, offset, len);
+
 	ret = path_cat_out(full_path, rctx->full_subvol_path, path);
 	if (ret < 0) {
 		error("write: path invalid: %s", path);
@@ -831,6 +835,11 @@ static int process_clone(const char *path, u64 offset, u64 len,
 	char full_clone_path[PATH_MAX];
 	int clone_fd = -1;
 
+	if (g_verbose >= 2)
+		fprintf(stderr,
+			"clone %s, offset %llu, len %llu, clone path %s, clone offset %llu\n",
+			path, offset, len, clone_path, clone_offset);
+
 	ret = path_cat_out(full_path, rctx->full_subvol_path, path);
 	if (ret < 0) {
 		error("clone: source path invalid: %s", path);
-- 
2.17.0

[PATCH v2 5/6] btrfs-progs: add total data size, fallocate to dump

[Howard McLauchlan]

From: Howard McLauchlan <hmclauchlan@fb.com>

Adding entries to dump for new commands (total data size, fallocate).

Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
---
 send-dump.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/send-dump.c b/send-dump.c
index 1591e0cc..c5a695a2 100644
--- a/send-dump.c
+++ b/send-dump.c
@@ -316,6 +316,21 @@ static int print_update_extent(const char *path, u64 offset, u64 len,
 			  offset, len);
 }
 
+static int print_total_data_size(u64 size, void *user)
+{
+	char path;
+
+	return PRINT_DUMP(user, &path, "total_data_size", "size=%llu", size);
+}
+
+static int print_fallocate(const char *path, u32 flags, u64 offset, u64 len,
+			   void *user)
+{
+	return PRINT_DUMP(user, path, "fallocate",
+			  "flags=%u offset=%llu len=%llu", flags, offset,
+			  len);
+}
+
 struct btrfs_send_ops btrfs_print_send_ops = {
 	.subvol = print_subvol,
 	.snapshot = print_snapshot,
@@ -337,5 +352,7 @@ struct btrfs_send_ops btrfs_print_send_ops = {
 	.chmod = print_chmod,
 	.chown = print_chown,
 	.utimes = print_utimes,
-	.update_extent = print_update_extent
+	.update_extent = print_update_extent,
+	.total_data_size = print_total_data_size,
+	.fallocate = print_fallocate,
 };
-- 
2.17.0

[PATCH v2 6/6] btrfs-progs: add chattr support for send/receive

[Howard McLauchlan]

From: Howard McLauchlan <hmclauchlan@fb.com>

Presently, btrfs send/receive does not propagate inode attribute flags;
all chattr operations are effectively discarded upon transmission.

This patch adds userspace support for inode attribute flags. Kernel
support can be found under the commit:

    btrfs: add chattr support for send/receive

An associated xfstest can also be found at:

    btrfs: verify chattr support for send/receive test

Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
---
 cmds-receive.c | 37 +++++++++++++++++++++++++++++++++++++
 send-dump.c    |  6 ++++++
 send-stream.c  |  5 +++++
 send-stream.h  |  1 +
 4 files changed, 49 insertions(+)

diff --git a/cmds-receive.c b/cmds-receive.c
index 20e593f7..2a841bfc 100644
--- a/cmds-receive.c
+++ b/cmds-receive.c
@@ -40,6 +40,7 @@
 #include <sys/xattr.h>
 #include <uuid/uuid.h>
 #include <linux/falloc.h>
+#include <linux/fs.h>
 
 #include "ctree.h"
 #include "ioctl.h"
@@ -1201,6 +1202,41 @@ out:
 	return ret;
 }
 
+static int process_chattr(const char *path, u64 flags, void *user)
+{
+	int ret = 0;
+	int fd = 0;
+	int _flags = flags;
+	struct btrfs_receive *rctx = user;
+	char full_path[PATH_MAX];
+
+	ret = path_cat_out(full_path, rctx->full_subvol_path, path);
+	if (ret < 0) {
+		error("chattr: path invalid: %s", path);
+		goto out;
+	}
+
+	if (g_verbose >= 2)
+		fprintf(stderr, "chattr %s - flags=0%o\n", path, (int)flags);
+
+	fd = open(full_path, O_RDONLY);
+	if (fd < 0) {
+		ret = -errno;
+		error("cannot open %s: %s", path, strerror(-ret));
+		goto out;
+	}
+
+	ret = ioctl(fd, FS_IOC_SETFLAGS, &_flags);
+	if (ret < 0) {
+		ret = -errno;
+		error("chattr %s failed: %s", path, strerror(-ret));
+		goto out;
+	}
+
+out:
+	return ret;
+}
+
 static struct btrfs_send_ops send_ops = {
 	.subvol = process_subvol,
 	.snapshot = process_snapshot,
@@ -1225,6 +1261,7 @@ static struct btrfs_send_ops send_ops = {
 	.update_extent = process_update_extent,
 	.total_data_size = process_total_data_size,
 	.fallocate = process_fallocate,
+	.chattr = process_chattr,
 };
 
 static int do_receive(struct btrfs_receive *rctx, const char *tomnt,
diff --git a/send-dump.c b/send-dump.c
index c5a695a2..15aea402 100644
--- a/send-dump.c
+++ b/send-dump.c
@@ -331,6 +331,11 @@ static int print_fallocate(const char *path, u32 flags, u64 offset, u64 len,
 			  len);
 }
 
+static int print_chattr(const char *path, u64 flags, void *user)
+{
+	return PRINT_DUMP(user, path, "chattr", "flags=%llu", flags);
+}
+
 struct btrfs_send_ops btrfs_print_send_ops = {
 	.subvol = print_subvol,
 	.snapshot = print_snapshot,
@@ -355,4 +360,5 @@ struct btrfs_send_ops btrfs_print_send_ops = {
 	.update_extent = print_update_extent,
 	.total_data_size = print_total_data_size,
 	.fallocate = print_fallocate,
+	.chattr = print_chattr,
 };
diff --git a/send-stream.c b/send-stream.c
index 74ec37dd..4f26fae3 100644
--- a/send-stream.c
+++ b/send-stream.c
@@ -470,6 +470,11 @@ static int read_and_process_cmd(struct btrfs_send_stream *sctx)
 						sctx->user);
 		}
 		break;
+	case BTRFS_SEND_C_CHATTR:
+		TLV_GET_STRING(sctx, BTRFS_SEND_A_PATH, &path);
+		TLV_GET_U64(sctx, BTRFS_SEND_A_CHATTR, &tmp);
+		ret = sctx->ops->chattr(path, tmp, sctx->user);
+		break;
 	case BTRFS_SEND_C_END:
 		ret = 1;
 		break;
diff --git a/send-stream.h b/send-stream.h
index 89e64043..a9f08d52 100644
--- a/send-stream.h
+++ b/send-stream.h
@@ -69,6 +69,7 @@ struct btrfs_send_ops {
 	int (*total_data_size)(u64 size, void *user);
 	int (*fallocate)(const char *path, u32 flags, u64 offset,
 			 u64 len, void *user);
+	int (*chattr)(const char *path, u64 flags, void *user);
 };
 
 int btrfs_read_and_process_send_stream(int fd,
-- 
2.17.0

Re: [RFC PATCH v2 0/6] btrfs send stream version 2

[David Sterba]

On Sun, May 27, 2018 at 10:14:29PM -0400, Howard McLauchlan wrote:
> This is v2 of send stream version 2. The goal is to provide proper
> versioning/compatibility as new features are implemented. v1 can be found here
> [1].

We need to decide the overall approach to the versioning updates. The
wiki page has several problems with v1 that I don't see addressed in
this patchset. So the question is:

1) fix everything we know about now in send protocol v2

2) incremental fixes (eg. this patchset) and version bumps as the missing
   features/bugs get fixed

I'd vote for 1 because 2 is likely to cause usability problems with
kernel and clients with different version support. But eg. rsync has
protocol version 30 and maybe more so handful of versions does not need
to be an issue in the end. We should be ready with version updates for
btrfs but at this moment I haven't thought about all the usability
issues for 2).