[PATCH] Btrfs: send - sparse file support for btrfs-send mechanism

[PATCH] Btrfs: send - sparse file support for btrfs-send mechanism

[Chen Yang]

when send/receive a sparse file, the holes of the original file
will be filled with zero. The holes will be sent as ZERO streams,
and it's unnecessary.

So, I improved this by skipping the hole of file while sending.

Signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
---
 fs/btrfs/send.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index e78b297..1e1d59a 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,
 	u64 pos = 0;
 	u64 len;
 	u32 l;
+	u64 bytenr;
 	u8 type;
 
 	ei = btrfs_item_ptr(path->nodes[0], path->slots[0],
@@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,
 		 */
 		len = PAGE_CACHE_ALIGN(len);
 	} else {
+		bytenr = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
+		if (bytenr == 0) {
+			ret = 0;
+			goto out;
+		}
 		len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
 	}
 
-- 
1.7.7.6

[PATCH] Btrfs: send - pre-allocated file support for btrfs-send mechanism

[Chen Yang]

>From 4222aa3a32cf4db161fcbbb87ad29b714f1bfaf2 Mon Sep 17 00:00:00 2001
From: Chen Yang <chenyang.fnst@cn.fujitsu.com>
Date: Thu, 29 Nov 2012 13:22:09 +0800
Subject: [PATCH] Btrfs: send - pre-allocated file  support for btrfs-send
 mechanism

when send or receive a file which has pre-allocated part,
the pre-allocated parts of the file will be sent as ZERO streams,
and it's unnecessary.

So we can improve this by skip these EMPTY parts.

Signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
---
this patch is based on
"Btrfs: send - sparse file support for btrfs-send mechanism"
---
 fs/btrfs/send.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 1e1d59a..2d23184 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -3732,13 +3732,18 @@ static int send_write_or_clone(struct send_ctx *sctx,
 		 * sure to send the whole thing
 		 */
 		len = PAGE_CACHE_ALIGN(len);
-	} else {
+	} else if (type == BTRFS_FILE_EXTENT_REG) {
 		bytenr = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
 		if (bytenr == 0) {
 			ret = 0;
 			goto out;
 		}
 		len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
+	} else if (type == BTRFS_FILE_EXTENT_PREALLOC) {
+		ret = 0;
+		goto out;
+	} else {
+		BUG();
 	}
 
 	if (offset + len > sctx->cur_inode_size)
-- 
1.7.7.6

Re: [PATCH] Btrfs: send - sparse file support for btrfs-send mechanism

[Alexander Block]

On Thu, Nov 29, 2012 at 4:11 AM, Chen Yang <chenyang.fnst@cn.fujitsu.com> wrote:
> when send/receive a sparse file, the holes of the original file
> will be filled with zero. The holes will be sent as ZERO streams,
> and it's unnecessary.
>
> So, I improved this by skipping the hole of file while sending.
>
> Signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
> ---
>  fs/btrfs/send.c |    6 ++++++
>  1 files changed, 6 insertions(+), 0 deletions(-)
>
> diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
> index e78b297..1e1d59a 100644
> --- a/fs/btrfs/send.c
> +++ b/fs/btrfs/send.c
> @@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,
>         u64 pos = 0;
>         u64 len;
>         u32 l;
> +       u64 bytenr;
>         u8 type;
>
>         ei = btrfs_item_ptr(path->nodes[0], path->slots[0],
> @@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,
>                  */
>                 len = PAGE_CACHE_ALIGN(len);
>         } else {
> +               bytenr = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
> +               if (bytenr == 0) {
> +                       ret = 0;
> +                       goto out;
> +               }
>                 len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
>         }
>
> --
> 1.7.7.6
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

This won't work for incremental sends if I understand it correctly. If
the hole got punched into the file after the initial send, the data
will be unchanged on the receiving side when receiving incrementally.

Re: [PATCH] Btrfs: send - pre-allocated file support for btrfs-send mechanism

[Alexander Block]

On Thu, Nov 29, 2012 at 7:06 AM, Chen Yang <chenyang.fnst@cn.fujitsu.com> wrote:
> From 4222aa3a32cf4db161fcbbb87ad29b714f1bfaf2 Mon Sep 17 00:00:00 2001
> From: Chen Yang <chenyang.fnst@cn.fujitsu.com>
> Date: Thu, 29 Nov 2012 13:22:09 +0800
> Subject: [PATCH] Btrfs: send - pre-allocated file  support for btrfs-send
>  mechanism
>
> when send or receive a file which has pre-allocated part,
> the pre-allocated parts of the file will be sent as ZERO streams,
> and it's unnecessary.
>
> So we can improve this by skip these EMPTY parts.
>
> Signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
> ---
> this patch is based on
> "Btrfs: send - sparse file support for btrfs-send mechanism"
> ---
>  fs/btrfs/send.c |    7 ++++++-
>  1 files changed, 6 insertions(+), 1 deletions(-)
>
> diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
> index 1e1d59a..2d23184 100644
> --- a/fs/btrfs/send.c
> +++ b/fs/btrfs/send.c
> @@ -3732,13 +3732,18 @@ static int send_write_or_clone(struct send_ctx *sctx,
>                  * sure to send the whole thing
>                  */
>                 len = PAGE_CACHE_ALIGN(len);
> -       } else {
> +       } else if (type == BTRFS_FILE_EXTENT_REG) {
>                 bytenr = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
>                 if (bytenr == 0) {
>                         ret = 0;
>                         goto out;
>                 }
>                 len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
> +       } else if (type == BTRFS_FILE_EXTENT_PREALLOC) {
> +               ret = 0;
> +               goto out;
> +       } else {
> +               BUG();
>         }
>
>         if (offset + len > sctx->cur_inode_size)
> --
> 1.7.7.6
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

The same as in my previous mail applies here.

Re: [PATCH] Btrfs: send - pre-allocated file support for btrfs-send mechanism

[Alex Lyakas]

Hi,
and also, I think, for PREALLOC extents, it is not enough just to skip
sending the data, but possibly need to perform preallocation on the
receive side, so a new SEND command might be needed.

Alex.

On Thu, Nov 29, 2012 at 10:55 AM, Alexander Block <ablock84@gmail.com> wrote:
> On Thu, Nov 29, 2012 at 7:06 AM, Chen Yang <chenyang.fnst@cn.fujitsu.com> wrote:
>> From 4222aa3a32cf4db161fcbbb87ad29b714f1bfaf2 Mon Sep 17 00:00:00 2001
>> From: Chen Yang <chenyang.fnst@cn.fujitsu.com>
>> Date: Thu, 29 Nov 2012 13:22:09 +0800
>> Subject: [PATCH] Btrfs: send - pre-allocated file  support for btrfs-send
>>  mechanism
>>
>> when send or receive a file which has pre-allocated part,
>> the pre-allocated parts of the file will be sent as ZERO streams,
>> and it's unnecessary.
>>
>> So we can improve this by skip these EMPTY parts.
>>
>> Signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
>> ---
>> this patch is based on
>> "Btrfs: send - sparse file support for btrfs-send mechanism"
>> ---
>>  fs/btrfs/send.c |    7 ++++++-
>>  1 files changed, 6 insertions(+), 1 deletions(-)
>>
>> diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
>> index 1e1d59a..2d23184 100644
>> --- a/fs/btrfs/send.c
>> +++ b/fs/btrfs/send.c
>> @@ -3732,13 +3732,18 @@ static int send_write_or_clone(struct send_ctx *sctx,
>>                  * sure to send the whole thing
>>                  */
>>                 len = PAGE_CACHE_ALIGN(len);
>> -       } else {
>> +       } else if (type == BTRFS_FILE_EXTENT_REG) {
>>                 bytenr = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
>>                 if (bytenr == 0) {
>>                         ret = 0;
>>                         goto out;
>>                 }
>>                 len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
>> +       } else if (type == BTRFS_FILE_EXTENT_PREALLOC) {
>> +               ret = 0;
>> +               goto out;
>> +       } else {
>> +               BUG();
>>         }
>>
>>         if (offset + len > sctx->cur_inode_size)
>> --
>> 1.7.7.6
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> The same as in my previous mail applies here.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: send - sparse file support for btrfs-send mechanism

[Alex Lyakas]

Hi Chen, Alexander, Jan,
how about the following patch: it works for disknr==0 extents (to
handle PREALLOC extents correctly, we need to add a new command, which
will break existing btrfs-progs, because they are updated slower than
the kernel side).

diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 5445454..2b46d92 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -3946,16 +3946,38 @@ static int process_extent(struct send_ctx *sctx,
        if (sctx->parent_root && !sctx->cur_inode_new) {
                ret = is_extent_unchanged(sctx, path, key);
                if (ret < 0)
                        goto out;
                if (ret) {
                        ret = 0;
                        goto out;
                }
+       } else {
+               struct extent_buffer *eb;
+               struct btrfs_file_extent_item *ei;
+               u8 extent_type;
+               u64 extent_disknr;
+
+               eb = path->nodes[0];
+               ei = btrfs_item_ptr(eb, path->slots[0],
+                               struct btrfs_file_extent_item);
+
+               extent_type = btrfs_file_extent_type(eb, ei);
+               extent_disknr = btrfs_file_extent_disk_bytenr(eb, ei);
+               if (extent_type == BTRFS_FILE_EXTENT_REG &&
extent_disknr == 0) {
+                       /*
+                        * This is disknr=0 extent in a full-send or a new inode
+                        * in a diff-send. Since we will send truncate command
+                        * in finish_inode_if_needed anyways, the
inode size will be
+                        * correct, and we don't have to send all-zero data.
+                        */
+                       ret = 0;
+                       goto out;
+               }
        }

        ret = find_extent_clone(sctx, path, key->objectid, key->offset,
                        sctx->cur_inode_size, &found_clone);
        if (ret != -ENOENT && ret < 0)
                goto out;

        ret = send_write_or_clone(sctx, path, key, found_clone);

Thanks,
Alex.


On Thu, Nov 29, 2012 at 10:54 AM, Alexander Block <ablock84@gmail.com> wrote:
> On Thu, Nov 29, 2012 at 4:11 AM, Chen Yang <chenyang.fnst@cn.fujitsu.com> wrote:
>> when send/receive a sparse file, the holes of the original file
>> will be filled with zero. The holes will be sent as ZERO streams,
>> and it's unnecessary.
>>
>> So, I improved this by skipping the hole of file while sending.
>>
>> Signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
>> ---
>>  fs/btrfs/send.c |    6 ++++++
>>  1 files changed, 6 insertions(+), 0 deletions(-)
>>
>> diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
>> index e78b297..1e1d59a 100644
>> --- a/fs/btrfs/send.c
>> +++ b/fs/btrfs/send.c
>> @@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,
>>         u64 pos = 0;
>>         u64 len;
>>         u32 l;
>> +       u64 bytenr;
>>         u8 type;
>>
>>         ei = btrfs_item_ptr(path->nodes[0], path->slots[0],
>> @@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,
>>                  */
>>                 len = PAGE_CACHE_ALIGN(len);
>>         } else {
>> +               bytenr = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
>> +               if (bytenr == 0) {
>> +                       ret = 0;
>> +                       goto out;
>> +               }
>>                 len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
>>         }
>>
>> --
>> 1.7.7.6
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> This won't work for incremental sends if I understand it correctly. If
> the hole got punched into the file after the initial send, the data
> will be unchanged on the receiving side when receiving incrementally.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: send - sparse file support for btrfs-send mechanism

[Alex Lyakas]

Hi,
I believe there are two more cases, where we can safely avoid shipping
disknr=0 extent. These are cases, in which the file grows between the
two snapshots, and some of the new extents are no-data extents. So the
whole patch becomes:


diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 5445454..0bd96fe 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -3839,17 +3839,18 @@ static int is_extent_unchanged(struct send_ctx *sctx,
 	/*
 	 * Handle special case where the right side has no extents at all.
 	 */
 	eb = path->nodes[0];
 	slot = path->slots[0];
 	btrfs_item_key_to_cpu(eb, &found_key, slot);
 	if (found_key.objectid != key.objectid ||
 	    found_key.type != key.type) {
-		ret = 0;
+	    /* No need to send a no-data extent it in this case */
+		ret = (left_disknr == 0) ? 1 : 0;
 		goto out;
 	}

 	/*
 	 * We're now on 2a, 2b or 7.
 	 */
 	key = found_key;
 	while (key.offset < ekey->offset + left_len) {
@@ -3865,17 +3866,18 @@ static int is_extent_unchanged(struct send_ctx *sctx,
 			goto out;
 		}

 		/*
 		 * Are we at extent 8? If yes, we know the extent is changed.
 		 * This may only happen on the first iteration.
 		 */
 		if (found_key.offset + right_len <= ekey->offset) {
-			ret = 0;
+			/* No need to send a no-data extent it in this case */
+			ret = (left_disknr == 0) ? 1 : 0;
 			goto out;
 		}

 		left_offset_fixed = left_offset;
 		if (key.offset < ekey->offset) {
 			/* Fix the right offset for 2a and 7. */
 			right_offset += ekey->offset - key.offset;
 		} else {
@@ -3946,16 +3948,38 @@ static int process_extent(struct send_ctx *sctx,
 	if (sctx->parent_root && !sctx->cur_inode_new) {
 		ret = is_extent_unchanged(sctx, path, key);
 		if (ret < 0)
 			goto out;
 		if (ret) {
 			ret = 0;
 			goto out;
 		}
+	} else {
+		struct extent_buffer *eb;
+		struct btrfs_file_extent_item *ei;
+		u8 extent_type;
+		u64 extent_disknr;
+
+		eb = path->nodes[0];
+		ei = btrfs_item_ptr(eb, path->slots[0],
+				struct btrfs_file_extent_item);
+
+		extent_type = btrfs_file_extent_type(eb, ei);
+		extent_disknr = btrfs_file_extent_disk_bytenr(eb, ei);
+		if (extent_type == BTRFS_FILE_EXTENT_REG && extent_disknr == 0) {
+			/*
+			 * This is disknr=0 extent in a full-send or a new inode
+			 * in a diff-send. Since we will send truncate command
+			 * in finish_inode_if_needed anyways, the inode size will be
+			 * correct, and we don't have to send all-zero data.
+			 */
+			ret = 0;
+			goto out;
+		}
 	}

 	ret = find_extent_clone(sctx, path, key->objectid, key->offset,
 			sctx->cur_inode_size, &found_clone);
 	if (ret != -ENOENT && ret < 0)
 		goto out;

 	ret = send_write_or_clone(sctx, path, key, found_clone);


Alex.


On Tue, Jan 8, 2013 at 6:06 PM, Alex Lyakas <alex@zadarastorage.com> wrote:
> Hi Chen, Alexander, Jan,
> how about the following patch: it works for disknr==0 extents (to
> handle PREALLOC extents correctly, we need to add a new command, which
> will break existing btrfs-progs, because they are updated slower than
> the kernel side).
>
> diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
> index 5445454..2b46d92 100644
> --- a/fs/btrfs/send.c
> +++ b/fs/btrfs/send.c
> @@ -3946,16 +3946,38 @@ static int process_extent(struct send_ctx *sctx,
>         if (sctx->parent_root && !sctx->cur_inode_new) {
>                 ret = is_extent_unchanged(sctx, path, key);
>                 if (ret < 0)
>                         goto out;
>                 if (ret) {
>                         ret = 0;
>                         goto out;
>                 }
> +       } else {
> +               struct extent_buffer *eb;
> +               struct btrfs_file_extent_item *ei;
> +               u8 extent_type;
> +               u64 extent_disknr;
> +
> +               eb = path->nodes[0];
> +               ei = btrfs_item_ptr(eb, path->slots[0],
> +                               struct btrfs_file_extent_item);
> +
> +               extent_type = btrfs_file_extent_type(eb, ei);
> +               extent_disknr = btrfs_file_extent_disk_bytenr(eb, ei);
> +               if (extent_type == BTRFS_FILE_EXTENT_REG &&
> extent_disknr == 0) {
> +                       /*
> +                        * This is disknr=0 extent in a full-send or a new inode
> +                        * in a diff-send. Since we will send truncate command
> +                        * in finish_inode_if_needed anyways, the
> inode size will be
> +                        * correct, and we don't have to send all-zero data.
> +                        */
> +                       ret = 0;
> +                       goto out;
> +               }
>         }
>
>         ret = find_extent_clone(sctx, path, key->objectid, key->offset,
>                         sctx->cur_inode_size, &found_clone);
>         if (ret != -ENOENT && ret < 0)
>                 goto out;
>
>         ret = send_write_or_clone(sctx, path, key, found_clone);
>
> Thanks,
> Alex.
>
>
> On Thu, Nov 29, 2012 at 10:54 AM, Alexander Block <ablock84@gmail.com> wrote:
>> On Thu, Nov 29, 2012 at 4:11 AM, Chen Yang <chenyang.fnst@cn.fujitsu.com> wrote:
>>> when send/receive a sparse file, the holes of the original file
>>> will be filled with zero. The holes will be sent as ZERO streams,
>>> and it's unnecessary.
>>>
>>> So, I improved this by skipping the hole of file while sending.
>>>
>>> Signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
>>> ---
>>>  fs/btrfs/send.c |    6 ++++++
>>>  1 files changed, 6 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
>>> index e78b297..1e1d59a 100644
>>> --- a/fs/btrfs/send.c
>>> +++ b/fs/btrfs/send.c
>>> @@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,
>>>         u64 pos = 0;
>>>         u64 len;
>>>         u32 l;
>>> +       u64 bytenr;
>>>         u8 type;
>>>
>>>         ei = btrfs_item_ptr(path->nodes[0], path->slots[0],
>>> @@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,
>>>                  */
>>>                 len = PAGE_CACHE_ALIGN(len);
>>>         } else {
>>> +               bytenr = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
>>> +               if (bytenr == 0) {
>>> +                       ret = 0;
>>> +                       goto out;
>>> +               }
>>>                 len = btrfs_file_extent_num_bytes(path->nodes[0], ei);
>>>         }
>>>
>>> --
>>> 1.7.7.6
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> This won't work for incremental sends if I understand it correctly. If
>> the hole got punched into the file after the initial send, the data
>> will be unchanged on the receiving side when receiving incrementally.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html