Receive on same subvolume

Receive on same subvolume

[Matthew Lai]

Hello,

Is this supposed to work? (/data is the root volume, /data/a is a subvolume)

btrfs subvolume snapshot /data/a /data/b
# make some changes in b
btrfs send -p /data/a /data/b > delta
btrfs receive /data/a < delta

I'm getting "ERROR: could not find parent subvolume" on receive.

What I'm trying to do is to back up using send/receive, but I don't have 
50% free space, and (please correct me if I'm wrong) since receive 
doesn't do deduplication, I want to use snapshot to do the initial 
bootstrapping, instead of send/receive without a parent.

Thanks!
Matthew

Re: Receive on same subvolume

[Felix Blanke]

Hi Matthew,

I'm not sure what you are trying to achive. Couldn't you simply do
another snapshot of the subvolume? I don't understand why you want to
use send/receive on the same subvolume to be honest.

Regards,
Felix

On Wed, Jan 29, 2014 at 10:26 PM, Matthew Lai <m@matthewlai.ca> wrote:
> Hello,
>
> Is this supposed to work? (/data is the root volume, /data/a is a subvolume)
>
> btrfs subvolume snapshot /data/a /data/b
> # make some changes in b
> btrfs send -p /data/a /data/b > delta
> btrfs receive /data/a < delta
>
> I'm getting "ERROR: could not find parent subvolume" on receive.
>
> What I'm trying to do is to back up using send/receive, but I don't have 50%
> free space, and (please correct me if I'm wrong) since receive doesn't do
> deduplication, I want to use snapshot to do the initial bootstrapping,
> instead of send/receive without a parent.
>
> Thanks!
> Matthew
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Receive on same subvolume

[Chris Murphy]

On Jan 29, 2014, at 2:26 PM, Matthew Lai <m@matthewlai.ca> wrote:

> Hello,
> 
> Is this supposed to work? (/data is the root volume, /data/a is a subvolume)
> 
> btrfs subvolume snapshot /data/a /data/b
> # make some changes in b
> btrfs send -p /data/a /data/b > delta
> btrfs receive /data/a < delta
> 
> I'm getting "ERROR: could not find parent subvolume" on receive.

> What I'm trying to do is to back up using send/receive, but I don't have 50% free space, and (please correct me if I'm wrong) since receive doesn't do deduplication, I want to use snapshot to do the initial bootstrapping, instead of send/receive without a parent.

I think you've oversimplified your commands, because it looks like you're using send/receive on the same file system. But if it's a backup, necessarily you'd have to be sending the subvolume(s) to another file system on another disk (either on the same system or remotely). So that needs some clarification.

Also, btrfs send requires subvolumes to be read only. Are they?

And btrfs incremental receive expects the identical parent already on the destination. Is it?

Also, while I'm not certain it matters, man page says to use -f to specify files. I haven't tested < and >. But then also the step where you create this intermediate snapshot file isn't necessary, just combine the send receive commands through pipe.


https://btrfs.wiki.kernel.org/index.php/Incremental_Backup


Chris Murphy

Re: Receive on same subvolume

[Matthew Lai]

Thanks. I should clarify what I'm trying to do.

I'm trying to use btrfs send for backup, without having another btrfs 
volume.

So the initial backup is a complete send, piped to Amazon Glacier (so my 
machine never has the whole file, and doesn't have space for one).

At the same time I'm keeping a snapshot of the current volume.

On the next incremental backup, I would use the first snapshot as the 
parent, and send the differences to Glacier again (without having the 
entire file on the system at any time).

It looks like the problem now is the sent file can't be applied to the 
original volume (for restore).

Thanks
Matthew

On 03/02/2014 9:30 AM, Chris Murphy wrote:
> On Jan 29, 2014, at 2:26 PM, Matthew Lai <m@matthewlai.ca> wrote:
>
>> Hello,
>>
>> Is this supposed to work? (/data is the root volume, /data/a is a subvolume)
>>
>> btrfs subvolume snapshot /data/a /data/b
>> # make some changes in b
>> btrfs send -p /data/a /data/b > delta
>> btrfs receive /data/a < delta
>>
>> I'm getting "ERROR: could not find parent subvolume" on receive.
>> What I'm trying to do is to back up using send/receive, but I don't have 50% free space, and (please correct me if I'm wrong) since receive doesn't do deduplication, I want to use snapshot to do the initial bootstrapping, instead of send/receive without a parent.
> I think you've oversimplified your commands, because it looks like you're using send/receive on the same file system. But if it's a backup, necessarily you'd have to be sending the subvolume(s) to another file system on another disk (either on the same system or remotely). So that needs some clarification.
>
> Also, btrfs send requires subvolumes to be read only. Are they?
>
> And btrfs incremental receive expects the identical parent already on the destination. Is it?
>
> Also, while I'm not certain it matters, man page says to use -f to specify files. I haven't tested < and >. But then also the step where you create this intermediate snapshot file isn't necessary, just combine the send receive commands through pipe.
>
>
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
>
>
> Chris Murphy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Receive on same subvolume

[Chris Murphy]

On Feb 3, 2014, at 11:19 AM, Matthew Lai <m@matthewlai.ca> wrote:

> Thanks. I should clarify what I'm trying to do.
> 
> I'm trying to use btrfs send for backup, without having another btrfs volume.
> 
> So the initial backup is a complete send, piped to Amazon Glacier (so my machine never has the whole file, and doesn't have space for one).

OK so you've use btrfs send piped to Glacier which creates a *file*, I'll call it "initial", not a navigable directory of files? Right?

> 
> At the same time I'm keeping a snapshot of the current volume.
> 
> On the next incremental backup, I would use the first snapshot as the parent, and send the differences to Glacier again (without having the entire file on the system at any time).

That's fine as long as the stdout from btrfs send ends up as a self contained file on Glacier. I'll call this "increment1"

> 
> It looks like the problem now is the sent file can't be applied to the original volume (for restore).

I'm counting two sent files: initial, increment1. I'm not sure which one you're applying. If you have the exact same read-only snapshot that the btrfs send file "initial" is based on, then you'd apply the increment1 to that read-only snapshot which will cause a new read-only snapshot to be created with the incremental data applied to it. The error you're getting sounds like the parent read-only snapshot isn't available?

Have you tried -vv flag to get more verbose error information when using btrfs receive?

Chris Murphy

Re: Receive on same subvolume

[Matthew Lai]

On 03/02/2014 11:26 AM, Chris Murphy wrote:
> On Feb 3, 2014, at 11:19 AM, Matthew Lai <m@matthewlai.ca> wrote:
>
>> Thanks. I should clarify what I'm trying to do.
>>
>> I'm trying to use btrfs send for backup, without having another btrfs volume.
>>
>> So the initial backup is a complete send, piped to Amazon Glacier (so my machine never has the whole file, and doesn't have space for one).
> OK so you've use btrfs send piped to Glacier which creates a *file*, I'll call it "initial", not a navigable directory of files? Right?
That is correct.
>> It looks like the problem now is the sent file can't be applied to the original volume (for restore).
> I'm counting two sent files: initial, increment1. I'm not sure which one you're applying. If you have the exact same read-only snapshot that the btrfs send file "initial" is based on, then you'd apply the increment1 to that read-only snapshot which will cause a new read-only snapshot to be created with the incremental data applied to it. The error you're getting sounds like the parent read-only snapshot isn't available?
>
That is also correct. There are 2 sent files. I am trying to apply 
increment1, on a snapshot of the parent (that was used to create 
increment1).

I added -vv. Here is the test script for reproducing this entire setup.

-----------------------------------------------------------------------
#!/bin/sh

btrfs subvolume create data
btrfs subvolume snapshot -r data first_backup
touch data/a
btrfs subvolume snapshot -r data second_backup
btrfs send -p first_backup second_backup > increment
btrfs subvolume snapshot first_backup first_backup_rw
btrfs receive -vv first_backup_rw < increment
-----------------------------------------------------------------------

Output:
-----------------------------------------------------------------------
Create subvolume './data'
Create a readonly snapshot of 'data' in './first_backup'
Create a readonly snapshot of 'data' in './second_backup'
At subvol second_backup
Create a snapshot of 'first_backup' in './first_backup_rw'
At snapshot second_backup
receiving snapshot second_backup 
uuid=e6159a2a-3430-344a-a23d-b9bb83851a63, ctransid=28 
parent_uuid=20c4ff66-a9ec-fc44-93c6-2c12637e95e6, parent_ctransid=26
ERROR: could not find parent subvolume
-----------------------------------------------------------------------

I would think applying the "patch" to first_backup_rw should succeed, 
because it's exactly the same as first_backup, which is the parent for 
the send, but it doesn't.

Thanks
Matthew

Re: Receive on same subvolume

[Chris Murphy]

On Feb 3, 2014, at 3:53 PM, Matthew Lai <m@matthewlai.ca> wrote:

> On 03/02/2014 11:26 AM, Chris Murphy wrote:
>> On Feb 3, 2014, at 11:19 AM, Matthew Lai <m@matthewlai.ca> wrote:
>> 
>>> Thanks. I should clarify what I'm trying to do.
>>> 
>>> I'm trying to use btrfs send for backup, without having another btrfs volume.
>>> 
>>> So the initial backup is a complete send, piped to Amazon Glacier (so my machine never has the whole file, and doesn't have space for one).
>> OK so you've use btrfs send piped to Glacier which creates a *file*, I'll call it "initial", not a navigable directory of files? Right?
> That is correct.
>>> It looks like the problem now is the sent file can't be applied to the original volume (for restore).
>> I'm counting two sent files: initial, increment1. I'm not sure which one you're applying. If you have the exact same read-only snapshot that the btrfs send file "initial" is based on, then you'd apply the increment1 to that read-only snapshot which will cause a new read-only snapshot to be created with the incremental data applied to it. The error you're getting sounds like the parent read-only snapshot isn't available?
>> 
> That is also correct. There are 2 sent files. I am trying to apply increment1, on a snapshot of the parent (that was used to create increment1).

I don't understand how you can apply increment1 to the snapshot of increment1; and also I don't understand how the parent is also increment1.




> 
> I added -vv. Here is the test script for reproducing this entire setup.
> 
> -----------------------------------------------------------------------
> #!/bin/sh
> 
> btrfs subvolume create data
> btrfs subvolume snapshot -r data first_backup
> touch data/a
> btrfs subvolume snapshot -r data second_backup
> btrfs send -p first_backup second_backup > increment
> btrfs subvolume snapshot first_backup first_backup_rw
> btrfs receive -vv first_backup_rw < increment
> -----------------------------------------------------------------------
> 
> Output:
> -----------------------------------------------------------------------
> Create subvolume './data'
> Create a readonly snapshot of 'data' in './first_backup'
> Create a readonly snapshot of 'data' in './second_backup'
> At subvol second_backup
> Create a snapshot of 'first_backup' in './first_backup_rw'
> At snapshot second_backup
> receiving snapshot second_backup uuid=e6159a2a-3430-344a-a23d-b9bb83851a63, ctransid=28 parent_uuid=20c4ff66-a9ec-fc44-93c6-2c12637e95e6, parent_ctransid=26
> ERROR: could not find parent subvolume
> -----------------------------------------------------------------------
> 
> I would think applying the "patch" to first_backup_rw should succeed, because it's exactly the same as first_backup, which is the parent for the send, but it doesn't.

btrfs sub snap -r subvol.1 subvol
btrfs send subvol.1 -f <toglacier>/subvol.1.btrfs
#write some more files to subvol
btrfs sub snap -r subvol.2 subvol
btrfs send -p subvol.1 subvol.2 -f <toglacier>/subvol.2.btrfs

#To make subvol.1 into subvol.2 by applying subvol.2.btrfs to subvol.1, the actual original subvol.1 must be present first or you need to "receive" it from subvol.1.btrfs first. And also, I'm pretty sure you can't have subvol.2 already present because receive must create it.

Again, I haven't tried > and < so I don't know they work. Have you tried -f to point to the files?



Chris Murphy

Re: Receive on same subvolume

[Matthew Lai]

On 03/02/2014 4:34 PM, Chris Murphy wrote:
> On Feb 3, 2014, at 3:53 PM, Matthew Lai <m@matthewlai.ca> wrote:
>
>> On 03/02/2014 11:26 AM, Chris Murphy wrote:
>>> On Feb 3, 2014, at 11:19 AM, Matthew Lai <m@matthewlai.ca> wrote:
>>>
>>>> Thanks. I should clarify what I'm trying to do.
>>>>
>>>> I'm trying to use btrfs send for backup, without having another btrfs volume.
>>>>
>>>> So the initial backup is a complete send, piped to Amazon Glacier (so my machine never has the whole file, and doesn't have space for one).
>>> OK so you've use btrfs send piped to Glacier which creates a *file*, I'll call it "initial", not a navigable directory of files? Right?
>> That is correct.
>>>> It looks like the problem now is the sent file can't be applied to the original volume (for restore).
>>> I'm counting two sent files: initial, increment1. I'm not sure which one you're applying. If you have the exact same read-only snapshot that the btrfs send file "initial" is based on, then you'd apply the increment1 to that read-only snapshot which will cause a new read-only snapshot to be created with the incremental data applied to it. The error you're getting sounds like the parent read-only snapshot isn't available?
>>>
>> That is also correct. There are 2 sent files. I am trying to apply increment1, on a snapshot of the parent (that was used to create increment1).
> I don't understand how you can apply increment1 to the snapshot of increment1; and also I don't understand how the parent is also increment1.
>
>
>
>
>> I added -vv. Here is the test script for reproducing this entire setup.
>>
>> -----------------------------------------------------------------------
>> #!/bin/sh
>>
>> btrfs subvolume create data
>> btrfs subvolume snapshot -r data first_backup
>> touch data/a
>> btrfs subvolume snapshot -r data second_backup
>> btrfs send -p first_backup second_backup > increment
>> btrfs subvolume snapshot first_backup first_backup_rw
>> btrfs receive -vv first_backup_rw < increment
>> -----------------------------------------------------------------------
>>
>> Output:
>> -----------------------------------------------------------------------
>> Create subvolume './data'
>> Create a readonly snapshot of 'data' in './first_backup'
>> Create a readonly snapshot of 'data' in './second_backup'
>> At subvol second_backup
>> Create a snapshot of 'first_backup' in './first_backup_rw'
>> At snapshot second_backup
>> receiving snapshot second_backup uuid=e6159a2a-3430-344a-a23d-b9bb83851a63, ctransid=28 parent_uuid=20c4ff66-a9ec-fc44-93c6-2c12637e95e6, parent_ctransid=26
>> ERROR: could not find parent subvolume
>> -----------------------------------------------------------------------
>>
>> I would think applying the "patch" to first_backup_rw should succeed, because it's exactly the same as first_backup, which is the parent for the send, but it doesn't.
> btrfs sub snap -r subvol.1 subvol
> btrfs send subvol.1 -f <toglacier>/subvol.1.btrfs
> #write some more files to subvol
> btrfs sub snap -r subvol.2 subvol
> btrfs send -p subvol.1 subvol.2 -f <toglacier>/subvol.2.btrfs
>
> #To make subvol.1 into subvol.2 by applying subvol.2.btrfs to subvol.1, the actual original subvol.1 must be present first or you need to "receive" it from subvol.1.btrfs first. And also, I'm pretty sure you can't have subvol.2 already present because receive must create it.
>
> Again, I haven't tried > and < so I don't know they work. Have you tried -f to point to the files?
According to the manpage, -f is the same as output redirection.
"Output is normally written to stdout. To write to a file, use this 
option.  An alternative would be to use pipes."

The reason why I can't use something like your sequence of commands 
(assuming the order of arguments for snap should be reversed) is that I 
want to be able to verify that the diff is correct, since there are 
still integrity problems with send/receive.

I was planning to do that by applying the "patch" to a snapshot of the 
parent right away, and make sure the patched volume is equal to the 
current snapshot (by trying another send, and making sure the output is 0).

Thanks
Matthew