send snapshot from snapshot incremental

send snapshot from snapshot incremental

[Jakob Schürz]

Hi there!

I asked one or two years ago for the ability of btrfs to use btrfs
send|receive with a parent-subvolume from a snapshot from a snapshot.

The thing is, if i take a snapshot from my system, transfer it with
btrfs send|receive  to an external HD on USB-Port, make some changes to
my system, which destroy my whole system, i can boot in the former taken
snapshot and work from there, as if nothing happened.

This part works.

BUT if i take a snapshot from the system, and want to transfer it to the
external HD, i can not set a parent subvolume, because there isn't any.

The problem was on my last question, there are missing
tansactional-numbers or something similar... I'm not a programmer, but i
use btrfs very extensive and wrote a fancy python-script and a
gnome-shell-extension and a fuse-filesystem to do enhanced and
userfriendly snapshots an backups with btrfs - also for non-programmer
users... something similar userfriendly as time-machine for apple does.

And one missing feature is the described problem above... booting into
an older snapshot and transfer a snapshot from this one lead to a full
transfer - even if there is a transferred snapshot on the external HD.

I hope, i described my problem understandable...

regards

Jakob

Re: send snapshot from snapshot incremental

[Hans van Kranenburg]

Hi Jakob,

On 03/25/2017 11:37 PM, Jakob Schürz wrote:
> Hi there!
> 
> I asked one or two years ago for the ability of btrfs to use btrfs
> send|receive with a parent-subvolume from a snapshot from a snapshot.
> 
> The thing is, if i take a snapshot from my system, transfer it with
> btrfs send|receive  to an external HD on USB-Port, make some changes to
> my system, which destroy my whole system, i can boot in the former taken
> snapshot and work from there, as if nothing happened.
> 
> This part works.
> 
> BUT if i take a snapshot from the system, and want to transfer it to the
> external HD, i can not set a parent subvolume, because there isn't any.
> 
> The problem was on my last question, there are missing
> tansactional-numbers or something similar... I'm not a programmer, but i
> use btrfs very extensive and wrote a fancy python-script and a
> gnome-shell-extension and a fuse-filesystem to do enhanced and
> userfriendly snapshots an backups with btrfs - also for non-programmer
> users... something similar userfriendly as time-machine for apple does.
> 
> And one missing feature is the described problem above... booting into
> an older snapshot and transfer a snapshot from this one lead to a full
> transfer - even if there is a transferred snapshot on the external HD.
> 
> I hope, i described my problem understandable...

Yes, if you make a writable snapshot, and then start creating a readonly
snapshot for send/receive from that, you actually start another
"timeline" in which the parent relationship for the snapshots is
replaced by the ID of the new writable one.

Send/receive also has a clone option, which lets you specify any other
subvolume that is present on the local and the remote system as well,
using the -c option. I'd recommend playing around and testing that
option, to see if, for the first send, you can refer to the latest
available snapshot of the previous "timeline". When getting this right,
it should prevent sending all data that was already present in the old
snapshot.

HTH,
-- 
Hans van Kranenburg

Re: send snapshot from snapshot incremental

[Peter Grandi]

[ ... ]
> BUT if i take a snapshot from the system, and want to transfer
> it to the external HD, i can not set a parent subvolume,
> because there isn't any.

Questions like this are based on incomplete understanding of
'send' and 'receive', and on IRC user "darkling" explained it
fairly well:

> When you use -c, you're telling the FS that it can expect to
> find a sent copy of that subvol on the receiving side, and
> that anything shared with it can be sent by reference. OK, so
> with -c on its own, you're telling the FS that "all the data
> in this subvol already exists on the remote".

> So, when you send your subvol, *all* of the subvol's metadata
> is sent, and where that metadata refers to an extent that's
> shared with the -c subvol, the extent data isn't sent, because
> it's known to be on the other end already, and can be shared
> directly from there.

> OK. So, with -p, there's a "base" subvol. The send subvol and
> the -p reference subvol are both snapshots of that base (at
> different times). The -p reference subvol, as with -c, is
> assumed to be on the remote FS. However, because it's known to
> be an earlier version of the same data, you can be more
> efficient in the sending by saying "start from the earlier
> version, and modify it in this way to get the new version"

> So, with -p, not all of the metadata is sent, because you know
> you've already got most of it on the remote in the form of the
> earlier version.

> So -p is "take this thing and apply these differences to it"
> and -c is "build this thing from scratch, but you can share
> some of the data with these sources"

Also here some additional details:

  http://logs.tvrrug.org.uk/logs/%23btrfs/2016-06-29.html#2016-06-29T22:39:59

The requirement for read-only is because in that way it is
pretty sure that the same stuff is on both origin and target
volume.

It may help to compare with RSYNC: it has to scan both the full
origin and target trees, because it cannot be told that there is
a parent tree that is the same on origin and target; but with
option '--link-dest' it can do something similar to 'send -c'.

Re: send snapshot from snapshot incremental

[Jakob Schürz]

Thanks for that explanation.

I'm sure, i didn't understand the -c option... and my english is pretty
good enough for the most things I need to know in Linux-things... but
not for this. :-(

Am 2017-03-26 um 22:07 schrieb Peter Grandi:
> [ ... ]
>> BUT if i take a snapshot from the system, and want to transfer
>> it to the external HD, i can not set a parent subvolume,
>> because there isn't any.
> 
> Questions like this are based on incomplete understanding of
> 'send' and 'receive', and on IRC user "darkling" explained it
> fairly well:
> 
>> When you use -c, you're telling the FS that it can expect to
>> find a sent copy of that subvol on the receiving side, and
>> that anything shared with it can be sent by reference. OK, so
>> with -c on its own, you're telling the FS that "all the data
>> in this subvol already exists on the remote".
> 
>> So, when you send your subvol, *all* of the subvol's metadata
>> is sent, and where that metadata refers to an extent that's
>> shared with the -c subvol, the extent data isn't sent, because
>> it's known to be on the other end already, and can be shared
>> directly from there.
> 
>> OK. So, with -p, there's a "base" subvol. The send subvol and
>> the -p reference subvol are both snapshots of that base (at
>> different times). The -p reference subvol, as with -c, is
>> assumed to be on the remote FS. However, because it's known to
>> be an earlier version of the same data, you can be more
>> efficient in the sending by saying "start from the earlier
>> version, and modify it in this way to get the new version"
> 
>> So, with -p, not all of the metadata is sent, because you know
>> you've already got most of it on the remote in the form of the
>> earlier version.
> 
>> So -p is "take this thing and apply these differences to it"
>> and -c is "build this thing from scratch, but you can share
>> some of the data with these sources"
> 
> Also here some additional details:
> 
>   http://logs.tvrrug.org.uk/logs/%23btrfs/2016-06-29.html#2016-06-29T22:39:59
> 
> The requirement for read-only is because in that way it is
> pretty sure that the same stuff is on both origin and target
> volume.
> 
> It may help to compare with RSYNC: it has to scan both the full
> origin and target trees, because it cannot be told that there is
> a parent tree that is the same on origin and target; but with
> option '--link-dest' it can do something similar to 'send -c'.

There is Subvolume A on the send- and the receive-side.
There is also Subvolume AA on the send-side from A.
The parent-ID from send-AA is the ID from A.
The received-ID from A on received-side A is the ID from A.

To send the AA, i use the command
btrfs send -p A AA|btrfs receive /path/to/receiveFS/

The received-ID from AA on received-side A is the ID from AA.


Now i take a snapshot from AA on the send-side -> Called AAA

If i try to send AAA to the receiving FS with
btrfs send -p AA AAA|btrfs receive /path/to/receiveFS/
no parent snapshot is found.
I should better take the -c Option.

btrfs send -c AA AAA|btrfs receive /path/to/receiveFS/

Am I right?
(Sorry, cannot test it now, i do not have my external Drive here)
I might remember, that this didn't work in the past...

Best regards

Jakob

Re: send snapshot from snapshot incremental

[J. Hart]

Don't be embarrassed.....
I'm a native speaker and still have trouble with most explanations.....:-)

On 03/28/2017 06:01 PM, Jakob Schürz wrote:
> Thanks for that explanation.
>
> I'm sure, i didn't understand the -c option... and my english is pretty
> good enough for the most things I need to know in Linux-things... but
> not for this. :-(
>
>

Re: send snapshot from snapshot incremental

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

On Tue, Mar 28, 2017 at 06:40:25PM -0400, J. Hart wrote:
> Don't be embarrassed.....
> I'm a native speaker and still have trouble with most explanations.....:-)

   You should try writing them. ;)

   Hugo ("darkling").

> 
> On 03/28/2017 06:01 PM, Jakob Schürz wrote:
> >Thanks for that explanation.
> >
> >I'm sure, i didn't understand the -c option... and my english is pretty
> >good enough for the most things I need to know in Linux-things... but
> >not for this. :-(
> >
> >
> 

-- 
Hugo Mills             | What's a Nazgûl like you doing in a place like this?
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                                Illiad

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: send snapshot from snapshot incremental

[Henk Slager]

On Wed, Mar 29, 2017 at 12:01 AM, Jakob Schürz
<wertstoffe@nurfuerspam.de> wrote:
[...]
> There is Subvolume A on the send- and the receive-side.
> There is also Subvolume AA on the send-side from A.
> The parent-ID from send-AA is the ID from A.
> The received-ID from A on received-side A is the ID from A.
>
> To send the AA, i use the command
> btrfs send -p A AA|btrfs receive /path/to/receiveFS/
>
> The received-ID from AA on received-side A is the ID from AA.
>
>
> Now i take a snapshot from AA on the send-side -> Called AAA
Is AAA readonly?

> If i try to send AAA to the receiving FS with
> btrfs send -p AA AAA|btrfs receive /path/to/receiveFS/
> no parent snapshot is found.
Something else is wrong, as this simply should work. I think the
subvol AAA was created on the send side like this:
btrfs sub snap -r AA AAA

then the send -p should work.

> I should better take the -c Option.
>
> btrfs send -c AA AAA|btrfs receive /path/to/receiveFS/
>
> Am I right?
> (Sorry, cannot test it now, i do not have my external Drive here)
> I might remember, that this didn't work in the past...

I once had a working (test) situation with the -c option where send
and receive were on the same machine. But I use send|receive mostly
over long distance and/or mobile (metered) links, so the -c option is
not preferred by me, as besides the diffs in data also way more
metadata exchange is done. And normally, in order to be able do
meaningful incremental differences checking, you need a single chain
of transactions.

It looks like you want or have have been changing snapshots on the
receive side. If you want that changes played back to the send side,
you somehow need still at least 1 older read-only pair on send and
receive side to do the increment from, not only for -p option but also
for -c option. And after that, you likely need to do some merging on
the send side.

I think 2 redundancy principles needs are mixed, that's why the
confusion w.r.t. send|receive:
1- redundancy on content level, e.g. you make some changes to a
script, C-source or whatever document: read-only snapshots within the
same file-system on the send-side alone should be OK for that.
2- redundancy on device level, a HDD or SSD might fail, so you want
backup: RAID is meant for that.

Using btrfs send|receice for both 1 and 2 is fine and works well I can
say, as long you keep strict rules of what your master/working
file-system/subvolume is and always that pair of identical older
read-only snapshots existing on the send and receive side.

Re: send snapshot from snapshot incremental

[Andrei Borzenkov]

On Wed, Mar 29, 2017 at 1:01 AM, Jakob Schürz <wertstoffe@nurfuerspam.de> wrote:
>
> There is Subvolume A on the send- and the receive-side.
> There is also Subvolume AA on the send-side from A.
> The parent-ID from send-AA is the ID from A.
> The received-ID from A on received-side A is the ID from A.
>
> To send the AA, i use the command
> btrfs send -p A AA|btrfs receive /path/to/receiveFS/
>
> The received-ID from AA on received-side A is the ID from AA.
>
>
> Now i take a snapshot from AA on the send-side -> Called AAA
>
> If i try to send AAA to the receiving FS with
> btrfs send -p AA AAA|btrfs receive /path/to/receiveFS/
> no parent snapshot is found.

I cannot reproduce it. Please show exact command invocations, ideally
starting with creation of A.

Re: send snapshot from snapshot incremental

[Giuseppe Della Bianca]

Hi.

>Jakob Schürz Tue, 28 Mar 2017 15:16:28 -0700

>Thanks for that explanation. 

>I'm sure, i didn't understand the -c option... and my english is pretty
>good enough for the most things I need to know in Linux-things... but
>not for this. :-(
]zac[

In one of my scripts I use this method:

subvolume            X
readonly snapshot    X1
readonly snapshot    X2
readonly snapshot    X3
....

send 1           btrfs send       X1
send 2           btrfs send -p X1 X2
send 3           btrfs send -p X2 X3
....


Any other sequence does not work, in the receive subvolume there must be the full parent (X1 then X2 then ...).


Gdb


https://sourceforge.net/projects/btrfsmanage/

Re: send snapshot from snapshot incremental

[Jakob Schürz]

Am 2017-03-26 um 22:07 schrieb Peter Grandi:
> [ ... ]
>> BUT if i take a snapshot from the system, and want to transfer
>> it to the external HD, i can not set a parent subvolume,
>> because there isn't any.
> 
> Questions like this are based on incomplete understanding of
> 'send' and 'receive', and on IRC user "darkling" explained it
> fairly well:
> 
>> When you use -c, you're telling the FS that it can expect to
>> find a sent copy of that subvol on the receiving side, and
>> that anything shared with it can be sent by reference. OK, so
>> with -c on its own, you're telling the FS that "all the data
>> in this subvol already exists on the remote".
> 
>> So, when you send your subvol, *all* of the subvol's metadata
>> is sent, and where that metadata refers to an extent that's
>> shared with the -c subvol, the extent data isn't sent, because
>> it's known to be on the other end already, and can be shared
>> directly from there.
> 
>> OK. So, with -p, there's a "base" subvol. The send subvol and
>> the -p reference subvol are both snapshots of that base (at
>> different times). The -p reference subvol, as with -c, is
>> assumed to be on the remote FS. However, because it's known to
>> be an earlier version of the same data, you can be more
>> efficient in the sending by saying "start from the earlier
>> version, and modify it in this way to get the new version"
> 
>> So, with -p, not all of the metadata is sent, because you know
>> you've already got most of it on the remote in the form of the
>> earlier version.
> 
>> So -p is "take this thing and apply these differences to it"
>> and -c is "build this thing from scratch, but you can share
>> some of the data with these sources"
> 

For now, I think i got it... (maybe).

I put the following logic into my script:
1) Search for all Subvolumes on local and remote side, where the
Received-UUID on the remote side is the same as the UUID on the local side
2) Take the parent-UUID from the Snapshot i want to transfer and search
in the list from 1) which snapshot (from the local side) has the same
parent UUID.
3) Take the younges Snapshot from 2) ans set it as parent for the btrfs
send-command
4) Search for snapshot local and remote, wich have the same name|path
ans "basename" as the snapshot i want to transfer
	basename means, my system-subvolume is called @debian
	it contains one subvolume @debian/var/spool
	the snapshotnames are @debian_$TIMESTAMP and
	@debian_$TIMESTAMP/var/spool
	The basename is @debian and @debian/var/spool
5) set all of the snapshots with the same basename as the snapshot to be
transferred as clones for btrfs send.

The final command involves the youngest "sister" from the snapshot i
want to transfer, which is on both sides, set as "parent", and a bunch
of snapshots wich are older (or even younger - is this a problem???)
than the snapshot i want to transfer wich contain modified and the same
data, set as clones

If there is no parent (In case of transferring a snapshot of a
snapshot...) then there are clones of this snapshot, so not all of the
data is to be sent again (and consumes the double space on the backup-media)
If there are no parents AND clones (similar snapshots), the subvolume
seems to be totally new, and the whole must be transferred.
If there is a parent and clones, both of them are used to minimize the
data for the transfer, and use as much as possible from the existing
data/metadata on the backup-media to build the new snapshot there

To use all of the similar snapshots (get by the snapshotname) as clones
seems to fasten the transfer in comparison to only use the parent (this
seems slower). Could this be, or is this only a "feeling?

Thanks for all your advices. This helped me a lot!!

regards Jakob