Using BtrFS and backup tools for keeping two systems in sync

Using BtrFS and backup tools for keeping two systems in sync

[Shriramana Sharma]

Hello. I see there are some backup tools taking advantage of BtrFS's
incremental send/receive feature:
https://btrfs.wiki.kernel.org/index.php/Incremental_Backup. [BTW Ames
Cornish's ButterSink (https://github.com/AmesCornish/buttersink) seems
to be missing from that page.]

Now I'd like to know if anyone has evolved some good practices w.r.t
maintaining the data of two systems in sync using this feature of
BtrFS. What I have in mind is: I work on my desktop by default, and
for ergonomics reasons only use my laptop when I need the mobility.
I'd like to keep the main data (documents I create, programs I write
etc) in sync between the two. (The profile data such as in the ~/.*
hidden folders had better stay separate though, I guess.)

I figure with the existing tools it would not be too difficult to
maintain a synced set of snapshots between the two systems if I only
use the desktop vs laptop alternatingly and sync at each switchover,
but the potential problem only would come if I modify both (something
like having to do git merge, I guess).

Has anyone come across this situation and evolved any policies to handle it?

-- 
Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा

Re: Using BtrFS and backup tools for keeping two systems in sync

[Shriramana Sharma]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 302 bytes --]

Hello can you elaborate a bit on the workflow as to how you use svn for that?

-- 
Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: Using BtrFS and backup tools for keeping two systems in sync

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

On Thu, Oct 08, 2015 at 08:05:09AM +0530, Shriramana Sharma wrote:
> Hello. I see there are some backup tools taking advantage of BtrFS's
> incremental send/receive feature:
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup. [BTW Ames
> Cornish's ButterSink (https://github.com/AmesCornish/buttersink) seems
> to be missing from that page.]
> 
> Now I'd like to know if anyone has evolved some good practices w.r.t
> maintaining the data of two systems in sync using this feature of
> BtrFS. What I have in mind is: I work on my desktop by default, and
> for ergonomics reasons only use my laptop when I need the mobility.
> I'd like to keep the main data (documents I create, programs I write
> etc) in sync between the two. (The profile data such as in the ~/.*
> hidden folders had better stay separate though, I guess.)
> 
> I figure with the existing tools it would not be too difficult to
> maintain a synced set of snapshots between the two systems if I only
> use the desktop vs laptop alternatingly and sync at each switchover,
> but the potential problem only would come if I modify both (something
> like having to do git merge, I guess).
> 
> Has anyone come across this situation and evolved any policies to handle it?

   You can't currently do this efficiently with send/receive. It
should be possible, but it needs a change to the send stream format.

   Hugo.

-- 
Hugo Mills             | UNIX: British manufacturer of modular shelving units
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Using BtrFS and backup tools for keeping two systems in sync

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]

On 2015-10-07 22:35, Shriramana Sharma wrote:
> Hello. I see there are some backup tools taking advantage of BtrFS's
> incremental send/receive feature:
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup. [BTW Ames
> Cornish's ButterSink (https://github.com/AmesCornish/buttersink) seems
> to be missing from that page.]
>
> Now I'd like to know if anyone has evolved some good practices w.r.t
> maintaining the data of two systems in sync using this feature of
> BtrFS. What I have in mind is: I work on my desktop by default, and
> for ergonomics reasons only use my laptop when I need the mobility.
> I'd like to keep the main data (documents I create, programs I write
> etc) in sync between the two. (The profile data such as in the ~/.*
> hidden folders had better stay separate though, I guess.)
>
> I figure with the existing tools it would not be too difficult to
> maintain a synced set of snapshots between the two systems if I only
> use the desktop vs laptop alternatingly and sync at each switchover,
> but the potential problem only would come if I modify both (something
> like having to do git merge, I guess).
>
> Has anyone come across this situation and evolved any policies to handle it?
>
Personally, while I use BTRFS on all of my systems, I usually use 
Dropbox for synchronizing data between them.  While the Linux client for 
it isn't perfect, it is significantly easier than something like a 
regularly scheduled rsync or btrfs send/receive.  It's also kind of nice 
because multiple clients bound to the same account will sync across the 
local network without needing to talk to the servers if they can avoid 
it.  Of course, it costs money if you want a decent amount of storage 
space, but it's pretty reasonable for the degree of reliability I've 
observed.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Using BtrFS and backup tools for keeping two systems in sync

[Duncan]

Hugo Mills posted on Thu, 08 Oct 2015 06:37:45 +0000 as excerpted:

> On Thu, Oct 08, 2015 at 08:05:09AM +0530, Shriramana Sharma wrote:
>> Hello. I see there are some backup tools taking advantage of BtrFS's
>> incremental send/receive feature:
>> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup. [BTW Ames
>> Cornish's ButterSink (https://github.com/AmesCornish/buttersink) seems
>> to be missing from that page.]
>> 
>> Now I'd like to know if anyone has evolved some good practices w.r.t
>> maintaining the data of two systems in sync using this feature of
>> BtrFS. What I have in mind is: I work on my desktop by default, and for
>> ergonomics reasons only use my laptop when I need the mobility.
>> I'd like to keep the main data (documents I create, programs I write
>> etc) in sync between the two. (The profile data such as in the ~/.*
>> hidden folders had better stay separate though, I guess.)
>> 
>> I figure with the existing tools it would not be too difficult to
>> maintain a synced set of snapshots between the two systems if I only
>> use the desktop vs laptop alternatingly and sync at each switchover,
>> but the potential problem only would come if I modify both (something
>> like having to do git merge, I guess).
>> 
>> Has anyone come across this situation and evolved any policies to
>> handle it?
> 
>    You can't currently do this efficiently with send/receive. It
> should be possible, but it needs a change to the send stream format.

Elucidating somewhat...

AFAIK (as a list regular but not a dev or a user, personally, of the send/
receive functionality), currently, btrfs send/receive incremental works 
only one way.  That is, the send-stream format provides sufficient 
information for incremental sends after an original send, but there's no 
way to reverse the process and sync the other way, from the original 
receiver back to the sender.  A full send can be done, but then it's no 
longer linked to the the original.

As Hugo says, the base functionality is available, but actually hooking 
it up to work will require a bump to the send stream format, as the 
required information simply isn't sent, ATM.  That send stream format 
bump is likely to eventually happen, but there's a very strong interest 
in keeping the number of formats that must be supported for backward 
compatibility to a minimum, and thus in a minimum number of format 
bumps.  So the devs want to delay the bump as long as possible, 
identifying anything else that might need to change in the mean time, and 
make, ideally, one final bump, including all changes discovered to be 
needed since the last one, and then no more.

So while this necessary change is known, it could be some time, yet, 
before it's actually done, with hopefully no further changes necessary or 
allowed after that.

Which back on the reoccurring theme of btrfs stability...

... is another point toward btrfs "definitely stabilizing now, but not 
yet fully stable and mature."  When the devs decide there are likely no 
further as-yet undiscovered necessary changes coming and finally do this 
bump, we'll know they really do consider btrfs to be settling down into 
stability.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman