Is anyone using btrfs send/receive for backups instead of rsync?

Is anyone using btrfs send/receive for backups instead of rsync?

[Marc MERLIN]

Currently, I have a nightly cronjob that rsyncs my SSD to my HD on my
laptop.

Of course, running rsync on an entire FS every night is slow, especially
if I happen to try to use the laptop at the same time.

Obviously, that's what btrfs send/receive is supposed to fix:
https://btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive
http://lwn.net/Articles/506244/

It's been around for about 1.5 years now, so I was curious if others had
converted to it for backups when indeed the wiki has the CYA 'if you use
for backups, you get what you deserve' :)

Is it still being worked on, or kind of frozen/good as is?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

   Yes, I am.

On Sat, Dec 28, 2013 at 09:19:43AM -0800, Marc MERLIN wrote:
> Currently, I have a nightly cronjob that rsyncs my SSD to my HD on my
> laptop.
> 
> Of course, running rsync on an entire FS every night is slow, especially
> if I happen to try to use the laptop at the same time.
> 
> Obviously, that's what btrfs send/receive is supposed to fix:
> https://btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive
> http://lwn.net/Articles/506244/
> 
> It's been around for about 1.5 years now, so I was curious if others had
> converted to it for backups when indeed the wiki has the CYA 'if you use
> for backups, you get what you deserve' :)
> 
> Is it still being worked on, or kind of frozen/good as is?

   It's almost feature-complete: there's one version bump expected in
the stream format, to handle files with holes in. There are probably
still awkward bugs in there (awkward in the sense that your backups
will fail noisily if you hit one), but for my current usage -- disk to
disk on the same system -- it seems to work OK.

   I'm trying at the moment to hack together a working send/receive
backup between two systems. As far as I know, the send/receive will
work OK, but I'm having problems with firewalls and ssh tunnels right
now. :)

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- A cross? Oy vey, have you picked the wrong vampire! ---       

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Emil Karlson]

There are also silent content missmatches on some subset of files that
are not linearly written (vm images, databases and such).

https://bugzilla.kernel.org/show_bug.cgi?id=66941

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Marc MERLIN]

On Sat, Dec 28, 2013 at 05:37:30PM +0000, Hugo Mills wrote:
> > Is it still being worked on, or kind of frozen/good as is?
> 
>    It's almost feature-complete: there's one version bump expected in
> the stream format, to handle files with holes in. There are probably
> still awkward bugs in there (awkward in the sense that your backups
> will fail noisily if you hit one), but for my current usage -- disk to
> disk on the same system -- it seems to work OK.
> 
>    I'm trying at the moment to hack together a working send/receive
> backup between two systems. As far as I know, the send/receive will
> work OK, but I'm having problems with firewalls and ssh tunnels right
> now. :)

Good to know, thanks.

Just curious: what happens if the destination you're syncing the
snapshot diff to, isn't quite like expected?

For instance, if I use an existing rsync destination to start syncing
btrfs snapshots to after that, and one file operation can't be applied
because let's say the destination file it's supposed to be applied to,
isn't there?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Marc MERLIN]

On Sat, Dec 28, 2013 at 10:07:58AM -0800, Marc MERLIN wrote:
> For instance, if I use an existing rsync destination to start syncing
> btrfs snapshots to after that, and one file operation can't be applied
> because let's say the destination file it's supposed to be applied to,
> isn't there?

I should have written more: I'm guessing what happens is that the btrfs
receive fails/aborts, I get an error, I then run a manual rsync to reset
everything to a good known state, and then continue the btrfs
send/receive after that?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Richard Michael]

On Sat, Dec 28, 2013 at 1:01 PM, Emil Karlson <jekarlson@gmail.com> wrote:
> There are also silent content missmatches on some subset of files that
> are not linearly written (vm images, databases and such).
>
> https://bugzilla.kernel.org/show_bug.cgi?id=66941

Can anyone elaborate on this?

Emil, your bug report mentions the subvolume journal is different, and
not anything about silent content differences on file data (images,
databases, etc.) as you mention above.  How are they related?

Also, how is the journal different on the receive-side?  Is it "just"
a size/allocation issue, as David commented on the bug report; or is
the content actually different? For example, can it still be used to
recover the filesystem post-crash?  What does "fixed in V2" mean:
BTRFSv2 ?

Backup by snapshot+send/receive is my primary BTRFS use-case, so I'm
curious about details/pitfalls.

Regards,
Richard

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Emil Karlson]

> Emil, your bug report mentions the subvolume journal is different, and
> not anything about silent content differences on file data (images,
> databases, etc.) as you mention above.  How are they related?

systemd-journald is a logging daemon in the userspace that has nothing
to do with filesystem journal.

> Also, how is the journal different on the receive-side?  Is it "just"
> a size/allocation issue, as David commented on the bug report; or is
> the content actually different? For example, can it still be used to
> recover the filesystem post-crash?  What does "fixed in V2" mean:
> BTRFSv2 ?

Send stream format specification V2.

I used sha1sum (stated also in the report) so contents are actually different.

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Richard Michael]

On Sat, Dec 28, 2013 at 2:52 PM, Emil Karlson <jekarlson@gmail.com> wrote:
>> Emil, your bug report mentions the subvolume journal is different, and
>> not anything about silent content differences on file data (images,
>> databases, etc.) as you mention above.  How are they related?
>
> systemd-journald is a logging daemon in the userspace that has nothing
> to do with filesystem journal.

Ah yes, of course.  I haven't used systemd very much, and was rather
focused on filesystem terminology. (It didn't help that I ignored the
path in your shasum output; but, yes I saw it.)  Sorry!

Though, I'm left wondering what the impact is to systemd.

Is the journal on the received side useless?  I'll read details ; but
I'm guessing the "difference" is that all the used extents were
copied, but that unused but pre-allocated extents were not ; so that
the journal is still has the data systemd has written, but is missing
the empty/unused extents?  However, in that case, I'm unclear about
what sha1sum is reading vs. what systemd is reading - I suppose
systemd must know to ignore extents it has not [yet] written.

>
>> Also, how is the journal different on the receive-side?  Is it "just"
>> a size/allocation issue, as David commented on the bug report; or is
>> the content actually different? For example, can it still be used to
>> recover the filesystem post-crash?  What does "fixed in V2" mean:
>> BTRFSv2 ?
>
> Send stream format specification V2.

Ok, thanks.

(I assume the "Send stream v2 draft", here:
https://btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive )

Regards,
Richard

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Murphy]

On Dec 28, 2013, at 1:34 PM, Richard Michael <rmichael@edgeofthenet.org> wrote:
> 
> Is the journal on the received side useless?

In my case, I'm unable to reproduce the problem. I get the same sha256sum of an existing 16M journal file (in the read only snapshot) as I do in the send/received snapshot. The difference is the original is compromised of 1976 extents, while the received is in a single extent.

This is kernel 3.12.5-301.fc20 and btrfs-progs 3.12.

I am slightly bugged about a 16MB file having nearly 2000 extents, basically it's being turned into a bunch of 8KB fragments. I know nothing of the pros and cons of how systemd is writing journals, but they don't seem very big so I don't understand why they're preallocated, which on btrfs appears instantly defeated by COW upon the journal being modified. It seems to me either the journal doesn't need to be preallocated (at least on btrfs) or maybe systemd should set xattr +C on /var/log/journal? That does disable checksumming though, along with data cow.


Chris Murphy

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Emil Karlson]

> In my case, I'm unable to reproduce the problem. I get the same sha256sum of an existing 16M journal file (in the read only snapshot) as I do in the send/received snapshot. The difference is the original is compromised of 1976 extents, while the received is in a single extent.

Was the send incremental? Did the files have actual changes? Did you
check more than one file?

I use rsync -cvan --delete "/.../${new}/" "/.../${new}/" to check the
full filesystem on every send (and sha1sum to confirm failures). Most
of the time at least one new file fails. My default time interval
between sends is about 16 days.

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Murphy]

On Dec 28, 2013, at 4:55 PM, Emil Karlson <jekarlson@gmail.com> wrote:

>> In my case, I'm unable to reproduce the problem. I get the same sha256sum of an existing 16M journal file (in the read only snapshot) as I do in the send/received snapshot. The difference is the original is compromised of 1976 extents, while the received is in a single extent.
> 
> Was the send incremental?

Oops no, and I bet that's the difference. So you are doing an incremental send/receive using -p? I've only done incremental snapshots with send -p while also using -f to write to a file; not through btrfs receive.


Chris Murphy

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Duncan]

Chris Murphy posted on Sat, 28 Dec 2013 16:11:37 -0700 as excerpted:

> I am slightly bugged about a 16MB file having nearly 2000 extents,
> basically it's being turned into a bunch of 8KB fragments. I know
> nothing of the pros and cons of how systemd is writing journals, but
> they don't seem very big so I don't understand why they're preallocated,
> which on btrfs appears instantly defeated by COW upon the journal being
> modified. It seems to me either the journal doesn't need to be
> preallocated (at least on btrfs) or maybe systemd should set xattr +C on
> /var/log/journal?
> 
> That does disable checksumming though, along with data cow.

While I don't (yet?) use systemd here, from what I understand of its 
journal, it's essentially a binary database, and isn't necessarily even 
sequentially written as is traditional with log files.  That would 
explain the preallocation.  And given your mention of 8KB fragments, I 
wonder if that's its record-size?

Meanwhile, as with any pre-allocated-then-written-into file, including VM 
images and pre-allocated bittorrent files, the systemd journal is a known 
worst-case for COW filesystems including btrfs.

And setting NOCOW on the file (xattr +C) before those write-intos is the 
knob btrfs exposes to deal with the problem... for all the above cases.

Yes, it does turn off checksumming as well as COW, but given the write-
into scenario, that's actually best anyway, because otherwise btrfs has 
to keep updating the checksums as the internal writes are occurring, and 
that's both CPU intensive and potentially rate-limited, and an invitation 
to race conditions since the writing application and btrfs' checksumming 
are constantly lock-fighting, the one to update the file, the other to 
update the checksum based on the new data.

But in all these cases, it's also quite common for the application doing 
the writing to have its own checksumming/error-detection and possible 
correction -- it pretty much comes with the territory -- in which case 
btrfs attempting to do the same is simply superfluous even if it weren't 
a race-condition trigger.  Certainly torrents include checksumming -- 
that automatically guaranteed download integrity is part of what makes 
the protocol as popular as it is.  And databases too.  I don't actually 
know enough about VMs to know if it's the case there or not, but 
certainly, unexpected bit-flipping is likely to corrupt and crash the VM, 
just as it tends to do with on-the-metal operating systems.

If/when the file reaches effective stasis, as a torrented file once it's 
fully downloaded, /then/ it's reasonable to kill the NOCOW and do a final 
(sequential-write) copy/move so btrfs has it checksummed too.  And 
database and VM backups... if they're not being actively used, btrfs 
checksumming can guard against bitrot there too.  Similarly systemd's 
binary journal, once those are taken out of active logging, yeah, let 
btrfs do its normal thing.

But for all these cases, as long as the files are being actively written 
into, NOCOW, including its NOSUM implications, is exactly and precisely 
what they SHOULD be when the filesystem hosting them is btrfs.

And I'm predicting that since btrfs is the assumed successor to the ext* 
series as the Linux default filesystem, and systemd is targeting Linux 
default initsystem status as well, it's only logical that at some point 
systemd will detect what filesystem it's logging to, and will 
automatically set NOCOW on the journal file when that filesystem is 
btrfs.  Most Linux-targeted databases and file-preallocating torrent 
clients will no doubt do exactly the same thing.  Either that, or in 
their documentation, they'll suggest setting NOCOW on the target 
directory when setting up the app in the first place.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: systemd-journal, nodatacow, was: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Murphy]

On Dec 29, 2013, at 5:39 AM, Duncan <1i5t5.duncan@cox.net> wrote:

> Yes, it does turn off checksumming as well as COW, but given the write-
> into scenario, that's actually best anyway, because otherwise btrfs has 
> to keep updating the checksums

On second thought, I'm less concerned with bitrot and checksumming being lost with nodatacow, than I am with significantly increasing the chance the journal is irreparably lost due to corruption during an unclean shutdown.


> But in all these cases, it's also quite common for the application doing 
> the writing to have its own checksumming/error-detection and possible 
> correction -- it pretty much comes with the territory -- in which case 
> btrfs attempting to do the same is simply superfluous even if it weren't 
> a race-condition trigger.

I don't know what kind of checksumming systemd performs on the journal, but whenever Btrfs has found corruption with the journal file(s), systemd-journald has also found corruption and starts a new log. So it makes sense to rely on its own mechanisms, than Btrfs's.

> And I'm predicting that since btrfs is the assumed successor to the ext* 
> series as the Linux default filesystem, and systemd is targeting Linux 
> default initsystem status as well, it's only logical that at some point 
> systemd will detect what filesystem it's logging to, and will 
> automatically set NOCOW on the journal file when that filesystem is 
> btrfs. 

Is this something that should be brought up on the systemd-devel@ list? Or maybe file it as an RFE against systemd at freedesktop.org?

Chris Murphy

Re: systemd-journal, nodatacow, was: Is anyone using btrfs send/receive for backups instead of rsync?

[Duncan]

Chris Murphy posted on Sun, 29 Dec 2013 17:38:23 -0700 as excerpted:

>> And I'm predicting that since btrfs is the assumed successor to the
>> ext*
>> series as the Linux default filesystem, and systemd is targeting Linux
>> default initsystem status as well, it's only logical that at some point
>> systemd will detect what filesystem it's logging to, and will
>> automatically set NOCOW on the journal file when that filesystem is
>> btrfs.
> 
> Is this something that should be brought up on the systemd-devel@ list?
> Or maybe file it as an RFE against systemd at freedesktop.org?

I don't know.

While I don't (yet?) run systemd personally, I'd have almost thought it'd 
be done by now (tho obviously it's not, at least in distro-current 
versions), but perhaps they've been waiting on word that btrfs or some API 
they plan to use for it is stabilizing before doing it.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: systemd-journal, nodatacow, was: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Mason]

On Sun, 2013-12-29 at 17:38 -0700, Chris Murphy wrote:
> On Dec 29, 2013, at 5:39 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> > Yes, it does turn off checksumming as well as COW, but given the write-
> > into scenario, that's actually best anyway, because otherwise btrfs has 
> > to keep updating the checksums
> 
> On second thought, I'm less concerned with bitrot and checksumming being lost with nodatacow, than I am with significantly increasing the chance the journal is irreparably lost due to corruption during an unclean shutdown.

So first, send/receive + nowcow aren't a great combination.  NOCOW won't
update the generation numbers send/receive needs to find changes.  The
best send/receive can do in that case is send over the entire file.

> 
> > But in all these cases, it's also quite common for the application doing 
> > the writing to have its own checksumming/error-detection and possible 
> > correction -- it pretty much comes with the territory -- in which case 
> > btrfs attempting to do the same is simply superfluous even if it weren't 
> > a race-condition trigger.
> 
> I don't know what kind of checksumming systemd performs on the journal, but whenever Btrfs has found corruption with the journal file(s), systemd-journald has also found corruption and starts a new log. So it makes sense to rely on its own mechanisms, than Btrfs's.
> 

The autodefrag mode was really made for the small databases like
systemd.  I'd prefer that we use that for systemd instead of suggesting
NOCOW.  I'm finally dusting off my work to improve db performance, so
hopefully we can do much better in the near future.

-chris

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Mason]

On Sat, 2013-12-28 at 10:20 -0800, Marc MERLIN wrote:
> On Sat, Dec 28, 2013 at 10:07:58AM -0800, Marc MERLIN wrote:
> > For instance, if I use an existing rsync destination to start syncing
> > btrfs snapshots to after that, and one file operation can't be applied
> > because let's say the destination file it's supposed to be applied to,
> > isn't there?
> 
> I should have written more: I'm guessing what happens is that the btrfs
> receive fails/aborts, I get an error, I then run a manual rsync to reset
> everything to a good known state, and then continue the btrfs
> send/receive after that?

Btrfs send/receive works by matching state between snapshots on the
sending and receiving end.  If you update the files manually on the
receiving end (say with rsync), it can't merge the states anymore.

-chris

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Marc MERLIN]

On Mon, Dec 30, 2013 at 04:05:03PM +0000, Chris Mason wrote:
> On Sat, 2013-12-28 at 10:20 -0800, Marc MERLIN wrote:
> > On Sat, Dec 28, 2013 at 10:07:58AM -0800, Marc MERLIN wrote:
> > > For instance, if I use an existing rsync destination to start syncing
> > > btrfs snapshots to after that, and one file operation can't be applied
> > > because let's say the destination file it's supposed to be applied to,
> > > isn't there?
> > 
> > I should have written more: I'm guessing what happens is that the btrfs
> > receive fails/aborts, I get an error, I then run a manual rsync to reset
> > everything to a good known state, and then continue the btrfs
> > send/receive after that?
> 
> Btrfs send/receive works by matching state between snapshots on the
> sending and receiving end.  If you update the files manually on the
> receiving end (say with rsync), it can't merge the states anymore.

I got that, but it wasn't quite my question :)

I understand that btrfs receive cannot apply file changes if the
destination filesystem isn't in a file state that's identical to the source
one.

I'm just not too sure how the destination FS needs to be configured so
that btrfs receive can work with it.

1) Does it need to be an exact byte for byte copy of the block device the
source was on?

2) Or can the destination be seeded with a full rsync or cp -a and can btrfs receive
take over from there?

3) Then, if I hit a bug where something doesn't get synced right, and I run
rsync to fix or verify that the two FS are indeed identical file-wise
like they're supposed to, if rsync fixes something, are you saying that
it'll stop btrfs receive from working after that?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Mason]

On Mon, 2013-12-30 at 08:17 -0800, Marc MERLIN wrote:
> On Mon, Dec 30, 2013 at 04:05:03PM +0000, Chris Mason wrote:
> > On Sat, 2013-12-28 at 10:20 -0800, Marc MERLIN wrote:
> > > On Sat, Dec 28, 2013 at 10:07:58AM -0800, Marc MERLIN wrote:
> > > > For instance, if I use an existing rsync destination to start syncing
> > > > btrfs snapshots to after that, and one file operation can't be applied
> > > > because let's say the destination file it's supposed to be applied to,
> > > > isn't there?
> > > 
> > > I should have written more: I'm guessing what happens is that the btrfs
> > > receive fails/aborts, I get an error, I then run a manual rsync to reset
> > > everything to a good known state, and then continue the btrfs
> > > send/receive after that?
> > 
> > Btrfs send/receive works by matching state between snapshots on the
> > sending and receiving end.  If you update the files manually on the
> > receiving end (say with rsync), it can't merge the states anymore.
> 
> I got that, but it wasn't quite my question :)
> 
> I understand that btrfs receive cannot apply file changes if the
> destination filesystem isn't in a file state that's identical to the source
> one.
> 
> I'm just not too sure how the destination FS needs to be configured so
> that btrfs receive can work with it.
> 
> 1) Does it need to be an exact byte for byte copy of the block device the
> source was on?
> 
No, in fact this doesn't help.

> 2) Or can the destination be seeded with a full rsync or cp -a and can btrfs receive
> take over from there?
> 

No, it has to be created by btrfs receive.

> 3) Then, if I hit a bug where something doesn't get synced right, and I run
> rsync to fix or verify that the two FS are indeed identical file-wise
> like they're supposed to, if rsync fixes something, are you saying that
> it'll stop btrfs receive from working after that?
> 

Yes, today anyway it won't work.  Send converts the changed items into
an intermediate format (we don't send btree blocks directly over the
wire) and then receive modifies the destination from userland.

At the end of the stream we update the destination root to say "you're
now version xxyyzz of uuid aabbcc".

We definitely could add a way to manually set this, but once a user does
it, it'll be very hard to debug any problems they might have had if
their copy wasn't actually up to date.

-chris

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Marc MERLIN]

On Mon, Dec 30, 2013 at 04:26:42PM +0000, Chris Mason wrote:
> > 1) Does it need to be an exact byte for byte copy of the block device the
> > source was on?
> > 
> No, in fact this doesn't help.
> 
> > 2) Or can the destination be seeded with a full rsync or cp -a and can btrfs receive
> > take over from there?
> 
> No, it has to be created by btrfs receive.

Aaah, I wasn't clear on that, thanks for clarifying.
So I need to make sure the target block device is at least as big as the
source one, and if necessary a few blocks bigger if the drives do not
allocate partitions of the exactly the same size.

Mmmh, this makes it less desirable for me to use this then since I use over
allocation on the backup servers and if I had to have as much space blocked
off for the full size of each filesystem backed up, I'm going to be short.

Bummer.
 
> > 3) Then, if I hit a bug where something doesn't get synced right, and I run
> > rsync to fix or verify that the two FS are indeed identical file-wise
> > like they're supposed to, if rsync fixes something, are you saying that
> > it'll stop btrfs receive from working after that?
> 
> Yes, today anyway it won't work.  Send converts the changed items into
> an intermediate format (we don't send btree blocks directly over the
> wire) and then receive modifies the destination from userland.
> 
> At the end of the stream we update the destination root to say "you're
> now version xxyyzz of uuid aabbcc".
> 
> We definitely could add a way to manually set this, but once a user does
> it, it'll be very hard to debug any problems they might have had if
> their copy wasn't actually up to date.

Understood. I dreamt that it was computing file differences and could just
apply them on top of any other btrfs filesystem, even if it were smaller and
had been created via rsync.

If one day, it could at least work on a subvolume level (only sync a
subvolume), then it would be more useful to me. Maybe later...

Thanks for clearing that up.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Murphy]

On Dec 30, 2013, at 10:10 AM, Marc MERLIN <marc@merlins.org> wrote:
> 
> If one day, it could at least work on a subvolume level (only sync a
> subvolume), then it would be more useful to me. Maybe later…

Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.


Chris Murphy

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Marc MERLIN]

On Mon, Dec 30, 2013 at 10:48:10AM -0700, Chris Murphy wrote:
> 
> On Dec 30, 2013, at 10:10 AM, Marc MERLIN <marc@merlins.org> wrote:
> > 
> > If one day, it could at least work on a subvolume level (only sync a
> > subvolume), then it would be more useful to me. Maybe later…
> 
> Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.

Never mind, I seem to be the one being dense. I mis-read that you needed
to create the filesystem with btrfs receive.
Indeed, it's on a subvolume level, so it's actually fine since it does
allow over provisionning afterall.

My bad, sorry :)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Murphy]

On Dec 30, 2013, at 10:57 AM, Marc MERLIN <marc@merlins.org> wrote:

> On Mon, Dec 30, 2013 at 10:48:10AM -0700, Chris Murphy wrote:
>> 
>> On Dec 30, 2013, at 10:10 AM, Marc MERLIN <marc@merlins.org> wrote:
>>> 
>>> If one day, it could at least work on a subvolume level (only sync a
>>> subvolume), then it would be more useful to me. Maybe later…
>> 
>> Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.
> 
> Never mind, I seem to be the one being dense. I mis-read that you needed
> to create the filesystem with btrfs receive.
> Indeed, it's on a subvolume level, so it's actually fine since it does
> allow over provisionning afterall.

Depending on resources and disaster recovery requirements, you might also consider using send -f without receive at all, to the backup destination. The first send file (which will be big) can then be put anywhere, even to tape, and use the backup storage just for the incremental send -f files.


Chris Murphy

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Marc MERLIN]

On Mon, Dec 30, 2013 at 09:57:40AM -0800, Marc MERLIN wrote:
> On Mon, Dec 30, 2013 at 10:48:10AM -0700, Chris Murphy wrote:
> > 
> > On Dec 30, 2013, at 10:10 AM, Marc MERLIN <marc@merlins.org> wrote:
> > > 
> > > If one day, it could at least work on a subvolume level (only sync a
> > > subvolume), then it would be more useful to me. Maybe later…
> > 
> > Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.
> 
> Never mind, I seem to be the one being dense. I mis-read that you needed
> to create the filesystem with btrfs receive.
> Indeed, it's on a subvolume level, so it's actually fine since it does
> allow over provisionning afterall.

Mmmh, but I just realized that on my laptop, I do boot the btrfs copy
(currently done with rsync) from time to time (i.e. emergency boot from
the HD the SSD was copied to).
If I do that, it'll change the filesystem that was created with btrfs
receive and break it, preventing further updates, correct?

If so, can I get around that by making a boot snapshot after each copy
and mount that snapshot for emergency boot instead of the main volume?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Is anyone using btrfs send/receive for backups instead of rsync?

[Chris Mason]

On Fri, 2014-01-03 at 12:15 -0800, Marc MERLIN wrote:
> On Mon, Dec 30, 2013 at 09:57:40AM -0800, Marc MERLIN wrote:
> > On Mon, Dec 30, 2013 at 10:48:10AM -0700, Chris Murphy wrote:
> > > 
> > > On Dec 30, 2013, at 10:10 AM, Marc MERLIN <marc@merlins.org> wrote:
> > > > 
> > > > If one day, it could at least work on a subvolume level (only sync a
> > > > subvolume), then it would be more useful to me. Maybe later…
> > > 
> > > Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.
> > 
> > Never mind, I seem to be the one being dense. I mis-read that you needed
> > to create the filesystem with btrfs receive.
> > Indeed, it's on a subvolume level, so it's actually fine since it does
> > allow over provisionning afterall.
> 
> Mmmh, but I just realized that on my laptop, I do boot the btrfs copy
> (currently done with rsync) from time to time (i.e. emergency boot from
> the HD the SSD was copied to).
> If I do that, it'll change the filesystem that was created with btrfs
> receive and break it, preventing further updates, correct?
> 
> If so, can I get around that by making a boot snapshot after each copy
> and mount that snapshot for emergency boot instead of the main volume?

Yes that will work.

-chris

Re: Is anyone using btrfs send/receive howto?

[Marc MERLIN]

I read different howtos on the wiki and oracle docs, but I can't get it
to work:

legolas:/mnt/btrfs_pool1# btrfs subvolume snapshot -r tmp tmp_read_only_new
Create a readonly snapshot of 'tmp' in './tmp_read_only_new'
legolas:/mnt/btrfs_pool1# sync
legolas:/mnt/btrfs_pool1# btrfs send tmp_read_only_new | btrfs receive /mnt/btrfs_pool2/
At subvol tmp_read_only_new
At subvol tmp_read_only_new

# Make a new snapshot later and try to sync it:
legolas:/mnt/btrfs_pool1# mv tmp_read_only_new tmp_read_only
legolas:/mnt/btrfs_pool1# btrfs subvolume snapshot -r tmp tmp_read_only_new
Create a readonly snapshot of 'tmp' in './tmp_read_only_new'
legolas:/mnt/btrfs_pool1# btrfs send -p tmp_read_only  tmp_read_only_new | btrfs receive /mnt/btrfs_pool2/
At subvol tmp_read_only_new
At snapshot tmp_read_only_new
ERROR: creating snapshot tmp_read_only_new -> tmp_read_only_new failed. File exists

This is where I get stuck:
Obviously /mnt/btrfs_pool2/tmp_read_only_new already exists since it's the reference backup volume.

What am I doing wrong?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Is anyone using btrfs send/receive howto?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

On Tue, Jan 07, 2014 at 02:49:51AM -0800, Marc MERLIN wrote:
> I read different howtos on the wiki and oracle docs, but I can't get it
> to work:
> 
> legolas:/mnt/btrfs_pool1# btrfs subvolume snapshot -r tmp tmp_read_only_new
> Create a readonly snapshot of 'tmp' in './tmp_read_only_new'
> legolas:/mnt/btrfs_pool1# sync
> legolas:/mnt/btrfs_pool1# btrfs send tmp_read_only_new | btrfs receive /mnt/btrfs_pool2/
> At subvol tmp_read_only_new
> At subvol tmp_read_only_new
> 
> # Make a new snapshot later and try to sync it:
> legolas:/mnt/btrfs_pool1# mv tmp_read_only_new tmp_read_only
> legolas:/mnt/btrfs_pool1# btrfs subvolume snapshot -r tmp tmp_read_only_new
> Create a readonly snapshot of 'tmp' in './tmp_read_only_new'
> legolas:/mnt/btrfs_pool1# btrfs send -p tmp_read_only  tmp_read_only_new | btrfs receive /mnt/btrfs_pool2/
> At subvol tmp_read_only_new
> At snapshot tmp_read_only_new
> ERROR: creating snapshot tmp_read_only_new -> tmp_read_only_new failed. File exists
> 
> This is where I get stuck:
> Obviously /mnt/btrfs_pool2/tmp_read_only_new already exists since it's the reference backup volume.
> 
> What am I doing wrong?

   You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
name as well. The send stream contains the name of the subvolume it
wants to create, so it's trying to create a subvolume called
"tmp_read_only_new" in /mnt/btrfs_pool2, and there's one there already.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- It's against my programming to impersonate a deity! ---       

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Is anyone using btrfs send/receive howto?

[Marc MERLIN]

On Tue, Jan 07, 2014 at 10:53:29AM +0000, Hugo Mills wrote:
>    You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
> name as well. The send stream contains the name of the subvolume it
> wants to create, so it's trying to create a subvolume called
> "tmp_read_only_new" in /mnt/btrfs_pool2, and there's one there already.

Aaah, got it, thanks.

I'll contribute back a shell script that makes this much easier when I'm
done with it.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)

[Marc MERLIN]

On Wed, Jan 08, 2014 at 12:02:06AM -0800, Marc MERLIN wrote:
> On Tue, Jan 07, 2014 at 10:53:29AM +0000, Hugo Mills wrote:
> >    You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
> > name as well. The send stream contains the name of the subvolume it
> > wants to create, so it's trying to create a subvolume called
> > "tmp_read_only_new" in /mnt/btrfs_pool2, and there's one there already.
> 
> Aaah, got it, thanks.
> 
> I'll contribute back a shell script that makes this much easier when I'm
> done with it.

I've now spent enough time on this and have enough testing to release it.

http://marc.merlins.org/perso/btrfs/post_2014-03-22_Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive.html
which points to
http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup


Just for the archives, here is a working copy (the blog post above explains
why it keeps multiple snapshots after the fact, and creates writeable ones
on the destination, but basically this allows you to automatically boot from
the latest snapshot (a writeable snapshot copy pointed to by a symlink tha
tis kept up to date)):
----------------------------------------------------------------------------
#!/bin/bash

# By Marc MERLIN <marc_soft@merlins.org>
# License: GPL-2 or BSD at your choice.

# Source: http://marc.merlins.org/linux/scripts/
# $Id: btrfs-subvolume-backup 958 2014-03-16 00:23:28Z svnuser $

# cron jobs might not have /sbin in their path.
export PATH="$PATH:/sbin"

set -o nounset
set -o errexit
set -o pipefail

# From https://btrfs.wiki.kernel.org/index.php/Incremental_Backup


# bash shortcut for `basename $0`
PROG=${0##*/}
lock=/var/run/$PROG

usage() {
    cat <<EOF
Usage: 
cd /mnt/source_btrfs_pool
$PROG [--init] [--keep|-k num] [--dest hostname] volume_name /mnt/backup_btrfs_pool

Options:
    --init:          Print this help message and exit.
    --keep num:      Keep the last snapshots for local backups (5 by default)
    --dest hostname: If present, ssh to that machine to make the copy.

This will snapshot volume_name in a btrfs pool, and send the diff
between it and the previous snapshot (volume_name.last) to another btrfs
pool (on other drives)

If your backup destination is another machine, you'll need to add a few
ssh commands this script

The num sanpshots to keep is to give snapshots you can recover data from 
and they get deleted after num runs. Set to 0 to disable (one snapshot will
be kept since it's required for the next diff to be computed).
EOF
    exit 0
}

die () {
    msg=${1:-}
    # don't loop on ERR
    trap '' ERR

    rm $lock

    echo "$msg" >&2
    echo >&2

    # This is a fancy shell core dumper
    if echo $msg | grep -q 'Error line .* with status'; then
	line=`echo $msg | sed 's/.*Error line \(.*\) with status.*/\1/'`
	echo " DIE: Code dump:" >&2
	nl -ba $0 | grep -3 "\b$line\b" >&2
    fi
    
    exit 1
}

# Trap errors for logging before we die (so that they can be picked up
# by the log checker)
trap 'die "Error line $LINENO with status $?"' ERR


init=""
# Keep the last 5 snapshots by default
keep=5
TEMP=$(getopt --longoptions help,usage,init,keep:,dest: -o h,k:,d: -- "$@") || usage
dest=localhost
ssh=""

# getopt quotes arguments with ' We use eval to get rid of that
eval set -- $TEMP

while :
do
    case "$1" in
        -h|--help|--usage)
            usage
            shift
            ;;

	--keep|-k)
	    shift
	    keep=$1
	    shift
	    ;;

	--dest|-d)
	    shift
	    dest=$1
	    ssh="ssh $dest"
	    shift
	    ;;

	--init)
	    init=1
	    shift
	    ;;

	--)
	    shift
	    break
	    ;;

        *) 
	    echo "Internal error!"
	    exit 1
	    ;;
    esac
done
[[ $keep < 1 ]] && die "Must keep at least one snapshot for things to work ($keep given)"

DATE="$(date '+%Y%m%d_%H:%M:%S')"

[[ $# != 2 ]] && usage
vol="$1"
dest_pool="$2"

# shlock (from inn) does the right thing and grabs a lock for a dead process
# (it checks the PID in the lock file and if it's not there, it
# updates the PID with the value given to -p)
if ! shlock -p $$ -f $lock; then
    echo "$lock held for $PROG, quitting" >&2
    exit
fi

if [[ -z "$init" ]]; then
    test -e "${vol}_last" \
	|| die "Cannot sync $vol, ${vol}_last missing. Try --init?"
    src_snap="$(readlink -e ${vol}_last)"
fi
src_newsnap="${vol}_ro.$DATE"
src_newsnaprw="${vol}_rw.$DATE"

$ssh test -d "$dest_pool/" || die "ABORT: $dest_pool not a directory (on $dest)"

btrfs subvolume snapshot -r "$vol" "$src_newsnap"

# There is currently an issue that the snapshots to be used with "btrfs send"
# must be physically on the disk, or you may receive a "stale NFS file handle"
# error. This is accomplished by "sync" after the snapshot
sync

if [[ -n "$init" ]]; then
    btrfs send "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
else
    btrfs send -p "$src_snap" "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
fi

# We make a read-write snapshot in case you want to use it for a chroot
# and some testing with a writeable filesystem or want to boot from a
# last good known snapshot.
btrfs subvolume snapshot "$src_newsnap" "$src_newsnaprw"
$ssh btrfs subvolume snapshot "$dest_pool/$src_newsnap" "$dest_pool/$src_newsnaprw"

# Keep track of the last snapshot to send a diff against.
ln -snf $src_newsnap ${vol}_last
# The rw version can be used for mounting with subvol=vol_last_rw
ln -snf $src_newsnaprw ${vol}_last_rw
$ssh ln -snf $src_newsnaprw $dest_pool/${vol}_last_rw

# How many snapshots to keep on the source btrfs pool (both read
# only and read-write).
ls -rd ${vol}_ro* | tail -n +$(( $keep + 1 ))| while read snap
do
    btrfs subvolume delete "$snap"
done
ls -rd ${vol}_rw* | tail -n +$(( $keep + 1 ))| while read snap
do
    btrfs subvolume delete "$snap"
done

# Same thing for destination (assume the same number of snapshots to keep,
# you can change this if you really want).
$ssh ls -rd $dest_pool/${vol}_ro* | tail -n +$(( $keep + 1 ))| while read snap
do
    $ssh btrfs subvolume delete "$snap"
done
$ssh ls -rd $dest_pool/${vol}_rw* | tail -n +$(( $keep + 1 ))| while read snap
do
    $ssh btrfs subvolume delete "$snap"
done

rm $lock
----------------------------------------------------------------------------
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)

[Brendan Hide]

On 2014/03/21 07:29 PM, Marc MERLIN wrote:
> On Wed, Jan 08, 2014 at 12:02:06AM -0800, Marc MERLIN wrote:
>> On Tue, Jan 07, 2014 at 10:53:29AM +0000, Hugo Mills wrote:
>>>     You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
>>> name as well. The send stream contains the name of the subvolume it
>>> wants to create, so it's trying to create a subvolume called
>>> "tmp_read_only_new" in /mnt/btrfs_pool2, and there's one there already.
>> Aaah, got it, thanks.
>>
>> I'll contribute back a shell script that makes this much easier when I'm
>> done with it.
> I've now spent enough time on this and have enough testing to release it.
>
> http://marc.merlins.org/perso/btrfs/post_2014-03-22_Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive.html
> which points to
> http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup
>
>
> Just for the archives, here is a working copy (the blog post above explains
> why it keeps multiple snapshots after the fact, and creates writeable ones
> on the destination, but basically this allows you to automatically boot from
> the latest snapshot (a writeable snapshot copy pointed to by a symlink tha
> tis kept up to date)):
> ----------------------------------------------------------------------------
> #!/bin/bash
>
> # By Marc MERLIN <marc_soft@merlins.org>
> # License: GPL-2 or BSD at your choice.
>
> # Source: http://marc.merlins.org/linux/scripts/
> # $Id: btrfs-subvolume-backup 958 2014-03-16 00:23:28Z svnuser $
>
> # cron jobs might not have /sbin in their path.
> export PATH="$PATH:/sbin"
>
> set -o nounset
> set -o errexit
> set -o pipefail
>
> # From https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
>
>
> # bash shortcut for `basename $0`
> PROG=${0##*/}
> lock=/var/run/$PROG
>
> usage() {
>      cat <<EOF
> Usage:
> cd /mnt/source_btrfs_pool
> $PROG [--init] [--keep|-k num] [--dest hostname] volume_name /mnt/backup_btrfs_pool
>
> Options:
>      --init:          Print this help message and exit.
>      --keep num:      Keep the last snapshots for local backups (5 by default)
>      --dest hostname: If present, ssh to that machine to make the copy.
>
> This will snapshot volume_name in a btrfs pool, and send the diff
> between it and the previous snapshot (volume_name.last) to another btrfs
> pool (on other drives)
>
> If your backup destination is another machine, you'll need to add a few
> ssh commands this script
>
> The num sanpshots to keep is to give snapshots you can recover data from
> and they get deleted after num runs. Set to 0 to disable (one snapshot will
> be kept since it's required for the next diff to be computed).
> EOF
>      exit 0
> }
>
> die () {
>      msg=${1:-}
>      # don't loop on ERR
>      trap '' ERR
>
>      rm $lock
>
>      echo "$msg" >&2
>      echo >&2
>
>      # This is a fancy shell core dumper
>      if echo $msg | grep -q 'Error line .* with status'; then
> 	line=`echo $msg | sed 's/.*Error line \(.*\) with status.*/\1/'`
> 	echo " DIE: Code dump:" >&2
> 	nl -ba $0 | grep -3 "\b$line\b" >&2
>      fi
>      
>      exit 1
> }
>
> # Trap errors for logging before we die (so that they can be picked up
> # by the log checker)
> trap 'die "Error line $LINENO with status $?"' ERR
>
>
> init=""
> # Keep the last 5 snapshots by default
> keep=5
> TEMP=$(getopt --longoptions help,usage,init,keep:,dest: -o h,k:,d: -- "$@") || usage
> dest=localhost
> ssh=""
>
> # getopt quotes arguments with ' We use eval to get rid of that
> eval set -- $TEMP
>
> while :
> do
>      case "$1" in
>          -h|--help|--usage)
>              usage
>              shift
>              ;;
>
> 	--keep|-k)
> 	    shift
> 	    keep=$1
> 	    shift
> 	    ;;
>
> 	--dest|-d)
> 	    shift
> 	    dest=$1
> 	    ssh="ssh $dest"
> 	    shift
> 	    ;;
>
> 	--init)
> 	    init=1
> 	    shift
> 	    ;;
>
> 	--)
> 	    shift
> 	    break
> 	    ;;
>
>          *)
> 	    echo "Internal error!"
> 	    exit 1
> 	    ;;
>      esac
> done
> [[ $keep < 1 ]] && die "Must keep at least one snapshot for things to work ($keep given)"
>
> DATE="$(date '+%Y%m%d_%H:%M:%S')"
>
> [[ $# != 2 ]] && usage
> vol="$1"
> dest_pool="$2"
>
> # shlock (from inn) does the right thing and grabs a lock for a dead process
> # (it checks the PID in the lock file and if it's not there, it
> # updates the PID with the value given to -p)
> if ! shlock -p $$ -f $lock; then
>      echo "$lock held for $PROG, quitting" >&2
>      exit
> fi
>
> if [[ -z "$init" ]]; then
>      test -e "${vol}_last" \
> 	|| die "Cannot sync $vol, ${vol}_last missing. Try --init?"
>      src_snap="$(readlink -e ${vol}_last)"
> fi
> src_newsnap="${vol}_ro.$DATE"
> src_newsnaprw="${vol}_rw.$DATE"
>
> $ssh test -d "$dest_pool/" || die "ABORT: $dest_pool not a directory (on $dest)"
>
> btrfs subvolume snapshot -r "$vol" "$src_newsnap"
>
> # There is currently an issue that the snapshots to be used with "btrfs send"
> # must be physically on the disk, or you may receive a "stale NFS file handle"
> # error. This is accomplished by "sync" after the snapshot
> sync
>
> if [[ -n "$init" ]]; then
>      btrfs send "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
> else
>      btrfs send -p "$src_snap" "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
> fi
>
> # We make a read-write snapshot in case you want to use it for a chroot
> # and some testing with a writeable filesystem or want to boot from a
> # last good known snapshot.
> btrfs subvolume snapshot "$src_newsnap" "$src_newsnaprw"
> $ssh btrfs subvolume snapshot "$dest_pool/$src_newsnap" "$dest_pool/$src_newsnaprw"
>
> # Keep track of the last snapshot to send a diff against.
> ln -snf $src_newsnap ${vol}_last
> # The rw version can be used for mounting with subvol=vol_last_rw
> ln -snf $src_newsnaprw ${vol}_last_rw
> $ssh ln -snf $src_newsnaprw $dest_pool/${vol}_last_rw
>
> # How many snapshots to keep on the source btrfs pool (both read
> # only and read-write).
> ls -rd ${vol}_ro* | tail -n +$(( $keep + 1 ))| while read snap
> do
>      btrfs subvolume delete "$snap"
> done
> ls -rd ${vol}_rw* | tail -n +$(( $keep + 1 ))| while read snap
> do
>      btrfs subvolume delete "$snap"
> done
>
> # Same thing for destination (assume the same number of snapshots to keep,
> # you can change this if you really want).
> $ssh ls -rd $dest_pool/${vol}_ro* | tail -n +$(( $keep + 1 ))| while read snap
> do
>      $ssh btrfs subvolume delete "$snap"
> done
> $ssh ls -rd $dest_pool/${vol}_rw* | tail -n +$(( $keep + 1 ))| while read snap
> do
>      $ssh btrfs subvolume delete "$snap"
> done
>
> rm $lock
> ----------------------------------------------------------------------------

Hi, Marc

Feel free to use ideas from my own script. Some aspects in my script are 
more mature and others are frankly pathetic. ;)

There are also quite a lot of TODOs throughout my script that aren't 
likely to get the urgent attention they deserve. It has been slowly 
evolving over the last two weeks.

http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)

[Brendan Hide]

On 2014/03/22 09:44 PM, Brendan Hide wrote:
> On 2014/03/21 07:29 PM, Marc MERLIN wrote:
>>
>
> Hi, Marc
>
> Feel free to use ideas from my own script. Some aspects in my script 
> are more mature and others are frankly pathetic. ;)
>
> There are also quite a lot of TODOs throughout my script that aren't 
> likely to get the urgent attention they deserve. It has been slowly 
> evolving over the last two weeks.
>
> http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup
>
I forgot to include some notes:

The script depends on a "config" file at /etc/btrfs-backup/paths.conf 
which is supposed to contain the paths as well as some parameters. At 
present the file consists solely of the following paths as these are all 
separate subvolumes in my test system:
"/
/home
/usr
/var
"

The snapshot names on source and backup are formatted as below. This 
way, daylight savings doesn't need any special treatment:
__2014-03-17-23h00m01s+0200
__2014-03-18-23h00m01s+0200
__2014-03-19-23h00m01s+0200
__2014-03-20-23h00m02s+0200
__2014-03-21-23h00m01s+0200
_home_2014-03-17-23h00m01s+0200
_home_2014-03-18-23h00m01s+0200
_home_2014-03-19-23h00m01s+0200
_home_2014-03-20-23h00m02s+0200
_home_2014-03-21-23h00m01s+0200
_usr_2014-03-17-23h00m01s+0200
_usr_2014-03-18-23h00m01s+0200
_usr_2014-03-19-23h00m01s+0200
_usr_2014-03-20-23h00m02s+0200
_usr_2014-03-21-23h00m01s+0200
_var_2014-03-17-23h00m01s+0200
_var_2014-03-18-23h00m01s+0200
_var_2014-03-19-23h00m01s+0200
_var_2014-03-20-23h00m02s+0200
_var_2014-03-21-23h00m01s+0200



-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)

[Marc MERLIN]

On Sat, Mar 22, 2014 at 09:44:05PM +0200, Brendan Hide wrote:
> Hi, Marc
> 
> Feel free to use ideas from my own script. Some aspects in my script are 
> more mature and others are frankly pathetic. ;)
> 
> There are also quite a lot of TODOs throughout my script that aren't 
> likely to get the urgent attention they deserve. It has been slowly 
> evolving over the last two weeks.
> 
> http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup

I figured I likely wasn't the only one working on a script like this :)

>From a quick read, it looks even more complex than mine :) but
- it doesn't do ssh to a destination for a remote backup
- it doesn't seem to keep a list of configurable snapshots not necessary for
send/restore but useful for getting historical data
- it doesn't seem to use a symlink to keep track of the last complete
snapshot on the source and destination, and does more work to compensate
when recovering from an incomplete backup/restore.
- it doesn't create writeable snapshots on the destination in case you want
to use the copy as a live filesystem

Things I noticed:
- I don't use ionice, maybe I should. Did you find that it actually made a
difference with send/receive?
- Your comments say shlock isn't safe and that's documented. I don't see
that in the man page
http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html
I'd love to have details on this if I shouldn't be using it
- Is set -o noclobber; echo $$ > $lockfile really atomic and safer than
shlock? If so, great, although I would then wonder why shlock even exists :)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)

[Brendan Hide]

On 2014/03/22 10:00 PM, Marc MERLIN wrote:
> On Sat, Mar 22, 2014 at 09:44:05PM +0200, Brendan Hide wrote:
>> Hi, Marc
>>
>> Feel free to use ideas from my own script. Some aspects in my script are
>> more mature and others are frankly pathetic. ;)
>>
>> There are also quite a lot of TODOs throughout my script that aren't
>> likely to get the urgent attention they deserve. It has been slowly
>> evolving over the last two weeks.
>>
>> http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup
> I figured I likely wasn't the only one working on a script like this :)
>
>  From a quick read, it looks even more complex than mine :) but
Well ... I did say some things are pathetic my side. ;) I also use a 
template (its about 3 years old now) when I make a new script, hence the 
options such as being able to ignore the mutex checks and also having a 
random delay at start. These obviously add some unnecessary complexity.
> - it doesn't do ssh to a destination for a remote backup
There should be a TODO for this on my side. Presently in testing I'm 
only using it for device-local backup to a separate disk and not to a 
"proper" remote backup.
> - it doesn't seem to keep a list of configurable snapshots not necessary for
> send/restore but useful for getting historical data
I'm not sure what this is useful for. :-/ If related, I plan on creating 
a separate script to move snapshots around into _var_daily_$date, 
_var_weekly_$date, etc.
> - it doesn't seem to use a symlink to keep track of the last complete
> snapshot on the source and destination, and does more work to compensate
> when recovering from an incomplete backup/restore.
Yes, a symlink would make this smoother.
> - it doesn't create writeable snapshots on the destination in case you want
> to use the copy as a live filesystem
One of the issues with doing writeable snapshots by default is that the 
backup is (ever so slightly) less safe from "fat-finger syndrome". If I 
want a writeable snapshot, I'll make it from the read-only snapshot, 
thereby reducing the chances of accidentally "tainting" or deleting data 
in the backup. I actually *did* accidentally delete my entire filesystem 
(hence the paranoid umounts). But, of course, my script *first* created 
read-only snapshots from which recovery took only a few minutes. ;)
> Things I noticed:
> - I don't use ionice, maybe I should. Did you find that it actually made a
> difference with send/receive?
This is just a habit I've developed over time in all my scripts. I 
figure that if I'm using the machine at the time and the snapshot has a 
large churn, I'd prefer the ionice. That said, the main test system is a 
desktop which is likely to have much less churn than a server. In the 
last two weeks the longest daily incremental backup took about 5 minutes 
to complete, while it typically takes about 30 seconds only.
> - Your comments say shlock isn't safe and that's documented. I don't see
> that in the man page
> http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html
That man page looks newer than the one I last looked at - specifically 
the part saying "improved by Berend Reitsma to solve a race condition." 
The previous documentation on shlock indicated it was safe for hourly 
crons - but not in the case where a cron might be executed twice 
simultaneously. Shlock was recommended by a colleague until I realised 
this potential issue, thus my template doesn't use it. I should update 
the comment with some updated information.

My only two worries then would be a) if it is outdated on other distros 
and b) that it appears that it is not installed by default. On my Arch 
desktop it seems to be available with "inn"[1] (Usenet server and 
related software) and nowhere else. It seems the same on Ubuntu (Google 
pointed me to inn2-dev). Do you have INN installed? If not, where did 
you get shlock from?
> I'd love to have details on this if I shouldn't be using it
> - Is set -o noclobber; echo $$ > $lockfile really atomic and safer than
> shlock? If so, great, although I would then wonder why shlock even exists :)
The part that brings about an atomic lock is "noclobber", which sets it 
so that we are not allowed to "clobber"/"overwrite" an existing file. 
Thus, if the file exists, the command fails. If it successfully creates 
the new file, the command returns true.

I'd consider changing this mostly for the fact that depending on INN is 
a very big dependency. There are other options as well, though I don't 
think they're as portable as noclobber.
>
> Thanks,
> Marc

Thanks for your input. It has already given me some direction. :)

[1] https://www.archlinux.org/packages/community/x86_64/inn/files/

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)

[Marc MERLIN]

Please consider adding a blank line between quotes, it makes them just a bit
more readable :)

On Sat, Mar 22, 2014 at 11:02:24PM +0200, Brendan Hide wrote:
> >- it doesn't create writeable snapshots on the destination in case you want
> >to use the copy as a live filesystem
> One of the issues with doing writeable snapshots by default is that the 
> backup is (ever so slightly) less safe from "fat-finger syndrome". If I 
> want a writeable snapshot, I'll make it from the read-only snapshot, 
> thereby reducing the chances of accidentally "tainting" or deleting data 
> in the backup. I actually *did* accidentally delete my entire filesystem 
> (hence the paranoid umounts). But, of course, my script *first* created 
> read-only snapshots from which recovery took only a few minutes. ;)

The writeable snapshot I create is on top of the read only one used by btrfs
receive. So, I can play with it, but it won't upset/break anything for
the backup.

The historical snapshots I keep give me cheap backups to go back to do get a
file I may have deleted 3 days ago and want back now even though my btrfs
send/receive runs hourly.

> >Things I noticed:
> >- I don't use ionice, maybe I should. Did you find that it actually made a
> >difference with send/receive?
> 
> This is just a habit I've developed over time in all my scripts. I 
> figure that if I'm using the machine at the time and the snapshot has a 
> large churn, I'd prefer the ionice. That said, the main test system is a 
> desktop which is likely to have much less churn than a server. In the 
> last two weeks the longest daily incremental backup took about 5 minutes 
> to complete, while it typically takes about 30 seconds only.

I'll consider that if I see too much load without ionice, thanks.

> >- Your comments say shlock isn't safe and that's documented. I don't see
> >that in the man page
> >http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html
> That man page looks newer than the one I last looked at - specifically 
> the part saying "improved by Berend Reitsma to solve a race condition." 
> The previous documentation on shlock indicated it was safe for hourly 
> crons - but not in the case where a cron might be executed twice 
> simultaneously. Shlock was recommended by a colleague until I realised 
> this potential issue, thus my template doesn't use it. I should update 
> the comment with some updated information.

It's not super important, it was more my curiosity.
If a simple lock program in C isn't atomic, what's the point of it?
I never looked at the source code, but maybe I should...

> >I'd love to have details on this if I shouldn't be using it
> >- Is set -o noclobber; echo $$ > $lockfile really atomic and safer than
> >shlock? If so, great, although I would then wonder why shlock even exists 
> >:)
> The part that brings about an atomic lock is "noclobber", which sets it 
> so that we are not allowed to "clobber"/"overwrite" an existing file. 
> Thus, if the file exists, the command fails. If it successfully creates 
> the new file, the command returns true.
 
I understand how it's supposed to work, I just wondered if it was really
atomic as it should be since there would be no reason for shlock to even
exist with that line of code you wrote.

> I'd consider changing this mostly for the fact that depending on INN is 
> a very big dependency. There are other options as well, though I don't 
> think they're as portable as noclobber.

I totally agree, bringing INN to get shlock sucks a bit. I have shlock
stashed in /usr/local/bin so that I don't have to :)

Thanks for the info
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)

[Brendan Hide]

On 2014/03/22 11:11 PM, Marc MERLIN wrote:
> Please consider adding a blank line between quotes, it makes them just a bit
> more readable :)

Np.

> On Sat, Mar 22, 2014 at 11:02:24PM +0200, Brendan Hide wrote:
>>> - it doesn't create writeable snapshots on the destination in case you want
>>> to use the copy as a live filesystem
>> One of the issues with doing writeable snapshots by default is that the
>> backup is (ever so slightly) less safe from "fat-finger syndrome". If I
>> want a writeable snapshot, I'll make it from the read-only snapshot,
>> thereby reducing the chances of accidentally "tainting" or deleting data
>> in the backup. I actually *did* accidentally delete my entire filesystem
>> (hence the paranoid umounts). But, of course, my script *first* created
>> read-only snapshots from which recovery took only a few minutes. ;)
> The writeable snapshot I create is on top of the read only one used by btrfs
> receive. So, I can play with it, but it won't upset/break anything for
> the backup.
>
> The historical snapshots I keep give me cheap backups to go back to do get a
> file I may have deleted 3 days ago and want back now even though my btrfs
> send/receive runs hourly.

Ah. In that case my comment is moot. I could add support for something 
like this but I'm unlikely to use it.

> [snip]
>>> - Your comments say shlock isn't safe and that's documented. I don't see
>>> that in the man page
>>> http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html
>> That man page looks newer than the one I last looked at - specifically
>> the part saying "improved by Berend Reitsma to solve a race condition."
>> The previous documentation on shlock indicated it was safe for hourly
>> crons - but not in the case where a cron might be executed twice
>> simultaneously. Shlock was recommended by a colleague until I realised
>> this potential issue, thus my template doesn't use it. I should update
>> the comment with some updated information.
> It's not super important, it was more my curiosity.
> If a simple lock program in C isn't atomic, what's the point of it?
> I never looked at the source code, but maybe I should...

Likely the INN devs needed something outside of a shell environment. 
Based on the man page, shlock should be atomic now.

>>> I'd love to have details on this if I shouldn't be using it
>>> - Is set -o noclobber; echo $$ > $lockfile really atomic and safer than
>>> shlock? If so, great, although I would then wonder why shlock even exists
>>> :)
>> The part that brings about an atomic lock is "noclobber", which sets it
>> so that we are not allowed to "clobber"/"overwrite" an existing file.
>> Thus, if the file exists, the command fails. If it successfully creates
>> the new file, the command returns true.
>   
> I understand how it's supposed to work, I just wondered if it was really
> atomic as it should be since there would be no reason for shlock to even
> exist with that line of code you wrote.

When I originally came across the feature I wasn't sure it would work 
and did extensive testing: For example, spawn 30 000 processes, each of 
which tried to take the lock. After the machine became responsive again 
;) only 1 lock ever turned out to have succeeded.

Since then its been in production use across various scripts on hundreds 
of servers. My guess (see above) is that the INN devs couldn't or didn't 
want to use it.

The original page where I learned about noclobber: 
http://www.davidpashley.com/articles/writing-robust-shell-scripts/

> Thanks for the info Marc 

No problem - and thanks. :)

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97