Announcement: buttersink - like rsync for btrfs snapshots

Announcement: buttersink - like rsync for btrfs snapshots

[Ames Cornish]

I've written a utility to help me with using btrfs send and receive for 
backups or other synchronization, and I'd love to get feedback on it.  

As of this release, buttersink will synchronize a set of read-only snapshots 
in a btrfs filesystem to an Amazon S3 bucket, and vice-versa.  It 
intelligently picks parent snapshots to "diff" from, so that a minimal amount 
of data needs to be sent over the wire and stored in the backend.  

The utility is on PyPi as "buttersink", and the GitHub page is here: 
  https://github.com/AmesCornish/buttersink.  

Thanks in advance for any feedback!

- Ames

Re: Announcement: buttersink - like rsync for btrfs snapshots

[Jim Salter]

How has it been for reliability?

I wrote a btrsync app a while back, and the app /itself/ worked fine, 
but the btrfs send / btrfs receive itself proved problematic.  Since 
btrfs would keep a partial receive - with no easy way to tell whether a 
receive WAS partial or full - I would inevitably end up with interrupted 
sends causing a problem that couldn't be resolved without manually 
deleting snapshots on the target end haphazardly until I nailed the 
incomplete one.

On 08/11/2014 01:49 PM, Ames Cornish wrote:
> I've written a utility to help me with using btrfs send and receive for
> backups or other synchronization, and I'd love to get feedback on it.
>
> As of this release, buttersink will synchronize a set of read-only snapshots
> in a btrfs filesystem to an Amazon S3 bucket, and vice-versa.  It
> intelligently picks parent snapshots to "diff" from, so that a minimal amount
> of data needs to be sent over the wire and stored in the backend.
>
> The utility is on PyPi as "buttersink", and the GitHub page is here:
>    https://github.com/AmesCornish/buttersink.
>
> Thanks in advance for any feedback!
>
> - Ames
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Announcement: buttersink - like rsync for btrfs snapshots

[Jim Salter]

To any core btrfs devs who are listening and care - the unreliability of 
btrfs send/receive is IMO the single biggest roadblock to adoption of 
btrfs as a serious next-gen FS.

I can live with occasional corner-case performance issues, I can even 
live with (very) occasional filesystem corruption... IF I can rely on 
replication to keep my data safe on another box.  Without the 
replication, there's just no reasonable case to be made to replace ZFS.

On 08/11/2014 02:05 PM, Ames Cornish wrote:
> Jim,
>
> btrfs send reliability has been an issue, though I've been able to 
> successfully use it for my backups.  buttersink usually detects the 
> errors and will either move the destination snapshot to mark it as 
> partial/failed (for btrfs), or cancel and delete the partial upload 
> (for S3).  I've also found that it helps to wait a while (e.g. 30 
> seconds) after any volume deletes before trying the send/sync.  I hope 
> btrfs-progs will get more reliable, too.
>
> - Ames

Re: Announcement: buttersink - like rsync for btrfs snapshots

[Clemens Eisserer]

Hi Jim,

> To any core btrfs devs who are listening and care - the unreliability of
> btrfs send/receive is IMO the single biggest roadblock to adoption of btrfs
> as a serious next-gen FS.

For you it are send/receive deficiencies, however there are many other
feature/enhancement requests having top priority for other users.

Fact is, just crying for improvements won't make things better - if
you want a feature/enhancement done, you have several options:
- Implement/fix it yourself
- Go to your distribution provider with whom you have a support
contract and make him implement what you would like to see
- Sponsor somebody yourself to get it sone.

> with (very) occasional filesystem corruption... IF I can rely on replication
> to keep my data safe on another box.  Without the replication, there's just
> no reasonable case to be made to replace ZFS.

... and now the usual ZFS comparison to make devs jealous ;)

Best regards, Clemens

Re: Announcement: buttersink - like rsync for btrfs snapshots

[Ames Cornish]

Jim,

btrfs send reliability has been an issue, though I've been able to
successfully use it for my backups.  buttersink usually detects the
errors and will either move the destination snapshot to mark it as
partial/failed (for btrfs), or cancel and delete the partial upload
(for S3).  I've also found that it helps to wait a while (e.g. 30
seconds) after any volume deletes before trying the send/sync.  I hope
btrfs-progs will get more reliable, too.

- Ames

Re: Announcement: buttersink - like rsync for btrfs snapshots

[Shriramana Sharma]

https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg36293.html
says ButterSink will detect errors with backups. Anyone else except
the author tried this?

I'd like to know if any experienced person (other than the author,
with all due respect) has tried this backup utility and would like
their review...

-- 
Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा