Encrypted software RAID1 with Debian Stretch

Encrypted software RAID1 with Debian Stretch

[commentsabout]

Hello,

(this is a cross-post from debian-users mailing list
https://groups.google.com/d/msg/linux.debian.user/jjdr6LXaOm8/MOoVVo0lAwAJ
)

Here is a picture of what I'm trying to achieve:
https://imgur.com/a/DAM8D (the "Today" column). 

I am trying to build a home backup system. The system (Debian Stretch) 
will be on a SSD. For the time being, I only have one pair of HDDs (the 
"Today" column in the picture) ; in the future (the "Future" column), I 
would like to add other pairs of HDD to store other kind of data. 

This backup system will only be turned on when needed, I don't plan on 
using it as some sort of server or a NAS. 

We are talking about software RAID1. 

I would like everything to be encrypted (FDE), from the system (/ and 
/swap) to the RAID1 drives. 

If possible, I would like to have different encryption keys for the 
system and the various RAID1 pairs (in the "Future" column in the 
picture, one for the system, one for "work", one for "family", one for 
"misc"). So that I can give the system encryption passphrase, "family" 
and "misc" ones to my wife and keep the "work" one for myself. 

I'm a complete noob when it comes to this kind of operations so I'm
looking for a step by step ELI5 explanation (I have tried to use the
Debian graphical installer to achieve this but have failed because I was
just messing around with the options trying to figure out what to do). 

Thank you in advance for your help :) 

CA 

Re: Encrypted software RAID1 with Debian Stretch

[Wols Lists]

On 01/09/17 00:58, commentsabout@riseup.net wrote:
> Hello,
> 
> (this is a cross-post from debian-users mailing list
> https://groups.google.com/d/msg/linux.debian.user/jjdr6LXaOm8/MOoVVo0lAwAJ
> )
> 
> Here is a picture of what I'm trying to achieve:
> https://imgur.com/a/DAM8D (the "Today" column). 
> 
Have you read the raid wiki? There's a few bits there about how to
install a raid system, and/or how to update a system to raid.

Bearing in mind you're installing debian on an SSD, and *then* adding
raid that makes your life simpler, you can install the system and then
setup the raid. I personally normally think booting off a rescue disk
and using the command line is simpler than trying to use a graphical
installer, but if you've already got a working system, that's not a problem.

> I am trying to build a home backup system. The system (Debian Stretch) 
> will be on a SSD. For the time being, I only have one pair of HDDs (the 
> "Today" column in the picture) ; in the future (the "Future" column), I 
> would like to add other pairs of HDD to store other kind of data. 
> 
> This backup system will only be turned on when needed, I don't plan on 
> using it as some sort of server or a NAS. 
> 
Okay. Personal preference (and I don't do it myself, but I'd have to
rebuild my system to do it) I would use btrfs for the filesystems. Yes
it has a bad rep for its inbuilt raid, but if all you're doing is backup
snapshots it should be great. Each backup cycle consists of "take a
snapshot, do an in-place rsync", so if only 10MB of live data has
changed, the backup only uses an extra 10MB on the backup drives.

> We are talking about software RAID1. 
> 
> I would like everything to be encrypted (FDE), from the system (/ and 
> /swap) to the RAID1 drives. 
> 
> If possible, I would like to have different encryption keys for the 
> system and the various RAID1 pairs (in the "Future" column in the 
> picture, one for the system, one for "work", one for "family", one for 
> "misc"). So that I can give the system encryption passphrase, "family" 
> and "misc" ones to my wife and keep the "work" one for myself. 
> 
> I'm a complete noob when it comes to this kind of operations so I'm
> looking for a step by step ELI5 explanation (I have tried to use the
> Debian graphical installer to achieve this but have failed because I was
> just messing around with the options trying to figure out what to do). 
> 
> Thank you in advance for your help :) 
> 
Okay, two-disk raid-1 is okay. When you add drives, DON'T go for another
mirror, go to raid-6. Whether you partition the drives first, or use raw
drives, is up to you, but hand over the entire disk to your raid.

Put lvm on top of the raid. Okay, you could move partitions around, it's
not too hard, but creating three partitions for work, family, and misc
on top of lvm makes everything a lot simpler.

These three partitions are btrfs. That means *everything* can easily be
expanded as you add capacity - when you add two new drives and go raid-6
you just say "mdadm --add new drives", then you increase the array
capacity - "mdadm --resize" ? can't remember - then you expand lvm, then
you expand your filesystems.

When you've run out of slots and want to use bigger drives, it's "mdadm
--replace" instead of --add. MAKE SURE you *always* have access to a
spare SATA slot for expansion or disaster recovery! Add-in SATA cards
are cheap.

Then personally, I'd just script the backup routine so either when the
server is powered on, or requiring you to log in and do it, it

Takes the snapshots
Runs the rsync
Does a smartctl over all drives and emails you the results
Does "cat /proc/mdstat" and emails you the results
Does an "mdadm --examine" and "mdadm --display" over all drives and
arrays (one command is for drives, the other for arrays) and emails you
the results
Does the btfs equivalent of "du" and emails you the results
Anything else you can think of ...

And then shut down.

Note also, that filling up a btrfs partition can easily trash a
partition. That why the "du", and MAKE SURE you check it!!! (Note that
the "du" command itself doesn't work properly on btrfs ... :-(

Note that I have NOT put "do a scrub" in there. You really must do
regular scrubs, but they scan the entire array and are rate-throttled.
Running one could easily take days so you don't want to automate it, but
you don't want to forget it, either!

Note also, I've really only covered the raid aspect. I don't know lvm, I
don't know btrfs. I don't know LUKS. But this is exactly how I would set
up a backup server.

Cheers,
Wol

Re: Encrypted software RAID1 with Debian Stretch

[Nix]

On 1 Sep 2017, Wols Lists stated:

> On 01/09/17 00:58, commentsabout@riseup.net wrote:
>> I am trying to build a home backup system. The system (Debian Stretch) 
>> will be on a SSD. For the time being, I only have one pair of HDDs (the 
>> "Today" column in the picture) ; in the future (the "Future" column), I 
>> would like to add other pairs of HDD to store other kind of data. 
>> 
>> This backup system will only be turned on when needed, I don't plan on 
>> using it as some sort of server or a NAS. 
>> 
> Okay. Personal preference (and I don't do it myself, but I'd have to
> rebuild my system to do it) I would use btrfs for the filesystems. Yes
> it has a bad rep for its inbuilt raid, but if all you're doing is backup
> snapshots it should be great. Each backup cycle consists of "take a
> snapshot, do an in-place rsync", so if only 10MB of live data has
> changed, the backup only uses an extra 10MB on the backup drives.

This sounds like downright dangerous advice to me. Surely what matters
for backup is stability? Use something old and boring and stable. btrfs
is the very last thing you should be thinking of for this application.
After all, you'll only need the backups when things are already going
wrong: the last thing you want to find is that you've been using a
filesystem that has betrayed you at the last.

ext2 is probably too old (it's not maintained much any more), but ext4
with the default mkfs options is certainly good enough, or xfs likewise.
If you want deduplicated backups, use something that's torture-tested
for that (there are many choices: my personal preference is bup, but
there are other perfectly good options now), not a bleeding-edge
filesystem that still has data loss bugs reported fairly frequently and
data corruption bugs reported at not-especially-rare intervals.

> Note also, I've really only covered the raid aspect. I don't know lvm, I
> don't know btrfs.

I can tell. if you did, you wouldn't be recommending it for this
application. btrfs is cool and all, but it's also new, and in filesystem
land new means dangerous.

> I don't know LUKS. But this is exactly how I would set up a backup
> server.

FWIW, my current backup configuration is a pair of encrypted USB disks.

One is attached to my largest server, one is attached to an odroid at
the top of the house that does nothing else and is exported via the
network block device. Shortly after server boot, on first interactive
ssh to the server, I do a 'ykchalresp -H -2 backup' to provide the
passphrase via OATH-HOTP from my Yubikey. (I have passphrases stored in
two keyslots, matching two Yubikeys, in case one key fails.). I store
the resulting passphrase in a root-only-readable ramfs (not tmpfs: I
don't want the thing swapped out). The drives are *not* decrypted at
this stage: attackers who get root on the machine can wipe the backup
drives, but can only decrypt them if this is a targetted attack and they
know where the ramfs is.

Each backup drive's decrypted content is an ext4 fs with metadata csums
enabled: at backup time, the main backup drive (the one attached to the
server) is mounted on /mnt/backup in a separate filesystem namespace (so
it never appears mounted to normal users) and a bup index / bup save is
run. There are three classes of backup:

 - hourly. This runs every three hours and backs up all directories with
   a .backup file in them (and all subdirs): the .backup file is a bup
   exclusions file (a set of Python regexps) which can prohibit hourly
   backup of subsets of that dir. Only files under 500MiB are
   considered. I use this for home directories: they get a separate
   index, and are recorded as separate backup sets. This takes about
   two minutes a time, even though one of my home directories has
   multiple kernel source trees, GCC source trees and the like in it.
   (Lots of RAM to cache the entire FS tree is what counts here.)

   Only the server with homedirs on it runs this type of backup.

 - daily. This runs once a day and backs up everything not denoted in
   /etc/backup-exclusions and which is not a network filesystem, tmpfs,
   etc, and is not on the transient store (a fast RAID-0 array I use for
   stuff I can easily regenerate). Again, only files under 500MiB are
   considered. This takes about half an hour a time, almost entirely
   filesystem walking and index merging (bup's least efficient chunk of
   code). It runs on every machine I own, even the home cinema.

 - weekly. This runs once a week (duh) and backs up everything the daily
   backup does plus files >500MiB in size. It does a second 'bup save'
   to the second backup drive sitting upstairs, which is my offsite
   drive and is regularly swapped out to another locale. If the house
   burns down, I'll at least have weekly backups starting at the last
   time I swapped this out. If the house floods, well, that's why it's
   sitting upstairs. The nice thing about bup is that even though this
   does two backups, it only needs to do one filesystem walk :)

Every day at 5am (or, for the offsite backup, every week), we wake up
and generate par2 redundancy information for all new backup files,
because the problem with deduplication is that one bitflip can be
disastrous. Adding a bit of controlled redundancy back helps there.

The only way I could make this better, I think, is something I've
blathered about before, which is to do all backups on another machine
that does nothing else other than run my authentication services and
that is not even sshable into without special measures (a bang on the
door with a yubikey, roughly), then have that machine pull backups from
the other boxes. Hopefully this would have a smaller attack surface and
thus be harder for a sufficiently malicious attacker to break into and
wipe the drives. (Even that attacker would find it hard to wipe the
swapped-out offsite drive: it's powered off!)

-- 
NULL && (void)

Re: Encrypted software RAID1 with Debian Stretch

[Reindl Harald]

Am 13.09.2017 um 01:30 schrieb Nix:
> On 1 Sep 2017, Wols Lists stated:
>> Okay. Personal preference (and I don't do it myself, but I'd have to
>> rebuild my system to do it) I would use btrfs for the filesystems. Yes
>> it has a bad rep for its inbuilt raid, but if all you're doing is backup
>> snapshots it should be great. Each backup cycle consists of "take a
>> snapshot, do an in-place rsync", so if only 10MB of live data has
>> changed, the backup only uses an extra 10MB on the backup drives.
> 
> This sounds like downright dangerous advice to me. Surely what matters
> for backup is stability? Use something old and boring and stable. btrfs
> is the very last thing you should be thinking of for this application.
> After all, you'll only need the backups when things are already going
> wrong: the last thing you want to find is that you've been using a
> filesystem that has betrayed you at the last.

well, SuSE is using it as default for their enterprise Linux an di am 
using it on an backup-vm for two years (two 2 TB virtual disks) now 
without any problem on top of LUKS with forcec compression enabled

> ext2 is probably too old (it's not maintained much any more)

sorry, but when you are talking about betrayed filesystems above and 
then take ext2, a non journaled filesystem which takes ages for fsck, in 
your mouth....

>> Note also, I've really only covered the raid aspect. I don't know lvm, I
>> don't know btrfs.
> 
> I can tell. if you did, you wouldn't be recommending it for this
> application. btrfs is cool and all, but it's also new, and in filesystem
> land new means dangerous

with that argumentation... no i better stop comment.....

Re: Encrypted software RAID1 with Debian Stretch

[Nix]

On 13 Sep 2017, Reindl Harald uttered the following:

> Am 13.09.2017 um 01:30 schrieb Nix:
>> ext2 is probably too old (it's not maintained much any more)
>
> sorry, but when you are talking about betrayed filesystems above and
> then take ext2, a non journaled filesystem which takes ages for fsck,

On slow links like USB2, the overhead of throwing heaps of metadata into
the journal across such a slow link often dominates any overhead from
fsck, particularly if the filesystem is usually unmounted (as is often
also true of backup filesystems) so is unlikely to benefit from a
journal anyway. Of course this depends on the backup model. bup doesn't
trigger many metadata updates (a few huge files): the late lamented
obnam and rdiff-backup trigger lots.

Nonetheless, fs/ext2 is almost unmaintained these days, and bugs are
slowly creeping back in. fs/ext4 can take up its duties perfectly well
these days, reading and writing both ext4-sans-journal and traditional
ext2 filesystems perfectly well.

One person having no problems with a filesystem as new as btrfs does not
mean the filesystem is reliable enough to use for backup. The
reliability bar for such filesystems is far higher than that for fses in
daily use! (The required-feature bar is often also much lower. All they
have to do is store stuff that rarely changes and not lose it!)

> in your mouth....

Oh yes, it's Reindl, destroyer of mailing list civility everywhere. How
wonderful it isn't to find you here.

I can't even tell what 'in your mouth' *means* in this context, but I'm
fairly sure it's meant to be offensive, simply because *everything* you
say is meant that way. Are you arguing that ext2 is well-maintained,
and thus should be used, or that it is unjournalled and thus should not
be used, or that it is unjournalled and thus less reliable, or that fsck
is slow and therefore the filesystem is unreliable?

God knows, and I'll never find out... because you are going back in my
otherwise-empty killfile. However *did* you get out? (Probably it
happened after you became the first person in the history of the
spamassassin list to go into a moderation queue.)

I encourage everyone else who's had enough of Reindl to killfile him as
well. 80% of what he says is wrong, spoken in tones of great authority,
but thankfully he undermines his misinformation by being so thoroughly
unpleasant that nobody wants to follow his instructions. The real
problem is that he's just as viciously nasty to newbies seeking help,
and they won't know to killfile him...

Re: Encrypted software RAID1 with Debian Stretch

[Wols Lists]

On 13/09/17 14:52, Nix wrote:
> Nonetheless, fs/ext2 is almost unmaintained these days, and bugs are
> slowly creeping back in. fs/ext4 can take up its duties perfectly well
> these days, reading and writing both ext4-sans-journal and traditional
> ext2 filesystems perfectly well.

ext2 as filesystem code in linux no longer exists. Likewise, I believe,
ext3. Both have been dropped and deleted. The only supported/maintained
ext driver now is ext4, which has backwards compatibility with 3 and 2.

Much like XFS I gather, which although nominally a single filesystem, is
actually full of compatibility code for earlier, incompatible variants.
> 
> One person having no problems with a filesystem as new as btrfs does not
> mean the filesystem is reliable enough to use for backup. The
> reliability bar for such filesystems is far higher than that for fses in
> daily use! ("The required-feature bar is often also much lower. All they
> have to do is store stuff that rarely changes and not lose it!")

"The required-feature bar is often also much lower. All they
have to do is store stuff that rarely changes and not lose it!"

Actually, btrfs is very good at that! PROVIDED you don't use the fancy
new experimental features (like raid :-) in btrfs, it works very well.
It's a nice, stable, very decent filesystem.

The only real bug left in basic functionality is IF you combine
snapshots with a disk full, it can be a disaster. Hence my comments
about monitoring disk space! Seriously, I would be very happy to set my
system up as I suggested and, apart from the learning curve (I'm
currently still on raid-1/ext on my home system, would love to upgrade
but can't afford it), I would be only to happy to entrust my backup to
btrfs.

The rule is simple - don't abuse your tools, and btrfs - USED WITHIN ITS
LIMITATIONS - is a powerful and reliable file system.

Cheers,
Wol

Re: Encrypted software RAID1 with Debian Stretch

[Nix]

On 13 Sep 2017, Wols Lists spake thusly:

> On 13/09/17 14:52, Nix wrote:
>> Nonetheless, fs/ext2 is almost unmaintained these days, and bugs are
>> slowly creeping back in. fs/ext4 can take up its duties perfectly well
>> these days, reading and writing both ext4-sans-journal and traditional
>> ext2 filesystems perfectly well.
>
> ext2 as filesystem code in linux no longer exists. Likewise, I believe,
> ext3. Both have been dropped and deleted. The only supported/maintained
> ext driver now is ext4, which has backwards compatibility with 3 and 2.

Oh! I failed to notice when that happened :)

... er, I just checked upstream master and ext2 still exists. ext3 was
removed in v4.3.

>> One person having no problems with a filesystem as new as btrfs does not
>> mean the filesystem is reliable enough to use for backup. The
>> reliability bar for such filesystems is far higher than that for fses in
>> daily use! ("The required-feature bar is often also much lower. All they
>> have to do is store stuff that rarely changes and not lose it!")
>
> "The required-feature bar is often also much lower. All they
> have to do is store stuff that rarely changes and not lose it!"
>
> Actually, btrfs is very good at that! PROVIDED you don't use the fancy
> new experimental features (like raid :-) in btrfs, it works very well.
> It's a nice, stable, very decent filesystem.

I lost two btrfses via "it had lots of space, now it's saying -ENOSPC,
and now I can't mount it any more" bugs. Both were rapidly fixed, but
that sort of thing dents your confidence :) mind you that was a couple
of years back and these things do improve with time.

> The rule is simple - don't abuse your tools, and btrfs - USED WITHIN ITS
> LIMITATIONS - is a powerful and reliable file system.

Yeah, but... if you avoid the advanced features, why use btrfs? In
particular, why use it *for a backup medium* (where such features are
distinctly less useful than on a non-backup medium)?

-- 
NULL && (void)

Re: Encrypted software RAID1 with Debian Stretch

[Wols Lists]

On 14/09/17 12:08, Nix wrote:
>> The rule is simple - don't abuse your tools, and btrfs - USED WITHIN ITS
>> > LIMITATIONS - is a powerful and reliable file system.

> Yeah, but... if you avoid the advanced features, why use btrfs? In
> particular, why use it *for a backup medium* (where such features are
> distinctly less useful than on a non-backup medium)?

Because, if you use snapshots and an "in-place rsync" (which overwrites
the part of files which have changed, rather than replacing the file by
default), then each snapshot is a full backup, but only uses the space
of an incremental.

The OP was building a backup server, so all their live data is
elsewhere, and provided you look after your backups, this will give you
a very cheap and effective backup system.

(Add to which I know nothing about XFS and ZFS, other than ZFS needs
gobs of ram and why would you want to over-provision a server that isn't
switched on unless you're backing up.)

Cheers,
Wol

Re: Encrypted software RAID1 with Debian Stretch

[Nix]

On 14 Sep 2017, Wols Lists uttered the following:

> On 14/09/17 12:08, Nix wrote:
>>> The rule is simple - don't abuse your tools, and btrfs - USED WITHIN ITS
>>> > LIMITATIONS - is a powerful and reliable file system.
>
>> Yeah, but... if you avoid the advanced features, why use btrfs? In
>> particular, why use it *for a backup medium* (where such features are
>> distinctly less useful than on a non-backup medium)?
>
> Because, if you use snapshots and an "in-place rsync" (which overwrites
> the part of files which have changed, rather than replacing the file by
> default), then each snapshot is a full backup, but only uses the space
> of an incremental.

Ah, so it's like bup only immediately accessible without Python and FUSE
installed, and probably less reliable (but hopefully this will change.)
Except it doesn't do full deduplication (if you change a file, it gets
backed up, even if you change it to the same contents as it had before:
yeah, perhaps this is a tad contrived).

> The OP was building a backup server, so all their live data is
> elsewhere, and provided you look after your backups, this will give you
> a very cheap and effective backup system.

Backups based on not-yet-reliable filesystems are, uh, less effective.
(I've lost backups to bad filesystems before due to turning on options
that weren't ready for prime time :( )

-- 
NULL && (void)

Re: Encrypted software RAID1 with Debian Stretch

[Roman Mamedov]

On Thu, 14 Sep 2017 14:08:15 +0100
Nix <nix@esperi.org.uk> wrote:

> Backups based on not-yet-reliable filesystems are, uh, less effective.

You're just grasping for straws at this point. Single-device Btrfs and
its core features like snapshotting and compression are very solid, it is
silly trying to "cast doubt" on those. Thousands of people use Btrfs for their
primary storage, it is certainly more than stable enough to be used for
backups (which by definition are just a copy of stuff also existing elsewhere).

And yeah rsync+Btrfs snapshotting are an amazing combination, it works really
well for backups, and the easiness of the workflow to access older versions of
a file or the whole dir tree is unsurpassed by anything imaginable.

-- 
With respect,
Roman

Re: Encrypted software RAID1 with Debian Stretch

[Nix]

On 14 Sep 2017, Roman Mamedov said:

> On Thu, 14 Sep 2017 14:08:15 +0100
> Nix <nix@esperi.org.uk> wrote:
>
>> Backups based on not-yet-reliable filesystems are, uh, less effective.
>
> You're just grasping for straws at this point. Single-device Btrfs and
> its core features like snapshotting and compression are very solid, it is
> silly trying to "cast doubt" on those.

I'm not grasping for straws, but I do think I'm a lot more conservative
than you where backups are concerned!

I've experienced total data loss on btrfs owing to -ENOSPC with
outstanding snapshots (saved by... backups! On a separate filesystem, of
course). It *was* two years ago, but backup-medium filesystems are
long-lived: two years isn't actually that long: some of mine are fifteen
years old. "Haven't lost data in the last two years" is not really good
enough for a backup filesystem in my view. Snapshotting was not solid
then: though it might be now, a mere two years' experience is not enough
to be sure enough of it for backups, at least not for me. (It's only
recently I moved off unjournalled ext2 for my backups, when the
five-year mark since I experienced data loss on ext4 passed -- and I was
unsure whether five years was too soon.)

>                                        Thousands of people use Btrfs for their
> primary storage, it is certainly more than stable enough to be used for
> backups (which by definition are just a copy of stuff also existing elsewhere).

Another way of putting it is that by the time you need backups they are
probably your *only copy* of your data and you have probably just
experienced some sort of data-loss disaster. It *is* certain that every
occurrence of such a disaster *will* be followed by a need for your
backups. They have to be *more* reliable than your primary filesystem,
not *less* reliable.

> And yeah rsync+Btrfs snapshotting are an amazing combination, it works really
> well for backups, and the easiness of the workflow to access older versions of
> a file or the whole dir tree is unsurpassed by anything imaginable.

I dunno, I find 'bup fuse mnt; cd mnt/backup/latest/blah/blah' to be
more or less the same, workflow-wise. (It just has more dependencies, to
wit FUSE.)

I would never consider snapshots on the same filesystem to be a backup
of anything, except possibly as a defence against 'oh whoops I rm'ed the
wrong tree'. If you btrfs send them somewhere else, perhaps... and of
course btrfs *does* make that easy, and you can store the result on any
filesyste you like, including via deduplicating backup programs that
don't rely on btrfs :) now *that* is a tempting model. btrfs send | bup split,
hmmm :)

... though of course if you do that it's useful only as a mass restore
system: you can't use it for individual-file restoration. However, bup
being what it is, you can combine a btrfs send | bup split with a normal
bup index/bup save at rarer intervals and get deduplication of the whole
lot, with restores of individual files at 'bup save' frequency (where
the backups pay the cost of a 'bup index' filesystem walk), and far
higher-frequency backups that are suitable for full image restores but
do not pay the cost of any sort of filesystem walk at all and can be
done every half hour if you wanted to :) with full deduplication against
all the other backups.

-- 
NULL && (void)

Re: Encrypted software RAID1 with Debian Stretch

[Roman Mamedov]

On Thu, 14 Sep 2017 16:02:31 +0100
Nix <nix@esperi.org.uk> wrote:

> I would never consider snapshots on the same filesystem to be a backup
> of anything, except possibly as a defence against 'oh whoops I rm'ed the
> wrong tree'.

No one proposed that. The scenario is that the backup server would have its
rsync destination dir (from multiple other systems) periodically snapshotted,
providing a historic view of what it contained 1-2-3 months ago. Just like
"bup" does in userspace, I guess? -- or like rdiff-backup did, which I used
before. But now we have that directly in filesystem, no need to cling to
userspace crutches anymore.

-- 
With respect,
Roman

Re: Encrypted software RAID1 with Debian Stretch

[Wols Lists]

On 14/09/17 14:08, Nix wrote:
> Ah, so it's like bup only immediately accessible without Python and FUSE
> installed, and probably less reliable (but hopefully this will change.)
> Except it doesn't do full deduplication (if you change a file, it gets
> backed up, even if you change it to the same contents as it had before:
> yeah, perhaps this is a tad contrived).

Actually, yes it does ... at least the way I suggested.

The whole point of an in-place rsync is it does a comparison, and only
writes the data IF IT CHANGES. So a "change it to itself" change will be
detected and ignored by rsync (okay, it may well update the inode, but
then that HAS changed :-)

Cheers,
Wol

Re: Encrypted software RAID1 with Debian Stretch

[Reindl Harald]

Am 14.09.2017 um 17:02 schrieb Nix:
> On 14 Sep 2017, Roman Mamedov said:
> 
>> On Thu, 14 Sep 2017 14:08:15 +0100
>> Nix <nix@esperi.org.uk> wrote:
>>
>>> Backups based on not-yet-reliable filesystems are, uh, less effective.
>>
>> You're just grasping for straws at this point. Single-device Btrfs and
>> its core features like snapshotting and compression are very solid, it is
>> silly trying to "cast doubt" on those.
> 
> I'm not grasping for straws, but I do think I'm a lot more conservative
> than you where backups are concerned!

depends also on the type of backup, normally you have more than once anyways

* microserver with NFS, dedicated to run a vmware backup-appliance
* that beast holding 31 days of anything is backuped
   once per month on a external disk, there are two identical
   external disks, one is always located off-house and both
   are containing the *supsended* backup appliance inclduding memory

so, and the btrfs backup machine is located on the other side of the 
town, does dauly backups and rnshshot for 7 days and is reachable via 
ssh/sftp and key-auth - that one is mostly for "i have deleted a single 
file or folder can you restore it"

why BTRFS?
because of the compression!

the other stuuf above is 1 time XFS and the external disks ext4

*that* is how a backup plan looks like because when you lose your 
complete backup because one damaged filesystem you have no backup plan 
at all (take it offeding if you want to do so...)

Re: Encrypted software RAID1 with Debian Stretch

[Nix]

On 14 Sep 2017, Roman Mamedov said:

> On Thu, 14 Sep 2017 16:02:31 +0100
> Nix <nix@esperi.org.uk> wrote:
>
>> I would never consider snapshots on the same filesystem to be a backup
>> of anything, except possibly as a defence against 'oh whoops I rm'ed the
>> wrong tree'.
>
> No one proposed that.

Oh I thought that was what people were proposing as the compelling btrfs
advantage.

>                       The scenario is that the backup server would have its
> rsync destination dir (from multiple other systems) periodically snapshotted,
> providing a historic view of what it contained 1-2-3 months ago. Just like
> "bup" does in userspace, I guess? -- or like rdiff-backup did, which I used
> before. But now we have that directly in filesystem, no need to cling to
> userspace crutches anymore.

I'm fairly sure that can't deduplicate anywhere near as effectively. No
dedup within files; no dedup across trees (very important if you have
duplicated data on multipl systems you are backing up); no dedup
anywhere except in immediate history. That's all rdiff-backup could do,
but the state of the art is better now.

-- 
NULL && (void)

Re: Encrypted software RAID1 with Debian Stretch

[Nix]

On 14 Sep 2017, Wols Lists spake thusly:

> On 14/09/17 14:08, Nix wrote:
>> Ah, so it's like bup only immediately accessible without Python and FUSE
>> installed, and probably less reliable (but hopefully this will change.)
>> Except it doesn't do full deduplication (if you change a file, it gets
>> backed up, even if you change it to the same contents as it had before:
>> yeah, perhaps this is a tad contrived).
>
> Actually, yes it does ... at least the way I suggested.
>
> The whole point of an in-place rsync is it does a comparison, and only
> writes the data IF IT CHANGES. So a "change it to itself" change will be
> detected and ignored by rsync (okay, it may well update the inode, but
> then that HAS changed :-)

That's not full deduplication. It can't even deduplicate the results of
a "cp -a", let alone changes of smoething from A to B and back to A
again, or a change of something to something else present elsewhere in
history, or a change of something to something in part containing
something seen elsewhere in history. The rsync rolling-hash algorithm is
the key to spotting all of these, but just because the algorithm can
chunk things properly doesn't mean you're going to spot duplicates
across time unless you try. And rsync (by design) doesn't do anything
like that.

... anyway, this has nothing to do with RAID anymore: I dragged things
so far off topic you need a telescope to see the topic now. Sorry about
that.

-- 
NULL && (void)