Home storage with btrfs

Home storage with btrfs

[Hérikz Nawarro]

Hello everyone,

Today is safe to use btrfs for home storage? No raid, just secure
storage for some files and create snapshots from it.

Thanks.

Re: Home storage with btrfs

[Juan Orti Alcaine]

2017-03-13 12:29 GMT+01:00 Hérikz Nawarro <herikz.nawarro@gmail.com>:
> Hello everyone,
>
> Today is safe to use btrfs for home storage? No raid, just secure
> storage for some files and create snapshots from it.
>

In my humble opinion, yes. I'm running a RAID1 btrfs at home for 5
years and I feel the most serious bugs have been fixed, because in the
last two years I have not experienced any issue.

Anyway, keeping your kernel and btrfs-progs updated is a must, and of
course, having good backups. I'm using Fedora and it's fine.

Kind regards.

Re: Home storage with btrfs

[Austin S. Hemmelgarn]

On 2017-03-13 07:52, Juan Orti Alcaine wrote:
> 2017-03-13 12:29 GMT+01:00 Hérikz Nawarro <herikz.nawarro@gmail.com>:
>> Hello everyone,
>>
>> Today is safe to use btrfs for home storage? No raid, just secure
>> storage for some files and create snapshots from it.
>>
>
> In my humble opinion, yes. I'm running a RAID1 btrfs at home for 5
> years and I feel the most serious bugs have been fixed, because in the
> last two years I have not experienced any issue.
In general, I'd agree.  I've not seen any issues resulting from BTRFS 
itself for the past 2.5 years (although it's helped me find quite a lot 
of marginal or failing hardware over that time), but I've also not used 
many of the less stable features (raid56, qgroups, and a handful of 
other things).

One piece of advice I will give though, try to keep the total number of 
snapshots to a reasonably small three digit number (ideally less than 
200, absolutely less than 300), otherwise performance is going to be 
horrible.
>
> Anyway, keeping your kernel and btrfs-progs updated is a must, and of
> course, having good backups. I'm using Fedora and it's fine.
Also agreed, Fedora is one of the best options for a traditional distro 
(they're very good about staying up to date and back-porting bug-fixes 
from the upstream kernel).  The other two I'd recommend are Arch (they 
actually use an almost upstream kernel and are generally the first 
distro to have new versions of any arbitrary software) and Gentoo 
(similar to Arch, but more maintenance intensive (although also more 
efficient (usually))).

Re: Home storage with btrfs

[Peter Becker]

I can confirm this. I have also no generell issues since the past 2
years with BTRFS in RAID1 and 6 Disks with different sizes and also no
issues with the DUP profile on a single disk.
Only some performance issues with deduplication and very large files.
But i also recommand to use a newer kernel (4.4 or higher) or better
the newest and build a newer version of btrfs progs form source.
I use Ubuntu 16.04 and kernel 4.9 + btrfs progs 4.9 currently.

2017-03-13 13:02 GMT+01:00 Austin S. Hemmelgarn <ahferroin7@gmail.com>:
> On 2017-03-13 07:52, Juan Orti Alcaine wrote:
>>
>> 2017-03-13 12:29 GMT+01:00 Hérikz Nawarro <herikz.nawarro@gmail.com>:
>>>
>>> Hello everyone,
>>>
>>> Today is safe to use btrfs for home storage? No raid, just secure
>>> storage for some files and create snapshots from it.
>>>
>>
>> In my humble opinion, yes. I'm running a RAID1 btrfs at home for 5
>> years and I feel the most serious bugs have been fixed, because in the
>> last two years I have not experienced any issue.
>
> In general, I'd agree.  I've not seen any issues resulting from BTRFS itself
> for the past 2.5 years (although it's helped me find quite a lot of marginal
> or failing hardware over that time), but I've also not used many of the less
> stable features (raid56, qgroups, and a handful of other things).
>
> One piece of advice I will give though, try to keep the total number of
> snapshots to a reasonably small three digit number (ideally less than 200,
> absolutely less than 300), otherwise performance is going to be horrible.
>>
>>
>> Anyway, keeping your kernel and btrfs-progs updated is a must, and of
>> course, having good backups. I'm using Fedora and it's fine.
>
> Also agreed, Fedora is one of the best options for a traditional distro
> (they're very good about staying up to date and back-porting bug-fixes from
> the upstream kernel).  The other two I'd recommend are Arch (they actually
> use an almost upstream kernel and are generally the first distro to have new
> versions of any arbitrary software) and Gentoo (similar to Arch, but more
> maintenance intensive (although also more efficient (usually))).
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Home storage with btrfs

[waxhead]

Same here, Have been using BTRFS for a 'scratch' disk since about 2014.
The disk have had quite some abuse and no issues yet.
I don't use compression, snapshots or any fancy features.
I have recently moved all of the root filesystem to BTRFS with 5x SSD 
disks set up in RAID1 and everything is (still) working fine, and I have 
been shuffling large amounts of data on this volume. I bet the SSD's 
will break before BTRFS does, so the real test is yet to come I guess...
I am on Debian GNU/Linux with kernel 4.9.0-2-amd64 (Debian 4.9.13-1) - 
btrfs-progs 4.7.3

However, keep in mind that backups is winning the fight against binary 
related traumas :)

Peter Becker wrote:
> I can confirm this. I have also no generell issues since the past 2
> years with BTRFS in RAID1 and 6 Disks with different sizes and also no
> issues with the DUP profile on a single disk.
> Only some performance issues with deduplication and very large files.
> But i also recommand to use a newer kernel (4.4 or higher) or better
> the newest and build a newer version of btrfs progs form source.
> I use Ubuntu 16.04 and kernel 4.9 + btrfs progs 4.9 currently.
>
> 2017-03-13 13:02 GMT+01:00 Austin S. Hemmelgarn <ahferroin7@gmail.com>:
>> On 2017-03-13 07:52, Juan Orti Alcaine wrote:
>>> 2017-03-13 12:29 GMT+01:00 Hérikz Nawarro <herikz.nawarro@gmail.com>:
>>>> Hello everyone,
>>>>
>>>> Today is safe to use btrfs for home storage? No raid, just secure
>>>> storage for some files and create snapshots from it.
>>>>
>>> In my humble opinion, yes. I'm running a RAID1 btrfs at home for 5
>>> years and I feel the most serious bugs have been fixed, because in the
>>> last two years I have not experienced any issue.
>> In general, I'd agree.  I've not seen any issues resulting from BTRFS itself
>> for the past 2.5 years (although it's helped me find quite a lot of marginal
>> or failing hardware over that time), but I've also not used many of the less
>> stable features (raid56, qgroups, and a handful of other things).
>>
>> One piece of advice I will give though, try to keep the total number of
>> snapshots to a reasonably small three digit number (ideally less than 200,
>> absolutely less than 300), otherwise performance is going to be horrible.
>>>
>>> Anyway, keeping your kernel and btrfs-progs updated is a must, and of
>>> course, having good backups. I'm using Fedora and it's fine.
>> Also agreed, Fedora is one of the best options for a traditional distro
>> (they're very good about staying up to date and back-porting bug-fixes from
>> the upstream kernel).  The other two I'd recommend are Arch (they actually
>> use an almost upstream kernel and are generally the first distro to have new
>> versions of any arbitrary software) and Gentoo (similar to Arch, but more
>> maintenance intensive (although also more efficient (usually))).
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Home storage with btrfs

[Duncan]

Hérikz Nawarro posted on Mon, 13 Mar 2017 08:29:32 -0300 as excerpted:

> Today is safe to use btrfs for home storage? No raid, just secure
> storage for some files and create snapshots from it.


I'll echo the others... but with emphasis on a few caveats the others 
mentioned but didn't give the emphasis I thought they deserved:

1) Btrfs is, as I repeatedly put it in post after post, "stabilizing, but 
not yet fully stable and mature."  In general, that means it's likely to 
work quite or even very well for you (as it has done for us) if you don't 
try the too unusual or get too cocky, but get too close to the edge and 
you just might find yourself over that edge.  Don't worry too much, tho, 
those edges are clearly marked if you're paying attention, and just by 
asking here, you're already paying way more attention than too many we 
see here... /after/ they've found themselves over the edge.  That's a 
_very_ good sign. =:^)

2) "Stabilizing, not fully stable and mature", means even more than ever, 
if you value your data more than the time, hassle and resources necessary 
to have backups, you HAVE them, tested and available for practical use 
should it be necessary.

Of course any sysadmin (and that's what you are for at least your own 
systems if you're making this choice) worth the name will tell you the 
value of the data is really defined by the number of backups it has, not 
by any arbitrary claims to value absent those backups.  No backups, you 
simply didn't value the data enough to have them, whatever claims of 
value you might otherwise try to make.  Backups, you /did/ value the data.

And of course the corollary to that first sysadmin's rule of backups is 
that an untested as restorable backup isn't yet a backup, only a 
potential backup, because the job isn't finished and it can't be properly 
called a backup until you know you can restore from it if necessary.

And lest anyone get the wrong idea, a snapshot is /not/ a backup for 
purposes of the above rules.  It's on the same filesystem and hardware 
media and if that goes down... you've lost it just the same.  And since 
that filesystem is still stabilizing, you really must be even more 
prepared for it to go down, even if the chances are still quite good it 
won't.

3) "Stabilizing, not fully stable and mature", also means that since the 
current best-practices code is still a moving target, you better be 
prepared to move with it.  The list-recommended kernels are the latest 
two releases of either the current or (mainline) LTS kernel series.  On 
the current track, 4.10 is out, so 4.10 and 4.9 are supported.  If you're 
still on 4.8 or earlier and can't point to a very specific known reason, 
you're behind.  On the LTS track, 4.9 is the latest LTS kernel as well, 
with 4.4 the one before that.  4.1's the one before that but that's a 
very long time ago in btrfs-development time, and while we'll generally 
still /try/ to help, honestly, our memory and thus our effectiveness at 
trying to help are going to be down dramatically from that of the 
recommended series.

If you prefer longer term "enterprise" or just Debian-stable distro 
support, fine, but honestly, the sort of stability they target doesn't 
have much in common with a still stabilizing btrfs, and the chances are 
/extremely/ high that either one or the other isn't a good match for your 
needs.   Either you want/need a more leading edge aka current distro 
which btrfs as still stabilizing fits in well with, or you want/need the 
stability of those longer term releases, and btrfs as still very actively 
stabilizing sticks out like a sore thumb and you're very likely to be 
better off on something that's actually considered stable, ext4 or xfs, 
perhaps, or my longer term stability favorite, reiserfs (which tends to 
be so stable in part because there's nobody screwing with it and messing 
things up, any more, reference the period when the mainline kernel devs 
switched the otherwise quite stable ext3 to the rather less stable 
data=writeback mode, for instance).

4) Keep the number of snapshots per subvolume under tight control as 
already suggested.  A few hundred, NOT a few thousand.  Easy enough if 
you do those snapshots manually, but easy enough to get thousands if 
you're not paying attention to thin out the old ones and using an 
automated tool such as snapper.

5) Stay away from quotas.  Either you need the feature and thus need a 
more mature filesystem where it's actually stable and does what it says 
on the label, or you don't, in which case you'll save yourself a /lot/ of 
headaches keeping them off.  Maybe someday...

6) Stay away from raid56 mode.  It has known problems ATM and is simply 
not ready.

FWIW, single-device and raid1 mode are the best tested and most reliable 
(within the single-device limitations for it, of course).  But even raid1 
mode has some caveats about rebuilding that it might be wise to 
familiarize yourself with /before/ they happen, if you're going to be 
using it, of course.

7) Keep your filesystem sizes manageable.  If it's going to take a week 
to btrfs check --repair, it's likely to be faster restoring from those 
backups (or simply doing a fresh mkfs and eating the lost data if it 
wasn't worth keeping backups to restore from).

I'm definitely on the low end here, but I have multiple independent btrfs 
on (partitioned) SSD, with none of the filesystems over 50 GiB.  For 
most, a more realistic size might be several hundred GiB to a TiB.  Above 
that, particularly since above that's likely to the the far slower 
spinning rust as opposed to the SSDs I'm using.  Yes, people use multi-TB 
btrfs, and yes, it works for many of them, but yes too, I've seen people 
posting maintenance times in the weeks and higher.  Is that really 
practical for you?  Maybe; it's definitely NOT for me, tho.  (FWIW, full 
balances, scrubs, checks, etc, on my btrfs', tend to take under a minute 
per filesystem.  I remember hours for md-raid repair back in the day, and 
it was acceptable then, but SSDs have spoiled me.  But days... even back 
then I think I'd have perhaps done it once, but after that, finding 
something or some way more efficient would have definitely been top of my 
priority list!)

Here is where quotas and thousands of snapshots will eat you alive, as 
simply put, neither one scales out very well.  But disabling quotas and 
keeping snapshots to 2-300 per subvolume will help *dramatically*.

8) If your use-case involves multi-gig VMs or DBs, consider nocow for 
them.  But do your research as nocow has some side-effects that make 
choosing nocow less straightforward than you might expect.

9) You may want to consider using the autodefrag mount option from the 
start, so the very first things you copy onto the filesystem are done 
with autodefrag.  In limited circumstances it may not be the right 
choice, but for general-purpose use, I'd recommend it.

10) Spend some time familiarizing yourself with the btrfs fi show/df/
usage commands.  They reveal a lot about the filesystem if you know how 
to read them, and posting their output is one of the first orders of 
business on any trouble report here.  (Note that btrfs fi show is perhaps 
the least informative of the three on its own, but is the only one that 
can be run on an unmounted filesystem.  It and btrfs fi df (not normal 
df, which is a bit quirky on btrfs) together provide about the same 
information as btrfs fi usage does, so usage is the preferred report if 
the filesystem is mountable.)

Similarly of course for the subvolume/snapshot report commands if you use 
those features.  My btrfs use-case doesn't (tho the feature is considered 
stable enough I certainly would if needed, just keep the numbers 
reasonable), so I don't personally know much about them.

And if you've not already discovered it, https://btrfs.wiki.kernel.org  
Spend some time reading up. =:^)

That's the whirlwind tour.  If it hasn't scared you off, welcome aboard! 
=:^)  If it has, that's fine; you can check back on btrfs in say five 
years, when it should have matured quite a bit. Meanwhile, there's no 
harm and for some quite some wisdom in sticking with something more 
mature for the time being. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Home storage with btrfs

[Kai Krakow]

Am Wed, 15 Mar 2017 07:55:51 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> Hérikz Nawarro posted on Mon, 13 Mar 2017 08:29:32 -0300 as excerpted:
> 
> > Today is safe to use btrfs for home storage? No raid, just secure
> > storage for some files and create snapshots from it.  
> 
> 
> I'll echo the others... but with emphasis on a few caveats the others 
> mentioned but didn't give the emphasis I thought they deserved:
> 
> 1) Btrfs is, as I repeatedly put it in post after post, "stabilizing,
> but not yet fully stable and mature."  In general, that means it's
> likely to work quite or even very well for you (as it has done for
> us) if you don't try the too unusual or get too cocky, but get too
> close to the edge and you just might find yourself over that edge.
> Don't worry too much, tho, those edges are clearly marked if you're
> paying attention, and just by asking here, you're already paying way
> more attention than too many we see here... /after/ they've found
> themselves over the edge.  That's a _very_ good sign. =:^)

Well, bugs can hit you with every filesystem. Nothing as complex as a
file system can ever be proven bug free (except FAT maybe). But as a
general-purpose-no-fancy-features-needed FS, btrfs should be on par
with other FS these days.

> 2) "Stabilizing, not fully stable and mature", means even more than
> ever, if you value your data more than the time, hassle and resources
> necessary to have backups, you HAVE them, tested and available for
> practical use should it be necessary.

This is totally not dependent on "stabilizing, not fully stable and
mature". If you data matters to you, do backups. It's that simple. If
you don't do backups, your data isn't important - by definition.

> Of course any sysadmin (and that's what you are for at least your own 
> systems if you're making this choice) worth the name will tell you
> the value of the data is really defined by the number of backups it
> has, not by any arbitrary claims to value absent those backups.  No
> backups, you simply didn't value the data enough to have them,
> whatever claims of value you might otherwise try to make.  Backups,
> you /did/ value the data.

Yes. :-)

> And of course the corollary to that first sysadmin's rule of backups
> is that an untested as restorable backup isn't yet a backup, only a 
> potential backup, because the job isn't finished and it can't be
> properly called a backup until you know you can restore from it if
> necessary.

Even more true. :-)

> And lest anyone get the wrong idea, a snapshot is /not/ a backup for 
> purposes of the above rules.  It's on the same filesystem and
> hardware media and if that goes down... you've lost it just the
> same.  And since that filesystem is still stabilizing, you really
> must be even more prepared for it to go down, even if the chances are
> still quite good it won't.

A good backup should follow the 3-2-1 rule: Have 3 different backup
copies, 2 different media, and store at least 1 copy external/off-site.

For customers, we usually deploy a strategy like this for Windows
machines: Do one local backup using Windows Image Backup to a local
NAS to backup from inside the VM, use a different software to do image
backups from outside of the VM to the local NAS, mirror the "outside
image" to a remote location (cloud storage). And keep some backup
history. Overwriting the one existing backup with a new one won't help
you anything. All involved software should be able to do efficient
delta backups, otherwise mirroring offsite may be no fun.

In linux, I'm using borgbackup and rsync to have something similar.
Using borgbackup to a local storage, and syncing it offsite with rsync
gives me the 2-1 rule part. You can get the third rule by using rsync
to also mirror the local FS off the machine. But that's usually
overkill for personal backups. Instead, I only have a third copy of
most valuable data like photos, dev stuff, documents, etc.

BTW: For me, different media also means different FS types. So a bug in
one FS wouldn't easily hit the other.

[snip]

> 4) Keep the number of snapshots per subvolume under tight control as 
> already suggested.  A few hundred, NOT a few thousand.  Easy enough
> if you do those snapshots manually, but easy enough to get thousands
> if you're not paying attention to thin out the old ones and using an 
> automated tool such as snapper.

Borgbackup is so fast and storage efficient that you could run it easily
multiple times per day. That in turn means I don't need to rely on
regular snapshots to undo mistakes. I only use snapshots before doing
some knowingly risky stuff to have fast recovery. But that's all,
nothing else should snapshots before (except you are doing more
advanced stuff like container cloning, VM instance spawning, ...).

> 5) Stay away from quotas.  Either you need the feature and thus need
> a more mature filesystem where it's actually stable and does what it
> says on the label, or you don't, in which case you'll save yourself
> a /lot/ of headaches keeping them off.  Maybe someday...

Meanwhile, one could work with underprovisioned LVM and create multiple
small btrfs on it, and expand when needed. Tho, personally I'd stick
with one big btrfs as btrfs still has issues when running out of space
with no easy way out.

In case, one such a split FS breaks down, your backups won't fit
anyways usually: When you restore only one volume back into your
system, it may not fit the rest of the system because packages on the
other volumes may be at another point in time than what you are
currently restoring.

So, I'd recommend to partition only for the sake of keeping root,
var/log and home separated. But it may come with a performance penalty
if all partitions share the same media, at least when media == HDD.

Better use a second partition for a small rescue system with the most
important tools (and keep that updated).

[snip]

> FWIW, single-device and raid1 mode are the best tested and most
> reliable (within the single-device limitations for it, of course).
> But even raid1 mode has some caveats about rebuilding that it might
> be wise to familiarize yourself with /before/ they happen, if you're
> going to be using it, of course.

Using 3-device btrfs mraid1 draid0 here on top of bcache, usually
without problems. Those problems I had are not related to btrfs raid
code I guess...

> 7) Keep your filesystem sizes manageable.  If it's going to take a
> week to btrfs check --repair, it's likely to be faster restoring from
> those backups (or simply doing a fresh mkfs and eating the lost data
> if it wasn't worth keeping backups to restore from).

Don't fix it, restore it, except you want to support the devs. A proper
FS shouldn't need a repair tool, a proper FS should stay in order and
fix smallish problems online and by itself. Fsck is not there to
recover data, that is what backups are for.

I think this is why btrfs was lacking a repair tool for so long: They
wanted to do it right. ;-)

> I'm definitely on the low end here, but I have multiple independent
> btrfs on (partitioned) SSD, with none of the filesystems over 50
> GiB.  For most, a more realistic size might be several hundred GiB to
> a TiB.  Above that, particularly since above that's likely to the the
> far slower spinning rust as opposed to the SSDs I'm using.  Yes,
> people use multi-TB btrfs, and yes, it works for many of them, but
> yes too, I've seen people posting maintenance times in the weeks and
> higher.  Is that really practical for you?  Maybe; it's definitely
> NOT for me, tho.  (FWIW, full balances, scrubs, checks, etc, on my
> btrfs', tend to take under a minute per filesystem.  I remember hours
> for md-raid repair back in the day, and it was acceptable then, but
> SSDs have spoiled me.  But days... even back then I think I'd have
> perhaps done it once, but after that, finding something or some way
> more efficient would have definitely been top of my priority list!)

Running on 3x 1TB here with 500GB SSD bcache. The HDDs are 1.8TB full.
A complete first time backup with borgbackup takes almost 24 hours,
successive runs take 15-20 minutes. A full restore takes almost 48
hours. I guess a btrfs repair could run longer (and probably not fix
most problems that occurred when I really need my data). So, restore
from backup is the first choice. It also gets you back a clean FS. The
repair tool may miss some corner cases and you still have corruptions
without knowing, maybe even overwriting good files in the backups with
broken files from such a repaired filesystem.

> 8) If your use-case involves multi-gig VMs or DBs, consider nocow for 
> them.  But do your research as nocow has some side-effects that make 
> choosing nocow less straightforward than you might expect.

For VMs I really recommend nocow as btrfs has bugs for me otherwise
(tho, I think this has been fixed according to one poster here).

> 9) You may want to consider using the autodefrag mount option from
> the start, so the very first things you copy onto the filesystem are
> done with autodefrag.  In limited circumstances it may not be the
> right choice, but for general-purpose use, I'd recommend it.

I can second this. Even on SSD, this helps.

[snip]

-- 
Regards,
Kai

Replies to list-only preferred.

Re: Home storage with btrfs

[Kai Krakow]

Am Wed, 15 Mar 2017 23:26:32 +0100
schrieb Kai Krakow <hurikhan77@gmail.com>:

> Well, bugs can hit you with every filesystem. Nothing as complex as a

Meh... I fooled myself. Find the mistake... ;-)

SPOILER:

"Nothing" should be "something".

-- 
Regards,
Kai

Replies to list-only preferred.

Re: Home storage with btrfs

[Kai Krakow]

Am Wed, 15 Mar 2017 23:41:41 +0100
schrieb Kai Krakow <hurikhan77@gmail.com>:

> Am Wed, 15 Mar 2017 23:26:32 +0100
> schrieb Kai Krakow <hurikhan77@gmail.com>:
> 
> > Well, bugs can hit you with every filesystem. Nothing as complex as
> > a  
> 
> Meh... I fooled myself. Find the mistake... ;-)
> 
> SPOILER:
> 
> "Nothing" should be "something".

*doublefacepalm*

Please forget what I wrote. The original sentence is correct.

I should get some coffee or go to bed. :-\

-- 
Regards,
Kai

Replies to list-only preferred.

Re: Home storage with btrfs

[Luca Citi]

I have had some troubles recently using btrfs with kernels >=4.8 when 
using the cfq scheduler.
Please see my post: http://www.spinics.net/lists/linux-btrfs/msg63599.html
I have no problem with kernels < 4.8 or when using the deadline 
scheduler (or none).

It is possible that the problem I see is a result of a major redesign 
that happened in March 2016.

So, contrary to most people advising to use very recent kernels, I 
suggest to try and strike a balance between a kernel that is recent 
enough to have known bugs fixed but old enough that any recently 
introduced change has been around for a while and tested.

Regards,
Luca