Re: state of btrfs snapshot limitations?

[Duncan <1i5t5.duncan@cox.net>]

James A. Robinson posted on Fri, 14 Sep 2018 14:05:29 -0700 as excerpted:

> The mail archive seems to indicate this list is appropriate for not only
> the technical coding issues, but also for user questions, so I wanted to
> pose a question here.  If I'm wrong about that, I apologize in advance.

User questions are fine here.  In fact, there are a number of non-dev 
regulars here who normally take the non-dev level questions.  I'm one of 
them. =:^)

> The page
> 
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
> 
> talks about the basic snapshot capabilities of btrfs and led me to look
> up what, if any, limits might apply.  I find some threads from a few
> years ago that talk about limiting the number of snapshots for a volume
> to 100.

Btrfs is optimized to make snapshotting very fast -- on an atomic copy-on-
write tree-based filesystem like btrfs it's pretty much just taking a new 
reference pointing at the current tree head so nothing in it disappears, 
and that's very fast -- but maintenance that works with existing 
snapshots (and other references) is often slower and doesn't always scale 
so nicely.  While from btrfs' perspective there's nothing "magical" about 
the number 100, in human terms it is of course easy to remember, and it's 
very roughly where the number of snapshots starts to take its toll on the 
time required for various filesystem maintenance tasks, including 
deleting snapshots, balance, fsck, quota maintenance, etc.

So the number of snapshots you can get away with depends primarily on 
three things:

1) Easiest and biggest factor:  If you don't need quotas, simply keeping 
that functionality turned off makes a big difference, and if you /do/ 
need them, turning them off temporarily for maintenance such as a 
rebalance, then doing a quota rescan when the balance is completed, can 
be the difference between a balance taking days or weeks with quotas on 
and constantly updating during the balance, vs. hours to a couple days 
turning quotas off during the balance.  There have been quite a number of 
people who have posted questions about balance not being practical (or 
even thinking it was hung) as it was taking "forever", that found simply 
turning quotas off (sometimes they didn't even know they were on, it was 
a distro setting) fixed the problem and that balance completed in a 
reasonable time after that.

(There have recently been patches to avoid some of the worst constant 
rescanning during balance, but as my own use-case doesn't require either 
quotas or snapshotting, I'm not following their status, and if quotas 
aren't required keeping them off will remain simplest and most efficient 
in any case.)

2) Use-case need for maintenance:  While (almost) any periodic-
snapshotting use-case is going to need snapshot thinning and thus 
snapshot removal as routine maintenance, some use-cases, particularly at 
the large scale, aren't going to find less routine maintenance tasks like 
full balance (converting between raid levels or adding/deleting devices 
to/from an existing filesystem) or check --repair, etc, useful; they'll 
simply swap in a hot-spare backup and mkfs the former working copy they 
would have otherwise needed maintenance on, because it's easier/simpler/
faster for them than trying to repair or change the device config of the 
existing filesystem, and their operating parameters already require the 
hot-spare resources for other reasons.

This is likely why a working fsck repair mechanism wasn't a high priority 
early on, and why it still has "holes" in the types of damage it can 
repair.  The big users such as facebook and oracle funding development 
simply don't find that sort of functionality useful as they hot-swap 
instead.  

But even for more "normal/personal" use-cases, if adding a device and 
rebalancing to make efficient use of it, or if repairing a broken 
filesystem when you already have the valuable stuff on it backed up 
anyway, is going to take days, with no guarantee all the problems will be 
fixed in any case for the repair case, even if it's going to take 
dropping by the local computer/electronics (super-)store for a new disk 
or three (remember the multi-device case), it may well make more sense to 
do that then to take days doing the repair/device-add with the existing 
filesystem.

Obviously if you aren't going to be repairing the filesystem or adding/
removing devices, the time that takes isn't a factor you need to worry 
about, and snapshot-deletion times are likely to be the only thing you 
need to worry about in terms of snapshot numbers scaling.

3) Backing-device speed, ssd vs. spinning-rust, etc, matters, but not as 
much as you might think, because for some filesystem maintenance 
operations, particularly with large numbers of snapshots/reflinks, parts 
of them are cpu- or memory-bound, not IO-bound.


So while 100 snapshots is a convenient number as a recommendation, it 
really depends.  On slow systems with quotas on and full-balances/fscks a 
necessary part of the use-case, 50 may even be high, while on fast 
systems with quotas off and mkfs and restore from backup preferable to 
full balances and check --repairs, the pain threshold for snapshot 
numbers may be 1000 or more, and indeed, the recommendation used to be 
under 300, which allows for a thinning scheme with a much nicer comfort 
margin than the newer under 100 recommendation.


> The reason I'm curious is I wanted to try and use the snapshot
> capability as a way of keeping a 'history' of a backup volume I
> maintain.  The backup doesn't change a lot overtime, but small changes
> are made to files within it daily.

Just keep in mind that "snapshots do not and cannot replace backups".  
You appear to be actually doing this /with/ a backup, not /as/ your 
backup, so you are likely fine, but if for no other reason than because 
I'll sleep better knowing I mentioned it explicitly... Don't make the 
mistake of thinking you're covered because you have it snapshotted, and 
then end up posting here when something happens to the filesystem or 
device(s) it's on, and all those snapshots are gone with the same 
filesystem damage that took out the working copy!

> With btrfs I was thinking perhaps I could more efficiently maintain the
> archive of changes over time using a snapshot. If this is an awful
> thought and I should just go away, please let me know.

This is actually a valid and quite common use-case...

> If the limit is 100 or less I'd need use a more complicated rotation
> scheme.  For example with a layout like the following:
> 
> min/<mm>
> hour/<hh>
> day/<dd>
> month/<mm>
> year/<yyy>
> 
> The idea being each bucket, min, hour, day, month, would be capped and
> older snapshots would be removed and replaced with newer ones over time.
> 
> so with a 15-minute snapshot cycle I'd end up with
> 
> min/[00,15,30,45]
> hour/[00-23]
> day/[01-31]
> month/[01-12]
> year/[2018,2019,...]
> 
> (72+ snapshots with room for a few years worth of yearly's).
> 
> But if things have changed with btrfs over the past few years and number
> of snapshots scales much higher, I would use the easier scheme:
> 
> /min/[00,15,30,45]
> /hourly/[00-23]
> /daily/<yyyy>/<mmdd>
> 
> with 365 snapshots added per additional year.

There's potentially at least two other snapshotting reasons to keep in 
mind as well, as they could add to the total...

* If you're planning to use btrfs send/receive, presumably for backups, 
that requires read-only snapshots, probably with at least some kept 
around as reference points for later incremental send/receives, as well.

* Some distros take pre-upgrade snapshots in ordered to allow rollbacks 
if necessary.

You can probably integrate your planned snapshotting scheme with both of 
the above, certainly with the first, but they are something you need to 
be aware of and keep in consideration if they apply.


Another possible caveat: With the current use-case being primarily 
backup, this likely doesn't apply, but snapshots limit the effectiveness 
of nocow, which effectively becomes cow1.  Look into that if it does 
apply.


As to your scheme...

Traditionally, our examples use a snapshot timestamp scheme, with 
snapshots taken at the minimum period (every 15 minutes in the above) and 
then thinned down, say deleting every other one to 30 minutes after an 
hour or two, again deleting every other one to an hour after say six, 
deleting 5 of six to every six hours after a day (or 30 hours, to give an 
overlap of six hours), deleting 6 of 7 days after a week or two... 
deleting every other week after say 6 weeks, deleting half to every 4th 
week after six months, deleting 2/3 to every 12th week (~quarterly) after 
a year...

And then to help stress the difference between snapshots and backups, and 
to help free space and with fragmentation caused by keeping references to 
otherwise long gone files locked up in ancient snapshots, after a year or 
two, rather than thinning to annual snapshots and keeping those, I at 
least, recommended taking backups to other media (tape, physically 
swapped out hard drives, etc) if it was considered necessary to keep 
history that far back at all, and deleting all snapshots beyond a year or 
two out.

However, as I was composing the above discussion of snapshot creation 
being nearly cost-free, with snapshot deletion and other filesystem 
maintenance being the real cost of snapshots, in the context of your 
separated time-based scheme above, it occurred to me that taking multiple 
separate snapshots at different period intervals, so for instance worst-
case 00/minute, hourly, daily, monthly, yearly, all at (nearly) the same 
time, and then simply deleting all in the appropriate directory beyond 
some cap time, instead of the thinning logic of the above traditional 
model, wouldn't actually be much less efficient in terms of snapshot 
taking, because snapshotting is /designed/ to be fast, while at the same 
time it would significantly simplify the logic of the deletion scripts 
since they could simply delete everything older than X, instead of having 
to do conditional thinning logic.

So your scheme with period slotting and capping as opposed to simply 
timestamping and thinning, is a new thought to me, but I like the idea 
for its simplicity, and as I said, it shouldn't really "cost" more, 
because taking snapshots is fast and relatively cost-free. =:^)

I'd still recommend taking it easy on the yearly, tho, perhaps beyond a 
year or two, preferring physically media swapping and archiving at the 
yearly level if yearly archiving is found necessary at all.  And 
depending on your particular needs, physical-swap archiving at six months 
or even quarterly might actually be appropriate, especially given that 
(with spinning rust at least, I guess ssds retain best with periodic 
power-up) on-the-shelf archiving should be more dependable as a last-
resort backup.

Or do similar online with for example Amazon Glacier (never used 
personally, tho I actually have the site open for reference as I write 
this and at US $0.004 per gig per month... so say $100 for a TB for 2 
years or a couple hundred gig for a decade, $10/yr with a much better 
chance at actually being able to use it after a fire/flood/etc that'd 
take out anything local, tho actually retrieving it would cost a bit 
too... I'm actually thinking perhaps I should consider it... obviously 
I'd well encrypt first... until now I'd always done onsite backup only, 
figuring if I had a fire or something that'd be the last thing I'd be 
worried about, but now I'm actually considering...)

OK, so I guess the bottom-line answer is "it depends."  But the above 
should give you more data to plugin for your specific use-case.

But if it's pure backup, you don't expect to expand to more devices in-
place and you can blow it away and don't have to consider check --repair, 
AND you can do a couple filesystems so as to keep your daily snapshots 
separate from the more frequent backups and thus avoid snapshot deletion, 
you may actually be able to do the 365 dailies for 2-3 years then swap-
out filesystems and devices without deleting snapshots, thus avoiding any 
of the maintenance-scaling issues that are the big limitation, and have 
it work just fine.

OTOH, if you're use-case is a bit more conventional, with more 
maintenance to have to worry about scaling, capping to 100 snapshots 
remains a reasonable recommendation, and if you need quotas as well and 
can't afford to disable them even temporarily for a balance, you may find 
under 50 snapshots to be your maintenance pain tolerance threshold.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman