5 _thousand_ snapshots? even 160? (was: device balance times)

5 _thousand_ snapshots? even 160? (was: device balance times)

[Robert White]

That's an unmanageably large and probably pointless number of snapshots 
guys.

I mean 150 is a heck of a lot, and 5000 is almost unfathomable in terms 
of possible usefulness.

Snapshots are cheap but they aren't free.

Each snapshot is effectively stapling down one version of your entire 
metadata tree, right? So imagine leaving tape spikes (little marks on 
the floor to keep track of where something is so you can put it back) 
for the last 150 or 5000 positions of the chair you are sitting in. At 
some point the clarity and purpose of those marks becomes the opposite 
of useful.

Hourly for a day, daily for a week, weekly for a month, monthly for a 
year. And it's not a "backup" if you haven't moved it to another device. 
If you have 5k snapshots of a file that didn't change, you are still 
just one bad disk sector away from never having that data again because 
there's only one copy of the actual data stapled down in all of those 
snapshots.

The ability to avoid fragmentation and cruft is diminished by excessive 
snapshots on a live media.

Go get a backup drive or whatever. Snapshot you live media, send the 
snapshot to that backup. If you want to hoard them, hoard them on the 
backup drive.

There is an old saying. If you haven't run the restore operation your 
backup scheme is untested. Have you _really_ considered how you would go 
about scavenging through 5k of snapshots? Have you really done the 
exercise-of-consideration about what you are safeguarding by having 156 
or more paths to the same single disk sector?

More than four snapshots on the live disk and you are playing with it 
(ha ha).

Excessive snapshotting _will_ complicate many operations because you are 
permuting the choices the system has to consider and you are leaving 
allocated the ghosts of long dead files (like old logs in /var/log and 
umpteen copies of your browser cookies and history, and copies of the 
window layout from the last several hundred times you logged out of your 
desktop).


I don't think balance will _ever_ move the contents of a read only 
snapshot. I could be wrong. I think you just end up with an endlessly 
fragmented storage space and balance has to take each chunk and search 
for someplace else it might better fit. Which explains why it took so long.

And just _forget_ single-extent large files at that point.

(Of course I could be wrong about the "never move" rule, but that would 
just make the checksums on the potentially hundreds or thousands of 
references need to be recalculated after a move, which would make 
incremental send/receive unfathomable.)


On 10/21/2014 01:44 PM, Arnaud Kapp wrote:
> Hello,
>
> I would like to ask if the balance time is related to the number of
> snapshot or if this is related only to data (or both).
>
> I currently have about 4TB of data and around 5k snapshots. I'm thinking
> of going raid1 instead of single. From the numbers I see this seems
> totally impossible as it would take *way* too long.
>
> Would destroying snapshots (those are hourly snapshots to prevent stupid
> error to happens, like `rm my_important_file`) help?
>
> Should I reconsider moving to raid1 because of the time it would take?
>
> Sorry if I'm somehow hijacking this thread, but it seemed related :)
>
> Thanks,
>
> On 10/21/2014 10:14 PM, Piotr Pawłow wrote:
>> On 21.10.2014 20:59, Tomasz Chmielewski wrote:
>>> FYI - after a failed disk and replacing it I've run a balance; it took
>>> almost 3 weeks to complete, for 120 GBs of data:
>>
>> Looks normal to me. Last time I started a balance after adding 6th
>> device to my FS, it took 4 days to move 25GBs of data. Some chunks took
>> 20 hours to move. I currently have 156 snapshots on this FS (nightly
>> rsync backups).
>>
>> I think it is so slow, because it's disassembling chunks piece by piece
>> and stuffing these pieces elsewhere, instead of moving chunks as a
>> whole. If you have a lot of little pieces (as I do), it will take a
>> while...
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 1664 bytes --]

On Tue, Oct 21, 2014 at 06:10:27PM -0700, Robert White wrote:
> That's an unmanageably large and probably pointless number of
> snapshots guys.
> 
> I mean 150 is a heck of a lot, and 5000 is almost unfathomable in
> terms of possible usefulness.
> 
> Snapshots are cheap but they aren't free.

This could be better documented.  Most of the documentation about btrfs
snapshots frankly just says how to make them and how awesome they are.
In small-scale tests they seem cheap--a little more expensive than
mkdir--but on larger scales they can become nightmares.

Things I learned the hard way about ~200 snapshots on a half-TB
filesystem:

btrfs balance cancel/pause take a long time.  Like 24+ hours.  Not the
entire balance operation--just one block group.  Reboot and mount with
skip_balance is a crude but effective way out.

Making new snapshots is cheap.  Deleting them is not.  btrfs-cleaner
will systematically disassemble each snapshot, sequentially, one extent
(or some similarly tiny fragment) at a time, for more than a week.
fsync() will take several seconds on that filesystem for the entire
time btrfs-cleaner is running.  There is no way to pause or stop the
btrfs-cleaner kernel thread and as far as I can tell it's more or less
immune to attempts to limit the amount of I/O bandwidth it demands with
cgroup blkio.weight or ionice.  Rebooting won't even stop it--it will
just resume the next time the filesystem is mounted.  mkfs and restore
the filesystem from a backup of the original subvolume--it's an order
of magnitude faster even with rsync.

2 or 3 are good numbers for snapshots.  0 or 1 are even better.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Duncan]

Robert White posted on Tue, 21 Oct 2014 18:10:27 -0700 as excerpted:

> Each snapshot is effectively stapling down one version of your entire
> metadata tree, right? So imagine leaving tape spikes (little marks on
> the floor to keep track of where something is so you can put it back)
> for the last 150 or 5000 positions of the chair you are sitting in. At
> some point the clarity and purpose of those marks becomes the opposite
> of useful.
> 
> Hourly for a day, daily for a week, weekly for a month, monthly for a
> year. And it's not a "backup" if you haven't moved it to another device.
> If you have 5k snapshots of a file that didn't change, you are still
> just one bad disk sector away from never having that data again because
> there's only one copy of the actual data stapled down in all of those
> snapshots.

Exactly.

I explain the same thing in different words:

(Note: "You" in this post is variously used to indicate the parent 
poster, and a "general you", including but not limited to the grandparent 
poster inquiring about his 5000 hourly snapshots.  As I'm not trying to 
write a book or a term paper I actively suppose it should be clear to 
which "you" I'm referring in each case based on context...)

Say you are taking hourly snapshots of a file, and you mistakenly delete 
it or need a copy from some time earlier.

If you figure that out a day later, yes, the hour the snapshot was taken 
can make a big difference.

If you don't figure it out until a month later, then is it going to be 
REALLY critical which HOUR you pick, or is simply picking one hour in the 
correct day (or possibly half-day) going to be as good, knowing that if 
you guess wrong you can always go back or forward another whole day?

And if it's a year later, is even the particular day going to matter, or 
will going forward or backward a week or a month going to be good enough?

And say it *IS* a year later, and the actual hour *DOES* matter.  A year 
later, exactly how are you planning to remember the EXACT hour you need, 
such that simply randomly picking just one out of the day or week is 
going to make THAT big a difference?

As you said but adjusted slightly to even out the weeks vs months, hourly 
for a day (or two), daily to complete the week (or two), weekly to 
complete the quarter (13 weeks), and if desired, quarterly for a year or 
two.

But as you also rightly pointed out, just as if it's not tested it's not 
a backup, if it's not on an entirely separate device and filesystem, it's 
not a backup.

And if you don't have real backups at least every quarter, why on earth 
are you worrying about a year's worth of hourly snapshots?  If disaster 
strikes and the filesystem blows up, without a separate backup, they're 
all gone, so why the trouble to keep them around in the first place?

And once you have that quarterly or whatever backup, then the advantage 
of continuing to lock down those 90-day-stale copies of all those files 
and metadata goes down dramatically, since if worse comes to worse, you 
simply retrieve it from backup, but meanwhile, all that stale locked down 
data and metadata is eating up room and dramatically complicating the job 
btrfs must do to manage it all!

Yes, there are use-cases and there are use-cases.  But if you aren't 
keeping at least quarterly backups, perhaps you better examine your 
backup plan and see if it really DOES match your use-case, ESPECIALLY if 
you're keeping thousands of snapshots around.  And once you DO have those 
quarterly or whatever backups, then do you REALLY need to keep around 
even quarterly snapshots covering the SAME period?

But let's say you do:

48 hourly snapshots, thinned after that to...

12 daily snapshots (2 weeks = 14, minus the two days of hourly), thinned 
after that to...

11 weekly snapshots (1 quarter = 13 weeks, minus the two weeks of daily), 
thinned after that to...

7 quarterly snapshots (2 years = 8 quarters, minus the quarter of weekly).

48 + 12 + 11 + 7 = ...

78 snapshots, appropriately spaced by age, covering two full years.

I've even done the math for the extreme case of per-minute snapshots.  
With reasonable thinning along the lines of the above, even per-minute 
snapshots ends up well under 300 snapshots being reasonably managed at 
any single time.

And keeping it under 300 snapshots really DOES help btrfs in terms of 
management task time-scaling.

If you're doing hourly, as I said, 78, tho killing the quarterly 
snapshots entirely because they're backed up reduces that to 71, but 
let's just say, EASILY under 100.

Tho that is of course per subvolume.  If you have multiple subvolumes on 
the same filesystem, that can still end up being a thousand or two 
snapshots per filesystem.  But those are all groups of something under 
300 (under 100 with hourly) highly connected to each other, with the 
interweaving inside each of those groups being the real complexity in 
terms of btrfs management.

But 5000 snapshots?

Why?  Are you *TRYING* to test btrfs until it breaks, or TRYING to 
demonstrate a balance taking an entire year?

Do a real backup (or more than one, using those snapshots) if you need 
to, then thin the snapshots to something reasonable.  As the above 
example shows, if it's a single subvolume being snapshotted, with hourly 
snapshots, 100 is /more/ than reasonable.

With some hard questions, keeping in mind the cost in extra maintenance 
time for each additional snapshot, you might even find that minimum 6-
hour snapshots (four per day) instead of 1-hour snapshots (24 per day) 
are fine.  Or you might find that you only need to keep hourly snapshots 
for 12 hours instead of the 48 I assumed above, and daily snapshots for a 
week instead of the two I assumed above.  Throwing in the nothing over a 
quarter because it's backed up assumption as well, that's...

8 4x-daily snapshots (2 days)

5 daily snapshots (a week, minus the two days above)

12 weekly snapshots (a quarter, minus the week above, then it's backed up 
to other storage)

8 + 5 + 12 = ...

25 snapshots total, 6-hours apart (four per day) at maximum frequency aka 
minimum spacing, reasonably spaced by age to no more than a week apart, 
with real backups taking over after a quarter.

Btrfs should be able to work thru that in something actually approaching 
reasonable time, even if you /are/ dealing with 4 TB of data. =:^)

Bonus hints:

Btrfs quotas significantly complicate management as well.  If you really 
need them, fine, but don't unnecessarily use them just because they are 
there.

Look into defrag.

If you don't have any half-gig plus VMs or databases or similar "internal 
rewrite pattern" files, consider the autodefrag mount option.  Note that 
if you haven't been using it and your files are highly fragmented, it can 
slow things down at first, but a manual defrag, possibly a directory tree 
at a time to split things up into reasonable size and timeframes, can 
help.

If you are running large VMs or databases or other half-gig-plus sized 
internal-rewrite-pattern files, the autodefrag mount option may not 
perform well for you.  There's other options for that, including separate 
subvolumes, setting nocow on those files, and setting up a scheduled 
defrag.  That's out of scope for this post, so do your research.  It has 
certainly been discussed enough on-list.

Meanwhile, do note that defrag is currently snapshot-aware-disabled, due 
to scaling issues.  IOW, if your files are highly fragmented as they may 
well be if you haven't been regularly defragging them, expect the defrag 
to eat a lot of space since it'll break the sharing with older snapshots 
as anything that defrag moves will be unshared.  However, if you've 
reduced snapshots to the quarter-max before off-filesystem backup as 
recommended above, a quarter from now all the undefragged snapshots will 
be expired and off the system and you'll have reclaimed that extra space. 
Meanwhile, your system should be /much/ easier to manage and will likely 
be snappier in its response as well.  =:^)

With all these points applied, balance performance should improve 
dramatically.  However, with 4 TB of data the shear data size will remain 
a factor.  Even in the best case typical thruput on spinning rust won't 
reach the ideal.  10 MiB/sec is a reasonable guide.  4 TiB/10 MiB/sec...

4*1024*1024 (MiB) /  10 MiB / sec = ...

nearly 420 thousand seconds ... / 60 sec/min = ...

7000 minutes ... / 60 min/hour = ...

nearly 120 hours or ...

a bit under 5 days.


So 4 TiB on spinning rust could reasonably take about 5 days to balance 
even under quite good conditions.  That's due to the simple mechanics of 
head seek to read, head seek again to write, on spinning rust, and the 
shear size of 4 TB of data and metadata (tho with a bit of luck some of 
that will disappear as you thin out those thousands of snapshots, and 
it'll be more like 3 TB than 4, or possibly even down to 2 TiB, by the 
time you actually do it).

IOW, it's not going to be instant, by any means.

But the good part of it is that you don't have to do it all at once.  You 
can use balance filters and balance start/pause/resume/cancel as 
necessary, to do only a portion of it at a time, and restart the balance 
using the convert,soft options so it doesn't redo already converted 
chunks when you have time to let it run.  As long as it completes at 
least one chunk each run it'll make progress.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Tomasz Chmielewski]

> But 5000 snapshots?
> 
> Why?  Are you *TRYING* to test btrfs until it breaks, or TRYING to
> demonstrate a balance taking an entire year?

Remember a given btrfs filesystem is not necessarily a backup 
destination for data from one source.

It can be, say, 30 or 60 daily snapshots, plus several monthly, for each 
data source * number of data sources.

So while it probably will make a difference (5000 snapshots from one 
source, vs 5000 snapshots made from many sources) for balance times, I 
wouldn't call a large number of snapshots that unusual.

-- 
Tomasz Chmielewski
http://www.sslrack.com

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Duncan]

Tomasz Chmielewski posted on Wed, 22 Oct 2014 09:14:14 +0200 as excerpted:

> Remember a given btrfs filesystem is not necessarily a backup
> destination for data from one source.
> 
> It can be, say, 30 or 60 daily snapshots, plus several monthly, for each
> data source * number of data sources.
> 
> So while it probably will make a difference (5000 snapshots from one
> source, vs 5000 snapshots made from many sources) for balance times, I
> wouldn't call a large number of snapshots that unusual.

That's what this paragraph, just above the paragraph you quoted, was all 
about:

>> Tho that is of course per subvolume.  If you have multiple subvolumes
>> on the same filesystem, that can still end up being a thousand or two
>> snapshots per filesystem.  But those are all groups of something under
>> 300 (under 100 with hourly) highly connected to each other, with the
>> interweaving inside each of those groups being the real complexity in
>> terms of btrfs management.

IOW, if you thin down the snapshots per subvolume to something reasonable 
(under 300 for sure, preferably under 100), then depending on the number 
of subvolumes you're snapshotting, you might have a thousand or two.  
However, of those couple thousand, btrfs will only have to deal with the 
under 300 and preferably well under a hundred in the same group, that are 
snapshots of the same thing and thus related to each other, at any given 
time.  The other snapshots will be there but won't be adding to the 
complexity near as much since they're of different subvolumes and aren't 
logically interwoven together with the ones being considered at that 
moment.

But even then, at say 250 snapshots per subvolume, 2000 snapshots is 8 
independent subvolumes.  That could happen.  But 5000 snapshots?  That'd 
be 20 independent subvolumes, which is heading toward the extreme again.  
Yes it could happen, but better if it does to cut down on the per-
subvolume snapshots further, to say the 25 per subvolume I mentioned, or 
perhaps even further.  25 snapshots per subvolume with those same 20 
subvolumes... 500 snapshots total instead of 5000. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 2597 bytes --]

On Wed, Oct 22, 2014 at 07:41:32AM +0000, Duncan wrote:
> Tomasz Chmielewski posted on Wed, 22 Oct 2014 09:14:14 +0200 as excerpted:
> >> Tho that is of course per subvolume.  If you have multiple subvolumes
> >> on the same filesystem, that can still end up being a thousand or two
> >> snapshots per filesystem.  But those are all groups of something under
> >> 300 (under 100 with hourly) highly connected to each other, with the
> >> interweaving inside each of those groups being the real complexity in
> >> terms of btrfs management.
> 
> IOW, if you thin down the snapshots per subvolume to something reasonable 
> (under 300 for sure, preferably under 100), then depending on the number 
> of subvolumes you're snapshotting, you might have a thousand or two.  
> However, of those couple thousand, btrfs will only have to deal with the 
> under 300 and preferably well under a hundred in the same group, that are 
> snapshots of the same thing and thus related to each other, at any given 
> time.  The other snapshots will be there but won't be adding to the 
> complexity near as much since they're of different subvolumes and aren't 
> logically interwoven together with the ones being considered at that 
> moment.
> 
> But even then, at say 250 snapshots per subvolume, 2000 snapshots is 8 
> independent subvolumes.  That could happen.  But 5000 snapshots?  That'd 
> be 20 independent subvolumes, which is heading toward the extreme again.  
> Yes it could happen, but better if it does to cut down on the per-
> subvolume snapshots further, to say the 25 per subvolume I mentioned, or 
> perhaps even further.  25 snapshots per subvolume with those same 20 
> subvolumes... 500 snapshots total instead of 5000. =:^)

If you have one subvolume per user and 1000 user directories on a server,
it's only 5 snapshots per user (last hour, last day, last week, last
month, and last year).  I hear this is a normal use case in the ZFS world.
It would certainly be attractive if there was working quota support.

I have datasets where I record 14000+ snapshots of filesystem directory
trees scraped from test machines and aggregated onto a single server
for deduplication...but I store each snapshot as a git commit, not as
a btrfs snapshot or even subvolume.

We do sometimes run queries like "in the last two years, how many times
did $CONDITION occur?" which will scan a handful files in all of the
snapshots.  The use case itself isn't unreasonable, although using the
filesystem instead of a more domain-specific tool to achieve it may be.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Robert White]

On 10/22/2014 01:08 PM, Zygo Blaxell wrote:
> I have datasets where I record 14000+ snapshots of filesystem directory
> trees scraped from test machines and aggregated onto a single server
> for deduplication...but I store each snapshot as a git commit, not as
> a btrfs snapshot or even subvolume.
>
> We do sometimes run queries like "in the last two years, how many times
> did $CONDITION occur?" which will scan a handful files in all of the
> snapshots.  The use case itself isn't unreasonable, although using the
> filesystem instead of a more domain-specific tool to achieve it may be.
>

Okay, sure. And as stated by others, there _are_ use cases that are 
exceptional.

But such an archival system most likely does not _need_ to be balanced 
etc with any frequency, or likely ever because it isn't experiencing 
churn from dynamic use.

In the world of trade-offs, trade-offs happen.

The guy who cited the 5000 snapshots said they were hourly and taken 
because he might remove an important file or something. This is _way_ 
more action than the feared condition.

ASIDE: While fixing someone's document archive RAID device (a Sun 
hardware device the size of a fridge) back in 1997 or so I discovered 
that they'd disabled _all_ the hardware cache features. When asked I was 
told that "the procedure for replacing a failed drive required the cache 
device to be cleared by pressing the red button" and they were afraid 
that, should that day come, someone would forget to press that button... 
so they'd turned off the feature entirely. This is a form of 
unreasonable paranoia. They were afraid that someone in the future would 
not follow the directions would be printed on both the machine and the 
new drive (these were _not_ commodity parts).

When an over-abundance of caution passes beyond reasonable expectations, 
performance will suffer. The system is immaterial, the rule holds.

What's worse is it becomes very like "security theater" only its "a 
backup show" where no actual backing up is really happening in any 
useful sense. And god save you picking which version of a file was the 
last "good one".

So in your use case, your git repository of snapshots isn't actually 
"live" on the production server you are archiving, right?

So too, it would be reasonable to btrfs send periodic snapshots to an 
archive system, retain lots and lots of them, and expect reasonable 
performance of your queries.

And you cold expect reasonable performance in your maintenance.

But "reasonable performance" in the maintenance case is massively 
different than reasonable performance in use cases. Indeed if you try to 
balance multiple terabytes of data spread across thousands of snapshots 
you'll be taking a lot of time. A _perfectly_ _reasonable_ lot of time 
for the operation at hand.

But if you expect to be able to do maintenance (like btrfsck your 
production box with its 5k snapshots) in just a few minutes when you've 
got logarithmic-rate meta data to shuffle through... well good luck with 
that.

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 4046 bytes --]

On Wed, Oct 22, 2014 at 01:37:15PM -0700, Robert White wrote:
> On 10/22/2014 01:08 PM, Zygo Blaxell wrote:
> >I have datasets where I record 14000+ snapshots of filesystem directory
> >trees scraped from test machines and aggregated onto a single server
> >for deduplication...but I store each snapshot as a git commit, not as
> >a btrfs snapshot or even subvolume.
> >
> >We do sometimes run queries like "in the last two years, how many times
> >did $CONDITION occur?" which will scan a handful files in all of the
> >snapshots.  The use case itself isn't unreasonable, although using the
> >filesystem instead of a more domain-specific tool to achieve it may be.
> 
> Okay, sure. And as stated by others, there _are_ use cases that are
> exceptional.
> 
> But such an archival system most likely does not _need_ to be
> balanced etc with any frequency, or likely ever because it isn't
> experiencing churn from dynamic use.
> 
> In the world of trade-offs, trade-offs happen.
> 
> The guy who cited the 5000 snapshots said they were hourly and taken
> because he might remove an important file or something. This is
> _way_ more action than the feared condition.
 
> When an over-abundance of caution passes beyond reasonable
> expectations, performance will suffer. The system is immaterial, the
> rule holds.
> 
> What's worse is it becomes very like "security theater" only its "a
> backup show" where no actual backing up is really happening in any
> useful sense. And god save you picking which version of a file was
> the last "good one".
> 
> So in your use case, your git repository of snapshots isn't actually
> "live" on the production server you are archiving, right?

I'm not sure if this question is directed at me or 5000-snapshots-guy.
5000-snapshots-guy was not using git.  I'd say he's using btrfs snapshots
to do a job git is much better at.

My 14K-snapshot git repo is on my production server.  The data in the repo
is aggregated to that server from many other smaller (tiny) machines.
Each smaller machine's data is replicated to the git working directory
and a git commit generated from it.  After the commit is created,
we replace the working tree with the next machine's data and repeat.

The entire git repo is replicated by git fetch/push to a couple of other
servers, so if the main git repo goes down we just swap it with one of
the clone servers in a few minutes.  No btrfs snapshots are involved.
One of the clone servers doesn't even have btrfs at all.

git's packing algorithm does a much more thorough job of compression and
deduplication than a mere filesystem can.  We can store 3,000 current
snapshots of 15GB of raw data in 10GB of disk space, and another 11,000
older snapshots in 25GB.  I don't think btrfs snapshots can do better,
or even get close, to that kind of storage efficiency.

The point I was making was that btrfs snapshots aren't the best tool,
or even a reasonably good tool, for this particular use case (5000
snapshots of a single subvolume on one server).

> So too, it would be reasonable to btrfs send periodic snapshots to
> an archive system, retain lots and lots of them, and expect
> reasonable performance of your queries.
> 
> And you cold expect reasonable performance in your maintenance.
> 
> But "reasonable performance" in the maintenance case is massively
> different than reasonable performance in use cases. Indeed if you
> try to balance multiple terabytes of data spread across thousands of
> snapshots you'll be taking a lot of time. A _perfectly_ _reasonable_
> lot of time for the operation at hand.

...unless you are balancing because you're replacing a RAID1 disk.
That's a good reason to be wary of too many snapshots even if the data
is fairly static.

> But if you expect to be able to do maintenance (like btrfsck your
> production box with its 5k snapshots) in just a few minutes when
> you've got logarithmic-rate meta data to shuffle through... well
> good luck with that.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Chris Murphy]

On Oct 22, 2014, at 4:08 PM, Zygo Blaxell <zblaxell@furryterror.org> wrote:
> 
> If you have one subvolume per user and 1000 user directories on a server,
> it's only 5 snapshots per user (last hour, last day, last week, last
> month, and last year). 


Sure. So if Btrfs is meant to address scalability, then perhaps at the moment it's falling short. As it's easy to add large drives and get very large multiple device volumes, the snapshotting needs to scale also.

I'd say per user, it's reasonable to have 24 hourly (one snapshot per hour for a day), 7 daily, 4 weekly, and 12 monthly snapshots, or 47 snapshots. That's 47,000 snapshots if it's sane for a single Btrfs volume to host 1000 users. Arguably, such a system is better off with a distributed fs: Gluster FS or GFS2 or Ceph.



Chris Murphy

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Robert White]

On 10/22/2014 09:30 PM, Chris Murphy wrote:
> Sure. So if Btrfs is meant to address scalability, then perhaps at the moment it's falling short. As it's easy to add large drives and get very large multiple device volumes, the snapshotting needs to scale also.
>
> I'd say per user, it's reasonable to have 24 hourly (one snapshot per hour for a day), 7 daily, 4 weekly, and 12 monthly snapshots, or 47 snapshots. That's 47,000 snapshots if it's sane for a single Btrfs volume to host 1000 users. Arguably, such a system is better off with a distributed fs: Gluster FS or GFS2 or Ceph.

Is one subvolume per user a rational expectation? Is it even 
particularly smart? Dooable, sure, but as a best practice it doesn't 
seem that useful because it multiplies the maintenace by the user base.

Presuming a linux standard base layout (which is very presumptive) 
having the 47 snapshots of /home instead of the 47,000 snapshots of 
/home/X(1000) is just as workable, if not moreso. A reflink recursive 
copy of /home/X(n) from /home_Backup_date/X(n) is only trivially longer 
than resnapshotting the individual user.

Again this gets into the question not of what exercises well to create 
the snapshot but what functions well during a restore.

People constantly create "backup solutions" without really looking at 
the restore path.

I can't get anybody here to answer the question about "btrfs fi li -s /" 
and setting/resetting the "snapshot" status of a subvolume. I've been 
told "snapshots are subvolumes" which is fine, but since there _is_ a 
classification mechanism things get all caca if you rely on the "-s" in 
your scripting and then promote a snapshot back into prime activity. 
(seriously compare the listing with and without -s, note its natural 
affinity for classifying subvolumes, then imagine the horror of needing 
to take /home_backup_date and make it /home.)

Similar problems obtain as soon as you consider the daunting task of 
shuffling through 47,000 snapshots instead of just 47.

And if you setup each user on their own snapshot what happens the first 
time two users want to hard-link a file betwixt them?

Excessive segmentation of storage is an evil unto itself.

YMMV, of course.

An orthoginal example:

If you give someone six disks and tell them to make an encrypted raid6 
via cryptsetup and mdadm, at least eight out of ten will encrypt the 
drives and then raid the result. But it's _massivly_ more efficent to 
raid the drives and then encrypt the result. Why? Because writing a 
block with the latter involves only one block being encrypted/decrypted. 
The former, if the raid is fine involves several encryptions/decryptions 
and _many_ if the raid is degraded.

The above is a mental constraint, a mistake, that is all to common 
because people expect encrytion to be "better" the closer you get to the 
spinning rust.

So too people expect that segmentation is somehow better if it most 
closely matches the abstract groupings (like per user) but in practical 
terms it is better matched to the modality, where, for instance, all 
users are one kind of thing, while all data stores are another kind of 
thing.

We were just talking about putting all your VMs and larger NOCOW files 
into a separate subvolume/domain because of their radically different 
write behaviors. Thats a sterling reason to subdivide the storage. So is 
/ vs. /var vs. /home as three different domains with radically different 
update profiles.

So while the natural impulse is to give each user its own subvolume it's 
not likely to be that great an idea in practice because... um... 47,000 
snapshots dude, and so on.

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Duncan]

Robert White posted on Wed, 22 Oct 2014 22:18:09 -0700 as excerpted:

> On 10/22/2014 09:30 PM, Chris Murphy wrote:
>> Sure. So if Btrfs is meant to address scalability, then perhaps at the
>> moment it's falling short. As it's easy to add large drives and get
>> very large multiple device volumes, the snapshotting needs to scale
>> also.

I believe it's a fair statement to say that many aspects of btrfs in 
general simply don't scale well at this point.  Many of the features are 
there; we're now getting to the point where many of those features are 
reasonably bug-free, altho that's definitely an ongoing thing; but pretty 
much wherever you turn and whatever you look at, btrfs is in general not 
yet optimized.

* The raid1 device-read-selection algorithm is simply even/odd-PID-
based.  That's great for a first implementation, since it's simple enough 
to implement and it works well enough to know that reading from either 
copy works, but it's horrible for a final, scalable implementation, since 
too many use-cases will be nearly all even or all odd pids.

* Btrfs multi-device-writes are nearly all serialized, one at a time, 
instead of scheduling writes to all devices at once in ordered to 
maximize bandwidth over the individual-device-speed bottleneck.

* Btrfs snapshot-aware-defrag was introduced to much fanfare, and then 
disabled a couple kernel series later when it became very apparent it 
simply didn't scale, and the lack of scaling meant it didn't work
/at/ /all/ for many users.

* The quota implementation was just recently pretty much entirely 
rewritten due to serious corner-case breakage and lack of scaling (one of 
the contributors to the defrag and balance scaling issues, as it happens).

* The autodefrag mount option doesn't scale well beyond a few hundred MiB 
or with frag-triggering file updates coming in faster than the entire 
file can be rewritten (there's plans to make this better, but the time to 
code and test simply hasn't been available yet).

* This thread is about the balance scaling issues, a good portion of 
which boil down to extremely poorly optimized quota and snapshot 
handling, and another problem set to no choice but extent-based 
operations, which are great for some things but don't work well when all 
you want to do is duplicate chunks in a conversion to raid1 mode, for 
instance.

That's what I think of off the top of my head.  I'm sure there's more.

However, specifically addressing snapshotting, while optimizing for speed 
and scale will certainly help, I'm not sure btrfs will ever be what might 
be called a speed demon in the area.

If that is indeed the case, and I don't know but it's certainly possible, 
then for the future, and regardless of the future, definitely for the 
present, that means it's absolutely /critical/ for the human side to 
optimize things to keep things like number of snapshots from growing out 
of reasonable management range.

...

>> I'd say per user, it's reasonable to have 24 hourly (one snapshot per
>> hour for a day), 7 daily, 4 weekly, and 12 monthly snapshots, or 47
>> snapshots. That's 47,000 snapshots if it's sane for a single Btrfs
>> volume to host 1000 users. Arguably, such a system is better off with a
>> distributed fs: Gluster FS or GFS2 or Ceph.
> 
> Is one subvolume per user a rational expectation? Is it even
> particularly smart? Dooable, sure, but as a best practice it doesn't
> seem that useful because it multiplies the maintenace by the user base.
> 
> Presuming a linux standard base layout (which is very presumptive)
> having the 47 snapshots of /home instead of the 47,000 snapshots of
> /home/X(1000) is just as workable, if not moreso. A reflink recursive
> copy of /home/X(n) from /home_Backup_date/X(n) is only trivially longer
> than resnapshotting the individual user.
> 
> Again this gets into the question not of what exercises well to create
> the snapshot but what functions well during a restore.
> 
> People constantly create "backup solutions" without really looking at
> the restore path.
> 

... Which is where this discussion comes in.

FWIW, over more than a decade of fine tuning and experience with a number 
of disaster and recovery cases here, I've come up with what is for me a 
reasonably close to ideal multiple partition layout.  That's actually one 
of the big reasons I don't use subvolumes here; I don't need to because I 
already have a nearly perfect... for my use-case... independent 
partitions layout.

Here's my point.  Via trial and error I concluded almost the exact same 
point that Chris is making about subvolumes, only for independent 
partitions.

The discussed example is subvolumes for individual users in /home, vs. 
one big subvolume for /home itself (or arguably, if there's a convenient 
user-role-based separation, perhaps a subvolume for say teacher-home and 
another for student-home, or for user-home, group-leader-home, upper-
management-home, etc).

The same lesson, however, applies to say all distro-update storage, in my 
(gentoo) case, the main distro package ebuild tree, the overlays, binpkgs 
for my 64-bit builds, binpkgs for my 32-bit builds, 64-bit ccache, 32-bit 
ccache, the separate (mainstream git) kernel repo and individual build 
dirs for 32-bit and 64-bit kernels, etc.

Back when I first split things up, most of those were in individual 
partitions.  Now I just have one partition with all those components in 
it, in separate subdirs, and symlinks from the various other locations in 
my layout to the various components in this partition.

Why?  Because managing all of them separately was a pain, and I tended to 
mount and unmount most of them together anyway, when I did system updates.

Similarly, in my original setup I had the traditional small /, with /etc, 
but with an independent /usr and /var, and /var/log independent of /var.

I still have /var/log independent of the others since limiting a runaway 
logging scenario to an independent log partition makes very good sense 
and logging is its own logical task, but /usr is on / now, as is /var 
itself, but with individual subdirs of for instance /var/lib symlinked 
elsewhere.

Why?  Because at one point I had an A/C failure, here in Phoenix in the 
middle of the summer, when I was gone.  I came home to a a 50C+ house, a 
heat-induced head-crashed disk and a frozen CPU.

Recovering from that disaster was a nightmare, because while I had 
backups, I ended up with a root backup from one date, a /usr backup from 
a different date, and a /var, including the record of what packages and 
files were installed, from a third date.  So my record (in /var/db) of 
what was installed didn't match /usr, which didn't match /!

So everything that the distro actually installs, including the database 
of what is actually installed, with one limited exception, is now all on 
the same partition, root!  If I have to recover from backup, whether I'm 
recovering from yesterday's backup or one made a year ago, there's one 
thing I'll be sure of, the database of what's installed will match what's 
actually there, because it's on the same system root partition and thus 
the same backup!

And that system partition is now mounted read-only by default.  I only 
mount it writable in ordered to update, either packages or system 
configuration.  When I'm done with the update I sudo systemctl emergency, 
hit ctrl-d to get back to normal mode without logging into emergency mode 
and log back into normal mode, systemctl daemon-reexec if systemd itself 
was updated (thus restarting all programs including init/systemd itself, 
so no stale and deleted libs are still in use), and remount ro /.

Which brings me to the limited /var exception I mentioned earlier.  
While /var is supposed to be variable/writable, in practice, many system 
processes only need to write to their /var subdir if they're updated.  
These can stay on /var.  The few others, the ones that really need a 
writable /var subdir, have that subdir actually linked back to a 
similarly named subdir in /home/var/, which is mounted writable by 
default.  That lets me keep /, including the rest of /var (with the 
exception of /var/tmp, which is tmpfs), read-only by default.

As it happens, these writable-necessary /var/ subdirs aren't critical to 
early boot, and if for some reason /home is unmountable (as it was along 
with /var/log at one point recently when the writable at crash /home 
btrfs refused to mount, but /, being read-only at crash, wasn't harmed), 
I can either do without temporarily, or reconstruct an empty or default 
/var subdir for them in tmpfs or whatever.

One critical thing that makes this all work is the existence of symlinks 
in various locations, pointing to the real locations on the partition 
where logical function-grouping places them.

But all this simply reinforces the point.  Logically function-group 
subdirs on subvolumes much as I logically function-group subdirs on 
independent partitions, and you won't be /dealing/ with 47K snapshots, 47 
snapshots each of 1000 individual user subvolumes.  You might be dealing 
with 47 snapshots of /home, with 1000 users on it, or you might function-
group a bit further and have 47 snapshots each of mgmt-home, user-home, 
and teamldr-home, so 141 snapshots total for all of homedirs, but that's 
still reasonable with btrfs' current scaling, while 47K snapshots, forget 
it!

And take it from me, it makes it FAR easier to deal with backup and 
testing, with disaster recovery should it be necessary, and with other 
sysadmin level maintenance as well.

Tho of course I'd personally argue that for reliability and 
recoverability reasons, each of those function-groups should be an 
independent partition, not just a subvolume that should the filesystem go 
down, it'll take all the subvolumes with it.  But that's an entirely 
different argument to be had...  Regardless of whether it's subvolumes or 
independent partitions and filesystems, however, the same point applies.

> I can't get anybody here to answer the question about "btrfs fi li -s /"
> and setting/resetting the "snapshot" status of a subvolume.

I wish I knew the answer.  But as I said above, I don't do subvolumes, or 
for that matter, snapshots, myself, preferring fully independent 
partitions, and fully independent "snapshot" backups to same-size backup 
partitions located elsewhere, so I can simply point the mount at the 
backup and mount it in place of the previously working copy for recovery 
should it be necessary.  So I've not had to investigate that for personal 
reasons, and while I've an academic interest as well as an interest in 
knowing it simply to help others here, I've seen nobody else post a 
satisfactory answer, so...

I share your frustration, tho at the academic and help-others level, not 
the personal installation operations level.


> I've been
> told "snapshots are subvolumes" which is fine, but since there _is_ a
> classification mechanism things get all caca if you rely on the "-s" in
> your scripting and then promote a snapshot back into prime activity.
> (seriously compare the listing with and without -s, note its natural
> affinity for classifying subvolumes, then imagine the horror of needing
> to take /home_backup_date and make it /home.)

By all means explain to me how this won't work if so, but it seems to me 
the following is a reasonably effective workaround that shouldn't take 
/too/ much more time...

1) According to the wiki, cross-subvolume reflinks now (since 3.6) work.  
See the explanatory text for the following (watch the linkwrap):

https://btrfs.wiki.kernel.org/index.php/
UseCases#Can_I_take_a_snapshot_of_a_directory.3F

Note that based on the above link, reflinks won't work if the subvolumes 
are separately mounted, that is, across separate mount-points.  However, 
as long as it's a single common "parent" mount, with the subvolumes 
simply accessed under it as if they were subdirs, reflink-copying should 
"just work".

Based on that...

2) Mount a parent (which might be the root subvolume) to both the backup-
snapshot and the intended target subvolume, creating the new target 
subvolume as necessary.

3) Reflink-copy recursively from the backup to the target, as if you were 
traditionally backup-restoring from a backup mounted elsewhere, except 
using the parent-subvolume mount paths so you don't cross mount-points, 
and using reflink-copying to dramatically speed the process.

4) When you are done, you should have a non-snapshot subvolume restored 
and ready for use, almost as if you were able to directly mount the 
snapshot in place of the non-snapshot original, removing its snapshot 
property in the process.

5) If desired, delete the backup snapshot, thus completing the parallel.  
Alternatively, keep it where it is.  After all, you needed to restore 
from it once, what's to say something else won't happen to kill the 
restored version, thus triggering the need for another restore?  Surely 
that'd be bad juju, but better to still have that backup snapshot on 
hand, then to have just moved it to production, and then lost it too.
=:^)

> Similar problems obtain as soon as you consider the daunting task of
> shuffling through 47,000 snapshots instead of just 47.
> 
> And if you setup each user on their own snapshot what happens the first
> time two users want to hard-link a file betwixt them?

See the above cross-subvol reflink discussion...

> Excessive segmentation of storage is an evil unto itself.

... But never-the-less, absolutely agreed. =:^)

> YMMV, of course.
> 
> An orthoginal example:
> 
> If you give someone six disks and tell them to make an encrypted raid6
> via cryptsetup and mdadm, at least eight out of ten will encrypt the
> drives and then raid the result. But it's _massivly_ more efficent to
> raid the drives and then encrypt the result. Why? Because writing a
> block with the latter involves only one block being encrypted/decrypted.
> The former, if the raid is fine involves several encryptions/decryptions
> and _many_ if the raid is degraded.
> 
> The above is a mental constraint, a mistake, that is all to common
> because people expect encrytion to be "better" the closer you get to the
> spinning rust.

This totally unexpected but useful jewel is part of why I'm addicted to 
newsgroups and mailing lists. (FWIW, I do this list as a newsgroup via 
gmane.org's list2news service.)  Totally unexpected "orthogonal 
examples", which can be immensely useful all on their own. =:^)

FWIW I haven't gotten much into encrypted storage here, but I keep 
thinking about it, and were I to have done so before reading this, I 
might have made exactly that mistake myself.

OTOH, with btrfs raid replacing mdraid, individually encrypted block 
devices are (currently) necessary, because btrfs merges the filesystem 
and raid levels.  Tho direct btrfs encryption support is apparently 
planned, and if/when that's implemented, one could expect they'll address 
your point and internally layer the encryption over the raid.  Tho I 
expect that'd be a weaker encryption implementation if done that way, 
because part of the advantage of btrfs raid is that the filesystem 
structures work down thru the raid level as well, so individual chunk 
structures appear at the device level below the raid.  If encryption is 
then built over the raid, that would mean the encryption would need to 
pass the individual chunk structures thru so btrfs raid could still use 
them, and that would be a critical information leak to the encrypted side.

So if btrfs does implement encryption, one would hope they'd either have 
a config option for above or below the raid level, or that they'd do it 
the less efficient multi-encryption way below the raid, thus not having 
to pass that information thru the encryption to the raid, leaking it in 
the process.

> So while the natural impulse is to give each user its own subvolume it's
> not likely to be that great an idea in practice because... um... 47,000
> snapshots dude, and so on.

Agreed. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 5233 bytes --]

On Wed, Oct 22, 2014 at 10:18:09PM -0700, Robert White wrote:
> On 10/22/2014 09:30 PM, Chris Murphy wrote:
> >Sure. So if Btrfs is meant to address scalability, then perhaps at the moment it's falling short. As it's easy to add large drives and get very large multiple device volumes, the snapshotting needs to scale also.
> >
> >I'd say per user, it's reasonable to have 24 hourly (one snapshot per hour for a day), 7 daily, 4 weekly, and 12 monthly snapshots, or 47 snapshots. That's 47,000 snapshots if it's sane for a single Btrfs volume to host 1000 users. Arguably, such a system is better off with a distributed fs: Gluster FS or GFS2 or Ceph.
> 
> Is one subvolume per user a rational expectation? Is it even
> particularly smart? Dooable, sure, but as a best practice it doesn't
> seem that useful because it multiplies the maintenace by the user
> base.

For snapshots alone it doesn't make much sense, but there are other
btrfs features that work in subvolume units.  Some people want quota
and send/receive to work on a per-user level too.

If 'btrfs subvolume' had a '-r' recursive option, it would make management
easier.  Even without -r, /home/* can be managed by a simple shell loop
with a wildcard:

makeSnapshot () {
	btrfs sub create "/snapshots/$1"
	for x in /home/*; do
		btrfs sub snap "$x" "/snapshots/$1/$x";
	done
}

makeSnapshot "/home/.snapshots/$(date +%Y-%m-%d-%H-%M-%S)"

> Presuming a linux standard base layout (which is very presumptive)
> having the 47 snapshots of /home instead of the 47,000 snapshots of
> /home/X(1000) is just as workable, if not moreso. A reflink
> recursive copy of /home/X(n) from /home_Backup_date/X(n) is only
> trivially longer than resnapshotting the individual user.

reflink copies are much slower than snapshots.  For that matter, making
a writable snapshot of the entire /home as one subvolume, then using
'rm -rf' to get rid of what we don't need for one particular snapshot
is *also* faster than reflink copies.

More precisely, the bulk of the total life-cycle execution time is at the
beginning with reflink copies (have to create shared extent ref items,
traverse the source directories, and allocate new directory trees, all
while racing against data modifications) and at the end with snapshots
(btrfs-cleaner has to remove unreferenced tree nodes and extents in the
background) or snap-then-trim (replace btrfs-cleaner with rm -rf).

> Again this gets into the question not of what exercises well to
> create the snapshot but what functions well during a restore.
> 
> People constantly create "backup solutions" without really looking
> at the restore path.

It's not all about backups.

> And if you setup each user on their own snapshot what happens the
> first time two users want to hard-link a file betwixt them?

One of the features of per-user subvolumes is that such things are
completely forbidden.  Security issues, user confusion, and all that.

Deduplication by extent sharing (and reflink copy) doesn't care about
subvolumes as long as you do the clone through a common parent of the
user subvolumes (i.e. /home).  The result isn't a hardlink which keeps
users happy, and shares underlying storage which keeps admins with
storage budget issues happy.

> Excessive segmentation of storage is an evil unto itself.
> 
> YMMV, of course.
> 
> An orthoginal example:
> 
> If you give someone six disks and tell them to make an encrypted
> raid6 via cryptsetup and mdadm, at least eight out of ten will
> encrypt the drives and then raid the result. But it's _massivly_
> more efficent to raid the drives and then encrypt the result. Why?

That seems...implausible.  They would need to enter the passphrases six
times too.

> Because writing a block with the latter involves only one block
> being encrypted/decrypted. The former, if the raid is fine involves
> several encryptions/decryptions and _many_ if the raid is degraded.

It would be the correct answer if you needed to keep the structure of
the storage array secret...or if you wanted to use btrfs to implement
the RAID layer, and needed the encrypted layer to be divided along
the same boundaries as the physical layer.

> The above is a mental constraint, a mistake, that is all to common
> because people expect encrytion to be "better" the closer you get to
> the spinning rust.
> 
> So too people expect that segmentation is somehow better if it most
> closely matches the abstract groupings (like per user) but in
> practical terms it is better matched to the modality, where, for
> instance, all users are one kind of thing, while all data stores are
> another kind of thing.
> 
> We were just talking about putting all your VMs and larger NOCOW
> files into a separate subvolume/domain because of their radically
> different write behaviors. Thats a sterling reason to subdivide the
> storage. So is / vs. /var vs. /home as three different domains with
> radically different update profiles.
> 
> So while the natural impulse is to give each user its own subvolume
> it's not likely to be that great an idea in practice because...
> um... 47,000 snapshots dude, and so on.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]