Scaling to 100k+ snapshots/subvolumes

Scaling to 100k+ snapshots/subvolumes

[Tristan Zajonc]

Hi,

In an early thread Duncan mentioned that btrfs does not scale well in
the number of subvolumes (including snapshots).  He recommended
keeping the total number under 1000.  I just wanted to understand this
limitation further.  Is this something that has been resolved or will
be resolved in the future or is it something inherent to the design of
btrfs?

We have an application that could easily generate 100k-1M snapshots
and 10s of thousands of subvolumes.  We use snapshots to track very
fine-grained filesystem histories and subvolumes to enforce quotas
across a large number of distinct projects.

Thanks,
Tristan

Duncan >
http://permalink.gmane.org/gmane.comp.file-systems.btrfs/43910

The question of number of subvolumes normally occurs in the context of
snapshots, since snapshots are a special kind of subvolume.  Ideally,
you'll want to keep the total number of subvolumes (including snapshots)
to under 1000, with the number of snapshots of any single subvolume
limited to 250-ish (say under 300).  However, even just four subvolumes
being snapshotted to this level will reach the thousand, and 2000-3000
total isn't /too/ bad as long as it's no more than 250-300 snapshots per
subvolume.  But DEFINITELY try to keep it under 3000, and preferably
under 2000, as the scaling really does start to go badly as the number of
subvolumes increases beyond that.  If you're dealing with 10k subvolumes/
snapshots, that's too many and you ARE likely to find yourself with
problems.

(With something like snapper, configuring it for say half-hour or hourly
snapshots at the shortest time, with twice-daily or daily being more
reasonable in many circumstances, and then thinning it down to say daily
after a few days and weekly after four weeks, goes quite a long way
toward reducing the number of snapshots per subvolume.  Keeping it near
250-ish per subvolume is WELL within reason, and considering that a month
or a year out, you're not likely to /care/ whether it's this hour or
that, just pick a day or a week and if it's not what you want, go back or
forward a day or a week, is actually likely to be more practical than
having hundreds of half-hourly snapshots a year old to choose from.  And
250-ish snapshots per subvolume really does turn out to be VERY
reasonable, provided you're doing reasonable thinning.)

Re: Scaling to 100k+ snapshots/subvolumes

[Michael Darling]

If someone can answer Tristan's question, can they also add in if
large volumes of frequently created and destroyed snapshots/subvolumes
will cause issues?  Or, if they're deleted quickly after being made,
is it just the number that exists at any given time that matters?
(Building source in chroot subvolumes with its "own" o/s install, as
in Arch's devtools scripts, if used to create *separate* subvolumes
under each source build, rather than a global shared one.)

On Tue, Aug 11, 2015 at 6:33 PM, Tristan Zajonc <tristan@sense.io> wrote:
>
> Hi,
>
> In an early thread Duncan mentioned that btrfs does not scale well in
> the number of subvolumes (including snapshots).  He recommended
> keeping the total number under 1000.  I just wanted to understand this
> limitation further.  Is this something that has been resolved or will
> be resolved in the future or is it something inherent to the design of
> btrfs?
>
> We have an application that could easily generate 100k-1M snapshots
> and 10s of thousands of subvolumes.  We use snapshots to track very
> fine-grained filesystem histories and subvolumes to enforce quotas
> across a large number of distinct projects.
>
> Thanks,
> Tristan
>
> Duncan >
> http://permalink.gmane.org/gmane.comp.file-systems.btrfs/43910
>
> The question of number of subvolumes normally occurs in the context of
> snapshots, since snapshots are a special kind of subvolume.  Ideally,
> you'll want to keep the total number of subvolumes (including snapshots)
> to under 1000, with the number of snapshots of any single subvolume
> limited to 250-ish (say under 300).  However, even just four subvolumes
> being snapshotted to this level will reach the thousand, and 2000-3000
> total isn't /too/ bad as long as it's no more than 250-300 snapshots per
> subvolume.  But DEFINITELY try to keep it under 3000, and preferably
> under 2000, as the scaling really does start to go badly as the number of
> subvolumes increases beyond that.  If you're dealing with 10k subvolumes/
> snapshots, that's too many and you ARE likely to find yourself with
> problems.
>
> (With something like snapper, configuring it for say half-hour or hourly
> snapshots at the shortest time, with twice-daily or daily being more
> reasonable in many circumstances, and then thinning it down to say daily
> after a few days and weekly after four weeks, goes quite a long way
> toward reducing the number of snapshots per subvolume.  Keeping it near
> 250-ish per subvolume is WELL within reason, and considering that a month
> or a year out, you're not likely to /care/ whether it's this hour or
> that, just pick a day or a week and if it's not what you want, go back or
> forward a day or a week, is actually likely to be more practical than
> having hundreds of half-hourly snapshots a year old to choose from.  And
> 250-ish snapshots per subvolume really does turn out to be VERY
> reasonable, provided you're doing reasonable thinning.)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Scaling to 100k+ snapshots/subvolumes

[Duncan]

Tristan Zajonc posted on Tue, 11 Aug 2015 11:33:45 -0700 as excerpted:

> In an early thread Duncan mentioned that btrfs does not scale well in
> the number of subvolumes (including snapshots).  He recommended keeping
> the total number under 1000.  I just wanted to understand this
> limitation further.  Is this something that has been resolved or will be
> resolved in the future or is it something inherent to the design of
> btrfs?

It is not resolved yet, but it's definitely on the radar.  I don't 
personally understand the details well enough to know if the problem is 
inherent to btrfs, or if some optimized rewrite down the road is likely 
to at least yield linear scaling.

On the practical side, one related thing I do know is that this is the 
reason snapshot-aware-defrag was disabled a few kernel cycles after being 
introduced -- it simply didn't scale, and the thought was, better a 
defrag that at least worked for the snapshot you pointed it at, even at 
the cost of increasing usage due to COW if other snapshots pointed at the 
same file extents, than a defrag that basically didn't work at all.

But the intent remains to at least get scaling working well enough to 
have snapshot-aware-defrag again.  So when snapshot-aware-defrag is 
enabled again, that's your clue that things should be scaling at least 
/reasonably/ well, and it's time to reexamine the situation.  Until then, 
I'd not recommend trying it.

> We have an application that could easily generate 100k-1M snapshots and
> 10s of thousands of subvolumes.  We use snapshots to track very
> fine-grained filesystem histories and subvolumes to enforce quotas
> across a large number of distinct projects.

Btrfs quotas... have been another sticky wicket on btrfs, both as earlier 
the code was simply broken (tho AFAIK that's fixed in general, now), and 
because due to the way it works, quota tracking multiplies the scaling 
issues several fold (certainly in the original code form).  AFAIK they've 
actually done at least two partial rewrites, so are on the third quota 
code version now.  The third-try quota code is fresh enough I don't think 
people know yet how well it's going to perform in deployment.

As a result of that quota code history, my recommendation has been that 
unless you're deliberately testing it, if you don't need quotas, keep it 
turned off on btrfs and avoid the issues it has been known, at least 
historically, to trigger.  As btrfs quota code is demonstrably not yet 
stable and reliable enough to use, if you *do* actually depend on quotas, 
you should definitely be on some other filesystem where the quota code is 
well tested and known to be dependable, as that simply doesn't describe 
btrfs quota code at this point.

But there's actually some pretty big effort going into the quota code at 
the moment, this the fact that we're on the third version now, and 
they're definitely planning on it actually working, or they'd not be 
sinking the effort into it that they are.

And as I said, the quota code was multiplying the scaling issues several 
fold, so getting quotas actually working well is a big part of getting 
the scaling issues fixed as well.

But beyond that; in particular, whether it's ever likely to work at the 
scales you mention above, is something you'd have to ask the devs, as I'm 
just a list regular and btrfs-using admin, with a use-case that doesn't 
directly involve either quotas or subvolumes/snapshotting to any great 
degree.  So while I can point to the current situation and the current 
trend and work areas, I have effectively no idea if scaling to the 
numbers you mention above is even technically possible, or not.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Scaling to 100k+ snapshots/subvolumes

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3402 bytes --]

On Tue, Aug 11, 2015 at 11:33:45AM -0700, Tristan Zajonc wrote:
> In an early thread Duncan mentioned that btrfs does not scale well in
> the number of subvolumes (including snapshots).  He recommended
> keeping the total number under 1000.  I just wanted to understand this
> limitation further.  Is this something that has been resolved or will
> be resolved in the future or is it something inherent to the design of
> btrfs?

   We've already discussed this on IRC yesterday, but I was reminded
this morning of one specific instance of poor scalability with lots of
snapshots, which is that balancing metadata scales poorly as the
number of copies of the same data increases. (So it's OK to have lots
of unrelated subvolumes, but many subvolumes that are snapshots of the
same thing would br problematic).

   I don't know what the exact scaling is, but it's superlinear, and
may be as bad as O(n^2).

> We have an application that could easily generate 100k-1M snapshots
> and 10s of thousands of subvolumes.  We use snapshots to track very
> fine-grained filesystem histories and subvolumes to enforce quotas
> across a large number of distinct projects.

   1e4 subvolumes and 1e6 snapshots will probably be OK, although
slow, when balancing. That's only 100 snapshots per subvolume, and
I've got that kind of scale here on my home machine -- it's very slow
(takes a couple of hours for some of the metadata chunks), but it's
not unusable.

   Hugo.

> Thanks,
> Tristan
> 
> Duncan >
> http://permalink.gmane.org/gmane.comp.file-systems.btrfs/43910
> 
> The question of number of subvolumes normally occurs in the context of
> snapshots, since snapshots are a special kind of subvolume.  Ideally,
> you'll want to keep the total number of subvolumes (including snapshots)
> to under 1000, with the number of snapshots of any single subvolume
> limited to 250-ish (say under 300).  However, even just four subvolumes
> being snapshotted to this level will reach the thousand, and 2000-3000
> total isn't /too/ bad as long as it's no more than 250-300 snapshots per
> subvolume.  But DEFINITELY try to keep it under 3000, and preferably
> under 2000, as the scaling really does start to go badly as the number of
> subvolumes increases beyond that.  If you're dealing with 10k subvolumes/
> snapshots, that's too many and you ARE likely to find yourself with
> problems.
> 
> (With something like snapper, configuring it for say half-hour or hourly
> snapshots at the shortest time, with twice-daily or daily being more
> reasonable in many circumstances, and then thinning it down to say daily
> after a few days and weekly after four weeks, goes quite a long way
> toward reducing the number of snapshots per subvolume.  Keeping it near
> 250-ish per subvolume is WELL within reason, and considering that a month
> or a year out, you're not likely to /care/ whether it's this hour or
> that, just pick a day or a week and if it's not what you want, go back or
> forward a day or a week, is actually likely to be more practical than
> having hundreds of half-hourly snapshots a year old to choose from.  And
> 250-ish snapshots per subvolume really does turn out to be VERY
> reasonable, provided you're doing reasonable thinning.)

-- 
Hugo Mills             | Some days, it's just not worth gnawing through the
hugo@... carfax.org.uk | straps
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Scaling to 100k+ snapshots/subvolumes

[Tristan Zajonc]

Thanks, this is helpful.

We are primarily scaling the number of snapshots. Unfortunately these
snapshots typically have very minor changes compared their parent, so
this sounds potentially problematic.

It sounds like I will need to do some testing of both snapshots and
quotas to determine scalability for our exact uses cases.  We can
control which tools / actions we take pretty carefully but do require
significant scalability along both the snapshot and quota subvolume
dimensions.

Any lessons here are helpful in terms of knowing what to test.

Thanks,
Tristan
---
Tristan Zajonc, PhD | Cofounder/CEO at Sense | https://sense.io |
@tristanzajonc | m: 617-953-2204


On Wed, Aug 12, 2015 at 12:31 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Tue, Aug 11, 2015 at 11:33:45AM -0700, Tristan Zajonc wrote:
>> In an early thread Duncan mentioned that btrfs does not scale well in
>> the number of subvolumes (including snapshots).  He recommended
>> keeping the total number under 1000.  I just wanted to understand this
>> limitation further.  Is this something that has been resolved or will
>> be resolved in the future or is it something inherent to the design of
>> btrfs?
>
>    We've already discussed this on IRC yesterday, but I was reminded
> this morning of one specific instance of poor scalability with lots of
> snapshots, which is that balancing metadata scales poorly as the
> number of copies of the same data increases. (So it's OK to have lots
> of unrelated subvolumes, but many subvolumes that are snapshots of the
> same thing would br problematic).
>
>    I don't know what the exact scaling is, but it's superlinear, and
> may be as bad as O(n^2).
>
>> We have an application that could easily generate 100k-1M snapshots
>> and 10s of thousands of subvolumes.  We use snapshots to track very
>> fine-grained filesystem histories and subvolumes to enforce quotas
>> across a large number of distinct projects.
>
>    1e4 subvolumes and 1e6 snapshots will probably be OK, although
> slow, when balancing. That's only 100 snapshots per subvolume, and
> I've got that kind of scale here on my home machine -- it's very slow
> (takes a couple of hours for some of the metadata chunks), but it's
> not unusable.
>
>    Hugo.
>
>> Thanks,
>> Tristan
>>
>> Duncan >
>> http://permalink.gmane.org/gmane.comp.file-systems.btrfs/43910
>>
>> The question of number of subvolumes normally occurs in the context of
>> snapshots, since snapshots are a special kind of subvolume.  Ideally,
>> you'll want to keep the total number of subvolumes (including snapshots)
>> to under 1000, with the number of snapshots of any single subvolume
>> limited to 250-ish (say under 300).  However, even just four subvolumes
>> being snapshotted to this level will reach the thousand, and 2000-3000
>> total isn't /too/ bad as long as it's no more than 250-300 snapshots per
>> subvolume.  But DEFINITELY try to keep it under 3000, and preferably
>> under 2000, as the scaling really does start to go badly as the number of
>> subvolumes increases beyond that.  If you're dealing with 10k subvolumes/
>> snapshots, that's too many and you ARE likely to find yourself with
>> problems.
>>
>> (With something like snapper, configuring it for say half-hour or hourly
>> snapshots at the shortest time, with twice-daily or daily being more
>> reasonable in many circumstances, and then thinning it down to say daily
>> after a few days and weekly after four weeks, goes quite a long way
>> toward reducing the number of snapshots per subvolume.  Keeping it near
>> 250-ish per subvolume is WELL within reason, and considering that a month
>> or a year out, you're not likely to /care/ whether it's this hour or
>> that, just pick a day or a week and if it's not what you want, go back or
>> forward a day or a week, is actually likely to be more practical than
>> having hundreds of half-hourly snapshots a year old to choose from.  And
>> 250-ish snapshots per subvolume really does turn out to be VERY
>> reasonable, provided you're doing reasonable thinning.)
>
> --
> Hugo Mills             | Some days, it's just not worth gnawing through the
> hugo@... carfax.org.uk | straps
> http://carfax.org.uk/  |
> PGP: E2AB1DE4          |