Re: btrfs-cleaner / snapshot performance analysis

["Ellis H. Wilson III" <ellisw@panasas.com>]

On 02/12/2018 11:02 AM, Austin S. Hemmelgarn wrote:
>> I will look into that if using built-in group capacity functionality 
>> proves to be truly untenable.  Thanks!
> As a general rule, unless you really need to actively prevent a 
> subvolume from exceeding it's quota, this will generally be more 
> reliable and have much less performance impact than using qgroups.

Ok ok :).  I will plan to go this route, but since I'll want to 
benchmark it either way, I'll include qgroups enabled in the benchmark 
and will report back.

> With qgroups involved, I really can't say for certain, as I've never 
> done much with them myself, but based on my understanding of how it all 
> works, I would expect multiple subvolumes with a small number of 
> snapshots each to not have as many performance issues as a single 
> subvolume with the same total number of snapshots.

Glad to hear that.  That was my expectation as well.

> BTRFS in general works fine at that scale, dependent of course on the 
> level of concurrent access you need to support.  Each tree update needs 
> to lock a bunch of things in the tree itself, and having large numbers 
> of clients writing to the same set of files concurrently can cause lock 
> contention issues because of this, especially if all of them are calling 
> fsync() or fdatasync() regularly.  These issues can be mitigated by 
> segregating workloads into their own subvolumes (each subvolume is a 
> mostly independent filesystem tree), but it sounds like you're already 
> doing that, so I don't think that would be an issue for you.
Hmm...I'll think harder about this.  There is potential for us to 
artificially divide access to files across subvolumes automatically 
because of the way we are using BTRFS as a backing store for our 
parallel file system.  So far even with around 1000 threads across about 
10 machines accessing BTRFS via our parallel filesystem over the wire 
we've not seen issues, but if we do I have some ways out I've not 
explored yet.  Thanks!

> Now, there are some other odd theoretical cases that may cause issues 
> when dealing with really big filesystems, but they're either really 
> specific edge cases (for example, starting with a really small 
> filesystem and gradually scaling it up in size as it gets full) or 
> happen at scales far larger than what you're talking about (on the order 
> of at least double digit petabyte scale).

Yea, our use case will be in the tens of TB to hundreds of TB for the 
foreseeable future, so I'm glad to hear this is relatively standard. 
That was my read of the situation as well.

Thanks!

ellis