Multiple btrfs-cleaner threads per volume

Multiple btrfs-cleaner threads per volume

[Martin Raiber]

Hi,

snapshot cleanup is a little slow in my case (50TB volume). Would it
help to have multiple btrfs-cleaner threads? The block layer underneath
would have higher throughput with more simultaneous read/write requests.

Regards,
Martin Raiber

Re: Multiple btrfs-cleaner threads per volume

[Austin S. Hemmelgarn]

On 2017-11-02 11:02, Martin Raiber wrote:
> Hi,
> 
> snapshot cleanup is a little slow in my case (50TB volume). Would it
> help to have multiple btrfs-cleaner threads? The block layer underneath
> would have higher throughput with more simultaneous read/write requests.
I think a bigger impact would be proper parallelization of IO requests. 
Right now, writes are serialized (they first get sent to one device, 
then the next, then the next, until they've been sent to all devices), 
and reads aren't inherently load-balanced across devices (ideally, other 
things than load, like where the last read came from on a rotational 
device, would be factored in, but even just regular load balancing would 
be an improvement right now).  As a result, multi-device BTRFS volumes 
in general don't perform as well as possible, which is then compounded 
by other issues (such as snapshot cleanup being somewhat expensive).

Re: Multiple btrfs-cleaner threads per volume

[Hans van Kranenburg]

Hi Martin,

On 11/02/2017 04:02 PM, Martin Raiber wrote:
> 
> snapshot cleanup is a little slow in my case (50TB volume). Would it
> help to have multiple btrfs-cleaner threads? The block layer underneath
> would have higher throughput with more simultaneous read/write requests.

Just curious:
* How many subvolumes/snapshots are you removing, and what's the
complexity level (like, how many other subvolumes/snapshots reference
the same data extents?)
* Do you see a lot of cpu usage, or mainly a lot of disk I/O? If it's
disk IO, is it mainly random read IO, or is it a lot of write traffic?
* What mount options are you running with (from /proc/mounts)?

-- 
Hans van Kranenburg

Re: Multiple btrfs-cleaner threads per volume

[Martin Raiber]

On 02.11.2017 16:10 Hans van Kranenburg wrote:
> On 11/02/2017 04:02 PM, Martin Raiber wrote:
>> snapshot cleanup is a little slow in my case (50TB volume). Would it
>> help to have multiple btrfs-cleaner threads? The block layer underneath
>> would have higher throughput with more simultaneous read/write requests.
> Just curious:
> * How many subvolumes/snapshots are you removing, and what's the
> complexity level (like, how many other subvolumes/snapshots reference
> the same data extents?)
> * Do you see a lot of cpu usage, or mainly a lot of disk I/O? If it's
> disk IO, is it mainly random read IO, or is it a lot of write traffic?
> * What mount options are you running with (from /proc/mounts)?

It is a single block device, so not a multi-device btrfs, so
optimizations in that area wouldn't help. It is a UrBackup system with
about 200 snapshots per client. 20009 snapshots total. UrBackup reflinks
files between them, but btrfs-cleaner doesn't use much CPU (so it
doesn't seem like the backref walking is the problem). btrfs-cleaner is
probably limited mainly by random read/write IO. The device has a cache,
so parallel accesses would help, as some of them may hit the cache.
Looking at the code it seems easy enough to do. Question is if there are
any obvious reasons why this wouldn't work (like some lock etc.).

Re: Multiple btrfs-cleaner threads per volume

[Hans van Kranenburg]

On 11/02/2017 04:26 PM, Martin Raiber wrote:
> On 02.11.2017 16:10 Hans van Kranenburg wrote:
>> On 11/02/2017 04:02 PM, Martin Raiber wrote:
>>> snapshot cleanup is a little slow in my case (50TB volume). Would it
>>> help to have multiple btrfs-cleaner threads? The block layer underneath
>>> would have higher throughput with more simultaneous read/write requests.
>> Just curious:
>> * How many subvolumes/snapshots are you removing, and what's the
>> complexity level (like, how many other subvolumes/snapshots reference
>> the same data extents?)
>> * Do you see a lot of cpu usage, or mainly a lot of disk I/O? If it's
>> disk IO, is it mainly random read IO, or is it a lot of write traffic?
>> * What mount options are you running with (from /proc/mounts)?

Can you paste the output from /proc/mounts for your filesystem? The
reason I'm asking is that the nossd/ssd/ssd_spread related mount options
can have a huge impact on subvolume removal performance for very large
filesystems, like your 50TB one.

> It is a single block device, so not a multi-device btrfs, so
> optimizations in that area wouldn't help. It is a UrBackup system with
> about 200 snapshots per client. 20009 snapshots total. UrBackup reflinks
> files between them, but btrfs-cleaner doesn't use much CPU (so it
> doesn't seem like the backref walking is the problem). btrfs-cleaner is
> probably limited mainly by random read/write IO.

Do you have some graphs, or iostat output? The question is what the
biggest part of the IO consists of. Is it on 100% random read IO and not
many writes, or is it 100% utilized because of many MiB/s of writes?

> The device has a cache,
> so parallel accesses would help, as some of them may hit the cache.
> Looking at the code it seems easy enough to do. Question is if there are
> any obvious reasons why this wouldn't work (like some lock etc.).

-- 
Hans van Kranenburg