Re: [RFC PATCH 0/6] btrfs: dynamic and periodic block_group reclaim

[David Sterba <dsterba@suse.cz>]

On Tue, Feb 06, 2024 at 02:07:52PM -0800, Boris Burkov wrote:
> On Tue, Feb 06, 2024 at 03:55:24PM +0100, David Sterba wrote:
> > On Fri, Feb 02, 2024 at 03:12:42PM -0800, Boris Burkov wrote:
> > A common workload on distros is regular system update (rolling distro)
> > with snapshots (snapper) and cleanup. This can create a lot of under
> > used block groups, both data and metadata. Reclaiming that preriodically
> > was one of the ground ideas for the btrfsmaintenance project.
> 
> I believe this is pretty similar to my workload 2 in spirit, except I
> haven't done much with snapshots. I would love to run this workload so
> I'll try to set it up with a VM. If you have a script for it already, or
> even tips for setting it up, I would be quite grateful :)
> 
> I think that the "lots of random deletes leave empty block groups"
> workload is the most interesting one in general for reclaim, and I
> think it's cool that it happens in the real world :)

As a simulation of that I'm using git based workload that randomly
checks out commits and does a snapshot. A once working script is
herehttps://github.com/kdave/testunion/blob/master/test-snapgit/startme
(I maybe have some updates but I'd have to find the most recent version).

The used git repo should provide large files too so it's closer to what
eg. rpm does.

> > The exact parameters of auto reclaim also depend on the storage type, an
> > NVMe would be probably fine with any amount of data, HDD not so much.
> 
> Good point, have only tested on NVMe. Definitely needs to be tunable to
> not abuse HDDs.

I think we'll need an automatic classification of devices, now it's
third type that I know could use it (raid mirror balancing, checksum
offload and this one).

There's more to reply to, I'll continue on another day.