Re: New file system with same issue (was: Again, no space left on device while rebalancing and recipe doesnt work)

[Andrew Vaughan <andrewjvaughan@gmail.com>]

Hi Marc

On 13 March 2016 at 22:58, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> Hi,
>
> On Sat, Mar 05, 2016 at 12:34:09PM -0700, Chris Murphy wrote:
>> The alternative if this can't be fixed, is to recreate the filesystem
>> because there's no practical way yet to migrate so many snapshots to a
>> new file system.
>
> I recreated the file system on March 7, with 200 GiB in size, using
> btrfs-tools 4.4. The snapshot-taking process has been running since
> then, but I also regularly cleaned up. The number of snapshots on the
> new filesystem has never exceeded 1000, with the current count being
> at 148.
>
<snip>


I'm not a dev, so I'll just thouw out a random, and possibly naive idea.

How much i/o load is this filesystem under?
What type of access pattern(s), how frequent and large are the changes?

Are you still making snapshots every 10m?
How often do you delete old snapshots?  Also every 10m, or do you
delete them in batchs every hour or so?

How long does "btrfs subvolume delete -c <one_snapshot>" take?
What does "time btrfs subvolume delete -C <one_snapshot> ; time btrfs
subvolume sync <mount_point>" print ?

The reason for asking is that even on a lightly loaded filesystem I
have seen btrfs subvolume delete take more than 30 seconds.  On a more
heavily load filesystem  I have seen 5+ minutes before btrfs subvolume
delete had finished.

If you have a high enough i/o load, plus large enough changes per
snapshot, it might be possible to get btrfs into a situation were it
never actually finishes cleaning up deleted snapshots.  (I'm also not
sure what happens if you shutdown or unmount whilst btrfs is still
cleaning up, but I expect the devs thought of that).

Andrew