no memory is freed after snapshots are deleted

no memory is freed after snapshots are deleted

[telsch]

Dear devs,

after my root partiton was full, i deleted the last monthly snapshots. however, no memory was freed.
so far rebalancing helped:

	btrfs balance start -v -musage=0 /
	btrfs balance start -v -dusage=0 /

i have deleted all snapshots, but no memory is being freed this time.

du -hcsx /
16G     /
16G     total

btrfs-progs v5.10.1
Linux arch-server 5.10.21-1-lts #1 SMP Sun, 07 Mar 2021 11:56:15 +0000 x86_64 GNU/Linux

btrfs fi show /
Label: none  uuid: 3d242677-6a15-4ce7-853a-5c82f0427769
        Total devices 1 FS bytes used 37.24GiB
        devid    1 size 39.95GiB used 39.95GiB path /dev/mapper/root

btrfs fi df /
Data, single: total=36.45GiB, used=35.86GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.72GiB, used=1.38GiB
GlobalReserve, single: total=215.94MiB, used=0.00B


any ideas how to solve this without recreating filesystem?

thx!

Re: no memory is freed after snapshots are deleted

[Graham Cobb]

On 10/03/2021 12:07, telsch wrote:
> Dear devs,
> 
> after my root partiton was full, i deleted the last monthly snapshots. however, no memory was freed.
> so far rebalancing helped:
> 
> 	btrfs balance start -v -musage=0 /
> 	btrfs balance start -v -dusage=0 /
> 
> i have deleted all snapshots, but no memory is being freed this time.

Don't forget that, in general, deleting a snapshot does nothing - if the
original files are still there (or any other snapshots of the same files
are still there). In my experience, if you *really* need space urgently
you are best of starting with deleting some big files *and* all the
snapshots containing them, rather than starting by deleting snapshots.

If you are doing balances with low space, I find it useful to watch
dmesg to see if the balance is hitting problems finding space to even
free things up.

However, one big advantage of btrfs is that you can easily temporarily
add a small amount of space while you sort things out. Just plug in a
USB memory stick, and add it to the filesystem using 'btrfs device add'.

I don't recommend leaving it as part of the filesystem for long - it is
too easy for the memory stick to fail, or for you remove it forgetting
how important it is, but it can be useful when you are trying to do
things like remove snapshots and files or run balance. Don't forget to
use btrfs device remove to remove it - not just unplugging it!

Aw: Re: no memory is freed after snapshots are deleted

[telsch]

> Don't forget that, in general, deleting a snapshot does nothing - if the
> original files are still there (or any other snapshots of the same files
> are still there). In my experience, if you *really* need space urgently
> you are best of starting with deleting some big files *and* all the
> snapshots containing them, rather than starting by deleting snapshots.
>
> If you are doing balances with low space, I find it useful to watch
> dmesg to see if the balance is hitting problems finding space to even
> free things up.
>
> However, one big advantage of btrfs is that you can easily temporarily
> add a small amount of space while you sort things out. Just plug in a
> USB memory stick, and add it to the filesystem using 'btrfs device add'.
>
> I don't recommend leaving it as part of the filesystem for long - it is
> too easy for the memory stick to fail, or for you remove it forgetting
> how important it is, but it can be useful when you are trying to do
> things like remove snapshots and files or run balance. Don't forget to
> use btrfs device remove to remove it - not just unplugging it!

Yes that's why i deleted all snapshots.
I had also added a ramdisk to work around the low memory problem during balance but without success.

Any other ideas to fix this?

Re: Aw: Re: no memory is freed after snapshots are deleted

[Remi Gauvin]

On 2021-03-10 10:49 a.m., telsch wrote:

> 
> Any other ideas to fix this?
> 


We can check that there are, in fact, no unexpected subvolumes.

btrfs sub list /

In particular, I wonder if you have subvolumes/snapshots hidden behind
the mounted subvolume.

Also, I don't think it's really the case here with only 16GB reported by
du,, but do you have any large, heavily fragmented files, such as VM
virtual disks?  Without defragmentation or compression, I've seen those
consume more than twice their reported file size on btrfs.

Re: Re: Re: no memory is freed after snapshots are deleted

[telsch]

> We can check that there are, in fact, no unexpected subvolumes.
>
> btrfs sub list /
>
> In particular, I wonder if you have subvolumes/snapshots hidden behind
> the mounted subvolume.
>
> Also, I don't think it's really the case here with only 16GB reported by
> du,, but do you have any large, heavily fragmented files, such as VM
> virtual disks?  Without defragmentation or compression, I've seen those
> consume more than twice their reported file size on btrfs.

thanks for the hint! i have only considered snapshots but no subvolumes. since them considered as file system boundary and i had set the option -x, --one-file-system, it is clear where I waste my storage space.
my fault - thank you!

Re: no memory is freed after snapshots are deleted

[Zygo Blaxell]

On Wed, Mar 10, 2021 at 01:07:47PM +0100, telsch wrote:
> Dear devs,
> 
> after my root partiton was full, i deleted the last monthly snapshots. however, no memory was freed.
> so far rebalancing helped:
> 
> 	btrfs balance start -v -musage=0 /
> 	btrfs balance start -v -dusage=0 /
> 
> i have deleted all snapshots, but no memory is being freed this time.
> 
> du -hcsx /
> 16G     /
> 16G     total
> 
> btrfs-progs v5.10.1
> Linux arch-server 5.10.21-1-lts #1 SMP Sun, 07 Mar 2021 11:56:15 +0000 x86_64 GNU/Linux
> 
> btrfs fi show /
> Label: none  uuid: 3d242677-6a15-4ce7-853a-5c82f0427769
>         Total devices 1 FS bytes used 37.24GiB
>         devid    1 size 39.95GiB used 39.95GiB path /dev/mapper/root
> 
> btrfs fi df /
> Data, single: total=36.45GiB, used=35.86GiB
> System, DUP: total=32.00MiB, used=16.00KiB
> Metadata, DUP: total=1.72GiB, used=1.38GiB
> GlobalReserve, single: total=215.94MiB, used=0.00B

Check

	btrfs sub list -d /

to make sure there are no deleted snapshots pending.  If a snapshot
has a single open file on it (or a bind mount or similar equivalent to
an open file), the cleaner will not delete it until the last open file
descriptor is closed.  You'll have to find the process with the open
file and convince it to close the file (or kill the process).  This can
be tricky since lsof and fuser are not able to identify open files on
deleted snapshots, so these tools are not usable.  Rebooting will force
all the files to be closed.

You can also use 'compsize' and measure the difference in size
between 'referenced' and 'usage' columns.  If referenced is below
usage then you have some big extents with small references (this can
be caused by prealloc and some database write patterns, or by using
a non-btrfs-extent-aware dedupe tool).  Defrag will get rid of those
if you have no snapshots.  You will have to start at the top of the
filesystem tree and work your way down until you find the offending files,
as compsize can only give you a summary.

> any ideas how to solve this without recreating filesystem?
> 
> thx!