How to force EXT4_MB_GRP_CLEAR_TRIMMED on a live ext4?

How to force EXT4_MB_GRP_CLEAR_TRIMMED on a live ext4?

[Laurent GUERBY]

Hi,

When using fstrim on an ext4 filesystem trim are not issued for
EXT4_MB_GRP_WAS_TRIMMED space which is a useful optimization.

Is there a way to force a complete trim on a mounted ext4 filesystem? 

My (limited) understanding of the code is that
EXT4_MB_GRP_CLEAR_TRIMMED should be called to do so.

My use case is having live migrated a virtual machine root disk from
one storage to another, the target supporting trim, but since fstrim in
the VM post migration does mostly nothing (assumes most space was
trimmed) I cannot release space to the new storage.

I tried mount -o remount but without effect. e2fsprogs don't seem to
have an option/tool to do this either.

I've seen suggestion that rebooting will do the job but the whole point
of live migration is to avoid reboot :).

I did end up creating dummy files to fill the filesystem and then
removing them, but this is far less efficient than what a filesystem
tool could do.

Thanks in advance for your help,

Sincerely,

Laurent

Re: How to force EXT4_MB_GRP_CLEAR_TRIMMED on a live ext4?

[Lukas Czerner]

On Sat, Oct 23, 2021 at 12:24:40PM +0200, Laurent GUERBY wrote:
> Hi,
> 
> When using fstrim on an ext4 filesystem trim are not issued for
> EXT4_MB_GRP_WAS_TRIMMED space which is a useful optimization.
> 
> Is there a way to force a complete trim on a mounted ext4 filesystem? 
> 
> My (limited) understanding of the code is that
> EXT4_MB_GRP_CLEAR_TRIMMED should be called to do so.
> 
> My use case is having live migrated a virtual machine root disk from
> one storage to another, the target supporting trim, but since fstrim in
> the VM post migration does mostly nothing (assumes most space was
> trimmed) I cannot release space to the new storage.

Hi,

interesting. I don't think this scenario was ever considered so no,
there is no way to bypass the optimization other than just unmount and
mount the file system again, which I can understand is inconvenient for
root file system ;)


> 
> I tried mount -o remount but without effect. e2fsprogs don't seem to
> have an option/tool to do this either.
> 
> I've seen suggestion that rebooting will do the job but the whole point
> of live migration is to avoid reboot :).
> 
> I did end up creating dummy files to fill the filesystem and then
> removing them, but this is far less efficient than what a filesystem
> tool could do.

Yeah, that's bad. The information is stored in the buddy cache in memory
and AFAIK is only dropped on unmount. I'll have to think about how to
clear either the cache or selectively just the flag.

What would be more convenient way of doing this for you, -o remount, or
using let's say tune2fs ? I am not promising anything yet, but I'll think
about how to implement it.


Meanwhile other than umount/mount, or actually writing to the dummy files,
you can try to use fallocate to allocate all the remaining space in the
file system and subsequently removing it. That should be more efficient,
but don't forget to sync after remove to make sure the space is released
before you call fstrim.

You could also force fsck on ro file system and use -E discard to trim the
free space but I can't say I recommend it.

-Lukas

> 
> Thanks in advance for your help,
> 
> Sincerely,
> 
> Laurent
> 

Re: How to force EXT4_MB_GRP_CLEAR_TRIMMED on a live ext4?

[Laurent GUERBY]

On Mon, 2021-10-25 at 11:42 +0200, Lukas Czerner wrote:
> On Sat, Oct 23, 2021 at 12:24:40PM +0200, Laurent GUERBY wrote:
> > 
> > I did end up creating dummy files to fill the filesystem and then
> > removing them, but this is far less efficient than what a
> > filesystem
> > tool could do.
> 
> Yeah, that's bad. The information is stored in the buddy cache in
> memory
> and AFAIK is only dropped on unmount. I'll have to think about how to
> clear either the cache or selectively just the flag.
> 
> What would be more convenient way of doing this for you, -o remount,
> or
> using let's say tune2fs ? I am not promising anything yet, but I'll
> think
> about how to implement it.
> 
> 
> Meanwhile other than umount/mount, or actually writing to the dummy
> files,
> you can try to use fallocate to allocate all the remaining space in
> the
> file system and subsequently removing it. That should be more
> efficient,
> but don't forget to sync after remove to make sure the space is
> released
> before you call fstrim.

Thanks for the advice on fallocate! It does work and is very fast.

I would prefer a specific tune2fs as remount forcing this TRIM cache
clearing behaviour might be unwanted.

> You could also force fsck on ro file system and use -E discard to
> trim the
> free space but I can't say I recommend it.

Thanks again for your help,

Sincerely,

Laurent

Re: How to force EXT4_MB_GRP_CLEAR_TRIMMED on a live ext4?

[Andreas Dilger]

On Oct 25, 2021, at 09:29, Laurent GUERBY <laurent@guerby.net> wrote:
> 
> On Mon, 2021-10-25 at 11:42 +0200, Lukas Czerner wrote:
>>> On Sat, Oct 23, 2021 at 12:24:40PM +0200, Laurent GUERBY wrote:
>>> 
>>> I did end up creating dummy files to fill the filesystem and then
>>> removing them, but this is far less efficient than what a
>>> filesystem
>>> tool could do.
>> 
>> Yeah, that's bad. The information is stored in the buddy cache in
>> memory
>> and AFAIK is only dropped on unmount. I'll have to think about how to
>> clear either the cache or selectively just the flag.
>> 
>> What would be more convenient way of doing this for you, -o remount,
>> or
>> using let's say tune2fs ? I am not promising anything yet, but I'll
>> think
>> about how to implement it.
>> 
>> 
>> Meanwhile other than umount/mount, or actually writing to the dummy
>> files,
>> you can try to use fallocate to allocate all the remaining space in
>> the
>> file system and subsequently removing it. That should be more
>> efficient,
>> but don't forget to sync after remove to make sure the space is
>> released
>> before you call fstrim.
> 
> Thanks for the advice on fallocate! It does work and is very fast.

It would be enough to allocate and free a block in each group (128MB)
of the filesystem. That can't be controlled directly by fallocate(),
but indirectly via the "goal inode", but if fallocate() if all free space is fast
enough it may not be worth the effort. 

> I would prefer a specific tune2fs as remount forcing this TRIM cache
> clearing behaviour might be unwanted.
> 
>> You could also force fsck on ro file system and use -E discard to
>> trim the
>> free space but I can't say I recommend it.

This would have a danger to corrupt the mounted filesystem, so should
not be allowed when doing a read-only e2fsck. The bitmaps that e2fsck
is using for the trim may be stale if the filesystem is modified since the
start of the run, which may be a long time in some cases (minutes, hours).

Cheers, Andreas