A possible way to reduce free space fragmentation?

A possible way to reduce free space fragmentation?

[Artem S. Tashkinov]

Hello,

ext4 has no free space defragmentation and at most you can use e4defrag
to defragment individual files. I now have a 24GB ext4 filesystem that
has only 7GB of space occupied however it has small files scattered all
over it and now bigger files occupy more than one extent and I cannot
reduce fragmentation to zero. One way to approach that would be to
shrink the volume and then defragment it but that will involve a ton of
disk writes and unnecessary tear and wear. Is it possible to modify the
e4degrag utility to move small defragmented files, so that they were
placed consecutively instead of being randomly spread all over the disk?

Regards,
Artem

Re: A possible way to reduce free space fragmentation?

[Andreas Dilger]

It should be possible to run "find $DIR -type f -size -1M | xargs e4defrag" to only defragment files below 1MB (or whatever you consider "small").

However, I don't recall if e4defrag will move a file if the new file has the same number of fragments as the original (presumably both "1") or leave it in place. That would be possible add an option to change. 

Alternately, just run the "find" above to find small files and then "cp $F $F.tmp && mv $F.tmp $F" to rewrite those files into new blocks, and hope mballoc will move them to a better location.

Cheers, Andreas

> On Jan 31, 2025, at 14:02, Artem S. Tashkinov <aros@gmx.com> wrote:
> 
> Hello,
> 
> ext4 has no free space defragmentation and at most you can use e4defrag
> to defragment individual files. I now have a 24GB ext4 filesystem that
> has only 7GB of space occupied however it has small files scattered all
> over it and now bigger files occupy more than one extent and I cannot
> reduce fragmentation to zero. One way to approach that would be to
> shrink the volume and then defragment it but that will involve a ton of
> disk writes and unnecessary tear and wear. Is it possible to modify the
> e4degrag utility to move small defragmented files, so that they were
> placed consecutively instead of being randomly spread all over the disk?
> 
> Regards,
> Artem
> 

Re: A possible way to reduce free space fragmentation?

[Artem S. Tashkinov]

On 2/1/25 3:38 PM, Andreas Dilger wrote:
> It should be possible to run "find $DIR -type f -size -1M | xargs e4defrag" to only defragment files below 1MB (or whatever you consider "small").

I have smaller files completely defragmented already.

The issue is a dozen of 50-250MB files that span multiple extents (up to
30).

>
> However, I don't recall if e4defrag will move a file if the new file has the same number of fragments as the original (presumably both "1") or leave it in place. That would be possible add an option to change.
>
> Alternately, just run the "find" above to find small files and then "cp $F $F.tmp && mv $F.tmp $F" to rewrite those files into new blocks, and hope mballoc will move them to a better location.

cp doesn't even support posix_fallocate(), rsync does but this process
will be a complete guesswork as I have no idea which files are worth
moving and which are not.

Considering there are holes that can include files in their entity, this
must be done by something that knows what it's doing.

Best regards,
Artem

>
> Cheers, Andreas
>
>> On Jan 31, 2025, at 14:02, Artem S. Tashkinov <aros@gmx.com> wrote:
>>
>> Hello,
>>
>> ext4 has no free space defragmentation and at most you can use e4defrag
>> to defragment individual files. I now have a 24GB ext4 filesystem that
>> has only 7GB of space occupied however it has small files scattered all
>> over it and now bigger files occupy more than one extent and I cannot
>> reduce fragmentation to zero. One way to approach that would be to
>> shrink the volume and then defragment it but that will involve a ton of
>> disk writes and unnecessary tear and wear. Is it possible to modify the
>> e4degrag utility to move small defragmented files, so that they were
>> placed consecutively instead of being randomly spread all over the disk?
>>
>> Regards,
>> Artem
>>

Re: A possible way to reduce free space fragmentation?

[Theodore Ts'o]

On Sat, Feb 01, 2025 at 03:48:16PM +0000, Artem S. Tashkinov wrote:
> 
> 
> On 2/1/25 3:38 PM, Andreas Dilger wrote:
> > It should be possible to run "find $DIR -type f -size -1M | xargs e4defrag" to only defragment files below 1MB (or whatever you consider "small").
> 
> I have smaller files completely defragmented already.
> 
> The issue is a dozen of 50-250MB files that span multiple extents (up to
> 30).

How big are the extents?  If you are performing large sequential
reads, a few seeks every few megabytes is really not a big deal from a
performance perspective, and it's certainly not worth the huge amount
of time that a perfect defragmentation would take (since that would
require moving smaller files out of the way to free up enough
contiguous space for a big file).

This is why Windows defraggers have mostly fallen out of faver, and
why no one has really found it worthwhile to invest more effort in
improving e4defrag (either the userspace program or the underlying
kernel infrastructure).

> > > ext4 has no free space defragmentation and at most you can use e4defrag
> > > to defragment individual files. I now have a 24GB ext4 filesystem that
> > > has only 7GB of space occupied however it has small files scattered all
> > > over it and now bigger files occupy more than one extent and I cannot
> > > reduce fragmentation to zero. One way to approach that would be to
> > > shrink the volume and then defragment it but that will involve a ton of
> > > disk writes and unnecessary tear and wear. Is it possible to modify the
> > > e4degrag utility to move small defragmented files, so that they were
> > > placed consecutively instead of being randomly spread all over the disk?

Anything is *possible*.  Whether anyone thinks its worth their
development time is a different question.  Many years ago, at a
face-to-face ext4 developer's get together, we had sketched out some
ideas for how we might do this.  It included ways to block certain
areas of the disk from being used for normal block allocation, and an
extended ext4-specific fallocate-like ioctl which e4defrag could use
to allocate blocks in a specific portion of the file system.

But no company has a business case where implementing this feature
would have a positive return on investment; no hobbyist has been
interested in doing in their free time; and unfortunately, this is too
complicated of a project for a Google Summer of Code, Outreachy, or
other Intern project.

If you're interested, I'm happy to chat.  But basically, this is a
"patches are welcome; send us code and we'll be happy to review them"
sort of situation.

Cheers,

						- Ted