Re: Feature requests: online backup - defrag - change RAID level

Zygo Blaxell
2019-09-14T23:40:25Z

On Sat, Sep 14, 2019 at 12:29:09PM -0600, Chris Murphy wrote:
> On Fri, Sep 13, 2019 at 5:04 AM Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
> >
> > Do you have a source for this claim of a 128MB max extent size? Because
> > everything I've seen indicates the max extent size is a full data chunk
> > (so 1GB for the common case, potentially up to about 5GB for really big
> > filesystems)
>
> Yeah a block group can be a kind of "super extent". I think the
> EXTENT_DATA maxes out at 128M but they are often contiguous, for
> example
>
> item 308 key (5741459 EXTENT_DATA 0) itemoff 39032 itemsize 53
> generation 241638 type 1 (regular)
> extent data disk byte 193851400192 nr 134217728
> extent data offset 0 nr 134217728 ram 134217728
> extent compression 0 (none)
> item 309 key (5741459 EXTENT_DATA 134217728) itemoff 38979 itemsize 53
> generation 241638 type 1 (regular)
> extent data disk byte 193985617920 nr 134217728
> extent data offset 0 nr 134217728 ram 134217728
> extent compression 0 (none)
> item 310 key (5741459 EXTENT_DATA 268435456) itemoff 38926 itemsize 53
> generation 241638 type 1 (regular)
> extent data disk byte 194119835648 nr 134217728
> extent data offset 0 nr 134217728 ram 134217728
> extent compression 0 (none)
>
> Where FIEMAP has a different view (via filefrag -v)
>
> ext: logical_offset: physical_offset: length: expected: flags:
> 0: 0.. 131071: 47327002.. 47458073: 131072:
> 1: 131072.. 294911: 47518701.. 47682540: 163840: 47458074:
> 2: 294912.. 360447: 50279681.. 50345216: 65536: 47682541:
> 3: 360448.. 499871: 50377984.. 50517407: 139424: 50345217: last,eof
> Fedora-Workstation-Live-x86_64-31_Beta-1.1.iso: 4 extents found
>
> Those extents are all bigger than 128M. But they're each made up of
> contiguous EXTENT_DATA items.
>
> Also, the EXTENT_DATA size goes to a 128K max for any compressed
> files, so you get an explosive number of EXTENT_DATA items on
> compressed file systems, and thus metadata to rewrite.

The compressed extents tend to be physically contiguous as well, so
quantitatively they aren't much of a problem. There's more space used in
metadata, but that is compensated by less space in data. In the subvol
trees, logically contiguous extents are always logically contiguous by
key order, so they are packed densely in subvol metadata pages.

In extent, csum, and free space trees, when there is physical
contiguity--or just proximity--the extent's items are packed into the
same metadata pages, keeping their costs down. That's true of all
small extents, not just compressed ones. Note that contiguity isn't
necessary for metadata space efficiency--the extents just have to be
close together, they don't need to be seamless, or in order (at least
not for this reason).

Writes that are separated in _time_ are a different problem, and
potentially much worse than the compression case. If you have a file that
consists of lots of extents that were written with significant allocations
to other files between them, that file becomes a metadata monster that
can create massive commit latencies when it is deleted or modified.
If you unpack tarballs or build sources or rsync backup trees or really
any two or more writing tasks at the same time on a big btrfs filesystem,
you can run into cases where the metadata:data ratio goes above 1.0 during
updates _and_ the metadata is randomly distributed physically. Commits
after a big delete run for hours.

> I wonder if instead of a rewrite of defragmenting, if there could be
> improvements to the allocator to write bigger extents. I guess the
> problem really comes from file appends? Smarter often means slower but
> perhaps it could be a variation on autodefrag?

Physically dispersed files can be fixed by defrag, but directory trees
are a little different. The current defrag doesn't look at the physical
distances between files, only the extents within a single file, so it
doesn't help when you have a big fragmented directory tree of many small
not-fragmented files. IOW defrag helps with 'rm -f' performance but not
'rm -rf' performance.

Other filesystems have allocator heuristics that reserve space near
growing files, or try to pre-divide the free space to spread out
files belonging to different directories or created by two processes.
This is an attempt to fix the problem before it occurs, and sometimes
it works; however, the heuristics have to match the reality or it just
makes things worse, and extra complexity breeds bugs, e.g. the fix
recently for a bug which tried to give every thread its own block group
for allocation--i.e. 20 threads writing 4K each could ENOSPC if there
was less than 20GB of unallocated space.

I think the best approach may be to attack the problem quantitatively
with an autodefrag agent: keep the write path fast and simple, but
detect areas where problems are occurring--i.e. where the ratio of extent
metadata locality to physical locality is low--and clean them up with
some minimal data relocation. Note that's somewhat different from what
the current kernel autodefrag does.

In absolute terms autodefrag is worse--ideally we'd just put the data
in the right place from the start, not write it in the wrong place
then spend more iops fixing it later--but not all iops have equal cost.
In some cases there is an opportunity to trade cheap iops at one time
for expensive iops at a different time, and a userspace agent can invest
more time, memory, and code complexity on that trade than the kernel.

Some back-of-the-envelope math says we don't need to do very much
post-processing work to deal with the very worst cases: keep extent
sizes over a few hundred KB, and keep small files not more than about
5-10 metadata items away from their logical neighbors, and we avoid the
worst-case 12.0 metadata-to-data ratios during updates. Compared to
those, other inefficiencies are trivial.

>
> --
> Chris Murphy
>