Endless "reclaiming chunk"/"relocating block group"

[Christoph Biedl <linux-kernel.bfrz@manchmal.in-ulm.de>]

Hello,

On some systems I observe a strange behaviour: After remounting a BTRFS
readwrite, a background process starts doing things on the disk,
messages look like

| BTRFS info (device nvme0n1p1): reclaiming chunk 21486669660160 with 100% used 0% unusable
| BTRFS info (device nvme0n1p1): relocating block group 21486669660160 flags data
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: move data extents
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: update data pointers

and (with differing numbers) this goes on for hours and days, at a
read/write rate of about 165/244 kbyte/sec. The filesystem, some 2.5
Gbyte total size, is filled to about 55%, so even if that process
touches each and every block, it should already have handled everything,
several times.

Now, I have no clue what is happening here, what triggers it, if it will
ever finish. Point is, this takes a measuarable amount of I/O and CPU,
and it delays other processes.


Some details, and things I've tested:

This behaviour is reproducible 100%, even with a btrfs created mere
moments ago.

The filesystem was created using the 5.10 and 6.0 version of the
btrfs-progs (both as provided by Debian stable and unstable resp.).

Using the grml rescue system (stable and daily, the latter kernel 5.19),
the system does not show this behaviour.

The group block number is constantly increasing (14 digits after two
days), in other words, I have not observed a wrap-around.

It was suggested in IRC to format using the --mixed parameter, no avail.

It was also suggested to set the various bg_reclaim_threshold to zero to
stop this process, no avail.

This is amd64 hardware without any unusual elements. I could easily
reproduce this on a fairly different platforms to make sure it's not
hardware specific.

Scrubbing did not show any errors, and the problem remained.


The host runs a hand-crafted kernel, currently 5.19, and I reckon this
is the source of the problem. Of course I've compared all the BTRFS
kernel options, they are identical. In the block device layer
configuration I couldn't see any difference that I can think would
relate to this issue. Likewise I compared all kernel configuration
options mentioned in src/fs/btrfs/, still nothing noteworthy.


So I'm a bit out of ideas. Unless there's something obvious from the
description above, perhaps you could give a hint to the following: The
process that emits the messages above, is there a way to stop it, or to
report completion percentage? Looking intobtrfs_reclaim_bgs_work
(block-group.c), it doesn't look like it. Are block group numbers really
*that* big, magnitudes over the size of the entire filesystem?

Regards,

    Christoph