Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

[Tine Mezgec]

Hi,

I have a Btrfs filesystem mounted at /media/storage with the following 
setup, that took minutes to mount:

Label: 'data'  uuid: 7e1efcb0-96c7-4e8f-861c-ee3edba3e028
Total devices 11  FS bytes used 40.33TiB

devid    2 size 10.91TiB used  3.79TiB path /dev/sdd
devid    3 size  7.28TiB used     0.00B path /dev/sdc
devid    4 size  7.28TiB used     0.00B path /dev/sdg
devid    5 size  7.28TiB used   215.00GiB path /dev/sda
devid    8 size 16.37TiB used  8.93TiB path /dev/sdk
devid    9 size 16.37TiB used  9.50TiB path /dev/sdh
devid   12 size 14.55TiB used  7.41TiB path /dev/sdi
devid   13 size 14.55TiB used  7.71TiB path /dev/sdj
devid   15 size 18.19TiB used 10.74TiB path /dev/sdf
devid   16 size 23.65TiB used 16.20TiB path /dev/sdb
devid   17 size 23.65TiB used 16.20TiB path /dev/sde

I ran (after unmounting):

btrfstune --convert-to-block-group-tree /dev/sdd
(using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system 
lost power during the conversion.

After reboot, rerunning the command gives:

ERROR: failed to find block group for bytenr 186297726533632
ERROR: failed to convert the filesystem to block group tree feature
ERROR: btrfstune failed

extent buffer leak: start 185256860958720 len 16384

Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.

The superblock flags now show:

0x4000000001 (WRITTEN | CHANGING_BG_TREE)

Attempting to revert:

btrfstune --convert-from-block-group-tree /dev/sdd
fails with:
ERROR: filesystem doesn't have block-group-tree feature
ERROR: btrfstune failed

So I'm stuck with a filesystem in a half-converted state — not fully 
converted, but marked as changing.

What’s the best way to either complete the conversion or revert/abort it 
cleanly?

Thanks,
-Tine

Re: Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

[Qu Wenruo]

在 2025/6/13 03:13, Tine Mezgec 写道:
> Hi,
> 
> I have a Btrfs filesystem mounted at /media/storage with the following 
> setup, that took minutes to mount:
> 
[...]
> btrfstune --convert-to-block-group-tree /dev/sdd
> (using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system 
> lost power during the conversion.

The progs is a little old, but I do not think there are bgt related 
fixes after that though.

> 
> After reboot, rerunning the command gives:

Rerunning the command is the correct way to resume, but something 
doesn't seem correct here.


> 
> ERROR: failed to find block group for bytenr 186297726533632

Please provide the following dump:

# btrfs ins dump-tree -t chunk <device>
# btrfs ins dump-tree -t extent <device>
   The above one is pretty large.

# btrfs ins dump-tree -t block-group <device>

I guess there is something wrong with the last converted bg checks, 
resulting btrfstune to trying to convert an already converted bg.

Thanks,
Qu


> ERROR: failed to convert the filesystem to block group tree feature
> ERROR: btrfstune failed
> 
> extent buffer leak: start 185256860958720 len 16384
> 
> Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.
> 
> The superblock flags now show:
> 
> 0x4000000001 (WRITTEN | CHANGING_BG_TREE)
> 
> Attempting to revert:
> 
> btrfstune --convert-from-block-group-tree /dev/sdd
> fails with:
> ERROR: filesystem doesn't have block-group-tree feature
> ERROR: btrfstune failed
> 
> So I'm stuck with a filesystem in a half-converted state — not fully 
> converted, but marked as changing.
> 
> What’s the best way to either complete the conversion or revert/abort it 
> cleanly?
> 
> Thanks,
> -Tine
> 

Re: Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

[Tine Mezgec]

Hi Qu,

Thanks for taking a look at this.

Here are the three tree dumps you asked for:

http://144.76.28.172/btrfs/block-group.dump.gz
http://144.76.28.172/btrfs/chunk.dump.gz
http://144.76.28.172/btrfs/extent.dump.gz

Let me know if you need anything else

Thanks,
-Tine

On 13/06/2025 00:29, Qu Wenruo wrote:
> 
> 
> 在 2025/6/13 03:13, Tine Mezgec 写道:
>> Hi,
>>
>> I have a Btrfs filesystem mounted at /media/storage with the following 
>> setup, that took minutes to mount:
>>
> [...]
>> btrfstune --convert-to-block-group-tree /dev/sdd
>> (using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system 
>> lost power during the conversion.
> 
> The progs is a little old, but I do not think there are bgt related 
> fixes after that though.
> 
>>
>> After reboot, rerunning the command gives:
> 
> Rerunning the command is the correct way to resume, but something 
> doesn't seem correct here.
> 
> 
>>
>> ERROR: failed to find block group for bytenr 186297726533632
> 
> Please provide the following dump:
> 
> # btrfs ins dump-tree -t chunk <device>
> # btrfs ins dump-tree -t extent <device>
>    The above one is pretty large.
> 
> # btrfs ins dump-tree -t block-group <device>
> 
> I guess there is something wrong with the last converted bg checks, 
> resulting btrfstune to trying to convert an already converted bg.
> 
> Thanks,
> Qu
> 
> 
>> ERROR: failed to convert the filesystem to block group tree feature
>> ERROR: btrfstune failed
>>
>> extent buffer leak: start 185256860958720 len 16384
>>
>> Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.
>>
>> The superblock flags now show:
>>
>> 0x4000000001 (WRITTEN | CHANGING_BG_TREE)
>>
>> Attempting to revert:
>>
>> btrfstune --convert-from-block-group-tree /dev/sdd
>> fails with:
>> ERROR: filesystem doesn't have block-group-tree feature
>> ERROR: btrfstune failed
>>
>> So I'm stuck with a filesystem in a half-converted state — not fully 
>> converted, but marked as changing.
>>
>> What’s the best way to either complete the conversion or revert/abort 
>> it cleanly?
>>
>> Thanks,
>> -Tine
>>
> 

Re: Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 3225 bytes --]



在 2025/6/13 15:42, Tine Mezgec 写道:
> Hi Qu,
> 
> Thanks for taking a look at this.
> 
> Here are the three tree dumps you asked for:
> 
> http://144.76.28.172/btrfs/block-group.dump.gz
> http://144.76.28.172/btrfs/chunk.dump.gz
> http://144.76.28.172/btrfs/extent.dump.gz
> 
> Let me know if you need anything else

Thanks a lot.

The block-group dump shows that all block groups after 186298800275456 
(including that bytenr) are converted to block group tree.

At the same time, chunk tree dump shows that block groups 
186297726533632 is indeed the block group before the converted one.

Furthermore in side the extent tree dump, it shows there is indeed a 
block group item for 186297726533632 in the extent tree.

So the bug looks very likely inside the mount process which doesn't 
handle the last block group in the old tree correctly.


I crafted and attached a small patch, that will skip the the 
optimization to do a full extent tree search for old block groups.
I hope this will allow you to resume the conversion.

You'll need to compile btrfs-progs with this patch applied, then use the 
compiled btrfstune to resume the conversion.
(Better using the latest release of btrfs-progs)

This patch will cause a warning about an unused function but you can 
safely ignore it.

Thanks,
Qu

> 
> Thanks,
> -Tine
> 
> On 13/06/2025 00:29, Qu Wenruo wrote:
>>
>>
>> 在 2025/6/13 03:13, Tine Mezgec 写道:
>>> Hi,
>>>
>>> I have a Btrfs filesystem mounted at /media/storage with the 
>>> following setup, that took minutes to mount:
>>>
>> [...]
>>> btrfstune --convert-to-block-group-tree /dev/sdd
>>> (using btrfs-progs 6.6.3-1.1build2 from Ubuntu 24.04), but the system 
>>> lost power during the conversion.
>>
>> The progs is a little old, but I do not think there are bgt related 
>> fixes after that though.
>>
>>>
>>> After reboot, rerunning the command gives:
>>
>> Rerunning the command is the correct way to resume, but something 
>> doesn't seem correct here.
>>
>>
>>>
>>> ERROR: failed to find block group for bytenr 186297726533632
>>
>> Please provide the following dump:
>>
>> # btrfs ins dump-tree -t chunk <device>
>> # btrfs ins dump-tree -t extent <device>
>>    The above one is pretty large.
>>
>> # btrfs ins dump-tree -t block-group <device>
>>
>> I guess there is something wrong with the last converted bg checks, 
>> resulting btrfstune to trying to convert an already converted bg.
>>
>> Thanks,
>> Qu
>>
>>
>>> ERROR: failed to convert the filesystem to block group tree feature
>>> ERROR: btrfstune failed
>>>
>>> extent buffer leak: start 185256860958720 len 16384
>>>
>>> Trying with kernel 6.15.2 and btrfs-progs 6.14 gives the same result.
>>>
>>> The superblock flags now show:
>>>
>>> 0x4000000001 (WRITTEN | CHANGING_BG_TREE)
>>>
>>> Attempting to revert:
>>>
>>> btrfstune --convert-from-block-group-tree /dev/sdd
>>> fails with:
>>> ERROR: filesystem doesn't have block-group-tree feature
>>> ERROR: btrfstune failed
>>>
>>> So I'm stuck with a filesystem in a half-converted state — not fully 
>>> converted, but marked as changing.
>>>
>>> What’s the best way to either complete the conversion or revert/abort 
>>> it cleanly?
>>>
>>> Thanks,
>>> -Tine
>>>
>>
> 

[-- Attachment #2: read_all_bgs_from_old.diff --]
[-- Type: text/x-patch, Size: 510 bytes --]

diff --git a/kernel-shared/extent-tree.c b/kernel-shared/extent-tree.c
index 2b7a962f294b..b02bba4630b6 100644
--- a/kernel-shared/extent-tree.c
+++ b/kernel-shared/extent-tree.c
@@ -2984,7 +2984,7 @@ static int read_converting_block_groups(struct btrfs_fs_info *fs_info)
 		return ret;
 	}
 
-	ret = read_old_block_groups_from_root(fs_info, old_root);
+	ret = read_block_groups_from_root(fs_info,  old_root);
 	if (ret < 0) {
 		error("failed to load block groups from the old root: %d", ret);
 		return ret;

Re: Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

[Tine Mezgec]

Hi Qu,

Your patch did the trick, the conversion completed cleanly, the 
CHANGING_BG_TREE flag is gone, and the incompat BG_TREE bit is now set. 
Mounting the 11‑device volume takes ~5 seconds instead of several 
minutes, so everything is back to normal or even better than before.

Thanks a lot for the quick diagnosis and the fix.

One thought: the optimisation that skips the full extent‑tree scan 
clearly helps in the common case, but when it backfires the only 
recovery path is recompiling progs with a patch. Would it make sense to 
expose a --full-scan (or --no-fast-scan) switch in btrfstune so an 
interrupted conversion can be resumed with stock binaries?

Either way, I really appreciate the help.

Best regards,
-Tine

Re: Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

[Qu Wenruo]

在 2025/6/13 19:01, Tine Mezgec 写道:
> Hi Qu,
> 
> Your patch did the trick, the conversion completed cleanly, the 
> CHANGING_BG_TREE flag is gone, and the incompat BG_TREE bit is now set. 
> Mounting the 11‑device volume takes ~5 seconds instead of several 
> minutes, so everything is back to normal or even better than before.
> 
> Thanks a lot for the quick diagnosis and the fix.
> 
> One thought: the optimisation that skips the full extent‑tree scan 
> clearly helps in the common case, but when it backfires the only 
> recovery path is recompiling progs with a patch. Would it make sense to 
> expose a --full-scan (or --no-fast-scan) switch in btrfstune so an 
> interrupted conversion can be resumed with stock binaries?

No need for a new option, because it's definitely a bug, and a bug 
should and will be hunt down.

The only problem is, I still didn't get a full chain on why the last 
block group in the old tree is skipped.

The old code seems correct, but it isn't. I'll keep digging and send out 
a proper fix later.

Thanks,
Qu

> 
> Either way, I really appreciate the help.
> 
> Best regards,
> -Tine

Re: Stuck in CHANGING_BG_TREE state after interrupted btrfstune --convert-to-block-group-tree

[Qu Wenruo]

在 2025/6/13 19:31, Qu Wenruo 写道:
> 
> 
> 在 2025/6/13 19:01, Tine Mezgec 写道:
>> Hi Qu,
>>
>> Your patch did the trick, the conversion completed cleanly, the 
>> CHANGING_BG_TREE flag is gone, and the incompat BG_TREE bit is now 
>> set. Mounting the 11‑device volume takes ~5 seconds instead of several 
>> minutes, so everything is back to normal or even better than before.
>>
>> Thanks a lot for the quick diagnosis and the fix.
>>
>> One thought: the optimisation that skips the full extent‑tree scan 
>> clearly helps in the common case, but when it backfires the only 
>> recovery path is recompiling progs with a patch. Would it make sense 
>> to expose a --full-scan (or --no-fast-scan) switch in btrfstune so an 
>> interrupted conversion can be resumed with stock binaries?
> 
> No need for a new option, because it's definitely a bug, and a bug 
> should and will be hunt down.
> 
> The only problem is, I still didn't get a full chain on why the last 
> block group in the old tree is skipped.

The final update, the proper fix is sent here:
https://lore.kernel.org/linux-btrfs/cover.1750223785.git.wqu@suse.com/

The final fix is just a single one line fix, for a stupid uninitialized 
structure.

But I also added an optimization so that for resume case, the fs opening 
would benefit a lot from the block group tree.

Thank you for the report and all the help pinning down the problematic 
function.

Thanks,
Qu

> 
> The old code seems correct, but it isn't. I'll keep digging and send out 
> a proper fix later.
> 
> Thanks,
> Qu
> 
>>
>> Either way, I really appreciate the help.
>>
>> Best regards,
>> -Tine
> 
>