Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

[Andrei Borzenkov]

bor@tw:~> sudo btrfs qgroup show .
ERROR: cannot find the qgroup 0/789
bor@tw:~>

Fine. This openSUSE with snapper which creates and automatically
destroys snapshots and apparently either kernel or snapper now also
remove corresponding qgroup. I played with snapshots and created several
top level qgroups that included snapshot qgroups existing at this time.
Now these snapshots are gone, their qgroups are gone ... and what can I
do? I have no way to even know what is wrong because the very command
that shows it fails immediately.

bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789
(0/789 QGROUP_RELATION 2/792)
(0/789 QGROUP_RELATION 2/793)
(0/789 QGROUP_RELATION 2/795)
(0/789 QGROUP_RELATION 2/799)
(0/789 QGROUP_RELATION 2/800)
(0/789 QGROUP_RELATION 2/803)
(0/789 QGROUP_RELATION 2/804)
(0/789 QGROUP_RELATION 2/805)
(0/789 QGROUP_RELATION 2/806)
(0/789 QGROUP_RELATION 2/807)
(0/789 QGROUP_RELATION 2/808)
(0/789 QGROUP_RELATION 2/809)
(0/789 QGROUP_RELATION 2/814)
(0/789 QGROUP_RELATION 2/818)
(0/789 QGROUP_RELATION 2/819)
(2/792 QGROUP_RELATION 0/789)
(2/793 QGROUP_RELATION 0/789)
(2/795 QGROUP_RELATION 0/789)
(2/799 QGROUP_RELATION 0/789)
(2/800 QGROUP_RELATION 0/789)
(2/803 QGROUP_RELATION 0/789)
(2/804 QGROUP_RELATION 0/789)
(2/805 QGROUP_RELATION 0/789)
(2/806 QGROUP_RELATION 0/789)
(2/807 QGROUP_RELATION 0/789)
(2/808 QGROUP_RELATION 0/789)
(2/809 QGROUP_RELATION 0/789)
(2/814 QGROUP_RELATION 0/789)
(2/818 QGROUP_RELATION 0/789)
(2/819 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

And even if I find it out, I cannot fix it anyway

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
ERROR: unable to assign quota group: Invalid argument
bor@tw:~/python-btrfs/examples>

I can remove parent qgroup, but it does not clean up parent-child
relationship

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 .
bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792
(0/789 QGROUP_RELATION 2/792)
(2/792 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

Re: Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2661 bytes --]



On 2019/8/3 上午2:08, Andrei Borzenkov wrote:
> bor@tw:~> sudo btrfs qgroup show .
> ERROR: cannot find the qgroup 0/789
> bor@tw:~>
> 
> Fine. This openSUSE with snapper which creates and automatically
> destroys snapshots and apparently either kernel or snapper now also
> remove corresponding qgroup. I played with snapshots and created several
> top level qgroups that included snapshot qgroups existing at this time.
> Now these snapshots are gone, their qgroups are gone ...

Kernel version please.

IIRC latest upstream kernel doesn't remove the level 0 qgroup.
It may be the userspace doing it improperly.

> and what can I
> do? I have no way to even know what is wrong because the very command
> that shows it fails immediately.
> 
> bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789
> (0/789 QGROUP_RELATION 2/792)
> (0/789 QGROUP_RELATION 2/793)
> (0/789 QGROUP_RELATION 2/795)
> (0/789 QGROUP_RELATION 2/799)
> (0/789 QGROUP_RELATION 2/800)
> (0/789 QGROUP_RELATION 2/803)
> (0/789 QGROUP_RELATION 2/804)
> (0/789 QGROUP_RELATION 2/805)
> (0/789 QGROUP_RELATION 2/806)
> (0/789 QGROUP_RELATION 2/807)
> (0/789 QGROUP_RELATION 2/808)
> (0/789 QGROUP_RELATION 2/809)
> (0/789 QGROUP_RELATION 2/814)
> (0/789 QGROUP_RELATION 2/818)
> (0/789 QGROUP_RELATION 2/819)
> (2/792 QGROUP_RELATION 0/789)
> (2/793 QGROUP_RELATION 0/789)
> (2/795 QGROUP_RELATION 0/789)
> (2/799 QGROUP_RELATION 0/789)
> (2/800 QGROUP_RELATION 0/789)
> (2/803 QGROUP_RELATION 0/789)
> (2/804 QGROUP_RELATION 0/789)
> (2/805 QGROUP_RELATION 0/789)
> (2/806 QGROUP_RELATION 0/789)
> (2/807 QGROUP_RELATION 0/789)
> (2/808 QGROUP_RELATION 0/789)
> (2/809 QGROUP_RELATION 0/789)
> (2/814 QGROUP_RELATION 0/789)
> (2/818 QGROUP_RELATION 0/789)
> (2/819 QGROUP_RELATION 0/789)
> bor@tw:~/python-btrfs/examples>
> 
> And even if I find it out, I cannot fix it anyway

Furthermore, latest kernel should automatically remove the relation when
deleting the qgroup.

Would you please provide the (minimal) script/reproducer causing the
situation and kernel version?

Thanks,
Qu

> 
> bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
> ERROR: unable to assign quota group: Invalid argument
> bor@tw:~/python-btrfs/examples>
> 
> I can remove parent qgroup, but it does not clean up parent-child
> relationship
> 
> bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 .
> bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792
> (0/789 QGROUP_RELATION 2/792)
> (2/792 QGROUP_RELATION 0/789)
> bor@tw:~/python-btrfs/examples>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

[Andrei Borzenkov]

[-- Attachment #1.1: Type: text/plain, Size: 3780 bytes --]

03.08.2019 2:09, Qu Wenruo пишет:
> 
> 
> On 2019/8/3 上午2:08, Andrei Borzenkov wrote:
>> bor@tw:~> sudo btrfs qgroup show .
>> ERROR: cannot find the qgroup 0/789
>> bor@tw:~>
>>
>> Fine. This openSUSE with snapper which creates and automatically
>> destroys snapshots and apparently either kernel or snapper now also
>> remove corresponding qgroup. I played with snapshots and created several
>> top level qgroups that included snapshot qgroups existing at this time.
>> Now these snapshots are gone, their qgroups are gone ...
> 
> Kernel version please.
> 
> IIRC latest upstream kernel doesn't remove the level 0 qgroup.

Yes?

> It may be the userspace doing it improperly.
> 

Not sure what "improperly" means here. snapper removes qgroup after
deleting snapshot. What is "improper" here?

Snapper obviously does not track every parent-child relationship beyond
what it itself cares about (single summary qrgoup that includes all
snapshots).

>> and what can I
>> do? I have no way to even know what is wrong because the very command
>> that shows it fails immediately.
>>
>> bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789
>> (0/789 QGROUP_RELATION 2/792)
>> (0/789 QGROUP_RELATION 2/793)
>> (0/789 QGROUP_RELATION 2/795)
>> (0/789 QGROUP_RELATION 2/799)
>> (0/789 QGROUP_RELATION 2/800)
>> (0/789 QGROUP_RELATION 2/803)
>> (0/789 QGROUP_RELATION 2/804)
>> (0/789 QGROUP_RELATION 2/805)
>> (0/789 QGROUP_RELATION 2/806)
>> (0/789 QGROUP_RELATION 2/807)
>> (0/789 QGROUP_RELATION 2/808)
>> (0/789 QGROUP_RELATION 2/809)
>> (0/789 QGROUP_RELATION 2/814)
>> (0/789 QGROUP_RELATION 2/818)
>> (0/789 QGROUP_RELATION 2/819)
>> (2/792 QGROUP_RELATION 0/789)
>> (2/793 QGROUP_RELATION 0/789)
>> (2/795 QGROUP_RELATION 0/789)
>> (2/799 QGROUP_RELATION 0/789)
>> (2/800 QGROUP_RELATION 0/789)
>> (2/803 QGROUP_RELATION 0/789)
>> (2/804 QGROUP_RELATION 0/789)
>> (2/805 QGROUP_RELATION 0/789)
>> (2/806 QGROUP_RELATION 0/789)
>> (2/807 QGROUP_RELATION 0/789)
>> (2/808 QGROUP_RELATION 0/789)
>> (2/809 QGROUP_RELATION 0/789)
>> (2/814 QGROUP_RELATION 0/789)
>> (2/818 QGROUP_RELATION 0/789)
>> (2/819 QGROUP_RELATION 0/789)
>> bor@tw:~/python-btrfs/examples>
>>
>> And even if I find it out, I cannot fix it anyway
> 
> Furthermore, latest kernel should automatically remove the relation when
> deleting the qgroup.
> 

Yes, seems to be the case now with 5.2.3.

> Would you please provide the (minimal) script/reproducer causing the
> situation and kernel version?
> 

I cannot reproduce it on purpose with kernel 5.2.3 and btrfsprogs 5.1. I
do not remember when I created the configuration in question, it
probably was late 4.x or early 5.x. System was periodically updated
since then and I noticed it only now.

Actually kernel should be dropping parent-child relation when removing
child since kernel 4.1 (commit
f5a6b1c53bdd44f79e3904c0f5e59f956b49b2c8). May be there is (was) some
other code path that skipped it.

OTOH this is not atomic. First qgroup item itself is removed, then
relation. If removing relation fails for whatever reason item itself is
already gone and we have situation above.

> Thanks,
> Qu
> 
>>
>> bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
>> ERROR: unable to assign quota group: Invalid argument
>> bor@tw:~/python-btrfs/examples>
>>
>> I can remove parent qgroup, but it does not clean up parent-child
>> relationship
>>
>> bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 .
>> bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792
>> (0/789 QGROUP_RELATION 2/792)
>> (2/792 QGROUP_RELATION 0/789)
>> bor@tw:~/python-btrfs/examples>
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4788 bytes --]



On 2019/8/3 下午1:31, Andrei Borzenkov wrote:
> 03.08.2019 2:09, Qu Wenruo пишет:
>>
>>
>> On 2019/8/3 上午2:08, Andrei Borzenkov wrote:
>>> bor@tw:~> sudo btrfs qgroup show .
>>> ERROR: cannot find the qgroup 0/789
>>> bor@tw:~>
>>>
>>> Fine. This openSUSE with snapper which creates and automatically
>>> destroys snapshots and apparently either kernel or snapper now also
>>> remove corresponding qgroup. I played with snapshots and created several
>>> top level qgroups that included snapshot qgroups existing at this time.
>>> Now these snapshots are gone, their qgroups are gone ...
>>
>> Kernel version please.
>>
>> IIRC latest upstream kernel doesn't remove the level 0 qgroup.
> 
> Yes?
> 
>> It may be the userspace doing it improperly.
>>
> 
> Not sure what "improperly" means here. snapper removes qgroup after
> deleting snapshot. What is "improper" here?

Doing without using the qgroup ioctl, but some extra flag in snapshot
creation/deletion, which can also add relation at subv/snapshot creation
time.

> 
> Snapper obviously does not track every parent-child relationship beyond
> what it itself cares about (single summary qrgoup that includes all
> snapshots).
> 
>>> and what can I
>>> do? I have no way to even know what is wrong because the very command
>>> that shows it fails immediately.
>>>
>>> bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789
>>> (0/789 QGROUP_RELATION 2/792)
>>> (0/789 QGROUP_RELATION 2/793)
>>> (0/789 QGROUP_RELATION 2/795)
>>> (0/789 QGROUP_RELATION 2/799)
>>> (0/789 QGROUP_RELATION 2/800)
>>> (0/789 QGROUP_RELATION 2/803)
>>> (0/789 QGROUP_RELATION 2/804)
>>> (0/789 QGROUP_RELATION 2/805)
>>> (0/789 QGROUP_RELATION 2/806)
>>> (0/789 QGROUP_RELATION 2/807)
>>> (0/789 QGROUP_RELATION 2/808)
>>> (0/789 QGROUP_RELATION 2/809)
>>> (0/789 QGROUP_RELATION 2/814)
>>> (0/789 QGROUP_RELATION 2/818)
>>> (0/789 QGROUP_RELATION 2/819)
>>> (2/792 QGROUP_RELATION 0/789)
>>> (2/793 QGROUP_RELATION 0/789)
>>> (2/795 QGROUP_RELATION 0/789)
>>> (2/799 QGROUP_RELATION 0/789)
>>> (2/800 QGROUP_RELATION 0/789)
>>> (2/803 QGROUP_RELATION 0/789)
>>> (2/804 QGROUP_RELATION 0/789)
>>> (2/805 QGROUP_RELATION 0/789)
>>> (2/806 QGROUP_RELATION 0/789)
>>> (2/807 QGROUP_RELATION 0/789)
>>> (2/808 QGROUP_RELATION 0/789)
>>> (2/809 QGROUP_RELATION 0/789)
>>> (2/814 QGROUP_RELATION 0/789)
>>> (2/818 QGROUP_RELATION 0/789)
>>> (2/819 QGROUP_RELATION 0/789)
>>> bor@tw:~/python-btrfs/examples>
>>>
>>> And even if I find it out, I cannot fix it anyway
>>
>> Furthermore, latest kernel should automatically remove the relation when
>> deleting the qgroup.
>>
> 
> Yes, seems to be the case now with 5.2.3.
> 
>> Would you please provide the (minimal) script/reproducer causing the
>> situation and kernel version?
>>
> 
> I cannot reproduce it on purpose with kernel 5.2.3 and btrfsprogs 5.1. I
> do not remember when I created the configuration in question, it
> probably was late 4.x or early 5.x. System was periodically updated
> since then and I noticed it only now.
> 
> Actually kernel should be dropping parent-child relation when removing
> child since kernel 4.1 (commit
> f5a6b1c53bdd44f79e3904c0f5e59f956b49b2c8). May be there is (was) some
> other code path that skipped it.
> 
> OTOH this is not atomic. First qgroup item itself is removed, then
> relation. If removing relation fails for whatever reason item itself is
> already gone and we have situation above.

You're right about the removal, although the term is not "atomic".

For qgroup item deletion, we can return -ENOENT or other serious tree
corruption error from btrfs_search_slot()/btrfs_del_item().

But even for -ENOENT case, we still tries to delete the relation item.
So that part is OK.

The problem is the del_qgroup_relation part, where if we hit even
-ENOENT we abort deleting even if we may have extra items needs to delete.

We need a little more polish on the error handler.

So is the btrfs_del_qgroup_relation() where it doesn't delete all
relations, but only one.

I'll soon send out patches to address your bugs.

Thanks,
Qu

> 
>> Thanks,
>> Qu
>>
>>>
>>> bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
>>> ERROR: unable to assign quota group: Invalid argument
>>> bor@tw:~/python-btrfs/examples>
>>>
>>> I can remove parent qgroup, but it does not clean up parent-child
>>> relationship
>>>
>>> bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 .
>>> bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792
>>> (0/789 QGROUP_RELATION 2/792)
>>> (2/792 QGROUP_RELATION 0/789)
>>> bor@tw:~/python-btrfs/examples>
>>>
>>
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

[Andrei Borzenkov]

[-- Attachment #1.1: Type: text/plain, Size: 1907 bytes --]

03.08.2019 9:17, Qu Wenruo пишет:
> 
> 
> On 2019/8/3 下午1:31, Andrei Borzenkov wrote:
>> 03.08.2019 2:09, Qu Wenruo пишет:
>>>
>>>
>>> On 2019/8/3 上午2:08, Andrei Borzenkov wrote:
>>>> bor@tw:~> sudo btrfs qgroup show .
>>>> ERROR: cannot find the qgroup 0/789
>>>> bor@tw:~>
>>>>
>>>> Fine. This openSUSE with snapper which creates and automatically
>>>> destroys snapshots and apparently either kernel or snapper now also
>>>> remove corresponding qgroup. I played with snapshots and created several
>>>> top level qgroups that included snapshot qgroups existing at this time.
>>>> Now these snapshots are gone, their qgroups are gone ...
>>>
>>> Kernel version please.
>>>
>>> IIRC latest upstream kernel doesn't remove the level 0 qgroup.
>>
>> Yes?
>>
>>> It may be the userspace doing it improperly.
>>>
>>
>> Not sure what "improperly" means here. snapper removes qgroup after
>> deleting snapshot. What is "improper" here?
> 
> Doing without using the qgroup ioctl, but some extra flag in snapshot
> creation/deletion, which can also add relation at subv/snapshot creation
> time.
> 

As far as I can tell, this is exactly what snapper does:

            if (qgroup != no_qgroup)
            {
                size_t size = sizeof(btrfs_qgroup_inherit) +
sizeof(((btrfs_qgroup_inherit*) 0)->qgroups[0]);
                vector<char> buffer(size, 0);
                struct btrfs_qgroup_inherit* inherit =
(btrfs_qgroup_inherit*) &buffer[0];

                inherit->num_qgroups = 1;
                inherit->num_ref_copies = 0;
                inherit->num_excl_copies = 0;
                inherit->qgroups[0] = qgroup;

                args_v2.flags |= BTRFS_SUBVOL_QGROUP_INHERIT;
                args_v2.size = size;
                args_v2.qgroup_inherit = inherit;
            }

Do you say it should not be doing it?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2139 bytes --]



On 2019/8/3 下午2:49, Andrei Borzenkov wrote:
> 03.08.2019 9:17, Qu Wenruo пишет:
>>
>>
>> On 2019/8/3 下午1:31, Andrei Borzenkov wrote:
>>> 03.08.2019 2:09, Qu Wenruo пишет:
>>>>
>>>>
>>>> On 2019/8/3 上午2:08, Andrei Borzenkov wrote:
>>>>> bor@tw:~> sudo btrfs qgroup show .
>>>>> ERROR: cannot find the qgroup 0/789
>>>>> bor@tw:~>
>>>>>
>>>>> Fine. This openSUSE with snapper which creates and automatically
>>>>> destroys snapshots and apparently either kernel or snapper now also
>>>>> remove corresponding qgroup. I played with snapshots and created several
>>>>> top level qgroups that included snapshot qgroups existing at this time.
>>>>> Now these snapshots are gone, their qgroups are gone ...
>>>>
>>>> Kernel version please.
>>>>
>>>> IIRC latest upstream kernel doesn't remove the level 0 qgroup.
>>>
>>> Yes?
>>>
>>>> It may be the userspace doing it improperly.
>>>>
>>>
>>> Not sure what "improperly" means here. snapper removes qgroup after
>>> deleting snapshot. What is "improper" here?
>>
>> Doing without using the qgroup ioctl, but some extra flag in snapshot
>> creation/deletion, which can also add relation at subv/snapshot creation
>> time.
>>
> 
> As far as I can tell, this is exactly what snapper does:
> 
>             if (qgroup != no_qgroup)
>             {
>                 size_t size = sizeof(btrfs_qgroup_inherit) +
> sizeof(((btrfs_qgroup_inherit*) 0)->qgroups[0]);
>                 vector<char> buffer(size, 0);
>                 struct btrfs_qgroup_inherit* inherit =
> (btrfs_qgroup_inherit*) &buffer[0];
> 
>                 inherit->num_qgroups = 1;
>                 inherit->num_ref_copies = 0;
>                 inherit->num_excl_copies = 0;
>                 inherit->qgroups[0] = qgroup;
> 
>                 args_v2.flags |= BTRFS_SUBVOL_QGROUP_INHERIT;
>                 args_v2.size = size;
>                 args_v2.qgroup_inherit = inherit;
>             }
> 
> Do you say it should not be doing it?
> 

For creation, that's OK, as long as it's not using the deprecated copy
version..

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]