About btrfs qgroup import/export command

About btrfs qgroup import/export command

[Miao Xie]

Hi, everyone.

As we know, there is no backup function for qgroup. when the problem
occurs, the users must recover qgroup configuration manually, it is not
convenient. And besides that, some users might want to import an existed
qgroup configuration into a new filesystem. Btrfs does not have such a
function,it can only be done manually.

So we want to implement btrfs qgroup import/export commands.
1)'btrfs qgroup export' commands will export qgroup tree
  into a user's specified file.(stdout by default)

2)user may modify the configuration file firstly and then
  import it into the filesystem.(by 'btrfs qgroup import' command)

The file may be formated as the following:

Qgroupid     is_compressed     is_exclusive   limited_size    parent
----------------------------------------------------------------------
 0/1            0                 0              10G            1/0
 1/0            1                 1              20G            ---
  
 If 'is_exclusive' is set, 'limited_size' corresponds to max exlusive size,
 else max referenced size. Here 'parent' exclude ancestral qgroups. 

Is there any comment about this idea? 

Thanks
Miao

Re: About btrfs qgroup import/export command

[Arne Jansen]

On 19.12.2012 12:25, Miao Xie wrote:
> Hi, everyone.
> 
> As we know, there is no backup function for qgroup. when the problem
> occurs, the users must recover qgroup configuration manually, it is not
> convenient. And besides that, some users might want to import an existed
> qgroup configuration into a new filesystem. Btrfs does not have such a
> function,it can only be done manually.
> 
> So we want to implement btrfs qgroup import/export commands.
> 1)'btrfs qgroup export' commands will export qgroup tree
>   into a user's specified file.(stdout by default)
> 
> 2)user may modify the configuration file firstly and then
>   import it into the filesystem.(by 'btrfs qgroup import' command)
> 
> The file may be formated as the following:
> 
> Qgroupid     is_compressed     is_exclusive   limited_size    parent
> ----------------------------------------------------------------------
>  0/1            0                 0              10G            1/0
>  1/0            1                 1              20G            ---
>   
>  If 'is_exclusive' is set, 'limited_size' corresponds to max exlusive size,
>  else max referenced size. Here 'parent' exclude ancestral qgroups. 
> 
> Is there any comment about this idea? 

The configuration only really makes sense in combination with the existing
subvolumes. Even if the target has subvolumes under the same name, they
might have different internal IDs. So it might make more sense to address
the level 0 qgroups by name.
Also it might be misleading to apply a configuration to an existing fs, as
it currently is not possible get a correct accounting if the fs is not
empty. Rescan is not yet implemented.
So instead of just saving and restoring the qgroup config, it might make
more sense to create a new filesystem including all subvolumes and quota
config from a config file.
But, I'm not completely convinced that this is a features that is needed
frequently. If I want a standard deployment, I simple write a script that
creates the fs + subvol + quota.

-Arne

> 
> Thanks
> Miao
> 
> 
> 
> 
> 
> 
> 
> 

Re: About btrfs qgroup import/export command

[Miao Xie]

On Wed, 19 Dec 2012 12:40:25 +0100, Arne Jansen wrote:
> On 19.12.2012 12:25, Miao Xie wrote:
>> Hi, everyone.
>>
>> As we know, there is no backup function for qgroup. when the problem
>> occurs, the users must recover qgroup configuration manually, it is not
>> convenient. And besides that, some users might want to import an existed
>> qgroup configuration into a new filesystem. Btrfs does not have such a
>> function,it can only be done manually.
>>
>> So we want to implement btrfs qgroup import/export commands.
>> 1)'btrfs qgroup export' commands will export qgroup tree
>>   into a user's specified file.(stdout by default)
>>
>> 2)user may modify the configuration file firstly and then
>>   import it into the filesystem.(by 'btrfs qgroup import' command)
>>
>> The file may be formated as the following:
>>
>> Qgroupid     is_compressed     is_exclusive   limited_size    parent
>> ----------------------------------------------------------------------
>>  0/1            0                 0              10G            1/0
>>  1/0            1                 1              20G            ---
>>   
>>  If 'is_exclusive' is set, 'limited_size' corresponds to max exlusive size,
>>  else max referenced size. Here 'parent' exclude ancestral qgroups. 
>>
>> Is there any comment about this idea? 
> 
> The configuration only really makes sense in combination with the existing
> subvolumes. Even if the target has subvolumes under the same name, they
> might have different internal IDs. So it might make more sense to address
> the level 0 qgroups by name.

Yeah. Thanks for your suggest.

> Also it might be misleading to apply a configuration to an existing fs, as
> it currently is not possible get a correct accounting if the fs is not
> empty. Rescan is not yet implemented.
> So instead of just saving and restoring the qgroup config, it might make
> more sense to create a new filesystem including all subvolumes and quota
> config from a config file.
> But, I'm not completely convinced that this is a features that is needed
> frequently. If I want a standard deployment, I simple write a script that
> creates the fs + subvol + quota.

But if you set a new value for some groups, you must modify your script at
the same time, it is a bit troublesome.

Thanks
Miao

Re: About btrfs qgroup import/export command

[Miao Xie]

Hi, Arne

On Wed, 19 Dec 2012 12:40:25 +0100, Arne Jansen wrote:
> On 19.12.2012 12:25, Miao Xie wrote:
>> As we know, there is no backup function for qgroup. when the problem
>> occurs, the users must recover qgroup configuration manually, it is not
>> convenient. And besides that, some users might want to import an existed
>> qgroup configuration into a new filesystem. Btrfs does not have such a
>> function,it can only be done manually.
>>
>> So we want to implement btrfs qgroup import/export commands.
>> 1)'btrfs qgroup export' commands will export qgroup tree
>>   into a user's specified file.(stdout by default)
>>
>> 2)user may modify the configuration file firstly and then
>>   import it into the filesystem.(by 'btrfs qgroup import' command)
>>
>> The file may be formated as the following:
>>
>> Qgroupid     is_compressed     is_exclusive   limited_size    parent
>> ----------------------------------------------------------------------
>>  0/1            0                 0              10G            1/0
>>  1/0            1                 1              20G            ---
>>   
>>  If 'is_exclusive' is set, 'limited_size' corresponds to max exlusive size,
>>  else max referenced size. Here 'parent' exclude ancestral qgroups. 
>>
>> Is there any comment about this idea? 
> 
> The configuration only really makes sense in combination with the existing
> subvolumes. Even if the target has subvolumes under the same name, they
> might have different internal IDs. So it might make more sense to address
> the level 0 qgroups by name.

Good idea.

> Also it might be misleading to apply a configuration to an existing fs, as
> it currently is not possible get a correct accounting if the fs is not
> empty. Rescan is not yet implemented.

Rescan will be implemented in the future, so it is not a main problem
to implement 'btrfs qgroup import/export' commands.

> So instead of just saving and restoring the qgroup config, it might make
> more sense to create a new filesystem including all subvolumes and quota
> config from a config file.
> But, I'm not completely convinced that this is a features that is needed
> frequently. If I want a standard deployment, I simple write a script that
> creates the fs + subvol + quota.

If users want to config some qgroups(reset the limited size,
modify its ancestral qgroups),i think it is more convenient and flexible
to use import/export commands than write a script.


Above all,our qgroup import/export commands will be implemented as follows:

qgroupid  is_compressed  is_exclusive  limited_size   parent   full_path
------------------------------------------------------------------------

And we may specify matching degree when we import the qgroup information.

<1>strict matching
	qgroup(level-0) matches a subvolume/snapshot 's objectid and full path exactly.
	If a qgroup fail to match, the process will exit.

<2>general matching
	It only require qgroup(level-0) to match a subvolume/snapshot 's full path.
	If the corresponding subvolume/snapshot does not exist,skip it. Otherwise,apply
	modifications to the corresponding subvolume/snapshot qgroup.

<3>weak matching
	It only require qgroup(level-0) to match a subvolume/snapshot 's full path.
	If the corresponding subvolume/snapshot does not exist,create the subvolume
	automatically(a tracking qgroup is also created automatically)and then apply
	modifications to the newly created tracking qgroup.


How do you think about the above idea?

Thanks
Miao

Re: About btrfs qgroup import/export command

[Arne Jansen]

On 09.01.2013 11:17, Miao Xie wrote:
> Hi, Arne
> 
> On Wed, 19 Dec 2012 12:40:25 +0100, Arne Jansen wrote:
>> On 19.12.2012 12:25, Miao Xie wrote:
>>> As we know, there is no backup function for qgroup. when the problem
>>> occurs, the users must recover qgroup configuration manually, it is not
>>> convenient. And besides that, some users might want to import an existed
>>> qgroup configuration into a new filesystem. Btrfs does not have such a
>>> function,it can only be done manually.
>>>
>>> So we want to implement btrfs qgroup import/export commands.
>>> 1)'btrfs qgroup export' commands will export qgroup tree
>>>   into a user's specified file.(stdout by default)
>>>
>>> 2)user may modify the configuration file firstly and then
>>>   import it into the filesystem.(by 'btrfs qgroup import' command)
>>>
>>> The file may be formated as the following:
>>>
>>> Qgroupid     is_compressed     is_exclusive   limited_size    parent
>>> ----------------------------------------------------------------------
>>>  0/1            0                 0              10G            1/0
>>>  1/0            1                 1              20G            ---
>>>   
>>>  If 'is_exclusive' is set, 'limited_size' corresponds to max exlusive size,
>>>  else max referenced size. Here 'parent' exclude ancestral qgroups. 
>>>
>>> Is there any comment about this idea? 
>>
>> The configuration only really makes sense in combination with the existing
>> subvolumes. Even if the target has subvolumes under the same name, they
>> might have different internal IDs. So it might make more sense to address
>> the level 0 qgroups by name.
> 
> Good idea.
> 
>> Also it might be misleading to apply a configuration to an existing fs, as
>> it currently is not possible get a correct accounting if the fs is not
>> empty. Rescan is not yet implemented.
> 
> Rescan will be implemented in the future, so it is not a main problem
> to implement 'btrfs qgroup import/export' commands.
> 
>> So instead of just saving and restoring the qgroup config, it might make
>> more sense to create a new filesystem including all subvolumes and quota
>> config from a config file.
>> But, I'm not completely convinced that this is a features that is needed
>> frequently. If I want a standard deployment, I simple write a script that
>> creates the fs + subvol + quota.
> 
> If users want to config some qgroups(reset the limited size,
> modify its ancestral qgroups),i think it is more convenient and flexible
> to use import/export commands than write a script.
> 
> 
> Above all,our qgroup import/export commands will be implemented as follows:
> 
> qgroupid  is_compressed  is_exclusive  limited_size   parent   full_path
> ------------------------------------------------------------------------
> 
> And we may specify matching degree when we import the qgroup information.
> 
> <1>strict matching
> 	qgroup(level-0) matches a subvolume/snapshot 's objectid and full path exactly.
> 	If a qgroup fail to match, the process will exit.
> 
> <2>general matching
> 	It only require qgroup(level-0) to match a subvolume/snapshot 's full path.
> 	If the corresponding subvolume/snapshot does not exist,skip it. Otherwise,apply
> 	modifications to the corresponding subvolume/snapshot qgroup.
> 
> <3>weak matching
> 	It only require qgroup(level-0) to match a subvolume/snapshot 's full path.
> 	If the corresponding subvolume/snapshot does not exist,create the subvolume
> 	automatically(a tracking qgroup is also created automatically)and then apply
> 	modifications to the newly created tracking qgroup.
> 
> 
> How do you think about the above idea?

I still have problems imagining a use case for this. In our setup we have
lots of subvolumes with a quota configuration that follows some rules, but
it won't be possible to just import/export from one machine to the other.
So you have to design the tool to your needs.
There's another essential tool that's still missing with regard to quota
which I'd love to see come to life:
Currently the configuration of the tracking qgroups is completely left to
the user. This requires a deep understanding how qgroups work from him.
It would be great if we could come up with a simple description language
where the user just describes what he wants to achieve and the tool calculates
the tracking groups itself. It could also contain a templating mechanism
that might cover your use case.
A description might contain information e.g. which subvols to group, from which
subvols the user intends to take snapshots in the future and in which groups
those snapshots will be put. My pdf gives some example use cases which should
be possible to cover.
That's not exactly what you have in mind, but maybe it is possible to cover
both needs with one tool.

-Arne

> 
> Thanks
> Miao
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html