Quota limit question

Quota limit question

[Christian Robottom Reis]

Hello there,

    I'm trying out btrfs on a machine we use to host a number of
containers. After a misbehaved process filled the partition allocated to
the containers, I decided to experiment with quotas to isolate the
containers from each other. But I've now run into an oddity with one of
the containers, which reports being out of space:

    # btrfs qgroup limit 2000m 0/261 . && touch x
    touch: cannot touch ‘x’: Disk quota exceeded

The strange thing is that it doesn't seem to be actually out of space:

    # btrfs qgroup show -p -r -e /var | grep 261
    0/261    1111810048  391114752   2097152000  0          ---   

which pretty-printed is 1.04G rfer and 0.36G excl (perhaps the qgroup
show command could take an option to display in other units?)

I can only get it to allow me to start using it again if I go over 5808M:

    # btrfs qgroup limit 5807m 0/261 . && rm -f x && touch x
    rm: cannot remove ‘x’: Disk quota exceeded
    # btrfs qgroup limit 5808m 0/261 . && rm -f x && touch x
    #

Why specifically 5808 I'm not sure, but I binary searched until I got to
that number. Does anyone have a clue as to why that might be happening,
and perhaps what I'm missing?

For completeness, some details on the filesystem and system:

    # btrfs fi show /var
    Label: var  uuid: 815b3280-e90f-483a-b244-1d2dfe9b6e67
            Total devices 2 FS bytes used 31.48GiB
            devid    1 size 80.00GiB used 55.91GiB path /dev/sda3
            devid    2 size 80.00GiB used 55.91GiB path /dev/sdb3

    root@riff:/var/lib/lxc/async-local-machine-2/rootfs# btrfs fi df /var
    Data, RAID1: total=53.88GiB, used=30.45GiB
    System, RAID1: total=32.00MiB, used=16.00KiB
    Metadata, RAID1: total=2.00GiB, used=1.03GiB

    # btrfs qgroup show -p -r -e /var
    qgroupid rfer        excl        max_rfer    max_excl   parent  
    -------- ----        ----        --------    --------   ------  
    0/5      1486852096  1252569088  0           0          ---     
    0/259    727175168   104947712   0           5368709120 ---     
    0/261    1111810048  391114752   2097152000  0          ---   
    0/265    1255923712  442871808   0           5368709120 ---     
    0/271    831856640   333189120   0           5368709120 ---     
    0/274    498761728   22827008    0           5368709120 ---     
    0/283    7666098176  6691426304  10737418240 0          ---     
    0/288    1118441472  348901376   0           5368709120 ---     
    0/289    11134029824 10498187264 16106127360 0          ---     
    0/290    1412505600  694210560   10737418240 0          ---     
    0/292    1131053056  333373440   0           5368709120 ---     
    0/293    1258176512  401141760   0           5368709120 ---     
    0/306    1430532096  656773120   0           5368709120 ---     
    0/318    9309212672  8509857792  10737418240 0          ---     
    0/320    860209152   837406720   0           5368709120 ---     
    0/323    1167962112  469741568   0           5368709120 ---     

    # btrfs --version
    Btrfs v3.12

    # uname -a
    Linux riff 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014
    x86_64 x86_64 x86_64 GNU/Linux

Thanks,
-- 
Christian Robottom Reis | [+55 16] 3376 0125   | http://async.com.br/~kiko
Async Open Source       | [+55 16] 9 9112 6430 | http://launchpad.net/~kiko

Re: Quota limit question

[Duncan]

Christian Robottom Reis posted on Tue, 16 Dec 2014 23:15:37 -0200 as
excerpted:

>     # btrfs qgroup limit 2000m 0/261 . && touch x
>     touch: cannot touch ‘x’: Disk quota exceeded
> 
> The strange thing is that it doesn't seem to be actually out of space:
> 
>     # btrfs qgroup show -p -r -e /var | grep 261
>     0/261    1111810048  391114752   2097152000  0          ---

>     # btrfs --version Btrfs v3.12
> 
>     # uname -a
>     Linux riff 3.13.0-43-generic #72-Ubuntu SMP
>     Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Does the problem still exist with a current 3.18 series kernel and 3.17 
series btrfs userspace?  The quota code in the versions you appear to be 
running had some (rather severe, quotes could go negative at times!) 
known bugs and was rewritten recently.  I don't use quotas here and don't 
know for sure whether the quota rewrite made it into kernel 3.17, but 
current 3.18 should have it, and current userspace is I believe 3.17.3, 
so you should consider upgrading it as well.

More in general, consider your use-case.  While btrfs is no longer the 
experimental filesystem it once was (tho the warnings weren't removed 
until after the old 3.12/3.13 you're running, AFAIK), it's still 
maturing, and running year-old code still means running code with bugs 
that are known and have been fixed for a year.  There's a place for being 
conservative and running old and known stable code, but btrfs is still 
young and maturing and doesn't match that use-case very well.  Either you 
need stable and mature and an equally stable and mature filesystem is a 
better fit, at least for a couple more years, or you want newer 
technology/code and running year-old stuff is behind the curve.  Either 
way, running year-old btrfs code, both kernel and userspace, doesn't make 
a lot of sense.  Perhaps in a couple years it will, after btrfs is more 
mature and stable and year-old code doesn't mean year-stale bugs, but not 
currently.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Quota limit question

[Christian Robottom Reis]

On Tue, Dec 16, 2014 at 11:15:37PM -0200, Christian Robottom Reis wrote:
>     # btrfs qgroup limit 2000m 0/261 . && touch x
>     touch: cannot touch ‘x’: Disk quota exceeded
> 
> The strange thing is that it doesn't seem to be actually out of space:
> 
>     # btrfs qgroup show -p -r -e /var | grep 261
>     0/261    1111810048  391114752   2097152000  0          ---   

Replying to myself as I had not yet been subscribed in time to receive a
reply; I just upgraded to 3.18.1 and am seeing the same issue on the
same subvolume (and on no others).

    root@riff:/etc# uname -a
    Linux riff 3.18.1-031801-generic #201412170637 SMP Wed Dec 17 11:38:50
    UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

It's quite odd that this specific subvolume acts up, given that there
are quite a few others that are closer to the quota:

    subvol                  group        total       unshared
    -------------------------------------------------------------
    (unknown)               0/5      1.37G / none  1.16G /   none
    lxc-template1/rootfs    0/259    0.68G / none  0.10G /  2.00G
    machine-2/rootfs        0/261    1.07G / none  0.40G /  2.00G
    machine-3/rootfs        0/265    1.17G / none  0.41G /  2.00G
    lxc-template2/rootfs    0/271    0.77G / none  0.31G /  2.00G
    lxc-template3/rootfs    0/274    0.46G / none  0.02G /  2.00G
    machine-4/rootfs        0/283    7.12G / none  6.21G / 10.00G
    machine-5/rootfs        0/288    1.05G / none  0.34G /  2.00G
    machine-6/rootfs        0/289   11.33G / none 10.74G / 15.00G
    machine-7/rootfs        0/290    1.30G / none  0.68G /  2.00G
    machine-8/rootfs        0/292    1.00G / none  0.33G /  2.00G
    machine-9/rootfs        0/293    1.17G / none  0.38G /  2.00G
    machine-10/rootfs       0/306    1.34G / none  0.62G /  2.00G
    machine-11/rootfs       0/318    9.49G / none  8.75G / 15.00G
    lxc-template4/rootfs    0/320    0.79G / none  0.78G /  2.00G
    machine-14/rootfs       0/323    1.10G / none  0.45G /  2.00G

The LWN article suggests that btrfs is quite conservative with quotas,
but shouldn't 265, 290, 306, 320 and 323 all be out of quota as well? Or
is there a lot else that goes into the calculation beyond the numbers
reported by btrfs qgroup show?

What could I do to help investigate further?
-- 
Christian Robottom Reis | [+55 16] 3376 0125   | http://async.com.br/~kiko
CEO, Async Open Source  | [+55 16] 9 9112 6430 | http://launchpad.net/~kiko

Re: Quota limit question

[Duncan]

Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as
excerpted:

> On Tue, Dec 16, 2014 at 11:15:37PM -0200, Christian Robottom Reis wrote:
>>     # btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch
>>     ‘x’: Disk quota exceeded
>> 
>> The strange thing is that it doesn't seem to be actually out of space:
>> 
>>     # btrfs qgroup show -p -r -e /var | grep 261
>>     0/261    1111810048  391114752   2097152000  0          ---
> 
> Replying to myself as I had not yet been subscribed in time to receive a
> reply; I just upgraded to 3.18.1 and am seeing the same issue on the
> same subvolume (and on no others).

Looking at the thread here on gmane.org (list2news and list2web gateway), 
it appears my reply was the only reply in any case, and it was general as 
I don't run quotas myself.

Basically I suggested upgrading, as the quota code as some rather huge 
bugs in it (quotas could go seriously negative!) with the old versions 
you were running.  But you've upgraded at least the kernel now (userspace 
you didn't say).

Here's a link to the thread on the gmane web interface for completeness, 
but the above about covers my reply, as I said the only one until your 
thread bump and my reply here, so there's not much new there unless 
someone posts further followups to this thread...


http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Quota limit question

[Christian Robottom Reis]

Just as a follow-up, I upgraded btrfs-tools and the kernel again. I
currently have a filesystem which reports 1G exclusive use:

root@riff# btrfs qg show -r -e /var -p -c
qgroupid         rfer         excl     max_rfer     max_excl parent child 
--------         ----         ----     --------     -------- ------ ----- 
0/261         1.52GiB      1.01GiB        0.00B    100.00GiB ---     ---  

This filesystem reports over quota, and removing the quota fixes that:

root@riff# touch x
touch: cannot touch ‘x’: Disk quota exceeded
root@riff# btrfs qg limit -e
none 261 /var
root@riff# touch x
root@riff# 

So at the moment quotas are pretty much unusable in kernel 3.18.6/tools
3.18.2, at least for my use case, and that's a bit surprising since
there isn't anything very interesting about it (other than it contains a
bunch of lxc-cloned rootfs).

I've proactively added Yang who has submitted a few patches on quota
checking recently just to let me know if he thinks that this should be
fixed with a trunk kernel, or if he'd like to investigate or consider
this further. Thanks!

On Wed, Dec 24, 2014 at 03:52:41AM +0000, Duncan wrote:
> Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as
> excerpted:
> 
> > On Tue, Dec 16, 2014 at 11:15:37PM -0200, Christian Robottom Reis wrote:
> >>     # btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch
> >>     ‘x’: Disk quota exceeded
> >> 
> >> The strange thing is that it doesn't seem to be actually out of space:
> >> 
> >>     # btrfs qgroup show -p -r -e /var | grep 261
> >>     0/261    1111810048  391114752   2097152000  0          ---
> > 
> > Replying to myself as I had not yet been subscribed in time to receive a
> > reply; I just upgraded to 3.18.1 and am seeing the same issue on the
> > same subvolume (and on no others).
> 
> Looking at the thread here on gmane.org (list2news and list2web gateway), 
> it appears my reply was the only reply in any case, and it was general as 
> I don't run quotas myself.
> 
> Basically I suggested upgrading, as the quota code as some rather huge 
> bugs in it (quotas could go seriously negative!) with the old versions 
> you were running.  But you've upgraded at least the kernel now (userspace 
> you didn't say).
> 
> Here's a link to the thread on the gmane web interface for completeness, 
> but the above about covers my reply, as I said the only one until your 
> thread bump and my reply here, so there's not much new there unless 
> someone posts further followups to this thread...
> 
> 
> http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
> 
> 
> -- 
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Christian Robottom Reis   | [+1] 612 888 4935    | http://launchpad.net/~kiko
Canonical VP Hyperscale   | [+55 16] 9 9112 6430

Re: Quota limit question

[Qu Wenruo]

-------- Original Message  --------
Subject: Re: Quota limit question
From: Christian Robottom Reis <kiko@canonical.com>
To: Duncan <1i5t5.duncan@cox.net>
Date: 2015年03月07日 05:44

> Just as a follow-up, I upgraded btrfs-tools and the kernel again. I
> currently have a filesystem which reports 1G exclusive use:
>
> root@riff# btrfs qg show -r -e /var -p -c
> qgroupid         rfer         excl     max_rfer     max_excl parent child
> --------         ----         ----     --------     -------- ------ -----
> 0/261         1.52GiB      1.01GiB        0.00B    100.00GiB ---     ---
It's recommended to sync the btrfs before 'qg show' command, since quota 
tree is not updated until commit transaction.
>
> This filesystem reports over quota, and removing the quota fixes that:
>
> root@riff# touch x
> touch: cannot touch ‘x’: Disk quota exceeded
> root@riff# btrfs qg limit -e
> none 261 /var
> root@riff# touch x
> root@riff#
>
> So at the moment quotas are pretty much unusable in kernel 3.18.6/tools
> 3.18.2, at least for my use case, and that's a bit surprising since
> there isn't anything very interesting about it (other than it contains a
> bunch of lxc-cloned rootfs).
Yes, Btrfs really has some problems, from yours to other annoying 
incorrect values.

[Some guess]
For your case, I'm afraid your reserved space in your qgroup takes too 
much space, and it's definitely a bug. Maybe some btrfs_qgroup_reserve() 
leaking?

[Reproducer]
Would you please describe your workload? Or some easy-to-reproduce scripts?

[Image dump/debug tree]
Another one which may help is your btrfs-debug-tree output on your quota 
tree.
You can dump it by "btrfs-debug-tree -t 8 <DEVICE>".
Or full disk metadata dump by "btrfs-image -c9 <DEVICE>"

Thanks,
Qu
>
> I've proactively added Yang who has submitted a few patches on quota
> checking recently just to let me know if he thinks that this should be
> fixed with a trunk kernel, or if he'd like to investigate or consider
> this further. Thanks!
>
> On Wed, Dec 24, 2014 at 03:52:41AM +0000, Duncan wrote:
>> Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as
>> excerpted:
>>
>>> On Tue, Dec 16, 2014 at 11:15:37PM -0200, Christian Robottom Reis wrote:
>>>>      # btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch
>>>>      ‘x’: Disk quota exceeded
>>>>
>>>> The strange thing is that it doesn't seem to be actually out of space:
>>>>
>>>>      # btrfs qgroup show -p -r -e /var | grep 261
>>>>      0/261    1111810048  391114752   2097152000  0          ---
>>>
>>> Replying to myself as I had not yet been subscribed in time to receive a
>>> reply; I just upgraded to 3.18.1 and am seeing the same issue on the
>>> same subvolume (and on no others).
>>
>> Looking at the thread here on gmane.org (list2news and list2web gateway),
>> it appears my reply was the only reply in any case, and it was general as
>> I don't run quotas myself.
>>
>> Basically I suggested upgrading, as the quota code as some rather huge
>> bugs in it (quotas could go seriously negative!) with the old versions
>> you were running.  But you've upgraded at least the kernel now (userspace
>> you didn't say).
>>
>> Here's a link to the thread on the gmane web interface for completeness,
>> but the above about covers my reply, as I said the only one until your
>> thread bump and my reply here, so there's not much new there unless
>> someone posts further followups to this thread...
>>
>>
>> http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
>>
>>
>> --
>> Duncan - List replies preferred.   No HTML msgs.
>> "Every nonfree program has a lord, a master --
>> and if you use the program, he is your master."  Richard Stallman
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quota limit question

[Dongsheng Yang]

Sorry about the late reply!!

On 03/10/2015 04:41 PM, Qu Wenruo wrote:
>
>
> -------- Original Message  --------
> Subject: Re: Quota limit question
> From: Christian Robottom Reis <kiko@canonical.com>
> To: Duncan <1i5t5.duncan@cox.net>
> Date: 2015年03月07日 05:44
>
>> Just as a follow-up, I upgraded btrfs-tools and the kernel again. I
>> currently have a filesystem which reports 1G exclusive use:
>>
>> root@riff# btrfs qg show -r -e /var -p -c
>> qgroupid         rfer         excl     max_rfer     max_excl parent 
>> child
>> --------         ----         ----     --------     -------- ------ 
>> -----
>> 0/261         1.52GiB      1.01GiB        0.00B    100.00GiB ---     ---
> It's recommended to sync the btrfs before 'qg show' command, since 
> quota tree is not updated until commit transaction.

Yes, sync is required be fore qg show now. There is no ioctl to get 
qgroup info from kernel space,
so we have to read the data from disk then we have to do a sync before 
we getting the correct information.
>>
>> This filesystem reports over quota, and removing the quota fixes that:
>>
>> root@riff# touch x
>> touch: cannot touch ‘x’: Disk quota exceeded
>> root@riff# btrfs qg limit -e
>> none 261 /var
>> root@riff# touch x
>> root@riff#
>>
>> So at the moment quotas are pretty much unusable in kernel 3.18.6/tools
>> 3.18.2, at least for my use case, and that's a bit surprising since
>> there isn't anything very interesting about it (other than it contains a
>> bunch of lxc-cloned rootfs).
> Yes, Btrfs really has some problems, from yours to other annoying 
> incorrect values.
>
> [Some guess]
> For your case, I'm afraid your reserved space in your qgroup takes too 
> much space, and it's definitely a bug. Maybe some 
> btrfs_qgroup_reserve() leaking?

Agreed, there must be a reserve leaking in your use case. And I have posted
some patches about qgroup.reserved. some of them is about reserved leaking.

Could you try the tree :

	btrfs:https://yangdongsheng@github.com/yangdongsheng/linux.git  qgroup_type
   btrfs-progs:https://yangdongsheng@github.com/yangdongsheng/btrfs-progs.git  qgroup_type

It could be working better but I can not guarantee your problem here 
will be solved by it.

>
> [Reproducer]
> Would you please describe your workload? Or some easy-to-reproduce 
> scripts?
>
> [Image dump/debug tree]
> Another one which may help is your btrfs-debug-tree output on your 
> quota tree.
> You can dump it by "btrfs-debug-tree -t 8 <DEVICE>".
> Or full disk metadata dump by "btrfs-image -c9 <DEVICE>"

A reproducer is great for investigation. "btrfs-debug-tree" is important 
to us.

Thanx
Yang
>
> Thanks,
> Qu
>>
>> I've proactively added Yang who has submitted a few patches on quota
>> checking recently just to let me know if he thinks that this should be
>> fixed with a trunk kernel, or if he'd like to investigate or consider
>> this further. Thanks!
>>
>> On Wed, Dec 24, 2014 at 03:52:41AM +0000, Duncan wrote:
>>> Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as
>>> excerpted:
>>>
>>>> On Tue, Dec 16, 2014 at 11:15:37PM -0200, Christian Robottom Reis 
>>>> wrote:
>>>>>      # btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot 
>>>>> touch
>>>>>      ‘x’: Disk quota exceeded
>>>>>
>>>>> The strange thing is that it doesn't seem to be actually out of 
>>>>> space:
>>>>>
>>>>>      # btrfs qgroup show -p -r -e /var | grep 261
>>>>>      0/261    1111810048  391114752   2097152000 0          ---
>>>>
>>>> Replying to myself as I had not yet been subscribed in time to 
>>>> receive a
>>>> reply; I just upgraded to 3.18.1 and am seeing the same issue on the
>>>> same subvolume (and on no others).
>>>
>>> Looking at the thread here on gmane.org (list2news and list2web 
>>> gateway),
>>> it appears my reply was the only reply in any case, and it was 
>>> general as
>>> I don't run quotas myself.
>>>
>>> Basically I suggested upgrading, as the quota code as some rather huge
>>> bugs in it (quotas could go seriously negative!) with the old versions
>>> you were running.  But you've upgraded at least the kernel now 
>>> (userspace
>>> you didn't say).
>>>
>>> Here's a link to the thread on the gmane web interface for 
>>> completeness,
>>> but the above about covers my reply, as I said the only one until your
>>> thread bump and my reply here, so there's not much new there unless
>>> someone posts further followups to this thread...
>>>
>>>
>>> http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
>>>
>>>
>>> -- 
>>> Duncan - List replies preferred.   No HTML msgs.
>>> "Every nonfree program has a lord, a master --
>>> and if you use the program, he is your master."  Richard Stallman
>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe 
>>> linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> .
>