Quota question

Quota question

[Cyril Scetbon]

Hi,

At https://btrfs.wiki.kernel.org/index.php/Quota_support it's said that "Using btrfs subvolume delete will break qgroup unshared space usage". Does it mean that if the qgroup associated with the subvolume is attached to another qgroup, this other qgroup won't be able to correctly apply its quota ? If yes, is there a workaround for that ?

Thanks
-- 
Cyril SCETBON

Quota question

[Cyril Scetbon]

Hi,

At https://btrfs.wiki.kernel.org/index.php/Quota_support it's said that "Using btrfs subvolume delete will break qgroup unshared space usage". Does it mean that if the qgroup associated with the subvolume is attached to another qgroup, this other qgroup won't be able to correctly apply its quota ? If yes, is there a workaround for that ?

Thanks
-- 
Cyril SCETBON

Re: Quota question

[Cyril Scetbon]

Besides the first question, I met an issue using parent groups (see http://pastebin.com/asT5ZFsi). I can't reproduce it all the time, but it seems to appear frequently. Is there any know BUG that can be the source of this error ? I'm using version 3.12 on Trusty

Thanks
-- 
Cyril SCETBON

> On 30 Oct 2014, at 19:37, Cyril Scetbon <cyril.scetbon@free.fr> wrote:
> 
> Hi,
> 
> At https://btrfs.wiki.kernel.org/index.php/Quota_support it's said that "Using btrfs subvolume delete will break qgroup unshared space usage". Does it mean that if the qgroup associated with the subvolume is attached to another qgroup, this other qgroup won't be able to correctly apply its quota ? If yes, is there a workaround for that ?
> 
> Thanks
> -- 
> Cyril SCETBON
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quota question

[Duncan]

Cyril Scetbon posted on Fri, 31 Oct 2014 09:45:23 +0100 as excerpted:

> Besides the first question, I met an issue using parent groups (see
> http://pastebin.com/asT5ZFsi). I can't reproduce it all the time, but it
> seems to appear frequently. Is there any know BUG that can be the source
> of this error ? I'm using version 3.12 on Trusty

I don't use quotas personally, but I know for quite some time they were 
essentially "broken" in certain instances.  My recommendation as a list 
regular then was, unless your goal is specific quota-feature-testing, if 
you /need/ quotas, use a more mature filesystem that can reliably provide 
them; if you don't need quotas, turn them off and avoid the quota related 
btrfs bugs.

Recently, around 3.16 timeframe I believe, there was quite a btrfs quota-
subsystem rewrite designed to tackle and eliminate these problems.  Now 
since I don't use quotas personally and I've not seen a definitive 
statement on-list one way or the other I'm not sure if the full rewrite 
is done yet or not, but I've not see further followup patches either, so 
barring further information to the contrary I'd guess it is.  However, 
it's still early in the history of the new code  and it's still just 
that, new.  While I've not seen many quota-related bug reports from it, 
it may well be that enough people took the general recommendation above 
that it simply hasn't gotten much testing.

So I honestly can't say what the state of the new quota code is, but one 
thing I can say for sure is that the quota code in 3.12 is the old code, 
known to be broken and now rewritten, dead quota code.  So for sure I'd 
say don't use it.  It's known to be broken and there's simply no reason 
to do so.

But with the rewrite you now have three choices instead of two.  If you 
need quotas and are up for testing relatively new code (and btrfs itself 
isn't exactly the maturest around, so if you're not up for testing new 
code, why are you running btrfs at all?), try a recent enough btrfs that 
you are running the new quota code and your tests and potentially bug 
reports will be actually worthwhile.  I just read today that Ubuntu is 
maintaining the 3.16 kernel for somewhat longer in coordination with 
their distro releases, after Greg recently released the last planned 
regular stable 3.16 release, so that's a reasonable kernel series to 
settle with.  I /think/ it has the new quota code, and certainly, the bad 
bug in the 3.15 series is fixed (in 3.16.2) and the bug that plagued 
3.17, now fixed in 3.18-rc2 and headed for 3.17 stable, wouldn't be in 
3.16 stable either.  Or you can of course follow current mainline.

The other two choices are as before, turn off quotas in btrfs, or switch 
to a more mature filesystem with reliable quota support.  But staying 
with 3.12 with btrfs quotas enabled simply doesn't make sense, because as 
I said, the quota code there is dead and known broken, so that's not a 
reasonable option at all.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Quota question

[Cyril Scetbon]

Hmm I chose btrfs because it's the only one supported by docker that supports quotas for the root user ...
Si i need to be sure it's really broken as I must use the version provided by distribution I will use, i.e Ubuntu trusty.

I tried simple quotas and it seems to work, but I have an issue with group of quotas 
I'd really appreciate if someone can tell me exactly what should work and what shouldn't 

Thanks

Cyril Scetbon

> On 01 Nov 2014, at 08:00, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> So I honestly can't say what the state of the new quota code is, but one 
> thing I can say for sure is that the quota code in 3.12 is the old code, 
> known to be broken and now rewritten, dead quota code.

Re: Quota question

[Chris Murphy]

On Nov 1, 2014, at 2:53 PM, Cyril Scetbon <cyril.scetbon@free.fr> wrote:

> Hmm I chose btrfs because it's the only one supported by docker that supports quotas for the root user …

https://github.com/docker/docker/issues/3804

I don't think Docker is supporting quotas at all on anything yet. They want to do this but it seems the work hasn't been done yet, and they want it to be agnostic so it can work with device mapper, XFS, EXT4, and Btrfs.


> I'd really appreciate if someone can tell me exactly what should work and what shouldn't 

Well I haven't tested this at all yet, but maybe XFS project quotas fits your use case better than group quotas? With project quotas you can still also use user quotas if necessary (but project and group quotas on XFS are mutually exclusive).

Chris Murphy

Re: Quota question

[Cyril Scetbon]

>> Hmm I chose btrfs because it's the only one supported by docker that supports quotas for the root user …
> 
> https://github.com/docker/docker/issues/3804
> 
> I don't think Docker is supporting quotas at all on anything yet. They want to do this but it seems the work hasn't been done yet, and they want it to be agnostic so it can work with device mapper, XFS, EXT4, and Btrfs.

What I'm saying is that btrfs is the only one that can apply quotas for root user (as docker needs to be launched as root user for some reason but it could change in the future). So Today, I apply quotas them after each container creation. However, you're right it could change in the next months but that's not the case today …

>> I'd really appreciate if someone can tell me exactly what should work and what shouldn't 
> 
> Well I haven't tested this at all yet, but maybe XFS project quotas fits your use case better than group quotas? With project quotas you can still also use user quotas if necessary (but project and group quotas on XFS are mutually exclusive).

I have only one user that creates/launches containers, root ! What I'm looking for is being able to apply quotas to a container and also to a group of containers. That's why I'm trying to use parent qgroups with btrfs.

-- 
Cyril SCETBON

Re: Quota question

[Chris Murphy]

On Nov 2, 2014, at 3:42 AM, Cyril Scetbon <cyril.scetbon@free.fr> wrote:

> 
>>> Hmm I chose btrfs because it's the only one supported by docker that supports quotas for the root user …
>> 
>> https://github.com/docker/docker/issues/3804
>> 
>> I don't think Docker is supporting quotas at all on anything yet. They want to do this but it seems the work hasn't been done yet, and they want it to be agnostic so it can work with device mapper, XFS, EXT4, and Btrfs.
> 
> What I'm saying is that btrfs is the only one that can apply quotas for root user (as docker needs to be launched as root user for some reason but it could change in the future).

OK no, XFS project quotas apply to root also which is why I suggested it.

[root@localhost project_quota_test1]# xfs_quota -c df
Filesystem           1K-blocks       Used  Available  Use% Pathname
/dev/sdb              83845120     157980   83687140    0% /xfs_local
/dev/sdb                102400     124928 9223372036854753280  122% /xfs_local/project_quota_test1
[root@localhost project_quota_test1]# dd if=/dev/zero of=test100MB bs=1M count=100
dd: error writing ‘test100MB’: No space left on device
79+0 records in
78+0 records out
81788928 bytes (82 MB) copied, 0.163849 s, 499 MB/s
[root@localhost project_quota_test1]# xfs_quota -c df
Filesystem           1K-blocks       Used  Available  Use% Pathname
/dev/sdb              83845120     237748   83607372    0% /xfs_local
/dev/sdb                102400     204800 9223372036854673408  200% /xfs_local/project_quota_test1


The available value is totally wonky, but once I reach the 200MB hard limit on this directory, even as root I get a no space left on device message.


> 
> 
>>> I'd really appreciate if someone can tell me exactly what should work and what shouldn't 
>> 
>> Well I haven't tested this at all yet, but maybe XFS project quotas fits your use case better than group quotas? With project quotas you can still also use user quotas if necessary (but project and group quotas on XFS are mutually exclusive).
> 
> I have only one user that creates/launches containers, root ! What I'm looking for is being able to apply quotas to a container and also to a group of containers. That's why I'm trying to use parent qgroups with btrfs.

It looks like XFS project quotas will do what you want.


Chris Murphy

Re: Quota question

[Cyril Scetbon]

oh cool to know !

It's weird that the man page says "limits are never enforced on the superuser (nor are they enforced for group and project ID zero)"

http://manpages.ubuntu.com/manpages/trusty/en/man8/xfs_quota.8.html

Thanks
-- 
Cyril SCETBON

> On 02 Nov 2014, at 22:48, Chris Murphy <lists@colorremedies.com> wrote:
> 
> 
> On Nov 2, 2014, at 3:42 AM, Cyril Scetbon <cyril.scetbon@free.fr> wrote:
> 
>> 
>>>> Hmm I chose btrfs because it's the only one supported by docker that supports quotas for the root user …
>>> 
>>> https://github.com/docker/docker/issues/3804
>>> 
>>> I don't think Docker is supporting quotas at all on anything yet. They want to do this but it seems the work hasn't been done yet, and they want it to be agnostic so it can work with device mapper, XFS, EXT4, and Btrfs.
>> 
>> What I'm saying is that btrfs is the only one that can apply quotas for root user (as docker needs to be launched as root user for some reason but it could change in the future).
> 
> OK no, XFS project quotas apply to root also which is why I suggested it.
> 
> [root@localhost project_quota_test1]# xfs_quota -c df
> Filesystem           1K-blocks       Used  Available  Use% Pathname
> /dev/sdb              83845120     157980   83687140    0% /xfs_local
> /dev/sdb                102400     124928 9223372036854753280  122% /xfs_local/project_quota_test1
> [root@localhost project_quota_test1]# dd if=/dev/zero of=test100MB bs=1M count=100
> dd: error writing ‘test100MB’: No space left on device
> 79+0 records in
> 78+0 records out
> 81788928 bytes (82 MB) copied, 0.163849 s, 499 MB/s
> [root@localhost project_quota_test1]# xfs_quota -c df
> Filesystem           1K-blocks       Used  Available  Use% Pathname
> /dev/sdb              83845120     237748   83607372    0% /xfs_local
> /dev/sdb                102400     204800 9223372036854673408  200% /xfs_local/project_quota_test1
> 
> 
> The available value is totally wonky, but once I reach the 200MB hard limit on this directory, even as root I get a no space left on device message.
> 
> 
>> 
>> 
>>>> I'd really appreciate if someone can tell me exactly what should work and what shouldn't 
>>> 
>>> Well I haven't tested this at all yet, but maybe XFS project quotas fits your use case better than group quotas? With project quotas you can still also use user quotas if necessary (but project and group quotas on XFS are mutually exclusive).
>> 
>> I have only one user that creates/launches containers, root ! What I'm looking for is being able to apply quotas to a container and also to a group of containers. That's why I'm trying to use parent qgroups with btrfs.
> 
> It looks like XFS project quotas will do what you want.
> 
> 
> Chris Murphy
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quota question

[Cyril Scetbon]

Hi guys,

I've made some tests with a newer version of btrfs, and I still have issues :(

http://pastebin.com/cycu08LG

I have 2 directories with quotas and a parent qgroup. All have quotas limits but something goes wrong. When the parent limit is exceeded I have a "Disk quota exceeded" which is what was expecting. However, after some tests I can exceed the limits (see line 80 and 84), but after that I can nolonger write to disk without getting an error and even if I just want to suppress the data !

FYI I'm using this kernel https://launchpad.net/ubuntu/+source/linux/3.16.0-25.33

Any idea ? Bug ?
-- 
Cyril SCETBON

> On 01 Nov 2014, at 08:00, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> Cyril Scetbon posted on Fri, 31 Oct 2014 09:45:23 +0100 as excerpted:
> 
>> Besides the first question, I met an issue using parent groups (see
>> http://pastebin.com/asT5ZFsi). I can't reproduce it all the time, but it
>> seems to appear frequently. Is there any know BUG that can be the source
>> of this error ? I'm using version 3.12 on Trusty
> 
> I don't use quotas personally, but I know for quite some time they were 
> essentially "broken" in certain instances.  My recommendation as a list 
> regular then was, unless your goal is specific quota-feature-testing, if 
> you /need/ quotas, use a more mature filesystem that can reliably provide 
> them; if you don't need quotas, turn them off and avoid the quota related 
> btrfs bugs.
> 
> Recently, around 3.16 timeframe I believe, there was quite a btrfs quota-
> subsystem rewrite designed to tackle and eliminate these problems.  Now 
> since I don't use quotas personally and I've not seen a definitive 
> statement on-list one way or the other I'm not sure if the full rewrite 
> is done yet or not, but I've not see further followup patches either, so 
> barring further information to the contrary I'd guess it is.  However, 
> it's still early in the history of the new code  and it's still just 
> that, new.  While I've not seen many quota-related bug reports from it, 
> it may well be that enough people took the general recommendation above 
> that it simply hasn't gotten much testing.
> 
> So I honestly can't say what the state of the new quota code is, but one 
> thing I can say for sure is that the quota code in 3.12 is the old code, 
> known to be broken and now rewritten, dead quota code.  So for sure I'd 
> say don't use it.  It's known to be broken and there's simply no reason 
> to do so.
> 
> But with the rewrite you now have three choices instead of two.  If you 
> need quotas and are up for testing relatively new code (and btrfs itself 
> isn't exactly the maturest around, so if you're not up for testing new 
> code, why are you running btrfs at all?), try a recent enough btrfs that 
> you are running the new quota code and your tests and potentially bug 
> reports will be actually worthwhile.  I just read today that Ubuntu is 
> maintaining the 3.16 kernel for somewhat longer in coordination with 
> their distro releases, after Greg recently released the last planned 
> regular stable 3.16 release, so that's a reasonable kernel series to 
> settle with.  I /think/ it has the new quota code, and certainly, the bad 
> bug in the 3.15 series is fixed (in 3.16.2) and the bug that plagued 
> 3.17, now fixed in 3.18-rc2 and headed for 3.17 stable, wouldn't be in 
> 3.16 stable either.  Or you can of course follow current mainline.
> 
> The other two choices are as before, turn off quotas in btrfs, or switch 
> to a more mature filesystem with reliable quota support.  But staying 
> with 3.12 with btrfs quotas enabled simply doesn't make sense, because as 
> I said, the quota code there is dead and known broken, so that's not a 
> reasonable option at all.
> 
> -- 
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quota question

[Christoph Hellwig]

On Wed, Nov 05, 2014 at 09:52:18PM +0100, Cyril Scetbon wrote:
> oh cool to know !
> 
> It's weird that the man page says "limits are never enforced on the superuser (nor are they enforced for group and project ID zero)"
> 
> http://manpages.ubuntu.com/manpages/trusty/en/man8/xfs_quota.8.html

This means it's never enforced for (uid|gid|prij) zero.  If you can come
up with a better wording were happy to update the man page.

Re: Quota question

[Duncan]

Cyril Scetbon posted on Fri, 07 Nov 2014 20:06:40 +0100 as excerpted:

> I've made some tests with a newer version of btrfs, and I still have
> issues :(
> 
> http://pastebin.com/cycu08LG
> 
> I have 2 directories with quotas and a parent qgroup. All have quotas
> limits but something goes wrong. When the parent limit is exceeded I
> have a "Disk quota exceeded" which is what was expecting. However, after
> some tests I can exceed the limits (see line 80 and 84), but after that
> I can nolonger write to disk without getting an error and even if I just
> want to suppress the data !
> 
> FYI I'm using this kernel
> https://launchpad.net/ubuntu/+source/linux/3.16.0-25.33
> 
> Any idea ? Bug ?

Again admitting that because I don't use quotas personally and don't 
follow them closely on btrfs, I'm not exactly the ideal person to be 
answering questions on them... but in the absence of someone more 
informed taking a shot (which I'd greet with great relief as I'm 
definitely out of my comfortable knowledge range at this point)...

Two comments:

1) As I said earlier I believe the btrfs quota rewrite was in the 3.16 
kernel series timeframe.  Unfortunately I don't know its current status.  
You say kernel 3.16.0 (-25.33, which means nothing to me, what I know is 
about the mainline Linus kernel and to a bit less of an extent the 
official mainline stable kernels), but is the rewrite in that, or in 
3.17, or in the coming 3.18, or not yet entirely there at all... that I 
simply don't know.

2) What I *DO* know and *CAN* say with quite some confidence is this:

With btrfs being in general a copy-on-write filesystem, even deletes 
normally require /some/ space, because the metadata pointing at the file 
in question must be rewritten, and that requires free space in ordered to 
happen.

IOW, regardless of whether it's quotas or full filesystem constraints 
that are limiting space and provoking ENOSPC errors, depending on how bad 
an out-of-space condition actually is, even deleting files in ordered to 
free more space may be impossible, because the act of deleting files 
requires a rewrite and thus triggers a copy-on-write of the metadata 
involved, which itself requires some space.

At least in the case of full filesystem ENOSPC errors, the space-related 
questions in the FAQ on the btrfs wiki suggest truncating a file first.  
Find an existing file that either doesn't matter or that you have a 
backup of elsewhere, and echo > filename (or echo >| filename if you have 
noclobber set), therefore truncating the existing file.  A few of those 
and with luck you'll free enough space to do some actual deletions.

While to some extent I'm guessing here since I'm not directly familiar 
with quotas myself, it sounds as if such a severe ENOSPC condition, due 
in your case to quotas instead of the general filesystem limits I'm more 
familiar with, might be what you are seeing, particularly since in your 
tests you first triggered the limit, then found a way to write a bit more 
anyway, and only THEN found yourself unable to write anything.

If that's the case, then try the truncate trick and see if it helps.

In the generic filesystem case, if the truncate trick doesn't help, 
another possible trick is to temporarily add another device of several 
gigs capacity to the filesystem, enough to let a balance work and 
hopefully reduce the data/metadata imbalance, after which there should be 
enough chunk-unallocated free space left on the original device, to do a 
btrfs device delete of the temporary device, generally still leaving 
space left, since the balance should have reclaimed some otherwise wasted 
chunk allocation.

I can't see how that'd apply to quotas, however, and suspect that the 
corresponding quota-system solution would be to temporarily up the quota 
for the affected user, only long enough to let them do the deletions they 
need to do.  But beyond that I can't go, since I simply don't know enough 
about the quota domain to further speculate, at this point.

While admittedly limited due to my lack of quota-domain knowledge, 
perhaps that sheds at least some light on what happened and why, and 
hopefully on how to get out of the situation.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Quota question

[Cyril Scetbon]

-- 
Cyril SCETBON

> 1)  but is the rewrite in that, or in 
> 3.17, or in the coming 3.18, or not yet entirely there at all... that I 
> simply don't know.

If anyone can answer it'd be great !

> 
> 2) What I *DO* know and *CAN* say with quite some confidence is this:
> 
> With btrfs being in general a copy-on-write filesystem, even deletes 
> normally require /some/ space, because the metadata pointing at the file 
> in question must be rewritten, and that requires free space in ordered to 
> happen.
ok
> 
> At least in the case of full filesystem ENOSPC errors, the space-related 
> questions in the FAQ on the btrfs wiki suggest truncating a file first.  
> Find an existing file that either doesn't matter or that you have a 
> backup of elsewhere, and echo > filename (or echo >| filename if you have 
> noclobber set), therefore truncating the existing file.  A few of those 
> and with luck you'll free enough space to do some actual deletions.
> 
> While to some extent I'm guessing here since I'm not directly familiar 
> with quotas myself, it sounds as if such a severe ENOSPC condition, due 
> in your case to quotas instead of the general filesystem limits I'm more 
> familiar with, might be what you are seeing, particularly since in your 
> tests you first triggered the limit, then found a way to write a bit more 
> anyway, and only THEN found yourself unable to write anything.
> 
> If that's the case, then try the truncate trick and see if it helps.
Unfortunately : http://pastebin.com/1Mw0yiUc
> 
> In the generic filesystem case, if the truncate trick doesn't help, 
> another possible trick is to temporarily add another device of several 
> gigs capacity to the filesystem, enough to let a balance work and 
> hopefully reduce the data/metadata imbalance, after which there should be 
> enough chunk-unallocated free space left on the original device, to do a 
> btrfs device delete of the temporary device, generally still leaving 
> space left, since the balance should have reclaimed some otherwise wasted 
> chunk allocation.
> 
> I can't see how that'd apply to quotas, however, and suspect that the 
> corresponding quota-system solution would be to temporarily up the quota 
> for the affected user, only long enough to let them do the deletions they 
> need to do.  But beyond that I can't go, since I simply don't know enough 
> about the quota domain to further speculate, at this point.

I'm pretty sure it'd work by setting a temporary higher quota, but my questions are : 

- why can I exceed the quotas (580MB when 500MB was set as the limit) ?

qgroupid  excl            max_excl
-------------------------------------------
1/101999 608632832 524288000

- I really think that the quota limit should take into account an amount of space for operations that would free space

For this last question I may think that this issue is caused by the fact that I have only one file taking all the space which won't really be the regular use case. but i'd like to get an answer :)
> 
> While admittedly limited due to my lack of quota-domain knowledge, 
> perhaps that sheds at least some light on what happened and why, and 
> hopefully on how to get out of the situation.

Thank you Duncan !

Re: Quota question

[Cyril Scetbon]

Hey,

I did some new tests and the issue with quota exceeded comes from the fact that the data is not sync to disk : http://pastebin.com/fZdh2YRu

-- 
Cyril SCETBON

> On 09 Nov 2014, at 18:55, Cyril Scetbon <cyril.scetbon@free.fr> wrote:
> 
> 
> -- 
> Cyril SCETBON
> 
>> 1)  but is the rewrite in that, or in 
>> 3.17, or in the coming 3.18, or not yet entirely there at all... that I 
>> simply don't know.
> 
> If anyone can answer it'd be great !
> 
>> 
>> 2) What I *DO* know and *CAN* say with quite some confidence is this:
>> 
>> With btrfs being in general a copy-on-write filesystem, even deletes 
>> normally require /some/ space, because the metadata pointing at the file 
>> in question must be rewritten, and that requires free space in ordered to 
>> happen.
> ok
>> 
>> At least in the case of full filesystem ENOSPC errors, the space-related 
>> questions in the FAQ on the btrfs wiki suggest truncating a file first.  
>> Find an existing file that either doesn't matter or that you have a 
>> backup of elsewhere, and echo > filename (or echo >| filename if you have 
>> noclobber set), therefore truncating the existing file.  A few of those 
>> and with luck you'll free enough space to do some actual deletions.
>> 
>> While to some extent I'm guessing here since I'm not directly familiar 
>> with quotas myself, it sounds as if such a severe ENOSPC condition, due 
>> in your case to quotas instead of the general filesystem limits I'm more 
>> familiar with, might be what you are seeing, particularly since in your 
>> tests you first triggered the limit, then found a way to write a bit more 
>> anyway, and only THEN found yourself unable to write anything.
>> 
>> If that's the case, then try the truncate trick and see if it helps.
> Unfortunately : http://pastebin.com/1Mw0yiUc
>> 
>> In the generic filesystem case, if the truncate trick doesn't help, 
>> another possible trick is to temporarily add another device of several 
>> gigs capacity to the filesystem, enough to let a balance work and 
>> hopefully reduce the data/metadata imbalance, after which there should be 
>> enough chunk-unallocated free space left on the original device, to do a 
>> btrfs device delete of the temporary device, generally still leaving 
>> space left, since the balance should have reclaimed some otherwise wasted 
>> chunk allocation.
>> 
>> I can't see how that'd apply to quotas, however, and suspect that the 
>> corresponding quota-system solution would be to temporarily up the quota 
>> for the affected user, only long enough to let them do the deletions they 
>> need to do.  But beyond that I can't go, since I simply don't know enough 
>> about the quota domain to further speculate, at this point.
> 
> I'm pretty sure it'd work by setting a temporary higher quota, but my questions are : 
> 
> - why can I exceed the quotas (580MB when 500MB was set as the limit) ?
> 
> qgroupid  excl            max_excl
> -------------------------------------------
> 1/101999 608632832 524288000
> 
> - I really think that the quota limit should take into account an amount of space for operations that would free space
> 
> For this last question I may think that this issue is caused by the fact that I have only one file taking all the space which won't really be the regular use case. but i'd like to get an answer :)
>> 
>> While admittedly limited due to my lack of quota-domain knowledge, 
>> perhaps that sheds at least some light on what happened and why, and 
>> hopefully on how to get out of the situation.
> 
> Thank you Duncan !--
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Quota question

[Cyril Scetbon]

Anyone on this ? There is an issue with quotas depending on the write rate. The more we can write before a sync, the more we can exceed quotas limits
-- 
Cyril SCETBON

> I did some new tests and the issue with quota exceeded comes from the fact that the data is not sync to disk : http://pastebin.com/fZdh2YRu

Re: Quota question

[Wang Shilong]

> Anyone on this ? There is an issue with quotas depending on the write rate. The more we can write before a sync, the more we can exceed quotas limits

Absolutely, this was a regression bug since Josef introduce delay refs merging for qgroup.
Josef, would you please spend some time tracking this problem down?


> -- 
> Cyril SCETBON
> 
>> I did some new tests and the issue with quota exceeded comes from the fact that the data is not sync to disk : http://pastebin.com/fZdh2YRu
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Best Regards,
Wang Shilong

Re: Quota question

[Dongsheng Yang]

On 11/12/2014 10:04 PM, Cyril Scetbon wrote:
> Anyone on this ? There is an issue with quotas depending on the write rate. The more we can write before a sync, the more we can exceed quotas limits

Hi Cyril, I attempted to reproduce the problem you reported in 
Linux.3.17, but failed.
It seems that the issue in this thread was fixed already. Could you test 
on Linux.3.17 or later version?

Below is my log:
[root@atest-guest linux_btrfs]# uname -a
Linux atest-guest 3.17.0+ #60 SMP Thu Nov 13 06:51:55 EST 2014 x86_64 
x86_64 x86_64 GNU/Linux
[root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
qgroupid rfer  excl  max_excl
-------- ----  ----  --------
0/5      16384 16384 0
[root@atest-guest linux_btrfs]# btrfs sub create /mnt/sub
Create subvolume '/mnt/sub'
[root@atest-guest linux_btrfs]# sync
[root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
qgroupid rfer  excl  max_excl
-------- ----  ----  --------
0/5      16384 16384 0
0/257    16384 16384 0
[root@atest-guest linux_btrfs]# btrfs qgroup limit -e 300M /mnt/sub
[root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
qgroupid rfer  excl  max_excl
-------- ----  ----  --------
0/5      16384 16384 0
0/257    16384 16384 314572800
[root@atest-guest linux_btrfs]# for((i=0;i<10;i++));do dd if=/dev/zero 
of=/mnt/sub/data$i bs=6M count=10; done
10+0 records in
10+0 records out
62914560 bytes (63 MB) copied, 0.0401096 s, 1.6 GB/s
10+0 records in
10+0 records out
62914560 bytes (63 MB) copied, 0.0431105 s, 1.5 GB/s
10+0 records in
10+0 records out
62914560 bytes (63 MB) copied, 0.0407304 s, 1.5 GB/s
10+0 records in
10+0 records out
62914560 bytes (63 MB) copied, 0.041441 s, 1.5 GB/s
dd: error writing \u2018/mnt/sub/data4\u2019: Disk quota exceeded
10+0 records in
9+0 records out
60817408 bytes (61 MB) copied, 0.0398122 s, 1.5 GB/s
dd: error writing \u2018/mnt/sub/data5\u2019: Disk quota exceeded
1+0 records in
0+0 records out
1703936 bytes (1.7 MB) copied, 0.00318793 s, 534 MB/s
dd: error writing \u2018/mnt/sub/data6\u2019: Disk quota exceeded
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00239674 s, 0.0 kB/s
dd: error writing \u2018/mnt/sub/data7\u2019: Disk quota exceeded
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.002271 s, 0.0 kB/s
dd: error writing \u2018/mnt/sub/data8\u2019: Disk quota exceeded
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00232063 s, 0.0 kB/s
dd: error writing \u2018/mnt/sub/data9\u2019: Disk quota exceeded
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00185336 s, 0.0 kB/s
[root@atest-guest linux_btrfs]# sync
[root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
qgroupid rfer      excl      max_excl
-------- ----      ----      --------
0/5      16384     16384     0
0/257    314195968 314195968 314572800

Re: Quota question

[Dongsheng Yang]

[-- Attachment #1: Type: text/plain, Size: 4327 bytes --]

On 11/13/2014 11:05 AM, Dongsheng Yang wrote:
> On 11/12/2014 10:04 PM, Cyril Scetbon wrote:
>> Anyone on this ? There is an issue with quotas depending on the write 
>> rate. The more we can write before a sync, the more we can exceed 
>> quotas limits
>
> Hi Cyril, I attempted to reproduce the problem you reported in 
> Linux.3.17, but failed.
> It seems that the issue in this thread was fixed already. Could you 
> test on Linux.3.17 or later version?

Hi Cyril, after some more investigation, there actually is a problem you 
described here.

*reason*
The logic in fallocate to update qgroup->excl here is:
1). btrfs_qgroup_reserve(). It reserves N_bytes.
2). btrfs_prealloc_file_range(). It record a qgroup ref into 
trans->qgroup_ref_list.
3). btrfs_qgroup_free(). It clear the N_bytes from reservation.
... ...
commit_transaction().

Similar logic in writing data.

Currently qgroup->excl is updated in qgroup_excl_accounting and it is 
called in commit_transation().
It means there probably is a *window* between btrfs_qgroup_free() and 
commit_transaction() where
the N_bytes is not accounted into qgroup. In this *window*, more data 
request will be accepted
even the total size of these requests exceeds the limit, and qgroup 
can't realize it.

There is a quick fix attached on this mail, it delays the reserve_free 
operation after updating qgroup->excl.

*NOTE*
It is just a quick fix for your problem here, could you help to test is 
on your box? Patch is based on
Linux 3.17 (bfe01a5ba2490f299e1d2d5508cbbbadd897bbe9).

More work about btrfs quota is on the TODO list.

Thanx
>
> Below is my log:
> [root@atest-guest linux_btrfs]# uname -a
> Linux atest-guest 3.17.0+ #60 SMP Thu Nov 13 06:51:55 EST 2014 x86_64 
> x86_64 x86_64 GNU/Linux
> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
> qgroupid rfer  excl  max_excl
> -------- ----  ----  --------
> 0/5      16384 16384 0
> [root@atest-guest linux_btrfs]# btrfs sub create /mnt/sub
> Create subvolume '/mnt/sub'
> [root@atest-guest linux_btrfs]# sync
> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
> qgroupid rfer  excl  max_excl
> -------- ----  ----  --------
> 0/5      16384 16384 0
> 0/257    16384 16384 0
> [root@atest-guest linux_btrfs]# btrfs qgroup limit -e 300M /mnt/sub
> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
> qgroupid rfer  excl  max_excl
> -------- ----  ----  --------
> 0/5      16384 16384 0
> 0/257    16384 16384 314572800
> [root@atest-guest linux_btrfs]# for((i=0;i<10;i++));do dd if=/dev/zero 
> of=/mnt/sub/data$i bs=6M count=10; done
> 10+0 records in
> 10+0 records out
> 62914560 bytes (63 MB) copied, 0.0401096 s, 1.6 GB/s
> 10+0 records in
> 10+0 records out
> 62914560 bytes (63 MB) copied, 0.0431105 s, 1.5 GB/s
> 10+0 records in
> 10+0 records out
> 62914560 bytes (63 MB) copied, 0.0407304 s, 1.5 GB/s
> 10+0 records in
> 10+0 records out
> 62914560 bytes (63 MB) copied, 0.041441 s, 1.5 GB/s
> dd: error writing \u2018/mnt/sub/data4\u2019: Disk quota exceeded
> 10+0 records in
> 9+0 records out
> 60817408 bytes (61 MB) copied, 0.0398122 s, 1.5 GB/s
> dd: error writing \u2018/mnt/sub/data5\u2019: Disk quota exceeded
> 1+0 records in
> 0+0 records out
> 1703936 bytes (1.7 MB) copied, 0.00318793 s, 534 MB/s
> dd: error writing \u2018/mnt/sub/data6\u2019: Disk quota exceeded
> 1+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.00239674 s, 0.0 kB/s
> dd: error writing \u2018/mnt/sub/data7\u2019: Disk quota exceeded
> 1+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.002271 s, 0.0 kB/s
> dd: error writing \u2018/mnt/sub/data8\u2019: Disk quota exceeded
> 1+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.00232063 s, 0.0 kB/s
> dd: error writing \u2018/mnt/sub/data9\u2019: Disk quota exceeded
> 1+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.00185336 s, 0.0 kB/s
> [root@atest-guest linux_btrfs]# sync
> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
> qgroupid rfer      excl      max_excl
> -------- ----      ----      --------
> 0/5      16384     16384     0
> 0/257    314195968 314195968 314572800
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> .
>


[-- Attachment #2: 0001-btrfs-free-quota-reserved-in-delay_ref-handler.patch --]
[-- Type: text/x-patch, Size: 3743 bytes --]

>From 8d203d87385fa56753832864ff462ad8b61b7c8f Mon Sep 17 00:00:00 2001
From: Dongsheng Yang <yangds.fnst@cn.fujitsu.com>
Date: Thu, 13 Nov 2014 12:56:06 -0500
Subject: [PATCH] btrfs: free quota reserved in delay_ref handler

Signed-off-by: Dongsheng Yang <yangds.fnst@cn.fujitsu.com>
---
 fs/btrfs/extent-tree.c |  2 +-
 fs/btrfs/file.c        |  2 +-
 fs/btrfs/qgroup.c      | 23 +++++++++++++++++++++--
 fs/btrfs/qgroup.h      |  1 +
 4 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 3efe1c3..7b27ff6 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -5300,7 +5300,7 @@ void btrfs_delalloc_release_metadata(struct inode *inode, u64 num_bytes)
 	trace_btrfs_space_reservation(root->fs_info, "delalloc",
 				      btrfs_ino(inode), to_free, 0);
 	if (root->fs_info->quota_enabled) {
-		btrfs_qgroup_free(root, num_bytes +
+		btrfs_qgroup_delayed_free(root, num_bytes +
 					dropped * root->leafsize);
 	}
 
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index ff1cc03..58b5237 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -2609,7 +2609,7 @@ static long btrfs_fallocate(struct file *file, int mode,
 out:
 	mutex_unlock(&inode->i_mutex);
 	if (root->fs_info->quota_enabled)
-		btrfs_qgroup_free(root, alloc_end - alloc_start);
+		btrfs_qgroup_delayed_free(root, alloc_end - alloc_start);
 out_reserve_fail:
 	/* Let go of our reservation. */
 	btrfs_free_reserved_data_space(inode, alloc_end - alloc_start);
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index ded5c60..1329500 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -73,6 +73,7 @@ struct btrfs_qgroup {
 	 * reservation tracking
 	 */
 	u64 reserved;
+	u64 delayed_release;
 
 	/*
 	 * lists
@@ -1409,6 +1410,11 @@ static int qgroup_excl_accounting(struct btrfs_fs_info *fs_info,
 	qgroup->excl += sign * oper->num_bytes;
 	qgroup->excl_cmpr += sign * oper->num_bytes;
 
+	if (qgroup->delayed_release) {
+		qgroup->reserved -= qgroup->delayed_release;
+		qgroup->delayed_release = 0;
+	}
+
 	qgroup_dirty(fs_info, qgroup);
 
 	/* Get all of the parent groups that contain this qgroup */
@@ -2440,7 +2446,7 @@ out:
 	return ret;
 }
 
-void btrfs_qgroup_free(struct btrfs_root *root, u64 num_bytes)
+static void qgroup_free(struct btrfs_root *root, u64 num_bytes, int delay)
 {
 	struct btrfs_root *quota_root;
 	struct btrfs_qgroup *qgroup;
@@ -2478,7 +2484,10 @@ void btrfs_qgroup_free(struct btrfs_root *root, u64 num_bytes)
 
 		qg = u64_to_ptr(unode->aux);
 
-		qg->reserved -= num_bytes;
+		if (delay)
+			qg->delayed_release += num_bytes;
+		else
+			qg->reserved -= num_bytes;
 
 		list_for_each_entry(glist, &qg->groups, next_group) {
 			ret = ulist_add(fs_info->qgroup_ulist,
@@ -2493,6 +2502,16 @@ out:
 	spin_unlock(&fs_info->qgroup_lock);
 }
 
+void btrfs_qgroup_free(struct btrfs_root *root, u64 num_bytes)
+{
+	return qgroup_free(root, num_bytes, 0);
+}
+
+void btrfs_qgroup_delayed_free(struct btrfs_root *root, u64 num_bytes)
+{
+	return qgroup_free(root, num_bytes, 1);
+}
+
 void assert_qgroups_uptodate(struct btrfs_trans_handle *trans)
 {
 	if (list_empty(&trans->qgroup_ref_list) && !trans->delayed_ref_elem.seq)
diff --git a/fs/btrfs/qgroup.h b/fs/btrfs/qgroup.h
index 18cc68c..1c50238 100644
--- a/fs/btrfs/qgroup.h
+++ b/fs/btrfs/qgroup.h
@@ -97,6 +97,7 @@ int btrfs_qgroup_inherit(struct btrfs_trans_handle *trans,
 			 struct btrfs_qgroup_inherit *inherit);
 int btrfs_qgroup_reserve(struct btrfs_root *root, u64 num_bytes);
 void btrfs_qgroup_free(struct btrfs_root *root, u64 num_bytes);
+void btrfs_qgroup_delayed_free(struct btrfs_root *root, u64 num_bytes);
 
 void assert_qgroups_uptodate(struct btrfs_trans_handle *trans);
 
-- 
1.8.3.1

Re: Quota question

[Cyril Scetbon]

Hey,

I'm using ubuntu kernels provided at http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17.1-utopic/. I asked them how to rebuild one with a patch and will try it when I got the answer.

Thanks

PS : good news and I hope it'll fix the issue
-- 
Cyril SCETBON

> On 13 Nov 2014, at 10:40, Dongsheng Yang <yangds.fnst@cn.fujitsu.com> wrote:
> 
> On 11/13/2014 11:05 AM, Dongsheng Yang wrote:
>> On 11/12/2014 10:04 PM, Cyril Scetbon wrote:
>>> Anyone on this ? There is an issue with quotas depending on the write rate. The more we can write before a sync, the more we can exceed quotas limits
>> 
>> Hi Cyril, I attempted to reproduce the problem you reported in Linux.3.17, but failed.
>> It seems that the issue in this thread was fixed already. Could you test on Linux.3.17 or later version?
> 
> Hi Cyril, after some more investigation, there actually is a problem you described here.
> 
> *reason*
> The logic in fallocate to update qgroup->excl here is:
> 1). btrfs_qgroup_reserve(). It reserves N_bytes.
> 2). btrfs_prealloc_file_range(). It record a qgroup ref into trans->qgroup_ref_list.
> 3). btrfs_qgroup_free(). It clear the N_bytes from reservation.
> ... ...
> commit_transaction().
> 
> Similar logic in writing data.
> 
> Currently qgroup->excl is updated in qgroup_excl_accounting and it is called in commit_transation().
> It means there probably is a *window* between btrfs_qgroup_free() and commit_transaction() where
> the N_bytes is not accounted into qgroup. In this *window*, more data request will be accepted
> even the total size of these requests exceeds the limit, and qgroup can't realize it.
> 
> There is a quick fix attached on this mail, it delays the reserve_free operation after updating qgroup->excl.
> 
> *NOTE*
> It is just a quick fix for your problem here, could you help to test is on your box? Patch is based on
> Linux 3.17 (bfe01a5ba2490f299e1d2d5508cbbbadd897bbe9).
> 
> More work about btrfs quota is on the TODO list.
> 
> Thanx
>> 
>> Below is my log:
>> [root@atest-guest linux_btrfs]# uname -a
>> Linux atest-guest 3.17.0+ #60 SMP Thu Nov 13 06:51:55 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
>> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
>> qgroupid rfer  excl  max_excl
>> -------- ----  ----  --------
>> 0/5      16384 16384 0
>> [root@atest-guest linux_btrfs]# btrfs sub create /mnt/sub
>> Create subvolume '/mnt/sub'
>> [root@atest-guest linux_btrfs]# sync
>> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
>> qgroupid rfer  excl  max_excl
>> -------- ----  ----  --------
>> 0/5      16384 16384 0
>> 0/257    16384 16384 0
>> [root@atest-guest linux_btrfs]# btrfs qgroup limit -e 300M /mnt/sub
>> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
>> qgroupid rfer  excl  max_excl
>> -------- ----  ----  --------
>> 0/5      16384 16384 0
>> 0/257    16384 16384 314572800
>> [root@atest-guest linux_btrfs]# for((i=0;i<10;i++));do dd if=/dev/zero of=/mnt/sub/data$i bs=6M count=10; done
>> 10+0 records in
>> 10+0 records out
>> 62914560 bytes (63 MB) copied, 0.0401096 s, 1.6 GB/s
>> 10+0 records in
>> 10+0 records out
>> 62914560 bytes (63 MB) copied, 0.0431105 s, 1.5 GB/s
>> 10+0 records in
>> 10+0 records out
>> 62914560 bytes (63 MB) copied, 0.0407304 s, 1.5 GB/s
>> 10+0 records in
>> 10+0 records out
>> 62914560 bytes (63 MB) copied, 0.041441 s, 1.5 GB/s
>> dd: error writing \u2018/mnt/sub/data4\u2019: Disk quota exceeded
>> 10+0 records in
>> 9+0 records out
>> 60817408 bytes (61 MB) copied, 0.0398122 s, 1.5 GB/s
>> dd: error writing \u2018/mnt/sub/data5\u2019: Disk quota exceeded
>> 1+0 records in
>> 0+0 records out
>> 1703936 bytes (1.7 MB) copied, 0.00318793 s, 534 MB/s
>> dd: error writing \u2018/mnt/sub/data6\u2019: Disk quota exceeded
>> 1+0 records in
>> 0+0 records out
>> 0 bytes (0 B) copied, 0.00239674 s, 0.0 kB/s
>> dd: error writing \u2018/mnt/sub/data7\u2019: Disk quota exceeded
>> 1+0 records in
>> 0+0 records out
>> 0 bytes (0 B) copied, 0.002271 s, 0.0 kB/s
>> dd: error writing \u2018/mnt/sub/data8\u2019: Disk quota exceeded
>> 1+0 records in
>> 0+0 records out
>> 0 bytes (0 B) copied, 0.00232063 s, 0.0 kB/s
>> dd: error writing \u2018/mnt/sub/data9\u2019: Disk quota exceeded
>> 1+0 records in
>> 0+0 records out
>> 0 bytes (0 B) copied, 0.00185336 s, 0.0 kB/s
>> [root@atest-guest linux_btrfs]# sync
>> [root@atest-guest linux_btrfs]# btrfs qgroup show -e /mnt
>> qgroupid rfer      excl      max_excl
>> -------- ----      ----      --------
>> 0/5      16384     16384     0
>> 0/257    314195968 314195968 314572800
>> 
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> .
>> 
> 
> <0001-btrfs-free-quota-reserved-in-delay_ref-handler.patch>