Disk quota exceeded

Disk quota exceeded

[Benoit GEORGELIN - Association Web4all]

Hi dear users of the list. 

I'm new to the BTRFS file-system and I am having some problems with quotas 
I would like to share with you what I'm facing about "Disk quota exceeded" it's quite strange to me. 

- Description of the environment 
I have a BTRFS volume "/var/lib/lxd" 
I have some subvolume into "/var/lib/lxd/containers" 

The volume have quota enable 
Every subvolume have their quota groupe created 
Limit of 10Go has been set to every single subvolumes 

-The problem 
Randomly I can't write to some of the subvolumes anymore without having datas added into the subvolume. 
I can't figure why and the subvolumes are randomly not usable with the error : Disk quota exceeded 

I did delete all the quota groups, disable quota on the volume, enable the quota group again on the volume, check if quota groups were back automatically as btrfs version 4 is supposed to do 
I added back the limit 10Go 
Rescan 
Everything work just fine but some time after, I'm back to over-quota on some subvolumes. 

Thanks for your help on this. 

Below, more details about the config and system point of view : 


uname -a 
Linux lxd-virt-01b 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26 00:58:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux 


btrfs --version 
btrfs-progs v4.4 

btrfs fi show 
Label: none uuid: b0b183ed-9970-4e54-b7d5-7b91f3bdea10 
Total devices 2 FS bytes used 21.01GiB 
devid 1 size 50.00GiB used 13.02GiB path /dev/sdb 
devid 2 size 50.00GiB used 13.01GiB path /dev/sdc 

btrfs fi df /var/lib/lxd 
Data, RAID0: total=22.00GiB, used=20.36GiB 
System, RAID1: total=8.00MiB, used=16.00KiB 
System, single: total=4.00MiB, used=0.00B 
Metadata, RAID1: total=2.00GiB, used=660.97MiB 
Metadata, single: total=8.00MiB, used=0.00B 
GlobalReserve, single: total=224.00MiB, used=0.00B 

# Simple script that check the content of /var/lib/lxd/containers were subvolumes stands 
# Try to touch a file into subvolume_name/rootfs/ 
./check_quotas_rw.sh 
Fichier écrit sur /var/lib/lxd/containers/angristan/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/ben/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
Fichier écrit sur /var/lib/lxd/containers/benltsp/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/debsid/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
Fichier écrit sur /var/lib/lxd/containers/dworak-vps/rootfs/tmp/test 
Fichier écrit sur /var/lib/lxd/containers/heimdaall/rootfs/tmp/test 
Fichier écrit sur /var/lib/lxd/containers/louisjo/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/onclben/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
touch: cannot touch ‘/var/lib/lxd/containers/testdisk/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
Fichier écrit sur /var/lib/lxd/containers/tyrower/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/vega/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 

# Script to check group id so you can correlate 
/btrfsQuota.py --unit M /var/lib/lxd 
subvol group total unshared 
------------------------------------------------------------------------------- 
(unknown) 0/5 649.89M 649.89M 
containers/ben 0/261 1297.45M 1297.45M 
images/0d07f11f3f2a0804f501967d278d72a8122b1ec49f01aae4483a41fd9fb546f3.btrfs 0/264 315.33M 315.33M 
containers/vega 0/336 3211.92M 3211.92M 
containers/onclben 0/391 384.45M 384.45M 
images/61ba8f528725bda896eb6ca007f29f2d90a3f375379292e6a31eb851eb33e5ab.btrfs 0/423 563.02M 563.02M 
containers/heimdaall 0/428 1936.61M 1653.59M 
containers/tyrower 0/429 384.33M 101.31M 
containers/angristan 0/435 1132.71M 846.59M 
containers/dworak-vps 0/437 384.74M 98.62M 
containers/louisjo 0/442 663.09M 377.49M 
containers/benltsp 0/443 9209.45M 8933.05M 
images/f452cda3bccb2903e56d53e402b9d35334b4276783d098a879be5d74b04e62e2.btrfs 0/448 631.32M 631.32M 
containers/testdisk 0/449 383.99M 383.99M 
containers/debsid 0/453 428.52M 428.52M 
images/96a08bb9c674e092fe99dd2f96d61f8f0a2a1ea80eb01d384274c1ff8efefce1.btrfs 0/457 328.57M 328.57M 
images/fca47fbd0bb34bd38fdbd67bcc90a00eb8f0420b29a048ff59125d5fc983016c.btrfs 0/458 315.15M 315.15M 
images/c61ac213e75fb99e40d5ac49157314f002db257f38a692fc98c8d4be867f18a6.btrfs 0/463 367.45M 367.45M 


btrfs qg show -r -e /var/lib/lxd -p -c 
qgroupid rfer excl max_rfer max_excl parent child 
-------- ---- ---- -------- -------- ------ ----- 
0/5 649.89MiB 649.89MiB none none --- --- 
0/261 1.27GiB 1.27GiB 10.00GiB none --- --- 
0/264 315.33MiB 315.33MiB none none --- --- 
0/336 3.14GiB 3.14GiB 10.00GiB none --- --- 
0/391 384.45MiB 384.45MiB 10.00GiB none --- --- 
0/423 563.02MiB 563.02MiB none none --- --- 
0/428 1.89GiB 1.61GiB 10.00GiB none --- --- 
0/429 384.33MiB 101.31MiB 10.00GiB none --- --- 
0/435 1.11GiB 846.59MiB 10.00GiB none --- --- 
0/437 384.74MiB 98.62MiB 10.00GiB none --- --- 
0/442 663.09MiB 377.49MiB 10.00GiB none --- --- 
0/443 8.99GiB 8.72GiB 10.00GiB none --- --- 
0/448 631.32MiB 631.32MiB none none --- --- 
0/449 383.99MiB 383.99MiB 10.00GiB none --- --- 
0/453 428.52MiB 428.52MiB 10.00GiB none --- --- 
0/457 328.57MiB 328.57MiB none none --- --- 
0/458 315.15MiB 315.15MiB none none --- --- 
0/463 367.45MiB 367.45MiB none none --- --- 


dmesg |grep -i btrfs 
[ 2.712185] Btrfs loaded 
[ 2.902305] BTRFS: device fsid b0b183ed-9970-4e54-b7d5-7b91f3bdea10 devid 2 transid 113894 /dev/sdc 
[ 2.903883] BTRFS: device fsid b0b183ed-9970-4e54-b7d5-7b91f3bdea10 devid 1 transid 113894 /dev/sdb 
[ 6.201318] BTRFS info (device sdb): disk space caching is enabled 
[ 6.201321] BTRFS: has skinny extents 
[ 39.732758] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 540.722625] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1247.758789] BTRFS info (device sdb): qgroup_rescan_init failed with -115 
[ 1265.879801] BTRFS info (device sdb): qgroup_rescan_init failed with -115 
[ 1289.854705] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1403.138613] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1564.685350] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1616.389847] BTRFS info (device sdb): The free space cache file (23651680256) is invalid. skip it 
[ 1765.145922] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[36615.633743] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared)

Cordialement, 

Benoît Georgelin - Expert Infrastructure 
Web 4 all Hébergeur associatif 
benoit.georgelin@web 4 all.fr 

Afin de contribuer au respect de l'environnement, merci de n'imprimer ce mail qu'en cas de nécessité

----- Mail original -----
De: "Benoit GEORGELIN - Association Web4all" <benoit.georgelin@web4all.fr>
À: "linux-btrfs" <linux-btrfs@vger.kernel.org>
Envoyé: Mercredi 29 Juin 2016 12:08:37
Objet: Disk quota exceeded


I'm new to the BTRFS file-system and I am having some problems with quotas 
I would like to share with you what I'm facing about "Disk quota exceeded" it's quite strange to me. 

- Description of the environment 
I have a BTRFS volume "/var/lib/lxd" 
I have some subvolume into "/var/lib/lxd/containers" 

The volume have quota enable 
Every subvolume have their quota groupe created 
Limit of 10Go has been set to every single subvolumes 

-The problem 
Randomly I can't write to some of the subvolumes anymore without having datas added into the subvolume. 
I can't figure why and the subvolumes are randomly not usable with the error : Disk quota exceeded 

I did delete all the quota groups, disable quota on the volume, enable the quota group again on the volume, check if quota groups were back automatically as btrfs version 4 is supposed to do 
I added back the limit 10Go 
Rescan 
Everything work just fine but some time after, I'm back to over-quota on some subvolumes. 

Thanks for your help on this. 

Below, more details about the config and system point of view : 


uname -a 
Linux lxd-virt-01b 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26 00:58:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux 


btrfs --version 
btrfs-progs v4.4 

btrfs fi show 
Label: none uuid: b0b183ed-9970-4e54-b7d5-7b91f3bdea10 
Total devices 2 FS bytes used 21.01GiB 
devid 1 size 50.00GiB used 13.02GiB path /dev/sdb 
devid 2 size 50.00GiB used 13.01GiB path /dev/sdc 

btrfs fi df /var/lib/lxd 
Data, RAID0: total=22.00GiB, used=20.36GiB 
System, RAID1: total=8.00MiB, used=16.00KiB 
System, single: total=4.00MiB, used=0.00B 
Metadata, RAID1: total=2.00GiB, used=660.97MiB 
Metadata, single: total=8.00MiB, used=0.00B 
GlobalReserve, single: total=224.00MiB, used=0.00B 

# Simple script that check the content of /var/lib/lxd/containers were subvolumes stands 
# Try to touch a file into subvolume_name/rootfs/ 
./check_quotas_rw.sh 
Fichier écrit sur /var/lib/lxd/containers/angristan/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/ben/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
Fichier écrit sur /var/lib/lxd/containers/benltsp/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/debsid/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
Fichier écrit sur /var/lib/lxd/containers/dworak-vps/rootfs/tmp/test 
Fichier écrit sur /var/lib/lxd/containers/heimdaall/rootfs/tmp/test 
Fichier écrit sur /var/lib/lxd/containers/louisjo/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/onclben/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
touch: cannot touch ‘/var/lib/lxd/containers/testdisk/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 
Fichier écrit sur /var/lib/lxd/containers/tyrower/rootfs/tmp/test 
touch: cannot touch ‘/var/lib/lxd/containers/vega/rootfs/tmp/test’: Disk quota exceeded 
Erreur de quota 

# Script to check group id so you can correlate 
/btrfsQuota.py --unit M /var/lib/lxd 
subvol group total unshared 
------------------------------------------------------------------------------- 
(unknown) 0/5 649.89M 649.89M 
containers/ben 0/261 1297.45M 1297.45M 
images/0d07f11f3f2a0804f501967d278d72a8122b1ec49f01aae4483a41fd9fb546f3.btrfs 0/264 315.33M 315.33M 
containers/vega 0/336 3211.92M 3211.92M 
containers/onclben 0/391 384.45M 384.45M 
images/61ba8f528725bda896eb6ca007f29f2d90a3f375379292e6a31eb851eb33e5ab.btrfs 0/423 563.02M 563.02M 
containers/heimdaall 0/428 1936.61M 1653.59M 
containers/tyrower 0/429 384.33M 101.31M 
containers/angristan 0/435 1132.71M 846.59M 
containers/dworak-vps 0/437 384.74M 98.62M 
containers/louisjo 0/442 663.09M 377.49M 
containers/benltsp 0/443 9209.45M 8933.05M 
images/f452cda3bccb2903e56d53e402b9d35334b4276783d098a879be5d74b04e62e2.btrfs 0/448 631.32M 631.32M 
containers/testdisk 0/449 383.99M 383.99M 
containers/debsid 0/453 428.52M 428.52M 
images/96a08bb9c674e092fe99dd2f96d61f8f0a2a1ea80eb01d384274c1ff8efefce1.btrfs 0/457 328.57M 328.57M 
images/fca47fbd0bb34bd38fdbd67bcc90a00eb8f0420b29a048ff59125d5fc983016c.btrfs 0/458 315.15M 315.15M 
images/c61ac213e75fb99e40d5ac49157314f002db257f38a692fc98c8d4be867f18a6.btrfs 0/463 367.45M 367.45M 


btrfs qg show -r -e /var/lib/lxd -p -c 
qgroupid rfer excl max_rfer max_excl parent child 
-------- ---- ---- -------- -------- ------ ----- 
0/5 649.89MiB 649.89MiB none none --- --- 
0/261 1.27GiB 1.27GiB 10.00GiB none --- --- 
0/264 315.33MiB 315.33MiB none none --- --- 
0/336 3.14GiB 3.14GiB 10.00GiB none --- --- 
0/391 384.45MiB 384.45MiB 10.00GiB none --- --- 
0/423 563.02MiB 563.02MiB none none --- --- 
0/428 1.89GiB 1.61GiB 10.00GiB none --- --- 
0/429 384.33MiB 101.31MiB 10.00GiB none --- --- 
0/435 1.11GiB 846.59MiB 10.00GiB none --- --- 
0/437 384.74MiB 98.62MiB 10.00GiB none --- --- 
0/442 663.09MiB 377.49MiB 10.00GiB none --- --- 
0/443 8.99GiB 8.72GiB 10.00GiB none --- --- 
0/448 631.32MiB 631.32MiB none none --- --- 
0/449 383.99MiB 383.99MiB 10.00GiB none --- --- 
0/453 428.52MiB 428.52MiB 10.00GiB none --- --- 
0/457 328.57MiB 328.57MiB none none --- --- 
0/458 315.15MiB 315.15MiB none none --- --- 
0/463 367.45MiB 367.45MiB none none --- --- 


dmesg |grep -i btrfs 
[ 2.712185] Btrfs loaded 
[ 2.902305] BTRFS: device fsid b0b183ed-9970-4e54-b7d5-7b91f3bdea10 devid 2 transid 113894 /dev/sdc 
[ 2.903883] BTRFS: device fsid b0b183ed-9970-4e54-b7d5-7b91f3bdea10 devid 1 transid 113894 /dev/sdb 
[ 6.201318] BTRFS info (device sdb): disk space caching is enabled 
[ 6.201321] BTRFS: has skinny extents 
[ 39.732758] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 540.722625] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1247.758789] BTRFS info (device sdb): qgroup_rescan_init failed with -115 
[ 1265.879801] BTRFS info (device sdb): qgroup_rescan_init failed with -115 
[ 1289.854705] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1403.138613] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1564.685350] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[ 1616.389847] BTRFS info (device sdb): The free space cache file (23651680256) is invalid. skip it 
[ 1765.145922] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 
[36615.633743] BTRFS info (device sdb): qgroup scan completed (inconsistency flag cleared) 

Re: Disk quota exceeded

[Duncan]

Benoit GEORGELIN - Association Web4all posted on Wed, 29 Jun 2016 18:15:53
+0200 as excerpted:

> Hi dear users of the list.
> 
> I'm new to the BTRFS file-system and I am having some problems with
> quotas I would like to share with you what I'm facing about "Disk quota
> exceeded" it's quite strange to me.

Three points.

1) Btrfs quotas have a long history of trouble, both in correctness and 
in scaling ability (btrfs balances and checks can take far longer with 
quotas enabled, often too long to be practical), and if they are now 
stable, it's only a very recent development.  As a result, my 
recommendation has been and remains not to use btrfs quotas unless you 
are specifically doing so to work with the devs to test and bugfix.  If 
you don't need them, simply turn them off and leave them off until they 
are known to work.  If you do actually need quota functionality, you're 
far better off using a filesystem where quotas are actually stable and 
work as intended.

> - Description of the environment I have a BTRFS volume "/var/lib/lxd"
> I have some subvolume into "/var/lib/lxd/containers"
> 
> The volume have quota enable Every subvolume have their quota groupe
> created Limit of 10Go has been set to every single subvolumes
> 
> -The problem Randomly I can't write to some of the subvolumes anymore
> without having datas added into the subvolume.
> I can't figure why and the subvolumes are randomly not usable with the
> error : Disk quota exceeded

2) I'm not sure of the current status here, but it's worth keeping in 
mind that unlike other filesystems, btrfs separates data and metadata, 
and it's possible for data to be under quota and still run out, if 
metadata takes you over quota.

There was some talk a kernel cycle or two ago suggesting separating data 
and metadata quotas and allowing setting them either separately or 
overall capping them, but as I said, I'm not sure what the status is on 
that.

> I did delete all the quota groups, disable quota on the volume, enable
> the quota group again on the volume, check if quota groups were back
> automatically as btrfs version 4 is supposed to do I added back the
> limit 10Go Rescan Everything work just fine but some time after, I'm
> back to over-quota on some subvolumes.
> 
> Thanks for your help on this.
> 
> Below, more details about the config and system point of view :
> 
> 
> uname -a Linux lxd-virt-01b 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26
> 00:58:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

3) Unfortunately, ubuntu chose what from upstream's viewpoint was a short-
term-stable kernel.  4.2 is no longer supported upstream, and while it's 
possible that ubuntu is backporting as appropriate, on this list we track 
upstream and don't know what individual distros have backported and what 
they haven't.

And I know for sure that 4.2 still had serious quota issues that are 
unlikely to have been fixed with backports because they never /did/ work 
right by 4.2, so the quota issues weren't regressions and thus patches to 
fix them would have been unlikely stable candidates.

You thus have a few choices:

4.1 and 4.4 are upstream LTS series, and continue to be supported (tho 
I'd suggest 4.4 if you'll be adopting one, as btrfs is still developing 
fast enough that we only tend to well support the last couple LTS, and 
beyond that, while critical patches will be backported to the extent 
possible, if you have a problem you're still likely to be asked to 
upgrade and see if the problem is still there if you're out of that 2 LTS 
series window, and 4.1 is getting a bit long in the tooth to be just 
upgrading to now).  So one choice is to move to the latest version of one 
of them.

Another choice would be to continue with 4.2, but get your support from 
your distro, who is better qualified to support it as they know what 
they've backported and what they haven't.

And another choice is upgrade to current kernels, where again we 
generally support the last two series, 4.6 and 4.5 ATM.


(FWIW, userspace version numbers are synced to kernelspace numbers.  
Backward compatibility is supported, so a good rule of thumb there is to 
run btrfs userspace at least as new as your kernel, which assuming you're 
staying in line with the latest two kernels of either current or LTS 
series, will mean your userspace doesn't get too far behind either.  But 
you're on 4.4 userspace already, so are fine there.)


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Disk quota exceeded

[Benoit GEORGELIN - Association Web4all]

Hi Duncan, 

Thanks a lot for your answer. This give me a lot of information and a better idea of what I should do. 
I just upgraded the kernel to 4.6.3-040603-generic and i will see if it's working better with this kernel. 

If not, I will have to look to another FS with a more stable quotas :) 

Cordialement, 

Benoît 

Afin de contribuer au respect de l'environnement, merci de n'imprimer ce mail qu'en cas de nécessité

----- Mail original -----
De: "Duncan" <1i5t5.duncan@cox.net>
À: "linux-btrfs" <linux-btrfs@vger.kernel.org>
Envoyé: Mercredi 29 Juin 2016 13:19:35
Objet: Re: Disk quota exceeded

Benoit GEORGELIN - Association Web4all posted on Wed, 29 Jun 2016 18:15:53 
+0200 as excerpted: 

> Hi dear users of the list. 
> 
> I'm new to the BTRFS file-system and I am having some problems with 
> quotas I would like to share with you what I'm facing about "Disk quota 
> exceeded" it's quite strange to me. 

Three points. 

1) Btrfs quotas have a long history of trouble, both in correctness and 
in scaling ability (btrfs balances and checks can take far longer with 
quotas enabled, often too long to be practical), and if they are now 
stable, it's only a very recent development. As a result, my 
recommendation has been and remains not to use btrfs quotas unless you 
are specifically doing so to work with the devs to test and bugfix. If 
you don't need them, simply turn them off and leave them off until they 
are known to work. If you do actually need quota functionality, you're 
far better off using a filesystem where quotas are actually stable and 
work as intended. 

> - Description of the environment I have a BTRFS volume "/var/lib/lxd" 
> I have some subvolume into "/var/lib/lxd/containers" 
> 
> The volume have quota enable Every subvolume have their quota groupe 
> created Limit of 10Go has been set to every single subvolumes 
> 
> -The problem Randomly I can't write to some of the subvolumes anymore 
> without having datas added into the subvolume. 
> I can't figure why and the subvolumes are randomly not usable with the 
> error : Disk quota exceeded 

2) I'm not sure of the current status here, but it's worth keeping in 
mind that unlike other filesystems, btrfs separates data and metadata, 
and it's possible for data to be under quota and still run out, if 
metadata takes you over quota. 

There was some talk a kernel cycle or two ago suggesting separating data 
and metadata quotas and allowing setting them either separately or 
overall capping them, but as I said, I'm not sure what the status is on 
that. 

> I did delete all the quota groups, disable quota on the volume, enable 
> the quota group again on the volume, check if quota groups were back 
> automatically as btrfs version 4 is supposed to do I added back the 
> limit 10Go Rescan Everything work just fine but some time after, I'm 
> back to over-quota on some subvolumes. 
> 
> Thanks for your help on this. 
> 
> Below, more details about the config and system point of view : 
> 
> 
> uname -a Linux lxd-virt-01b 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26 
> 00:58:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux 

3) Unfortunately, ubuntu chose what from upstream's viewpoint was a short- 
term-stable kernel. 4.2 is no longer supported upstream, and while it's 
possible that ubuntu is backporting as appropriate, on this list we track 
upstream and don't know what individual distros have backported and what 
they haven't. 

And I know for sure that 4.2 still had serious quota issues that are 
unlikely to have been fixed with backports because they never /did/ work 
right by 4.2, so the quota issues weren't regressions and thus patches to 
fix them would have been unlikely stable candidates. 

You thus have a few choices: 

4.1 and 4.4 are upstream LTS series, and continue to be supported (tho 
I'd suggest 4.4 if you'll be adopting one, as btrfs is still developing 
fast enough that we only tend to well support the last couple LTS, and 
beyond that, while critical patches will be backported to the extent 
possible, if you have a problem you're still likely to be asked to 
upgrade and see if the problem is still there if you're out of that 2 LTS 
series window, and 4.1 is getting a bit long in the tooth to be just 
upgrading to now). So one choice is to move to the latest version of one 
of them. 

Another choice would be to continue with 4.2, but get your support from 
your distro, who is better qualified to support it as they know what 
they've backported and what they haven't. 

And another choice is upgrade to current kernels, where again we 
generally support the last two series, 4.6 and 4.5 ATM. 


(FWIW, userspace version numbers are synced to kernelspace numbers. 
Backward compatibility is supported, so a good rule of thumb there is to 
run btrfs userspace at least as new as your kernel, which assuming you're 
staying in line with the latest two kernels of either current or LTS 
series, will mean your userspace doesn't get too far behind either. But 
you're on 4.4 userspace already, so are fine there.) 


-- 
Duncan - List replies preferred. No HTML msgs. 
"Every nonfree program has a lord, a master -- 
and if you use the program, he is your master." Richard Stallman 

-- 
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

Disk quota exceeded

[Matthew Jurgens]

I've create a subvolume
CMD: btrfs quota enable /export/shared
CMD: btrfs subvol create /export/shared/TV/files
Create subvolume '/export/shared/TV/files'
CMD: btrfs quota rescan /export/shared/TV/files

I've assigned a small quota to it:
CMD: btrfs qgroup limit 100M /export/shared/TV/files

I've also disabled copy on write:
CMD: chattr -R -V +C /export/shared/TV/files
chattr 1.42.12 (29-Aug-2014)
Flags of /export/shared/TV/files set as ---------------C

CMD: sync; btrfs qgroup show -r /export/shared/TV/files
qgroupid         rfer         excl     max_rfer
--------         ----         ----     --------
0/5           2.11TiB      2.11TiB         none
0/1894       16.00KiB     16.00KiB    100.00MiB

I created one file until I exceeded the quota using:
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded
50+0 records in
49+0 records out
51380224 bytes (51 MB) copied, 0.0238218 s, 2.2 GB/s

then I removed the file
CMD: rm /export/shared/TV/files/junk1

I tried to create it again (and only got 900k written before the quota 
was exceeded):
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded
1+0 records in
0+0 records out
917504 bytes (918 kB) copied, 0.000568984 s, 1.6 GB/s

then I removed the file
CMD: rm /export/shared/TV/files/junk1

I tried to create it again (and this time got 900k again written before 
the quota was exceeded):
dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded
1+0 records in
0+0 records out
917504 bytes (918 kB) copied, 0.000943652 s, 972 MB/s

The usage looks ok:
CMD: sync; btrfs qgroup show -r /export/shared/TV/files
qgroupid         rfer         excl     max_rfer
--------         ----         ----     --------
0/5           3.10TiB      3.10TiB         none
0/1894      912.00KiB    912.00KiB    100.00MiB

But I can not create any more files:
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: failed to open ‘/export/shared/TV/files/junk1’: Disk quota exceeded

CMD: lsattr /export/shared/TV/files
---------------C /export/shared/TV/files/junk1

During other tests I have managed to get it to the point where I can't 
delete the file (since disk quota exceeded).
I've waited many minutes to ensure commits etc. The only way I can 
seemingly consistently get the ability to write to the directory again 
is to reboot. Immediately after a reboot I can create files as expected.

It seems like copy on write is still functioning for this directory? or 
quota results have delays?
I've tried mounting the subvolume to another location, but same results.

Is this happening because quotas are not yet considered stable?
Is there something I am doing wrong?




Other Info:
The fstab entry:
LABEL=SHARED            /export/shared          btrfs 
defaults,compress=lzo,relatime,nofail 0 1

uname -a
Linux gw.lambert.rd.to 4.2.7-200.fc22.x86_64 #1 SMP Thu Dec 10 03:28:47 
UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

btrfs --version
btrfs-progs v4.2.2

btrfs fi show
Label: 'SHARED'  uuid: 199672d4-69ef-4ecf-865a-851bed0167d5
         Total devices 5 FS bytes used 3.10TiB
         devid    1 size 2.73TiB used 1.27TiB path /dev/sdc
         devid    2 size 2.73TiB used 1.27TiB path /dev/sdf
         devid    3 size 2.73TiB used 1.27TiB path /dev/sdh
         devid    4 size 2.73TiB used 1.27TiB path /dev/sde
         devid    5 size 2.73TiB used 1.27TiB path /dev/sdd

btrfs fi df /export/shared
Data, RAID10: total=3.16TiB, used=3.09TiB
System, RAID10: total=64.00MiB, used=352.00KiB
Metadata, RAID10: total=9.00GiB, used=7.61GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

dmesg:
http://edcint.co.nz/tmp/dmesg_20151219.log

Re: Disk quota exceeded

[Duncan]

Matthew Jurgens posted on Sat, 19 Dec 2015 21:38:50 +1100 as excerpted:

> I've create a subvolume [and] assigned a small quota to it:
> CMD: btrfs qgroup limit 100M /export/shared/TV/files

> I've also disabled copy on write:
> CMD: chattr -R -V +C /export/shared/TV/files

[wrote only ~50 M of files, can't write more]

> It seems like copy on write is still functioning for this directory? or
> quota results have delays?

> Is this happening because quotas are not yet considered stable?
> Is there something I am doing wrong?

> uname [-r] 4.2.7-200.fc22.x86_64

> btrfs --version btrfs-progs v4.2.2

[Disclaimer: List regular and btrfs user.  Not a dev.  My own use-case 
doesn't involve quotas or even subvolumes, so the following is based 
purely on what I've seen on-list.]

Quotas are definitely not stable, thru 4.3 at least, and you're running 
4.2.

IIRC They've on the product of the second major rewrite (so third try) 
now and there have always been issues.  While there's hard work going 
into fixing the issues, full fixes have remained at least a kernel cycle 
or two out for quite some time now.  AFAIK, 4.4 is supposed to fix some 
more issues, but I'm not sure if it fixes all known issues yet or if 
there's more remaining, and even if it fixes all known issues, I'd 
suggest waiting at least a complete kernel cycle and preferably two with 
no known issues, before considering quotas actually reasonably 
dependable, as given the history, there's a fair chance more issues will 
popup.

Additionally, quotas thru at least 4.3 still have bad scaling issues that 
slow down maintenance such as balance.  (I'm not sure if check is 
affected or not, but scrub shouldn't be.)

So I'd strongly suggest leaving quotas off thru the 4.4 cycle at least.  
If 4.4 and the 4.5 development cycle ends up with no further issues, you 
may wish to consider enabling them for 4.5, tho more conservative users 
will wait another cycle and see if there's no further issues reported 
thru the 4.6 development cycle as well, enabling them once they're on it.

(Since 4.4 will be an LTS kernel, if you're sticking to it, you could 
consider enabling quotas on updates after 4.5 or 4.6 release, if there 
have been no reports of quota issues on 4.4 by then.)


Meanwhile, a few more quota related factors to consider:

1) As I'm not a quota user I can't say for sure on this, but it seems 
mighty coincidental that it failed at just over half the set quota... and 
you're running btrfs raid10 data, so btrfs is keeping two copies and 
doubling the usage.  Maybe it's tracking the actual doubled due to the 
raid10 usage instead of the single-copy value?  (This actually sounds 
like some of the sort of internal logic bugs they've been fixing tho I'm 
unaware of one exactly like this, so you might try a current kernel 4.4-rc 
and see if the results are different, and/or try it on a different btrfs 
with replication set to single mode.)

2) Btrfs metadata is always copy-on-write.  That can't be disabled.  
Disabling COW only disables it for data.

I remember some discussion of whether quotas should track data only or 
include metadata as well, or possibly make it optional to track both or 
set quotas separately for each, but IDR the results of those discussions 
and as I don't use them myself...

That would explain not being able to delete the file at times, because 
doing so would change the metadata and thus require COWing it, and if 
it's quota tracked and reporting over quota errors...

3) While this shouldn't apply to the scenario above because you followed 
the recommendation and set nocow on the dir and /then/ created the file, 
so the file should have inherited the nocow, and I suppose you know this 
already, but for the benefit of others following alone, it should be 
noted that nocow doesn't necessarily take effect on files that already 
have content -- they need to be zero-sized for it to take effect.

The recommended procedure to avoid the problem is exactly what you did, 
set nocow on the dir and take care to either create new files (as you 
did) or copy/move them into place without reflinking, so they're actually 
created before they have any content, so the nocow inherited from the dir 
will actually take effect.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Disk quota exceeded

[Qu Wenruo]

Matthew Jurgens wrote on 2015/12/19 21:38 +1100:
> I've create a subvolume
> CMD: btrfs quota enable /export/shared
> CMD: btrfs subvol create /export/shared/TV/files
> Create subvolume '/export/shared/TV/files'
> CMD: btrfs quota rescan /export/shared/TV/files
>
> I've assigned a small quota to it:
> CMD: btrfs qgroup limit 100M /export/shared/TV/files
>
> I've also disabled copy on write:
> CMD: chattr -R -V +C /export/shared/TV/files
> chattr 1.42.12 (29-Aug-2014)
> Flags of /export/shared/TV/files set as ---------------C
>
> CMD: sync; btrfs qgroup show -r /export/shared/TV/files
> qgroupid         rfer         excl     max_rfer
> --------         ----         ----     --------
> 0/5           2.11TiB      2.11TiB         none
> 0/1894       16.00KiB     16.00KiB    100.00MiB
>
> I created one file until I exceeded the quota using:
> CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
> CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
> dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded
> 50+0 records in
> 49+0 records out
> 51380224 bytes (51 MB) copied, 0.0238218 s, 2.2 GB/s
>
> then I removed the file
> CMD: rm /export/shared/TV/files/junk1
>
> I tried to create it again (and only got 900k written before the quota
> was exceeded):
> CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
> dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded
> 1+0 records in
> 0+0 records out
> 917504 bytes (918 kB) copied, 0.000568984 s, 1.6 GB/s
>
> then I removed the file
> CMD: rm /export/shared/TV/files/junk1
>
> I tried to create it again (and this time got 900k again written before
> the quota was exceeded):
> dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
> dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded
> 1+0 records in
> 0+0 records out
> 917504 bytes (918 kB) copied, 0.000943652 s, 972 MB/s
>
> The usage looks ok:
> CMD: sync; btrfs qgroup show -r /export/shared/TV/files
> qgroupid         rfer         excl     max_rfer
> --------         ----         ----     --------
> 0/5           3.10TiB      3.10TiB         none
> 0/1894      912.00KiB    912.00KiB    100.00MiB

The excl/rfer numbers will always be accurate, as that what 4.2 quota 
rework does.

>
> But I can not create any more files:
> CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
> dd: failed to open ‘/export/shared/TV/files/junk1’: Disk quota exceeded

That's what exactly I fixed in 4.3-rc1.
And the operation to trigger the bug is almost the same with xfstest 
test case: btrfs/099.

Btrfs doesn't handle quota reserved space well, and caused reserved 
space leak. (fixed in 4.3-rc1, but more test would be better)

As long as you did overwrite, no matter whether you enabled or disabled 
CoW, it will leak reserved space, and the reserved space won't be freed.

Finally eating up your quota space.

You can either try latest 4.3-rc, which does not only include the quota 
fix, but also a lot of delayed ref fixes.

Or, there is a workaround:
Umount the fs and mount it again.
This will free the leaked space and make it work again for a short time,
until your next overwrite.

Thanks,
Qu
>
> CMD: lsattr /export/shared/TV/files
> ---------------C /export/shared/TV/files/junk1
>
> During other tests I have managed to get it to the point where I can't
> delete the file (since disk quota exceeded).
> I've waited many minutes to ensure commits etc. The only way I can
> seemingly consistently get the ability to write to the directory again
> is to reboot. Immediately after a reboot I can create files as expected.
>
> It seems like copy on write is still functioning for this directory? or
> quota results have delays?
> I've tried mounting the subvolume to another location, but same results.
>
> Is this happening because quotas are not yet considered stable?
> Is there something I am doing wrong?
>
>
>
>
> Other Info:
> The fstab entry:
> LABEL=SHARED            /export/shared          btrfs
> defaults,compress=lzo,relatime,nofail 0 1
>
> uname -a
> Linux gw.lambert.rd.to 4.2.7-200.fc22.x86_64 #1 SMP Thu Dec 10 03:28:47
> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> btrfs --version
> btrfs-progs v4.2.2
>
> btrfs fi show
> Label: 'SHARED'  uuid: 199672d4-69ef-4ecf-865a-851bed0167d5
>          Total devices 5 FS bytes used 3.10TiB
>          devid    1 size 2.73TiB used 1.27TiB path /dev/sdc
>          devid    2 size 2.73TiB used 1.27TiB path /dev/sdf
>          devid    3 size 2.73TiB used 1.27TiB path /dev/sdh
>          devid    4 size 2.73TiB used 1.27TiB path /dev/sde
>          devid    5 size 2.73TiB used 1.27TiB path /dev/sdd
>
> btrfs fi df /export/shared
> Data, RAID10: total=3.16TiB, used=3.09TiB
> System, RAID10: total=64.00MiB, used=352.00KiB
> Metadata, RAID10: total=9.00GiB, used=7.61GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> dmesg:
> http://edcint.co.nz/tmp/dmesg_20151219.log
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>