[BUG] "Quota Ignored On write" problem still exist with 3.16-rc5

[BUG] "Quota Ignored On write" problem still exist with 3.16-rc5

[Satoru Takeuchi]

Hi Josef, Chris,

I found "Quota Ignored On write" problem still exist with 3.16-rc5,
which Kevin reported before.

Kevin's report:
https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35292.html

The result of bisect:
https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35304.html

> I bisected and found the bad commit is the following patch.
> 
> ===============================================================================
> commit fcebe4562dec83b3f8d3088d77584727b09130b2
> Author: Josef Bacik <jba...@fb.com>
> Date:   Tue May 13 17:30:47 2014 -0700
> 
>     Btrfs: rework qgroup accounting
> ===============================================================================
> 
> Josef, please take a look at this patch.

Reproducer:
https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35299.html

Could you tell me the progress of fixing this bug?
In addition, could you fix it by 3.16?

command log:
===============================================================================
# ./test.sh
+ uname -a
Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux
+ df -T /test7
Filesystem     Type  1K-blocks  Used Available Use% Mounted on
/dev/sdc7      btrfs  29296640  1536  27169536   1% /test7
+ btrfs quota ena /test7
+ cd /test7
+ btrfs sub cre test
Create subvolume './test'
+ btrfs sub l -a /test7
ID 270 gen 66 top level 5 path test
+ btrfs qg lim 1G test                          # limit test subvol to 1GB
+ btrfs qg show -pcre /test7
qgroupid rfer  excl  max_rfer   max_excl parent  child
-------- ----  ----  --------   -------- ------  -----
0/5      16384 16384 0          0        ---     ---
0/270    16384 16384 1073741824 0        ---     ---
+ dd if=/dev/zero of=test/file0 bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s   # write 2GB. It's a bug!
+ sync
+ ls -lisaR /test7
/test7:
total 20
256 16 drwxr-xr-x   1 root root    8 Jul 18 15:12 .
  2  4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 ..
256  0 drwxr-xr-x   1 root root   10 Jul 18 15:17 test

/test7/test:
total 2048016
256       0 drwxr-xr-x 1 root root         10 Jul 18 15:17 .
256      16 drwxr-xr-x 1 root root          8 Jul 18 15:12 ..
257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
+ btrfs qg show -pcre /test7
qgroupid rfer       excl       max_rfer   max_excl parent  child
-------- ----       ----       --------   -------- ------  -----
0/5      16384      16384      0          0        ---     ---
0/270    2097168384 2097168384 1073741824 0        ---     ---
+ btrfs quota dis /test7
+ btrfs sub del test
Transaction commit: none (default)
Delete subvolume '/test7/test'
+ set +x
#
===============================================================================

  NOTE:
  The reproducer here (./test.sh) is a bit different from above-mentioned
  one because of some reason.

Thanks,
Satoru

Re: [BUG] "Quota Ignored On write" problem still exist with 3.16-rc5

[Wang Shilong]

On 07/18/2014 04:45 PM, Satoru Takeuchi wrote:
> Hi Josef, Chris,
>
> I found "Quota Ignored On write" problem still exist with 3.16-rc5,
> which Kevin reported before.
>
> Kevin's report:
> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35292.html
>
> The result of bisect:
> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35304.html
I guess this is because Josef's patch delayed qgroup accounting, it will
cause @refer and @excl updating very late...

The patch maybe optimize to merge some delayed refs(for example), but
it updates qgroup accounting when commiting transaction which will be
very late,
we may have accumulated many data..

Thanks,
Wang
>
>> I bisected and found the bad commit is the following patch.
>>
>> ===============================================================================
>> commit fcebe4562dec83b3f8d3088d77584727b09130b2
>> Author: Josef Bacik <jba...@fb.com>
>> Date:   Tue May 13 17:30:47 2014 -0700
>>
>>     Btrfs: rework qgroup accounting
>> ===============================================================================
>>
>> Josef, please take a look at this patch.
> Reproducer:
> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35299.html
>
> Could you tell me the progress of fixing this bug?
> In addition, could you fix it by 3.16?
>
> command log:
> ===============================================================================
> # ./test.sh
> + uname -a
> Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux
> + df -T /test7
> Filesystem     Type  1K-blocks  Used Available Use% Mounted on
> /dev/sdc7      btrfs  29296640  1536  27169536   1% /test7
> + btrfs quota ena /test7
> + cd /test7
> + btrfs sub cre test
> Create subvolume './test'
> + btrfs sub l -a /test7
> ID 270 gen 66 top level 5 path test
> + btrfs qg lim 1G test                          # limit test subvol to 1GB
> + btrfs qg show -pcre /test7
> qgroupid rfer  excl  max_rfer   max_excl parent  child
> -------- ----  ----  --------   -------- ------  -----
> 0/5      16384 16384 0          0        ---     ---
> 0/270    16384 16384 1073741824 0        ---     ---
> + dd if=/dev/zero of=test/file0 bs=1M count=2000
> 2000+0 records in
> 2000+0 records out
> 2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s   # write 2GB. It's a bug!
> + sync
> + ls -lisaR /test7
> /test7:
> total 20
> 256 16 drwxr-xr-x   1 root root    8 Jul 18 15:12 .
>   2  4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 ..
> 256  0 drwxr-xr-x   1 root root   10 Jul 18 15:17 test
>
> /test7/test:
> total 2048016
> 256       0 drwxr-xr-x 1 root root         10 Jul 18 15:17 .
> 256      16 drwxr-xr-x 1 root root          8 Jul 18 15:12 ..
> 257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
> + btrfs qg show -pcre /test7
> qgroupid rfer       excl       max_rfer   max_excl parent  child
> -------- ----       ----       --------   -------- ------  -----
> 0/5      16384      16384      0          0        ---     ---
> 0/270    2097168384 2097168384 1073741824 0        ---     ---
> + btrfs quota dis /test7
> + btrfs sub del test
> Transaction commit: none (default)
> Delete subvolume '/test7/test'
> + set +x
> #
> ===============================================================================
>
>   NOTE:
>   The reproducer here (./test.sh) is a bit different from above-mentioned
>   one because of some reason.
>
> Thanks,
> Satoru
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> .
>

Re: [BUG] "Quota Ignored On write" problem still exist with 3.16-rc5

[Satoru Takeuchi]

Hi Wang,

(2014/07/18 19:29), Wang Shilong wrote:
> On 07/18/2014 04:45 PM, Satoru Takeuchi wrote:
>> Hi Josef, Chris,
>>
>> I found "Quota Ignored On write" problem still exist with 3.16-rc5,
>> which Kevin reported before.
>>
>> Kevin's report:
>> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35292.html
>>
>> The result of bisect:
>> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35304.html
> I guess this is because Josef's patch delayed qgroup accounting, it will
> cause @refer and @excl updating very late...
> 
> The patch maybe optimize to merge some delayed refs(for example), but
> it updates qgroup accounting when commiting transaction which will be
> very late,
> we may have accumulated many data..

Thank you for your comment. I know of the code logic which caused
this problem. However, what I want to say here is that this
problem should be fixed as soon as possible. It is a important
regression problem and we've already know the root cause.

If it's impossible to fix it by releasing 3.16, I consider this
patch should be reverted.

Thanks,
Satoru

> 
> Thanks,
> Wang
>>
>>> I bisected and found the bad commit is the following patch.
>>>
>>> ===============================================================================
>>> commit fcebe4562dec83b3f8d3088d77584727b09130b2
>>> Author: Josef Bacik <jba...@fb.com>
>>> Date:   Tue May 13 17:30:47 2014 -0700
>>>
>>>      Btrfs: rework qgroup accounting
>>> ===============================================================================
>>>
>>> Josef, please take a look at this patch.
>> Reproducer:
>> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35299.html
>>
>> Could you tell me the progress of fixing this bug?
>> In addition, could you fix it by 3.16?
>>
>> command log:
>> ===============================================================================
>> # ./test.sh
>> + uname -a
>> Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux
>> + df -T /test7
>> Filesystem     Type  1K-blocks  Used Available Use% Mounted on
>> /dev/sdc7      btrfs  29296640  1536  27169536   1% /test7
>> + btrfs quota ena /test7
>> + cd /test7
>> + btrfs sub cre test
>> Create subvolume './test'
>> + btrfs sub l -a /test7
>> ID 270 gen 66 top level 5 path test
>> + btrfs qg lim 1G test                          # limit test subvol to 1GB
>> + btrfs qg show -pcre /test7
>> qgroupid rfer  excl  max_rfer   max_excl parent  child
>> -------- ----  ----  --------   -------- ------  -----
>> 0/5      16384 16384 0          0        ---     ---
>> 0/270    16384 16384 1073741824 0        ---     ---
>> + dd if=/dev/zero of=test/file0 bs=1M count=2000
>> 2000+0 records in
>> 2000+0 records out
>> 2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s   # write 2GB. It's a bug!
>> + sync
>> + ls -lisaR /test7
>> /test7:
>> total 20
>> 256 16 drwxr-xr-x   1 root root    8 Jul 18 15:12 .
>>    2  4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 ..
>> 256  0 drwxr-xr-x   1 root root   10 Jul 18 15:17 test
>>
>> /test7/test:
>> total 2048016
>> 256       0 drwxr-xr-x 1 root root         10 Jul 18 15:17 .
>> 256      16 drwxr-xr-x 1 root root          8 Jul 18 15:12 ..
>> 257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
>> + btrfs qg show -pcre /test7
>> qgroupid rfer       excl       max_rfer   max_excl parent  child
>> -------- ----       ----       --------   -------- ------  -----
>> 0/5      16384      16384      0          0        ---     ---
>> 0/270    2097168384 2097168384 1073741824 0        ---     ---
>> + btrfs quota dis /test7
>> + btrfs sub del test
>> Transaction commit: none (default)
>> Delete subvolume '/test7/test'
>> + set +x
>> #
>> ===============================================================================
>>
>>    NOTE:
>>    The reproducer here (./test.sh) is a bit different from above-mentioned
>>    one because of some reason.
>>
>> Thanks,
>> Satoru
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> .
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [BUG] "Quota Ignored On write" problem still exist with 3.16-rc5

[Wang Shilong]

Hi Satoru-san,

On 07/23/2014 08:53 AM, Satoru Takeuchi wrote:
> Hi Wang,
>
> (2014/07/18 19:29), Wang Shilong wrote:
>> On 07/18/2014 04:45 PM, Satoru Takeuchi wrote:
>>> Hi Josef, Chris,
>>>
>>> I found "Quota Ignored On write" problem still exist with 3.16-rc5,
>>> which Kevin reported before.
>>>
>>> Kevin's report:
>>> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35292.html
>>>
>>> The result of bisect:
>>> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35304.html
>> I guess this is because Josef's patch delayed qgroup accounting, it will
>> cause @refer and @excl updating very late...
>>
>> The patch maybe optimize to merge some delayed refs(for example), but
>> it updates qgroup accounting when commiting transaction which will be
>> very late,
>> we may have accumulated many data..
> Thank you for your comment. I know of the code logic which caused
> this problem. However, what I want to say here is that this
> problem should be fixed as soon as possible. It is a important
> regression problem and we've already know the root cause.
>
> If it's impossible to fix it by releasing 3.16, I consider this
> patch should be reverted.
Since Btrfs Quota function is under heavy development, and should be
considered as *broken*.

I think we'd better close Btrfs quota function(Like snapshot-aware function)
until we really sit down and solve everything.

Thanks,
Wang
>
> Thanks,
> Satoru
>
>> Thanks,
>> Wang
>>>> I bisected and found the bad commit is the following patch.
>>>>
>>>> ===============================================================================
>>>> commit fcebe4562dec83b3f8d3088d77584727b09130b2
>>>> Author: Josef Bacik <jba...@fb.com>
>>>> Date:   Tue May 13 17:30:47 2014 -0700
>>>>
>>>>      Btrfs: rework qgroup accounting
>>>> ===============================================================================
>>>>
>>>> Josef, please take a look at this patch.
>>> Reproducer:
>>> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35299.html
>>>
>>> Could you tell me the progress of fixing this bug?
>>> In addition, could you fix it by 3.16?
>>>
>>> command log:
>>> ===============================================================================
>>> # ./test.sh
>>> + uname -a
>>> Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux
>>> + df -T /test7
>>> Filesystem     Type  1K-blocks  Used Available Use% Mounted on
>>> /dev/sdc7      btrfs  29296640  1536  27169536   1% /test7
>>> + btrfs quota ena /test7
>>> + cd /test7
>>> + btrfs sub cre test
>>> Create subvolume './test'
>>> + btrfs sub l -a /test7
>>> ID 270 gen 66 top level 5 path test
>>> + btrfs qg lim 1G test                          # limit test subvol to 1GB
>>> + btrfs qg show -pcre /test7
>>> qgroupid rfer  excl  max_rfer   max_excl parent  child
>>> -------- ----  ----  --------   -------- ------  -----
>>> 0/5      16384 16384 0          0        ---     ---
>>> 0/270    16384 16384 1073741824 0        ---     ---
>>> + dd if=/dev/zero of=test/file0 bs=1M count=2000
>>> 2000+0 records in
>>> 2000+0 records out
>>> 2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s   # write 2GB. It's a bug!
>>> + sync
>>> + ls -lisaR /test7
>>> /test7:
>>> total 20
>>> 256 16 drwxr-xr-x   1 root root    8 Jul 18 15:12 .
>>>    2  4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 ..
>>> 256  0 drwxr-xr-x   1 root root   10 Jul 18 15:17 test
>>>
>>> /test7/test:
>>> total 2048016
>>> 256       0 drwxr-xr-x 1 root root         10 Jul 18 15:17 .
>>> 256      16 drwxr-xr-x 1 root root          8 Jul 18 15:12 ..
>>> 257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
>>> + btrfs qg show -pcre /test7
>>> qgroupid rfer       excl       max_rfer   max_excl parent  child
>>> -------- ----       ----       --------   -------- ------  -----
>>> 0/5      16384      16384      0          0        ---     ---
>>> 0/270    2097168384 2097168384 1073741824 0        ---     ---
>>> + btrfs quota dis /test7
>>> + btrfs sub del test
>>> Transaction commit: none (default)
>>> Delete subvolume '/test7/test'
>>> + set +x
>>> #
>>> ===============================================================================
>>>
>>>    NOTE:
>>>    The reproducer here (./test.sh) is a bit different from above-mentioned
>>>    one because of some reason.
>>>
>>> Thanks,
>>> Satoru
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> .
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> .
>

Re: [BUG] "Quota Ignored On write" problem still exist with 3.16-rc5

[Josef Bacik]

On 07/22/2014 08:53 PM, Satoru Takeuchi wrote:
> Hi Wang,
> 
> (2014/07/18 19:29), Wang Shilong wrote:
>> On 07/18/2014 04:45 PM, Satoru Takeuchi wrote:
>>> Hi Josef, Chris,
>>>
>>> I found "Quota Ignored On write" problem still exist with 3.16-rc5,
>>> which Kevin reported before.
>>>
>>> Kevin's report:
>>> https://urldefense.proofpoint.com/v1/url?u=https://www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg35292.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=HiJFypfWVhg2Jy2US%2BNG1tF8q4%2BYUryNKec6yPkYJM4%3D%0A&s=9d1740f5e1701e4906fc04b77d9117052e84d6e16c30971b261d1acf41c10910
>>>
>>> The result of bisect:
>>> https://urldefense.proofpoint.com/v1/url?u=https://www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg35304.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=HiJFypfWVhg2Jy2US%2BNG1tF8q4%2BYUryNKec6yPkYJM4%3D%0A&s=75ca6ce24c6d8e64fccdb8f61e519c59b1c6d027140c4f0f5eac24942a53a55d
>> I guess this is because Josef's patch delayed qgroup accounting, it will
>> cause @refer and @excl updating very late...
>>
>> The patch maybe optimize to merge some delayed refs(for example), but
>> it updates qgroup accounting when commiting transaction which will be
>> very late,
>> we may have accumulated many data..
> 
> Thank you for your comment. I know of the code logic which caused
> this problem. However, what I want to say here is that this
> problem should be fixed as soon as possible. It is a important
> regression problem and we've already know the root cause.
> 
> If it's impossible to fix it by releasing 3.16, I consider this
> patch should be reverted.
> 

That's not happening.  Qgroups have always been broken in one way or
another, now with my stuff the accounting is always correct and we don't
have a huge performance impact with or without qgroups.  We'll fix it
but there's no way we're reverting this work.  Thanks,

Josef

Re: [BUG] "Quota Ignored On write" problem still exist with 3.16-rc5

[Kevin Brandstatter]

On 07/23/2014 08:23 AM, Josef Bacik wrote:
> On 07/22/2014 08:53 PM, Satoru Takeuchi wrote:
>> Hi Wang,
>>
>> (2014/07/18 19:29), Wang Shilong wrote:
>>> On 07/18/2014 04:45 PM, Satoru Takeuchi wrote:
>>>> Hi Josef, Chris,
>>>>
>>>> I found "Quota Ignored On write" problem still exist with 3.16-rc5,
>>>> which Kevin reported before.
>>>>
>>>> Kevin's report:
>>>> https://urldefense.proofpoint.com/v1/url?u=https://www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg35292.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=HiJFypfWVhg2Jy2US%2BNG1tF8q4%2BYUryNKec6yPkYJM4%3D%0A&s=9d1740f5e1701e4906fc04b77d9117052e84d6e16c30971b261d1acf41c10910
>>>>
>>>> The result of bisect:
>>>> https://urldefense.proofpoint.com/v1/url?u=https://www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg35304.html&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=HiJFypfWVhg2Jy2US%2BNG1tF8q4%2BYUryNKec6yPkYJM4%3D%0A&s=75ca6ce24c6d8e64fccdb8f61e519c59b1c6d027140c4f0f5eac24942a53a55d
>>> I guess this is because Josef's patch delayed qgroup accounting, it will
>>> cause @refer and @excl updating very late...
>>>
>>> The patch maybe optimize to merge some delayed refs(for example), but
>>> it updates qgroup accounting when commiting transaction which will be
>>> very late,
>>> we may have accumulated many data..
>> Thank you for your comment. I know of the code logic which caused
>> this problem. However, what I want to say here is that this
>> problem should be fixed as soon as possible. It is a important
>> regression problem and we've already know the root cause.
>>
>> If it's impossible to fix it by releasing 3.16, I consider this
>> patch should be reverted.
>>
> That's not happening.  Qgroups have always been broken in one way or
> another, now with my stuff the accounting is always correct and we don't
> have a huge performance impact with or without qgroups.  We'll fix it
> but there's no way we're reverting this work.  Thanks,
>
> Josef
>
How/when does the accounting happen? Obviously we want the accounting to
be correct, but the overrun seems to indicate that the accounting isn't
happening early enough. From briefly looking at the patches, it looks to
be that the intent is to account when a new extent is allocated, but it
seems that this isn't happening if its still allowed to allocate post
quota, or is the accounting postponed till after a specified time, etc?

-Kevin