[PATCH] common/quota: fix to wait for all inodes been evicted in _check_quota_usage()

[PATCH] common/quota: fix to wait for all inodes been evicted in _check_quota_usage()

[Chao Yu]

generic/233  3s ... - output mismatch (see /share/git/fstests/results//generic/233.out.bad)
    --- tests/generic/233.out   2025-01-12 21:57:40.259440359 +0800
    +++ /share/git/fstests/results//generic/233.out.bad 2025-12-04 03:02:26.000000000 +0800
    @@ -4,4 +4,12 @@

     seed = S
     Comparing user usage
    +4c4
    +< #1000     --   31476   32000   32000            994  1000  1000
    +---
    +> #1000     --   31476   32000   32000            944  1000  1000
    ...
    (Run 'diff -u /share/git/fstests/tests/generic/233.out /share/git/fstests/results//generic/233.out.bad'  to see the entire diff)
Ran: generic/233
Failures: generic/233
Failed 1 of 1 tests

Sometimes, generic/233 will fail due to it founds inode count is mismatched
in between in-memory one and on-disk one.

The reason is cgroup v2 implementation requires to increase inode reference,
and then attach it to thread related cgroup writeback structure, once it
needs to switch once write owner changes of target inode, a kernel thread
will process it and then release inode reference via evict_inode().

So, sync & drop_cache may not guarantee all inodes being evicted, as cgroup
has one more refernece on inodes during the time.

If inode has not been evicted, dquot inode reference will not be release, it
will lead to inode quota mismatch.

Let's add a delay to wait for cgroup switching completion before quota check.

Cc: Jaegeuk Kim <jaegeuk@kernel.org>
Signed-off-by: Chao Yu <chao@kernel.org>
---
 common/quota | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/common/quota b/common/quota
index a51386b1..de7e84e8 100644
--- a/common/quota
+++ b/common/quota
@@ -333,6 +333,9 @@ _check_quota_usage()
 	# XXX: really need an ioctl instead of this big hammer
 	echo 3 > /proc/sys/vm/drop_caches
 
+	# wait for inode_switch_wbs_wor_fn() to release inodes
+	sleep 3
+
 	VFS_QUOTA=0
 	case $FSTYP in
 	ext2|ext3|ext4|f2fs|gfs2|bcachefs)
-- 
2.49.0

Re: [PATCH] common/quota: fix to wait for all inodes been evicted in _check_quota_usage()

[Darrick J. Wong]

On Thu, Dec 04, 2025 at 03:31:47AM +0800, Chao Yu wrote:
> generic/233  3s ... - output mismatch (see /share/git/fstests/results//generic/233.out.bad)
>     --- tests/generic/233.out   2025-01-12 21:57:40.259440359 +0800
>     +++ /share/git/fstests/results//generic/233.out.bad 2025-12-04 03:02:26.000000000 +0800
>     @@ -4,4 +4,12 @@
> 
>      seed = S
>      Comparing user usage
>     +4c4
>     +< #1000     --   31476   32000   32000            994  1000  1000
>     +---
>     +> #1000     --   31476   32000   32000            944  1000  1000
>     ...
>     (Run 'diff -u /share/git/fstests/tests/generic/233.out /share/git/fstests/results//generic/233.out.bad'  to see the entire diff)
> Ran: generic/233
> Failures: generic/233
> Failed 1 of 1 tests
> 
> Sometimes, generic/233 will fail due to it founds inode count is mismatched
> in between in-memory one and on-disk one.
> 
> The reason is cgroup v2 implementation requires to increase inode reference,
> and then attach it to thread related cgroup writeback structure, once it
> needs to switch once write owner changes of target inode, a kernel thread
> will process it and then release inode reference via evict_inode().
> 
> So, sync & drop_cache may not guarantee all inodes being evicted, as cgroup
> has one more refernece on inodes during the time.
> 
> If inode has not been evicted, dquot inode reference will not be release, it
> will lead to inode quota mismatch.
> 
> Let's add a delay to wait for cgroup switching completion before quota check.
> 
> Cc: Jaegeuk Kim <jaegeuk@kernel.org>
> Signed-off-by: Chao Yu <chao@kernel.org>
> ---
>  common/quota | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/common/quota b/common/quota
> index a51386b1..de7e84e8 100644
> --- a/common/quota
> +++ b/common/quota
> @@ -333,6 +333,9 @@ _check_quota_usage()
>  	# XXX: really need an ioctl instead of this big hammer
>  	echo 3 > /proc/sys/vm/drop_caches
>  
> +	# wait for inode_switch_wbs_wor_fn() to release inodes
> +	sleep 3

Odd -- I've never seen this problem crop up on ext4 or XFS.  What's
different about f2fs such that writeback is still in progress?

Oh, XFS cycles the mount and ext4 does ... quotaoff???

Perhaps the sleep 3 thing should only be done for f2fs?  Or do you see
it on other filesystems as well?

--D

> +
>  	VFS_QUOTA=0
>  	case $FSTYP in
>  	ext2|ext3|ext4|f2fs|gfs2|bcachefs)
> -- 
> 2.49.0
> 
> 

Re: [PATCH] common/quota: fix to wait for all inodes been evicted in _check_quota_usage()

[Chao Yu]

On 2025/12/5 05:59, Darrick J. Wong wrote:
> On Thu, Dec 04, 2025 at 03:31:47AM +0800, Chao Yu wrote:
>> generic/233  3s ... - output mismatch (see /share/git/fstests/results//generic/233.out.bad)
>>      --- tests/generic/233.out   2025-01-12 21:57:40.259440359 +0800
>>      +++ /share/git/fstests/results//generic/233.out.bad 2025-12-04 03:02:26.000000000 +0800
>>      @@ -4,4 +4,12 @@
>>
>>       seed = S
>>       Comparing user usage
>>      +4c4
>>      +< #1000     --   31476   32000   32000            994  1000  1000
>>      +---
>>      +> #1000     --   31476   32000   32000            944  1000  1000
>>      ...
>>      (Run 'diff -u /share/git/fstests/tests/generic/233.out /share/git/fstests/results//generic/233.out.bad'  to see the entire diff)
>> Ran: generic/233
>> Failures: generic/233
>> Failed 1 of 1 tests
>>
>> Sometimes, generic/233 will fail due to it founds inode count is mismatched
>> in between in-memory one and on-disk one.
>>
>> The reason is cgroup v2 implementation requires to increase inode reference,
>> and then attach it to thread related cgroup writeback structure, once it
>> needs to switch once write owner changes of target inode, a kernel thread
>> will process it and then release inode reference via evict_inode().
>>
>> So, sync & drop_cache may not guarantee all inodes being evicted, as cgroup
>> has one more refernece on inodes during the time.
>>
>> If inode has not been evicted, dquot inode reference will not be release, it
>> will lead to inode quota mismatch.
>>
>> Let's add a delay to wait for cgroup switching completion before quota check.
>>
>> Cc: Jaegeuk Kim <jaegeuk@kernel.org>
>> Signed-off-by: Chao Yu <chao@kernel.org>
>> ---
>>   common/quota | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/common/quota b/common/quota
>> index a51386b1..de7e84e8 100644
>> --- a/common/quota
>> +++ b/common/quota
>> @@ -333,6 +333,9 @@ _check_quota_usage()
>>   	# XXX: really need an ioctl instead of this big hammer
>>   	echo 3 > /proc/sys/vm/drop_caches
>>   
>> +	# wait for inode_switch_wbs_wor_fn() to release inodes
>> +	sleep 3
> 
> Odd -- I've never seen this problem crop up on ext4 or XFS.  What's
> different about f2fs such that writeback is still in progress?
> 
> Oh, XFS cycles the mount and ext4 does ... quotaoff???

f2fs does quotaoff like ext4 as well, but the difference ext4_quota_off() will call
sync_filesystem() before turning off quota, f2fs doesn't, not sure this is the key
to wait/speedup inode_switch_wbs_wor_fn() or not.

 > > Perhaps the sleep 3 thing should only be done for f2fs?

Yes, I think so, let me revise the patch.

> Or do you see it on other filesystems as well?

I tested ext4, it doesn't fail.

Thanks,

> 
> --D
> 
>> +
>>   	VFS_QUOTA=0
>>   	case $FSTYP in
>>   	ext2|ext3|ext4|f2fs|gfs2|bcachefs)
>> -- 
>> 2.49.0
>>
>>