Weird behaviour with project quotas

Weird behaviour with project quotas

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]


Hello,

As far as I understand, and from my tests, on folders on which a
project quota is applied, either the available quota or the actually
avialable space should be reported when using "df". However on a
running system (Debian 12, kernel 6.1 Debian) I have incoherent results:

The volume /mnt/raid is 100 TB and has 500GB free.

There are several folders like /mnt/raid/project1, /mnt/raid/project2
etc with various quotas (20TB, 30TB, etc).

Though there are only 500 GB free on the whole filesystem, "df
/mnt/raid/project1" etc reports several TB available, which is the
remaining quota for the folder. Of course the users are led to believe
there is more space available than what's actually the case (in fact
that's why the volume is down to so little space) which causes all sort
of problems.

What's strange is that I can't reproduce this behaviour on another
system. Every time I try to reproduce it, available space reported by
df on a project quota folder matches actual available space.

Any idea on what could be going wrong ?

-- 
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
 

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Weird behaviour with project quotas

[Dave Chinner]

On Thu, Nov 28, 2024 at 05:14:58PM +0100, Emmanuel Florac wrote:
> 
> Hello,
> 
> As far as I understand, and from my tests, on folders on which a
> project quota is applied, either the available quota or the actually
> avialable space should be reported when using "df".

Only if you are using project quotas as directories quotas. i.e.
the directory you are querying with df needs to have the
XFS_DIFLAG_PROJINHERIT flag set on it for df to behave this way.

> However on a
> running system (Debian 12, kernel 6.1 Debian) I have incoherent results:

32 bit or 64 bit architecture?

> The volume /mnt/raid is 100 TB and has 500GB free.
> 
> There are several folders like /mnt/raid/project1, /mnt/raid/project2
> etc with various quotas (20TB, 30TB, etc).

Output of df and a project quota report showing usage and limits
would be useful here.

Then, for each of the top level project directories you are querying
with df, also run `xfs_io -rxc "stat" <dir>` and post the output.
This will tell us if the project quota is set up correctly for df to
report quota limits for them.

It would also be useful to know if the actual quota usage is correct
- having the output of `du -s /mnt/raid/project1` to count the
blocks and `find /mnt/raid/project1 -print |wc -l` to count the
files in quota controlled directories. That'll give us some idea if
there's a quota accounting issue.

> Any idea on what could be going wrong ?

No. There's nothing obvious that stands out in the current TOT
code...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Weird behaviour with project quotas

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 2311 bytes --]

Le Fri, 29 Nov 2024 08:07:09 +1100
Dave Chinner <david@fromorbit.com> écrivait:

> > As far as I understand, and from my tests, on folders on which a
> > project quota is applied, either the available quota or the actually
> > avialable space should be reported when using "df".  
> 
> Only if you are using project quotas as directories quotas. i.e.
> the directory you are querying with df needs to have the
> XFS_DIFLAG_PROJINHERIT flag set on it for df to behave this way.

Interesting, and how is this set ? I basically set up quotas using
something like

xfs_quota -x -c 'project -s -p /mnt/raid/project1 10' /mnt/raid

xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid


> > However on a
> > running system (Debian 12, kernel 6.1 Debian) I have incoherent
> > results:  
> 
> 32 bit or 64 bit architecture?

AMD64, the most common one.
 
> > The volume /mnt/raid is 100 TB and has 500GB free.
> > 
> > There are several folders like /mnt/raid/project1,
> > /mnt/raid/project2 etc with various quotas (20TB, 30TB, etc).  
> 
> Output of df and a project quota report showing usage and limits
> would be useful here.
> 
> Then, for each of the top level project directories you are querying
> with df, also run `xfs_io -rxc "stat" <dir>` and post the output.
> This will tell us if the project quota is set up correctly for df to
> report quota limits for them.
> 
> It would also be useful to know if the actual quota usage is correct
> - having the output of `du -s /mnt/raid/project1` to count the
> blocks and `find /mnt/raid/project1 -print |wc -l` to count the
> files in quota controlled directories. That'll give us some idea if
> there's a quota accounting issue.
> 

OK, I'll run these as soon as I have a connection to the system. There
seemed to be no error with the reported used space though, only
available remaining space in quota'ed directories; however df reports
actually available space for directories without quota set.


-- 
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
------------------------------------------------------------------------
 

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Weird behaviour with project quotas

[Dave Chinner]

On Fri, Nov 29, 2024 at 10:33:32AM +0100, Emmanuel Florac wrote:
> Le Fri, 29 Nov 2024 08:07:09 +1100
> Dave Chinner <david@fromorbit.com> écrivait:
> 
> > > As far as I understand, and from my tests, on folders on which a
> > > project quota is applied, either the available quota or the actually
> > > avialable space should be reported when using "df".  
> > 
> > Only if you are using project quotas as directories quotas. i.e.
> > the directory you are querying with df needs to have the
> > XFS_DIFLAG_PROJINHERIT flag set on it for df to behave this way.
> 
> Interesting, and how is this set ? I basically set up quotas using
> something like
> 
> xfs_quota -x -c 'project -s -p /mnt/raid/project1 10' /mnt/raid
> 
> xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid

That should set it up appropriately, hence the need to check if it
has actually been set up correctly on disk.

> > > However on a
> > > running system (Debian 12, kernel 6.1 Debian) I have incoherent
> > > results:  
> > 
> > 32 bit or 64 bit architecture?
> 
> AMD64, the most common one.
>  
> > > The volume /mnt/raid is 100 TB and has 500GB free.
> > > 
> > > There are several folders like /mnt/raid/project1,
> > > /mnt/raid/project2 etc with various quotas (20TB, 30TB, etc).  
> > 
> > Output of df and a project quota report showing usage and limits
> > would be useful here.
> > 
> > Then, for each of the top level project directories you are querying
> > with df, also run `xfs_io -rxc "stat" <dir>` and post the output.
> > This will tell us if the project quota is set up correctly for df to
> > report quota limits for them.
> > 
> > It would also be useful to know if the actual quota usage is correct
> > - having the output of `du -s /mnt/raid/project1` to count the
> > blocks and `find /mnt/raid/project1 -print |wc -l` to count the
> > files in quota controlled directories. That'll give us some idea if
> > there's a quota accounting issue.

iAnother thought occurred to me - can you also check that
/etc/projid and /etc/projects is similar on all machines, and post
the contents of them from the bad machine?

> OK, I'll run these as soon as I have a connection to the system. There
> seemed to be no error with the reported used space though, only
> available remaining space in quota'ed directories; however df reports
> actually available space for directories without quota set.

Ok - it would be helpful to post the commands you run and the output
so that we both know exactly what behaviour you are seeing.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Weird behaviour with project quotas

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 3992 bytes --]

Le Sat, 30 Nov 2024 09:14:20 +1100
Dave Chinner <david@fromorbit.com> écrivait:

> > xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid  
> 
> That should set it up appropriately, hence the need to check if it
> has actually been set up correctly on disk.
> 

Unfortunately in the meantime the users did some cleanup, therefore the
displayed information is coherent again (as there is more free space on
the filesystem as a whole as any remaining allocated quota).

xfs_quota -x -c "report -p"
Project quota on /mnt/raid (/dev/mapper/vg0-raid)
                         Blocks  
Project ID       Used       Soft       Hard    Warn/Grace

----------- ------------------------------------------------------

<snip>
#40        10795758244          0 16106127360     00 [--------]


> > > Output of df and a project quota report showing usage and limits
> > > would be useful here.

looking at the corresponding folder :

/dev/mapper/vg0-raid    15T     11T  5,0T  68% /mnt/raid/pad


 du -s /mnt/raid/pad
10795758244	/mnt/raid/pad

# find /mnt/raid/pad -print | wc -l
39086

> > > Then, for each of the top level project directories you are
> > > querying with df, also run `xfs_io -rxc "stat" <dir>` and post
> > > the output. This will tell us if the project quota is set up
> > > correctly for df to report quota limits for them.
> > > 

Starting with "pad" :

# xfs_io -rxc "stat" pad
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 6442662464
stat.type = directory
stat.size = 4096
stat.blocks = 16
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 40
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 2
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136

# xfs_io -rxc "stat" rush
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 142
stat.type = directory
stat.size = 283
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 10
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136

# xfs_io -rxc "stat" labo
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 2147695168
stat.type = directory
stat.size = 310
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 20
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136

# xfs_io -rxc "stat" prods
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 4295178816
stat.type = directory
stat.size = 319
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 30
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136

> > > It would also be useful to know if the actual quota usage is
> > > correct
> > > - having the output of `du -s /mnt/raid/project1` to count the
> > > blocks and `find /mnt/raid/project1 -print |wc -l` to count the
> > > files in quota controlled directories. That'll give us some idea
> > > if there's a quota accounting issue.  
> 
> iAnother thought occurred to me - can you also check that
> /etc/projid and /etc/projects is similar on all machines, and post
> the contents of them from the bad machine?
> 

Hum, actually they didn't set up neither projid nor projects. Of course
I did create these during my tests, but could this be the culprit ?

-- 
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
 

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Weird behaviour with project quotas

[Darrick J. Wong]

On Thu, Dec 12, 2024 at 04:33:51PM +0100, Emmanuel Florac wrote:
> Le Sat, 30 Nov 2024 09:14:20 +1100
> Dave Chinner <david@fromorbit.com> écrivait:
> 
> > > xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid  
> > 
> > That should set it up appropriately, hence the need to check if it
> > has actually been set up correctly on disk.
> > 
> 
> Unfortunately in the meantime the users did some cleanup, therefore the
> displayed information is coherent again (as there is more free space on
> the filesystem as a whole as any remaining allocated quota).
> 
> xfs_quota -x -c "report -p"
> Project quota on /mnt/raid (/dev/mapper/vg0-raid)
>                          Blocks  
> Project ID       Used       Soft       Hard    Warn/Grace
> 
> ----------- ------------------------------------------------------
> 
> <snip>
> #40        10795758244          0 16106127360     00 [--------]
> 
> 
> > > > Output of df and a project quota report showing usage and limits
> > > > would be useful here.
> 
> looking at the corresponding folder :
> 
> /dev/mapper/vg0-raid    15T     11T  5,0T  68% /mnt/raid/pad
> 
> 
>  du -s /mnt/raid/pad
> 10795758244	/mnt/raid/pad
> 
> # find /mnt/raid/pad -print | wc -l
> 39086
> 
> > > > Then, for each of the top level project directories you are
> > > > querying with df, also run `xfs_io -rxc "stat" <dir>` and post
> > > > the output. This will tell us if the project quota is set up
> > > > correctly for df to report quota limits for them.
> > > > 
> 
> Starting with "pad" :
> 
> # xfs_io -rxc "stat" pad
> fd.path = "."
> fd.flags = non-sync,non-direct,read-only
> stat.ino = 6442662464
> stat.type = directory
> stat.size = 4096
> stat.blocks = 16
> fsxattr.xflags = 0x200 \[--------P--------\]
> fsxattr.projid = 40
> fsxattr.extsize = 0
> fsxattr.cowextsize = 0
> fsxattr.nextents = 2
> fsxattr.naextents = 0
> dioattr.mem = 0x200
> dioattr.miniosz = 512
> dioattr.maxiosz = 2147483136
> 
> # xfs_io -rxc "stat" rush
> fd.path = "."
> fd.flags = non-sync,non-direct,read-only
> stat.ino = 142
> stat.type = directory
> stat.size = 283
> stat.blocks = 0
> fsxattr.xflags = 0x200 \[--------P--------\]
> fsxattr.projid = 10
> fsxattr.extsize = 0
> fsxattr.cowextsize = 0
> fsxattr.nextents = 0
> fsxattr.naextents = 0
> dioattr.mem = 0x200
> dioattr.miniosz = 512
> dioattr.maxiosz = 2147483136
> 
> # xfs_io -rxc "stat" labo
> fd.path = "."
> fd.flags = non-sync,non-direct,read-only
> stat.ino = 2147695168
> stat.type = directory
> stat.size = 310
> stat.blocks = 0
> fsxattr.xflags = 0x200 \[--------P--------\]
> fsxattr.projid = 20
> fsxattr.extsize = 0
> fsxattr.cowextsize = 0
> fsxattr.nextents = 0
> fsxattr.naextents = 0
> dioattr.mem = 0x200
> dioattr.miniosz = 512
> dioattr.maxiosz = 2147483136
> 
> # xfs_io -rxc "stat" prods
> fd.path = "."
> fd.flags = non-sync,non-direct,read-only
> stat.ino = 4295178816
> stat.type = directory
> stat.size = 319
> stat.blocks = 0
> fsxattr.xflags = 0x200 \[--------P--------\]
> fsxattr.projid = 30
> fsxattr.extsize = 0
> fsxattr.cowextsize = 0
> fsxattr.nextents = 0
> fsxattr.naextents = 0
> dioattr.mem = 0x200
> dioattr.miniosz = 512
> dioattr.maxiosz = 2147483136
> 
> > > > It would also be useful to know if the actual quota usage is
> > > > correct
> > > > - having the output of `du -s /mnt/raid/project1` to count the
> > > > blocks and `find /mnt/raid/project1 -print |wc -l` to count the
> > > > files in quota controlled directories. That'll give us some idea
> > > > if there's a quota accounting issue.  
> > 
> > iAnother thought occurred to me - can you also check that
> > /etc/projid and /etc/projects is similar on all machines, and post
> > the contents of them from the bad machine?
> > 
> 
> Hum, actually they didn't set up neither projid nor projects. Of course
> I did create these during my tests, but could this be the culprit ?

Does this recreate the symptoms?

# mkfs.xfs -f /dev/sda
meta-data=/dev/sda               isize=512    agcount=4, agsize=1298176 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=0   metadir=0
data     =                       bsize=4096   blocks=5192704, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=0
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
         =                       rgcount=0    rgsize=0 extents
Discarding blocks...Done.
# mount /dev/sda /mnt -o prjquota
# xfs_quota  -x -c 'limit -p bhard=2G 55' /mnt
# mkdir /mnt/dir
# xfs_io -c 'chproj 55' -c 'chattr +P' -c 'stat -vvvv' /mnt/dir
fd.path = "/mnt/dir"
fd.flags = non-sync,non-direct,read-write
stat.ino = 132
stat.type = directory
stat.size = 6
stat.blocks = 0
stat.atime = Thu Dec 12 12:07:53 2024
stat.mtime = Thu Dec 12 12:07:53 2024
stat.ctime = Thu Dec 12 12:08:12 2024
fsxattr.xflags = 0x200 [proj-inherit]
fsxattr.projid = 55
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G  420M   20G   3% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt
# fallocate -l 19g /mnt/a
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  345M  99% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt

Clearly, df should be reporting 345M available for both cases, since we
haven't actually used any of project 55's blocks.

# xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv' /mnt/dir/fork
wrote 1048576/1048576 bytes at offset 0
1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec)
fd.path = "/mnt/dir/fork"
fd.flags = non-sync,non-direct,read-write
stat.ino = 134
stat.type = regular file
stat.size = 1048576
stat.blocks = 2048
stat.atime = Thu Dec 12 12:11:06 2024
stat.mtime = Thu Dec 12 12:11:06 2024
stat.ctime = Thu Dec 12 12:11:06 2024
fsxattr.xflags = 0x0 []
fsxattr.projid = 55
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 1
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.0M  2.0G   1% /mnt

I think this behavior comes from xfs_fill_statvfs_from_dquot, which does
this:

	limit = blkres->softlimit ?
		blkres->softlimit :
		blkres->hardlimit;
	if (limit && statp->f_blocks > limit) {
		statp->f_blocks = limit;
		statp->f_bfree = statp->f_bavail =
			(statp->f_blocks > blkres->reserved) ?
			 (statp->f_blocks - blkres->reserved) : 0;
	}

I think the f_bfree/f_bavail assignment is wrong because it doesn't
handle the case where f_bfree was less than (limit - reserved).

	if (limit) {
		uint64_t	remaining = 0;

		if (statp->f_blocks > limit)
			statp->f_blocks = limit;
		if (limit > blkres->reserved)
			remaining = limit - blkres->reserved;
		statp->f_bfree = min(statp->f_bfree, remaining);
		statp->f_bavail = min(statp->f_bavail, remaining);
	}

This fixes the df output a bit:
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.7G  344M  84% /mnt

Though the "used" column is nonsense now.  But I guess that's why statfs
only defines total blocks and free/available blocks.

--D


> -- 
> ------------------------------------------------------------------------
>    Emmanuel Florac     |   Direction technique
> ------------------------------------------------------------------------
>    https://intellique.com
>    +33 6 16 30 15 95
> ------------------------------------------------------------------------
>  

Re: Weird behaviour with project quotas

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 3165 bytes --]

Le Thu, 12 Dec 2024 12:25:47 -0800
"Darrick J. Wong" <djwong@kernel.org> écrivait:


> Does this recreate the symptoms?
> 
<snip>
> # df /mnt /mnt/dir
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda         20G  420M   20G   3% /mnt
> /dev/sda        2.0G     0  2.0G   0% /mnt
> # fallocate -l 19g /mnt/a
> # df /mnt /mnt/dir
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda         20G   20G  345M  99% /mnt
> /dev/sda        2.0G     0  2.0G   0% /mnt
> 
> Clearly, df should be reporting 345M available for both cases, since
> we haven't actually used any of project 55's blocks.
> 
> # xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv'
> /mnt/dir/fork wrote 1048576/1048576 bytes at offset 0
> 1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec)
> fd.path = "/mnt/dir/fork"
> fd.flags = non-sync,non-direct,read-write
> stat.ino = 134
> stat.type = regular file
> stat.size = 1048576
> stat.blocks = 2048
> stat.atime = Thu Dec 12 12:11:06 2024
> stat.mtime = Thu Dec 12 12:11:06 2024
> stat.ctime = Thu Dec 12 12:11:06 2024
> fsxattr.xflags = 0x0 []
> fsxattr.projid = 55
> fsxattr.extsize = 0
> fsxattr.cowextsize = 0
> fsxattr.nextents = 1
> fsxattr.naextents = 0
> dioattr.mem = 0x200
> dioattr.miniosz = 512
> dioattr.maxiosz = 2147483136
> # df /mnt /mnt/dir
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda         20G   20G  344M  99% /mnt
> /dev/sda        2.0G  1.0M  2.0G   1% /mnt
> 
> I think this behavior comes from xfs_fill_statvfs_from_dquot, which
> does this:
> 
> 	limit = blkres->softlimit ?
> 		blkres->softlimit :
> 		blkres->hardlimit;
> 	if (limit && statp->f_blocks > limit) {
> 		statp->f_blocks = limit;
> 		statp->f_bfree = statp->f_bavail =
> 			(statp->f_blocks > blkres->reserved) ?
> 			 (statp->f_blocks - blkres->reserved) : 0;
> 	}
> 
> I think the f_bfree/f_bavail assignment is wrong because it doesn't
> handle the case where f_bfree was less than (limit - reserved).
> 
> 	if (limit) {
> 		uint64_t	remaining = 0;
> 
> 		if (statp->f_blocks > limit)
> 			statp->f_blocks = limit;
> 		if (limit > blkres->reserved)
> 			remaining = limit - blkres->reserved;
> 		statp->f_bfree = min(statp->f_bfree, remaining);
> 		statp->f_bavail = min(statp->f_bavail, remaining);
> 	}
> 
> This fixes the df output a bit:
> # df /mnt /mnt/dir
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda         20G   20G  344M  99% /mnt
> /dev/sda        2.0G  1.7G  344M  84% /mnt
> 
> Though the "used" column is nonsense now.  But I guess that's why
> statfs only defines total blocks and free/available blocks.

Yep, that looks exactly like the problem we've met. Does the fact that
not all folders have project quota change something in that case ?

-- 
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
 

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Weird behaviour with project quotas

[Darrick J. Wong]

On Fri, Dec 13, 2024 at 04:42:51PM +0100, Emmanuel Florac wrote:
> Le Thu, 12 Dec 2024 12:25:47 -0800
> "Darrick J. Wong" <djwong@kernel.org> écrivait:
> 
> 
> > Does this recreate the symptoms?
> > 
> <snip>
> > # df /mnt /mnt/dir
> > Filesystem      Size  Used Avail Use% Mounted on
> > /dev/sda         20G  420M   20G   3% /mnt
> > /dev/sda        2.0G     0  2.0G   0% /mnt
> > # fallocate -l 19g /mnt/a
> > # df /mnt /mnt/dir
> > Filesystem      Size  Used Avail Use% Mounted on
> > /dev/sda         20G   20G  345M  99% /mnt
> > /dev/sda        2.0G     0  2.0G   0% /mnt
> > 
> > Clearly, df should be reporting 345M available for both cases, since
> > we haven't actually used any of project 55's blocks.
> > 
> > # xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv'
> > /mnt/dir/fork wrote 1048576/1048576 bytes at offset 0
> > 1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec)
> > fd.path = "/mnt/dir/fork"
> > fd.flags = non-sync,non-direct,read-write
> > stat.ino = 134
> > stat.type = regular file
> > stat.size = 1048576
> > stat.blocks = 2048
> > stat.atime = Thu Dec 12 12:11:06 2024
> > stat.mtime = Thu Dec 12 12:11:06 2024
> > stat.ctime = Thu Dec 12 12:11:06 2024
> > fsxattr.xflags = 0x0 []
> > fsxattr.projid = 55
> > fsxattr.extsize = 0
> > fsxattr.cowextsize = 0
> > fsxattr.nextents = 1
> > fsxattr.naextents = 0
> > dioattr.mem = 0x200
> > dioattr.miniosz = 512
> > dioattr.maxiosz = 2147483136
> > # df /mnt /mnt/dir
> > Filesystem      Size  Used Avail Use% Mounted on
> > /dev/sda         20G   20G  344M  99% /mnt
> > /dev/sda        2.0G  1.0M  2.0G   1% /mnt
> > 
> > I think this behavior comes from xfs_fill_statvfs_from_dquot, which
> > does this:
> > 
> > 	limit = blkres->softlimit ?
> > 		blkres->softlimit :
> > 		blkres->hardlimit;
> > 	if (limit && statp->f_blocks > limit) {
> > 		statp->f_blocks = limit;
> > 		statp->f_bfree = statp->f_bavail =
> > 			(statp->f_blocks > blkres->reserved) ?
> > 			 (statp->f_blocks - blkres->reserved) : 0;
> > 	}
> > 
> > I think the f_bfree/f_bavail assignment is wrong because it doesn't
> > handle the case where f_bfree was less than (limit - reserved).
> > 
> > 	if (limit) {
> > 		uint64_t	remaining = 0;
> > 
> > 		if (statp->f_blocks > limit)
> > 			statp->f_blocks = limit;
> > 		if (limit > blkres->reserved)
> > 			remaining = limit - blkres->reserved;
> > 		statp->f_bfree = min(statp->f_bfree, remaining);
> > 		statp->f_bavail = min(statp->f_bavail, remaining);
> > 	}
> > 
> > This fixes the df output a bit:
> > # df /mnt /mnt/dir
> > Filesystem      Size  Used Avail Use% Mounted on
> > /dev/sda         20G   20G  344M  99% /mnt
> > /dev/sda        2.0G  1.7G  344M  84% /mnt
> > 
> > Though the "used" column is nonsense now.  But I guess that's why
> > statfs only defines total blocks and free/available blocks.
> 
> Yep, that looks exactly like the problem we've met. Does the fact that
> not all folders have project quota change something in that case ?

No, I don't think that changes anything.  If you can build your own
kernel, can you try this out?

--D

xfs: don't over-report free space or inodes in statvfs

Emmanual Florac reports a strange occurrence when project quota limits
are enabled, free space is lower than the remaining quota, and someone
runs statvfs:

  # mkfs.xfs -f /dev/sda
  # mount /dev/sda /mnt -o prjquota
  # xfs_quota  -x -c 'limit -p bhard=2G 55' /mnt
  # mkdir /mnt/dir
  # xfs_io -c 'chproj 55' -c 'chattr +P' -c 'stat -vvvv' /mnt/dir
  # fallocate -l 19g /mnt/a
  # df /mnt /mnt/dir
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda         20G   20G  345M  99% /mnt
  /dev/sda        2.0G     0  2.0G   0% /mnt

I think the bug here is that xfs_fill_statvfs_from_dquot unconditionally
assigns to f_bfree without checking that the filesystem has enough free
space to fill the remaining project quota.  However, this is a
longstanding behavior of xfs so it's unclear what to do here.

Cc: <stable@vger.kernel.org> # v2.6.18
Fixes: 932f2c323196c2 ("[XFS] statvfs component of directory/project quota support, code originally by Glen.")
Reported-by: Emmanuel Florac <eflorac@intellique.com>
Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 fs/xfs/xfs_qm_bhv.c |   27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/fs/xfs/xfs_qm_bhv.c b/fs/xfs/xfs_qm_bhv.c
index 847ba29630e9d8..db5b8afd9d1b97 100644
--- a/fs/xfs/xfs_qm_bhv.c
+++ b/fs/xfs/xfs_qm_bhv.c
@@ -32,21 +32,28 @@ xfs_fill_statvfs_from_dquot(
 	limit = blkres->softlimit ?
 		blkres->softlimit :
 		blkres->hardlimit;
-	if (limit && statp->f_blocks > limit) {
-		statp->f_blocks = limit;
-		statp->f_bfree = statp->f_bavail =
-			(statp->f_blocks > blkres->reserved) ?
-			 (statp->f_blocks - blkres->reserved) : 0;
+	if (limit) {
+		uint64_t	remaining = 0;
+
+		if (limit > blkres->reserved)
+			remaining = limit - blkres->reserved;
+
+		statp->f_blocks = min(statp->f_blocks, limit);
+		statp->f_bfree = min(statp->f_bfree, remaining);
+		statp->f_bavail = min(statp->f_bavail, remaining);
 	}
 
 	limit = dqp->q_ino.softlimit ?
 		dqp->q_ino.softlimit :
 		dqp->q_ino.hardlimit;
-	if (limit && statp->f_files > limit) {
-		statp->f_files = limit;
-		statp->f_ffree =
-			(statp->f_files > dqp->q_ino.reserved) ?
-			 (statp->f_files - dqp->q_ino.reserved) : 0;
+	if (limit) {
+		uint64_t	remaining = 0;
+
+		if (limit > dqp->q_ino.reserved)
+			remaining = limit - dqp->q_ino.reserved;
+
+		statp->f_files = min(statp->f_files, limit);
+		statp->f_ffree = min(statp->f_ffree, remaining);
 	}
 }
 

Re: Weird behaviour with project quotas

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

Le Fri, 13 Dec 2024 09:15:37 -0800
"Darrick J. Wong" <djwong@kernel.org> écrivait:

> No, I don't think that changes anything.  If you can build your own
> kernel, can you try this out?
> 
> --D
> 
> xfs: don't over-report free space or inodes in statvfs

I'll give it a try, but that looks like a patch for old weird RedHat
kernel, I'm running plain vanilla generally, and much higher versions,
I'll see how it applies :)

-- 
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
 

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Weird behaviour with project quotas

[Darrick J. Wong]

On Mon, Dec 16, 2024 at 11:18:51PM +0100, Emmanuel Florac wrote:
> Le Fri, 13 Dec 2024 09:15:37 -0800
> "Darrick J. Wong" <djwong@kernel.org> écrivait:
> 
> > No, I don't think that changes anything.  If you can build your own
> > kernel, can you try this out?
> > 
> > --D
> > 
> > xfs: don't over-report free space or inodes in statvfs
> 
> I'll give it a try, but that looks like a patch for old weird RedHat
> kernel, I'm running plain vanilla generally, and much higher versions,
> I'll see how it applies :)

That's from 6.13-rc3; I don't do RH kernels.

--D

> -- 
> ------------------------------------------------------------------------
>    Emmanuel Florac     |   Direction technique
> ------------------------------------------------------------------------
>    https://intellique.com
>    +33 6 16 30 15 95
> ------------------------------------------------------------------------
>  

Re: Weird behaviour with project quotas

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

Le Tue, 17 Dec 2024 08:50:42 -0800
"Darrick J. Wong" <djwong@kernel.org> écrivait:

> > > xfs: don't over-report free space or inodes in statvfs  
> > 
> > I'll give it a try, but that looks like a patch for old weird RedHat
> > kernel, I'm running plain vanilla generally, and much higher
> > versions, I'll see how it applies :)  
> 
> That's from 6.13-rc3; I don't do RH kernels.

Sorry, I've seen the line 

> Cc: <stable@vger.kernel.org> # v2.6.18

And thought it was about some RH7.x kernel or something :)

-- 
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
------------------------------------------------------------------------
 

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Weird behaviour with project quotas

[Darrick J. Wong]

On Wed, Dec 18, 2024 at 06:47:32PM +0100, Emmanuel Florac wrote:
> Le Tue, 17 Dec 2024 08:50:42 -0800
> "Darrick J. Wong" <djwong@kernel.org> écrivait:
> 
> > > > xfs: don't over-report free space or inodes in statvfs  
> > > 
> > > I'll give it a try, but that looks like a patch for old weird RedHat
> > > kernel, I'm running plain vanilla generally, and much higher
> > > versions, I'll see how it applies :)  
> > 
> > That's from 6.13-rc3; I don't do RH kernels.
> 
> Sorry, I've seen the line 
> 
> > Cc: <stable@vger.kernel.org> # v2.6.18
> 
> And thought it was about some RH7.x kernel or something :)

Ah.  Nope, cc:stable and RHEL kernels are not related.

--D

> -- 
> ------------------------------------------------------------------------
>    Emmanuel Florac     |   Direction technique
> ------------------------------------------------------------------------
>    https://intellique.com
> ------------------------------------------------------------------------
>