du result is smaller than xfs_quota report

du result is smaller than xfs_quota report

[YeYin]

[-- Attachment #1.1: Type: text/plain, Size: 1576 bytes --]

Hi, all,
I encountered some problems when I used XFS with project quota:


# cat /etc/projid 
docker-15091421396426:65061
docker-15092311522409:26286
docker-15101215637294:14162



# cat /etc/projects 
65061:/data/docker-volumes/docker-15091421396426
26286:/data/docker-volumes/docker-15092311522409
14162:/data/docker-volumes/docker-15101215637294



# xfs_quota -xc 'report -h' /data          
Project quota on /data (/dev/sda4)
                        Blocks              
Project ID   Used   Soft   Hard Warn/Grace   
---------- --------------------------------- 
docker-15091421396426   1.0G      0   100G  00 [------]
docker-15092311522409 371.5G      0   700G  00 [------]
docker-15101215637294 920.8M      0   400G  00 [------]



# du -sh /data/docker-volumes/docker-15092311522409 
200G    /data/docker-volumes/docker-15092311522409



As we can see,  du's result(200G) is smaller than xfs_quota report.


# xfs_db -xr -c 'dquot -p 26286' -c p /dev/sda4
diskdq.magic = 0x4451
diskdq.version = 0x1
diskdq.flags = 0x2
diskdq.id = 26286
diskdq.blk_hardlimit = 183500800
diskdq.blk_softlimit = 0
diskdq.ino_hardlimit = 0
diskdq.ino_softlimit = 0
diskdq.bcount = 97049008
diskdq.icount = 11725
diskdq.itimer = 0
diskdq.btimer = 0
diskdq.iwarns = 0
diskdq.bwarns = 0
diskdq.rtb_hardlimit = 0
diskdq.rtb_softlimit = 0
diskdq.rtbcount = 0
diskdq.rtbtimer = 0
diskdq.rtbwarns = 0



It seems that some blocks owned by project wasn't be freed. Anyone can give me some suggestions to find the reason?


Thanks,
Ye

[-- Attachment #1.2: Type: text/html, Size: 2372 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: du result is smaller than xfs_quota report

[Emmanuel Florac]

Le Tue, 20 Oct 2015 17:09:58 +0800
"YeYin" <eyniy@qq.com> écrivait:

> It seems that some blocks owned by project wasn't be freed. Anyone
> can give me some suggestions to find the reason?
> 

What does say

ls -l /data/docker-volumes/docker-15092311522409

is it coherent with du, or xfs_quota?
-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: du result is smaller than xfs_quota report

[YeYin]

[-- Attachment #1.1: Type: text/plain, Size: 1266 bytes --]

>What does say
>
>ls -l /data/docker-volumes/docker-15092311522409
>
>is it coherent with du, or xfs_quota?


/data/docker-volumes/docker-15092311522409 is a directory which have a lot of subdirectories,  run 'ls -l' is meaningless. 
'ls -l' and 'du' used  fstat or lstat similar function to get file's size, so ls should be coherent with du.


Thanks,
Ye



------------------ Original ------------------
From:  "Emmanuel Florac";<eflorac@intellique.com>;
Date:  Tue, Oct 20, 2015 07:31 PM
To:  "YeYin"<eyniy@qq.com>; 
Cc:  "xfs"<xfs@oss.sgi.com>; 
Subject:  Re: du result is smaller than xfs_quota report



Le Tue, 20 Oct 2015 17:09:58 +0800
"YeYin" <eyniy@qq.com> écrivait:

> It seems that some blocks owned by project wasn't be freed. Anyone
> can give me some suggestions to find the reason?
> 

What does say

ls -l /data/docker-volumes/docker-15092311522409

is it coherent with du, or xfs_quota?
-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #1.2: Type: text/html, Size: 2160 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: du result is smaller than xfs_quota report

[Eric Sandeen]

> On Oct 20, 2015, at 4:09 AM, YeYin <eyniy@qq.com> wrote:
> 
> Hi, all,
> I encountered some problems when I used XFS with project quota:
> 
> # cat /etc/projid 
> docker-15091421396426:65061
> docker-15092311522409:26286
> docker-15101215637294:14162
> 
> # cat /etc/projects 
> 65061:/data/docker-volumes/docker-15091421396426
> 26286:/data/docker-volumes/docker-15092311522409
> 14162:/data/docker-volumes/docker-15101215637294
> 
> # xfs_quota -xc 'report -h' /data          
> Project quota on /data (/dev/sda4)
>                         Blocks              
> Project ID   Used   Soft   Hard Warn/Grace   
> ---------- --------------------------------- 
> docker-15091421396426   1.0G      0   100G  00 [------]
> docker-15092311522409 371.5G      0   700G  00 [------]
> docker-15101215637294 920.8M      0   400G  00 [------]
> 
> # du -sh /data/docker-volumes/docker-15092311522409 
> 200G    /data/docker-volumes/docker-15092311522409
> 
> As we can see,  du's result(200G) is smaller than xfs_quota report.
> 
Is anything mounted under that directory hierarchy, and hiding files from du's view?

Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: du result is smaller than xfs_quota report

[YeYin]

[-- Attachment #1.1: Type: text/plain, Size: 3276 bytes --]

> Is anything mounted under that directory hierarchy, and hiding files from du's view?


No, Thanks. I found some unlinked inodes, but I can't get some useful information:


# xfs_db -xr -c 'agi 1' -c p /dev/sda4  
magicnum = 0x58414749
versionnum = 1
seqno = 1
length = 89150592
count = 9472
root = 3
level = 1
freecount = 1902
newino = 997952
dirino = null
unlinked[0-63] = 0:78272 1:98561 2:98562 3:78275 4:77060 5:78277 6:77062 7:78279 8:98568 9:98569 10:98570 11:77067 12:77068 13:98573 14:98574 15:78287 16:98576 17:98577 18:78290 19:98515 20:98068 21:78165 22:78294 23:78295 24:78168 25:888985 26:78170 27:98075 28:78172 29:78301 30:78302 31:98079 32:888992 33:76129 34:98082 35:76131 36:98084 37:98277 38:98278 39:910503 40:98280 41:98281 42:98282 43:98091 44:98092 45:98541 46:98094 47:98095 48:76976 49:75953 50:75954 51:98547 52:98484 53:98485 54:98550 55:98551 56:98488 57:98489 58:98106 59:98491 60:98492 61:889021 62:98558 63:98495





# xfs_db -xr -c 'inode 78272' -c p /dev/sda4
core.magic = 0
core.mode = 0
core.version = 0
core.format = 0 (dev)
core.uid = 0
core.gid = 0
core.flushiter = 0
core.atime.sec = Thu Jan  1 08:00:00 1970
core.atime.nsec = 000000000
core.mtime.sec = Thu Jan  1 08:00:00 1970
core.mtime.nsec = 000000000
core.ctime.sec = Thu Jan  1 08:00:00 1970
core.ctime.nsec = 000000000
core.size = 0
core.nblocks = 0
core.extsize = 0
core.nextents = 0
core.naextents = 0
core.forkoff = 0
core.aformat = 0 (dev)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 0
next_unlinked = 0
u.dev = 0



Thanks,
Ye


------------------ Original ------------------
From:  "Eric Sandeen";<sandeen@sandeen.net>;
Date:  Wed, Oct 21, 2015 03:35 AM
To:  "YeYin"<eyniy@qq.com>; 
Cc:  "xfs"<xfs@oss.sgi.com>; 
Subject:  Re: du result is smaller than xfs_quota report




> On Oct 20, 2015, at 4:09 AM, YeYin <eyniy@qq.com> wrote:
> 
> Hi, all,
> I encountered some problems when I used XFS with project quota:
> 
> # cat /etc/projid 
> docker-15091421396426:65061
> docker-15092311522409:26286
> docker-15101215637294:14162
> 
> # cat /etc/projects 
> 65061:/data/docker-volumes/docker-15091421396426
> 26286:/data/docker-volumes/docker-15092311522409
> 14162:/data/docker-volumes/docker-15101215637294
> 
> # xfs_quota -xc 'report -h' /data          
> Project quota on /data (/dev/sda4)
>                         Blocks              
> Project ID   Used   Soft   Hard Warn/Grace   
> ---------- --------------------------------- 
> docker-15091421396426   1.0G      0   100G  00 [------]
> docker-15092311522409 371.5G      0   700G  00 [------]
> docker-15101215637294 920.8M      0   400G  00 [------]
> 
> # du -sh /data/docker-volumes/docker-15092311522409 
> 200G    /data/docker-volumes/docker-15092311522409
> 
> As we can see,  du's result(200G) is smaller than xfs_quota report.
> 
Is anything mounted under that directory hierarchy, and hiding files from du's view?

Eric

[-- Attachment #1.2: Type: text/html, Size: 4783 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: du result is smaller than xfs_quota report

[Dave Chinner]

On Wed, Oct 21, 2015 at 02:37:04PM +0800, YeYin wrote:
> > Is anything mounted under that directory hierarchy, and hiding files from du's view?
> 
> 
> No, Thanks. I found some unlinked inodes, but I can't get some useful information:
> 
> 
> # xfs_db -xr -c 'agi 1' -c p /dev/sda4  
....
> unlinked[0-63] = 0:78272 ....

> # xfs_db -xr -c 'inode 78272' -c p /dev/sda4

That's because 78272 isn't an real inode number. It's a AG relative
inode number ("agino") of 78272. The global inode number folds in
the AG number to it, and this is from AG 1. So:

xfs_db> convert agno 1 agino 78272 ino
<inum>
xfs_db> inode <inum>
xfs_db> p
....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: du result is smaller than xfs_quota report

[YeYin]

[-- Attachment #1.1: Type: text/plain, Size: 5302 bytes --]

Hi, Dave,
Thank you very much. I got some results:
# xfs_db -xr  /dev/sda4                        
xfs_db> convert agno 1 agino 78272 ino
0x800131c0 (2147561920)
xfs_db> inode 2147561920
xfs_db> p
core.magic = 0x494e
core.mode = 0100600
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 0
core.onlink = 0
core.projid_lo = 26286
core.projid_hi = 0
core.uid = 30000
core.gid = 100
core.flushiter = 30
core.atime.sec = Tue Oct 20 13:51:58 2015
core.atime.nsec = 104938956
core.mtime.sec = Tue Oct 20 13:53:58 2015
core.mtime.nsec = 824941455
core.ctime.sec = Tue Oct 20 14:21:59 2015
core.ctime.nsec = 062660227
core.size = 41041920
core.nblocks = 10020
core.extsize = 0
core.nextents = 2
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 1
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1194777008
next_unlinked = 75712
u.bmx[0-1] = [startoff,startblock,blockcount,extentflag] 0:[0,135330882,32,0] 1:[32,146117410,9988,0]





However, I want to ask how this situation to happen?


From the output,  we can see "core.nlinkv2 = 0". But I can't reapper this:


# dd if=/dev/zero of=/data/test.data bs=4k count=1024
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 0.00476946 s, 879 MB/s


# stat /data/test.data 
  File: `/data/test.data'
  Size: 4194304         Blocks: 8192       IO Block: 4096   regular file
Device: 804h/2052d      Inode: 29409295    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-10-21 16:05:45.201679018 +0800
Modify: 2015-10-21 16:05:45.205679018 +0800
Change: 2015-10-21 16:05:45.205679018 +0800


# xfs_db -xr -c 'inode 29409295' -c p /dev/sda4                                      
core.magic = 0x494e
core.mode = 0100644
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 0
core.gid = 0
core.flushiter = 12
core.atime.sec = Wed Oct 21 16:05:45 2015
core.atime.nsec = 201679018
core.mtime.sec = Wed Oct 21 16:05:45 2015
core.mtime.nsec = 205679018
core.ctime.sec = Wed Oct 21 16:05:45 2015
core.ctime.nsec = 205679018
core.size = 4194304
core.nblocks = 1024
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 3722739155
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,5943235,1024,0]


# tail -f /data/test.data &
[1] 27598
# unlink /data/test.data 


# xfs_db -xr -c 'inode 29409295' -c p /dev/sda4
core.magic = 0x494e
core.mode = 0100644
core.version = 2
core.format = 2 (extents)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 0
core.gid = 0
core.flushiter = 12
core.atime.sec = Wed Oct 21 16:05:45 2015
core.atime.nsec = 201679018
core.mtime.sec = Wed Oct 21 16:05:45 2015
core.mtime.nsec = 205679018
core.ctime.sec = Wed Oct 21 16:05:45 2015
core.ctime.nsec = 205679018
core.size = 4194304
core.nblocks = 1024
core.extsize = 0
core.nextents = 1
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 3722739155
next_unlinked = null
u.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,5943235,1024,0]



As we can see, core.nlinkv2 is still "1" after I unlink /data/test.data.


Thanks,
Ye


------------------ Original ------------------
From:  "Dave Chinner";<david@fromorbit.com>;
Date:  Wed, Oct 21, 2015 03:13 PM
To:  "YeYin"<eyniy@qq.com>; 
Cc:  "Eric Sandeen"<sandeen@sandeen.net>; "xfs"<xfs@oss.sgi.com>; 
Subject:  Re: du result is smaller than xfs_quota report



On Wed, Oct 21, 2015 at 02:37:04PM +0800, YeYin wrote:
> > Is anything mounted under that directory hierarchy, and hiding files from du's view?
> 
> 
> No, Thanks. I found some unlinked inodes, but I can't get some useful information:
> 
> 
> # xfs_db -xr -c 'agi 1' -c p /dev/sda4  
....
> unlinked[0-63] = 0:78272 ....

> # xfs_db -xr -c 'inode 78272' -c p /dev/sda4

That's because 78272 isn't an real inode number. It's a AG relative
inode number ("agino") of 78272. The global inode number folds in
the AG number to it, and this is from AG 1. So:

xfs_db> convert agno 1 agino 78272 ino
<inum>
xfs_db> inode <inum>
xfs_db> p
....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[-- Attachment #1.2: Type: text/html, Size: 7952 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: du result is smaller than xfs_quota report

[Dave Chinner]

On Wed, Oct 21, 2015 at 04:09:20PM +0800, YeYin wrote:
> Hi, Dave,
> Thank you very much. I got some results:
> # xfs_db -xr  /dev/sda4                        
> xfs_db> convert agno 1 agino 78272 ino
> 0x800131c0 (2147561920)
> xfs_db> inode 2147561920
> xfs_db> p
> core.magic = 0x494e
> core.mode = 0100600
> core.version = 2
> core.format = 2 (extents)
> core.nlinkv2 = 0
....
> However, I want to ask how this situation to happen?

Either O_TMPFILE files that are still open, or files that have been
unlinked but still have an open file descriptor referencing them.
They won't be unlinked until the last reference goes away.

> # tail -f /data/test.data &
> [1] 27598
> # unlink /data/test.data 
> 
> 
> # xfs_db -xr -c 'inode 29409295' -c p /dev/sda4
> core.magic = 0x494e
> core.mode = 0100644
> core.version = 2
> core.format = 2 (extents)
> core.nlinkv2 = 1

Inode hasn't been written back to disk yet. That happens
asynchronously, usually based on time, sometimes sooner due to log
space demands, sometimes longer due to repeated modifications.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs