df bigger than ls?

From: Brian Candler <B.Candler@pobox.com>
To: xfs@oss.sgi.com
Subject: df bigger than ls?
Date: Wed, 7 Mar 2012 15:54:39 +0000	[thread overview]
Message-ID: <20120307155439.GA23360@nsrc.org> (raw)

I have created some files spread across 12 XFS filesystems, using a
glusterfs "striped" volume.  This writes files with lots of holes(*) - so I
would expect the space reported by 'du' to be less than the space reported
by 'ls'.

However it's the other way round - du is reporting more space used than ls! 
Here's what I mean:  I'm looking at the files directly on the underlying
disk mount point, not via glusterfs at all.

$ ls -lh /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1.1G 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ ls -lk /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1059968 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -h /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -k /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2096392	/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff

And here's what xfs_bmap reports:

root@storage1:~# xfs_bmap -vp /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff:
 EXT: FILE-OFFSET         BLOCK-RANGE            AG AG-OFFSET              TOTAL FLAGS
   0: [0..255]:           2933325744..2933325999  2 (3059152..3059407)       256 00000
   1: [256..3071]:        hole                                              2816
   2: [3072..3327]:       2933326000..2933326255  2 (3059408..3059663)       256 00000
   3: [3328..6143]:       hole                                              2816
   4: [6144..8191]:       2933326472..2933328519  2 (3059880..3061927)      2048 00000
   5: [8192..9215]:       hole                                              1024
   6: [9216..13311]:      2933369480..2933373575  2 (3102888..3106983)      4096 00000
   7: [13312..15359]:     hole                                              2048
   8: [15360..23551]:     2933375624..2933383815  2 (3109032..3117223)      8192 00000
   9: [23552..24575]:     hole                                              1024
  10: [24576..40959]:     2933587168..2933603551  2 (3320576..3336959)     16384 00000
  11: [40960..43007]:     hole                                              2048
  12: [43008..75775]:     2933623008..2933655775  2 (3356416..3389183)     32768 00000
  13: [75776..76799]:     hole                                              1024
  14: [76800..142335]:    2933656800..2933722335  2 (3390208..3455743)     65536 00000
  15: [142336..144383]:   hole                                              2048
  16: [144384..275455]:   2933724384..2933855455  2 (3457792..3588863)    131072 00000
  17: [275456..276479]:   hole                                              1024
  18: [276480..538623]:   2935019808..2935281951  2 (4753216..5015359)    262144 00000
  19: [538624..540671]:   hole                                              2048
  20: [540672..1064959]:  2935284000..2935808287  2 (5017408..5541695)    524288 00000
  21: [1064960..1065983]: hole                                              1024
  22: [1065984..2114559]: 2935809312..2936857887  2 (5542720..6591295)   1048576 00000
  23: [2114560..2116607]: hole                                              2048
  24: [2116608..2119935]: 2943037984..2943041311  2 (12771392..12774719)    3328 00000
root@storage1:~# 

Given that these values are all in 512-byte disk blocks, the total file size
is (2119935 + 1) * 512 which agrees with ls. And some proportion of it
is holes, so du should report less than this, shouldn't it?

(Aside: it starts off being 11/12th holes as expected, but after a while
isn't.  This may be a different bug, possibly in glusterfs itself)

I guess 'du' gets its info from stat(), and stat() also says the file is
using 4192784 blocks which is 2096392 KB:

$ stat /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
  File: `/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff'
  Size: 1085407232	Blocks: 4192784    IO Block: 4096   regular file
Device: 810h/2064d	Inode: 8595657903  Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/   brian)   Gid: ( 1000/   brian)
Access: 2012-03-07 15:20:36.044365215 +0000
Modify: 2012-03-07 15:19:33.640364277 +0000
Change: 2012-03-07 15:19:33.640364277 +0000

Finally, here is xfs_db dump of the inode:

# ls -i /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
8595657903 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
...
xfs_db> inode 8595657903
xfs_db> p
core.magic = 0x494e
core.mode = 0100664
core.version = 2
core.format = 3 (btree)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 1000
core.gid = 1000
core.flushiter = 231
core.atime.sec = Wed Mar  7 15:20:36 2012
core.atime.nsec = 044365215
core.mtime.sec = Wed Mar  7 15:19:33 2012
core.mtime.nsec = 640364277
core.ctime.sec = Wed Mar  7 15:19:33 2012
core.ctime.nsec = 640364277
core.size = 1085407232
core.nblocks = 262370
core.extsize = 0
core.nextents = 13
core.naextents = 1
core.forkoff = 15
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1875303423
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:537294687
a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,537253238,1,0]
xfs_db> 

Platform: ubuntu 11.10
Linux 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Have I missed something obvious, or is there a bug of some sort here?

Regards,

Brian.

(*) http://gluster.org/community/documentation//index.php/GlusterFS_Technical_FAQ#Stripe_behavior_not_working_as_expected

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs