df bigger than ls?

Brian Candler
2012-03-07T15:54:40Z

I have created some files spread across 12 XFS filesystems, using a
glusterfs "striped" volume. This writes files with lots of holes(*) - so I
would expect the space reported by 'du' to be less than the space reported
by 'ls'.

However it's the other way round - du is reporting more space used than ls!
Here's what I mean: I'm looking at the files directly on the underlying
disk mount point, not via glusterfs at all.

$ ls -lh /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1.1G 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ ls -lk /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
-rw-rw-r-- 1 brian brian 1059968 2012-03-07 15:19 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -h /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2.0G /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
$ du -k /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
2096392 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff

And here's what xfs_bmap reports:

root@storage1:~# xfs_bmap -vp /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0: [0..255]: 2933325744..2933325999 2 (3059152..3059407) 256 00000
1: [256..3071]: hole 2816
2: [3072..3327]: 2933326000..2933326255 2 (3059408..3059663) 256 00000
3: [3328..6143]: hole 2816
4: [6144..8191]: 2933326472..2933328519 2 (3059880..3061927) 2048 00000
5: [8192..9215]: hole 1024
6: [9216..13311]: 2933369480..2933373575 2 (3102888..3106983) 4096 00000
7: [13312..15359]: hole 2048
8: [15360..23551]: 2933375624..2933383815 2 (3109032..3117223) 8192 00000
9: [23552..24575]: hole 1024
10: [24576..40959]: 2933587168..2933603551 2 (3320576..3336959) 16384 00000
11: [40960..43007]: hole 2048
12: [43008..75775]: 2933623008..2933655775 2 (3356416..3389183) 32768 00000
13: [75776..76799]: hole 1024
14: [76800..142335]: 2933656800..2933722335 2 (3390208..3455743) 65536 00000
15: [142336..144383]: hole 2048
16: [144384..275455]: 2933724384..2933855455 2 (3457792..3588863) 131072 00000
17: [275456..276479]: hole 1024
18: [276480..538623]: 2935019808..2935281951 2 (4753216..5015359) 262144 00000
19: [538624..540671]: hole 2048
20: [540672..1064959]: 2935284000..2935808287 2 (5017408..5541695) 524288 00000
21: [1064960..1065983]: hole 1024
22: [1065984..2114559]: 2935809312..2936857887 2 (5542720..6591295) 1048576 00000
23: [2114560..2116607]: hole 2048
24: [2116608..2119935]: 2943037984..2943041311 2 (12771392..12774719) 3328 00000
root@storage1:~#

Given that these values are all in 512-byte disk blocks, the total file size
is (2119935 + 1) * 512 which agrees with ls. And some proportion of it
is holes, so du should report less than this, shouldn't it?

(Aside: it starts off being 11/12th holes as expected, but after a while
isn't. This may be a different bug, possibly in glusterfs itself)

I guess 'du' gets its info from stat(), and stat() also says the file is
using 4192784 blocks which is 2096392 KB:

$ stat /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
File: `/disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff'
Size: 1085407232 Blocks: 4192784 IO Block: 4096 regular file
Device: 810h/2064d Inode: 8595657903 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 1000/ brian) Gid: ( 1000/ brian)
Access: 2012-03-07 15:20:36.044365215 +0000
Modify: 2012-03-07 15:19:33.640364277 +0000
Change: 2012-03-07 15:19:33.640364277 +0000

Finally, here is xfs_db dump of the inode:

# ls -i /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
8595657903 /disk1/scratch2/work/PRSRA1/PRSRA1.1.0.bff
...
xfs_db> inode 8595657903
xfs_db> p
core.magic = 0x494e
core.mode = 0100664
core.version = 2
core.format = 3 (btree)
core.nlinkv2 = 1
core.onlink = 0
core.projid_lo = 0
core.projid_hi = 0
core.uid = 1000
core.gid = 1000
core.flushiter = 231
core.atime.sec = Wed Mar 7 15:20:36 2012
core.atime.nsec = 044365215
core.mtime.sec = Wed Mar 7 15:19:33 2012
core.mtime.nsec = 640364277
core.ctime.sec = Wed Mar 7 15:19:33 2012
core.ctime.nsec = 640364277
core.size = 1085407232
core.nblocks = 262370
core.extsize = 0
core.nextents = 13
core.naextents = 1
core.forkoff = 15
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 1875303423
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:537294687
a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,537253238,1,0]
xfs_db>

Platform: ubuntu 11.10
Linux 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Have I missed something obvious, or is there a bug of some sort here?

Regards,

Brian.

(*) http://gluster.org/community/documentation//index.php/GlusterFS_Technical_FAQ#Stripe_behavior_not_working_as_expected