"No space left on device" from hell

Arkadiusz Miśkiewicz
2014-09-24T07:03:56Z

Hello.

3.10.40 kernel and creating files causes "No space left on device". After
deleting some files I can create new files again until the problem happens
again.

Any ideas what is going on and, what is more important, how to fix the
problem?

# df -h
/dev/mapper/vgsys-home 600G 368G 232G 62% /home

# df -i
/dev/mapper/vgsys-home 629145600 7982895 621162705 2% /home

# xfs_info /home
meta-data=/dev/mapper/vgsys-home isize=256 agcount=8, agsize=19660800 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=157286400, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=38400, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=

imaxpct=25% but current inodes take only about 2GB, much less that 25% of
600G, so that shouldn't be problem

/dev/mapper/vgsys-home /home xfs
rw,nosuid,nodev,relatime,attr2,inode64,usrquota,prjquota 0 0

so inode64 is not a problem

# xfs_quota -c "quota -p 0" /home
Disk quotas for Project #0 (0)
Filesystem Blocks Quota Limit Warn/Time Mounted
on
/dev/mapper/vgsys-home
27848804 0 0 00 [--------] /home
# xfs_quota -c "quota -u 0" /home
Disk quotas for User root (0)
Filesystem Blocks Quota Limit Warn/Time Mounted
on
/dev/mapper/vgsys-home
103484 0 0 00 [--------] /home
# xfs_quota -c "quota -i 0" /home
Disk quotas for User root (0)
Filesystem Files Quota Limit Warn/Time Mounted
on
/dev/mapper/vgsys-home
8336 0 0 00 [--------] /home

so quota is not a problem, too.

# xfs_db -r "-c freesp -s" /dev/mapper/vgsys-home
from to extents blocks pct
1 1 3426 3426 0.01
2 3 3319 8026 0.01
4 7 3694 19027 0.03
8 15 23225 330378 0.54
16 31 396 8180 0.01
32 63 75 3256 0.01
64 127 44 3443 0.01
128 255 26 4892 0.01
256 511 32 10737 0.02
512 1023 5 3867 0.01
1024 2047 8 13576 0.02
2048 4095 5 13893 0.02
4096 8191 3 16896 0.03
8192 16383 1 15532 0.03
16384 32767 1 17731 0.03
1048576 2097151 1 1414952 2.33
16777216 19660800 3 58913965 96.90
total free extents 34264
total free blocks 60801777
average free extent size 1774.51

unfortunately no idea how to interpret this. Man page isn't helpful.

http://sprunge.us/IVjE
(trace-cmd of touch /home/x ... but looks to be partial only)

--
Arkadiusz Miśkiewicz, arekm / maven.pl